donaldpipowitch · November 13, 2025 13:43 · Oct 18, 2023 · Oct 18, 2023 · Oct 18, 2023 · Oct 18, 2023
diff --git a/.storybook test-runner.js b/.storybook test-runner.js
@@ -33,11 +33,19 @@ module.exports = {
     if (!context.title.includes('Components/')) return;
 
     // storyContext.parameters gives you access to the parameters defined in the story and more
-    // (left it here to add per story filtering, custom delays or custom MatchImageSnapshot
-    // configs per story in the future)
+    // supported API:
+    // {
+    //   image: {
+    //     waitTime?: number; // wait that long before taking the screenshot
+    //     skip?: boolean; // do NOT take a screenshot
+    //     snapshotOptions?: MatchImageSnapshotOptions; // override our default options (https://github.com/americanexpress/jest-image-snapshot#%EF%B8%8F-api)
+    //   }
+    // }
     const storyContext = await getStoryContext(page, context);
     const imageParameters = storyContext.parameters?.image || {};
 
+    if (imageParameters.skip) return;
+
     // Make sure assets (images, fonts) are loaded and ready
     await page.waitForLoadState('domcontentloaded');
     await page.waitForLoadState('load');
@@ -55,7 +63,8 @@ module.exports = {
     expect(image).toMatchImageSnapshot({
       customSnapshotsDir: '.storybook-images',
       customSnapshotIdentifier: context.id,
-      storeReceivedOnFailure: true,
+      storeReceivedOnFailure: true, // sadly this currently doesn't work for new images: https://github.com/americanexpress/jest-image-snapshot/issues/331
+      ...imageParameters.snapshotOptions,
     });
   },
 };
diff --git a/README.md b/README.md
@@ -36,6 +36,8 @@ Just shows the "small" production build for the image generation.
 
 Sadly we needed to add a custom `prepare` function here. This is only needed, because of `TARGET_URL`. Usually this tells the Storybook Test Runner where it can find the Storybook instance (e.g. `http://127.0.0.1:58414`), but we'd actually need a custom `PLAYWRIGHT_TARGET_URL` in case the Playwright Server needs to reach the Storybook instance outside of the image (e.g. `http://host.docker.internal:58414`).
 
-Besides that we have our custom 
+Besides that we have our custom `postRender`. Here we want to take an image for all stories within a `Components` directory. Besides that we can add custom settings _per story_. Super nice! I also use the `storeReceivedOnFailure: true` option, because I like to check and download images right away from the failed job in the Gitlab CI. Sadly this seems to only work for image conflicts, [not new images](https://github.com/americanexpress/jest-image-snapshot/issues/331).
 
-## `.gitlab-ci.yml`
+## `.gitlab-ci.yml`
+
+Nothing suprising here. Uses the Playwright Docker image, builds Storybook (_all_ stories) and runs the Storybook Test Runner. Generated images and diffs are persisted as artifacts.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -0,0 +1,15 @@
+storybook:test-runner8:
+  image: mcr.microsoft.com/playwright:v1.39.0-jammy
+  stage: build-and-test
+  artifacts:
+    expire_in: 2 weeks
+    when: always
+    paths:
+      - .storybook-images/__diff_output__/
+      - .storybook-images/__received_output__/
+  before_script:
+    - npm install -g pnpm@$PNPM_VERSION
+    - pnpm config set store-dir .pnpm-store
+    - pnpm install --frozen-lockfile
+  script:
+    - STORYBOOK_TEST_RUNNER_CI=true pnpm test-storybook:build-and-run
diff --git a/README.md b/README.md
@@ -18,13 +18,24 @@ As a general note: We have a lot of stories in Storybook, but we only want to ta
 
 ## `package.json`
 
-...
+Here we have custom `"scripts"`. The idea is the following:
+
+In case we want to update images during local development we run `$ pnpm start-playwright-server` to start the Playwright server which will be reachable on `ws://127.0.0.1:3000`. Then we run `$ pnpm test-storybook:generate-images` which will create a "small" (not optimized, only relevant stories) production build of Storybook and then runs the Storybook Test Runner which will generate new images.
+
+The Gitlab CI will later run `$ test-storybook:build-and-run` within the Playwright Docker image (so no need for `$ pnpm start-playwright-server`). This command will be run on _all_ stories.
 
 ## `test-runner-jest.config.js`
 
-..
+Here we say to always use `ws://127.0.0.1:3000` in order to connect to Playwright. (Exception: if `STORYBOOK_TEST_RUNNER_CI` is set in the Gitlab CI.)
 
 ## `.storybook/main.ts`
 
-..
+Just shows the "small" production build for the image generation.
+
+## `.storybook/test-runner-jest.config.js`
+
+Sadly we needed to add a custom `prepare` function here. This is only needed, because of `TARGET_URL`. Usually this tells the Storybook Test Runner where it can find the Storybook instance (e.g. `http://127.0.0.1:58414`), but we'd actually need a custom `PLAYWRIGHT_TARGET_URL` in case the Playwright Server needs to reach the Storybook instance outside of the image (e.g. `http://host.docker.internal:58414`).
+
+Besides that we have our custom 
 
+## `.gitlab-ci.yml`
diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@ The benefits:
 - it's **way** faster
 - it has better official support
 - it does _more_ (component smoke tests, `play` tests, extensibility for more like a11y tests)
-- it's **way** more stable (actually it looks like there is not a _single_ flaky test right now 🤯)
+- it's **way** more stable
 
 But the setup might be non-trivial and there are some rough edges. The biggest downside: while it is way faster it could be even more faster, if we could just use generate images against the Storybook Dev Server. Sadly this was _very_ flaky and error boundaries would always fail (I assume this is because of Reacts unfortunate decision to re-throw errors in _Dev Mode_). See also [this issue](https://github.com/storybookjs/test-runner/issues/218).
 

diff --git a/README.md b/README.md
@@ -14,6 +14,8 @@ What I do now in order to generate images locally is a custom "production" build
 
 The other tricky part was the Playwright setup. In order to get the same results across machines we want to run Playwright in a Docker container. For various reasons I decided to use the Network API of Playwright. That means I'll _only_ run Playwright in the Docker container, but I keep my running Storybook instance on the host. (At least in the local setup. Within the Gitlab CI we run everything inside Docker.)
 
+As a general note: We have a lot of stories in Storybook, but we only want to take images of stories that are inside a `components/` directory.
+
 ## `package.json`
 
 ...

diff --git a/.storybook main.ts b/.storybook main.ts
@@ -0,0 +1,42 @@
+import fs from 'fs';
+import path from 'path';
+import { StorybookConfig } from '@storybook/react-vite';
+
+// custom stuff...
+
+const config: StorybookConfig = {
+  stories: process.env.GENERATE_IMAGES_LOCALLY
+    ? ['../stories/**/components/**/*.stories.@(js|jsx|ts|tsx)']
+    : [
+        '../stories/**/*.stories.mdx',
+        '../stories/**/*.stories.@(js|jsx|ts|tsx)',
+      ],
+  addons: [
+    '@storybook/addon-links',
+    '@storybook/addon-essentials',
+    '@storybook/addon-interactions',
+    '@storybook/addon-styling',
+    '@storybook/addon-a11y',
+    '@storybook/test-runner',
+  ],
+  viteFinal: async (config) => {
+    config.resolve = config.resolve ?? {};
+    config.resolve.alias = {
+      ...config.resolve.alias,
+      stories: path.resolve(__dirname, '../stories'),
+      '.storybook': path.resolve(__dirname),
+    };
+
+    if (process.env.STORYBOOK_NOT_MINIFIED === 'true') {
+      config.build = config.build ?? {};
+      config.build.minify = false;
+    }
+
+    // custom stuff...
+
+    return config;
+  },
+  // custom stuff...
+};
+
+export default config;
diff --git a/.storybook test-runner.js b/.storybook test-runner.js
@@ -0,0 +1,61 @@
+const { getStoryContext } = require('@storybook/test-runner');
+const { toMatchImageSnapshot } = require('jest-image-snapshot');
+
+module.exports = {
+  setup() {
+    expect.extend({ toMatchImageSnapshot });
+  },
+  // https://github.com/storybookjs/test-runner#prepare
+  // https://github.com/storybookjs/test-runner/blob/next/src/setup-page.ts#L12
+  async prepare({ page, browserContext, testRunnerConfig }) {
+    // this line is customized!
+    const targetURL = process.env.STORYBOOK_TEST_RUNNER_CI
+      ? 'http://127.0.0.1:58414'
+      : 'http://host.docker.internal:58414';
+    const iframeURL = new URL('iframe.html', targetURL).toString();
+
+    if (testRunnerConfig?.getHttpHeaders) {
+      const headers = await testRunnerConfig.getHttpHeaders(iframeURL);
+      await browserContext.setExtraHTTPHeaders(headers);
+    }
+
+    await page.goto(iframeURL, { waitUntil: 'load' }).catch((err) => {
+      if (err.message?.includes('ERR_CONNECTION_REFUSED')) {
+        const errorMessage = `Could not access the Storybook instance at ${targetURL}. Are you sure it's running?\n\n${err.message}`;
+        throw new Error(errorMessage);
+      }
+
+      throw err;
+    });
+  },
+  // context = { id, title, name }
+  async postRender(page, context) {
+    if (!context.title.includes('Components/')) return;
+
+    // storyContext.parameters gives you access to the parameters defined in the story and more
+    // (left it here to add per story filtering, custom delays or custom MatchImageSnapshot
+    // configs per story in the future)
+    const storyContext = await getStoryContext(page, context);
+    const imageParameters = storyContext.parameters?.image || {};
+
+    // Make sure assets (images, fonts) are loaded and ready
+    await page.waitForLoadState('domcontentloaded');
+    await page.waitForLoadState('load');
+    await page.waitForLoadState('networkidle');
+    await page.evaluate(() => document.fonts.ready);
+    if (imageParameters.waitTime)
+      await new Promise((resolve) =>
+        setTimeout(resolve, imageParameters.waitTime)
+      );
+
+    const image = await page.screenshot({
+      animations: 'disabled',
+      fullPage: true,
+    });
+    expect(image).toMatchImageSnapshot({
+      customSnapshotsDir: '.storybook-images',
+      customSnapshotIdentifier: context.id,
+      storeReceivedOnFailure: true,
+    });
+  },
+};
diff --git a/README.md b/README.md
@@ -16,4 +16,13 @@ The other tricky part was the Playwright setup. In order to get the same results
 
 ## `package.json`
 
-...
+...
+
+## `test-runner-jest.config.js`
+
+..
+
+## `.storybook/main.ts`
+
+..
+
diff --git a/test-runner-jest.config.js b/test-runner-jest.config.js
@@ -0,0 +1,23 @@
+const { getJestConfig } = require('@storybook/test-runner');
+
+/**
+ * @type {import('@jest/types').Config.InitialOptions}
+ */
+module.exports = {
+  // The default configuration comes from @storybook/test-runner
+  ...getJestConfig(),
+  /** Add your own overrides below
+   * @see https://jestjs.io/docs/configuration
+   */
+  testEnvironmentOptions: {
+    'jest-playwright': process.env.STORYBOOK_TEST_RUNNER_CI
+      ? undefined
+      : {
+          connectOptions: {
+            chromium: {
+              wsEndpoint: 'ws://127.0.0.1:3000',
+            },
+          },
+        },
+  },
+};
diff --git a/package.json b/package.json
@@ -3,8 +3,8 @@
     "test-storybook:run": "DEBUG_PRINT_LIMIT=0 test-storybook --index-json",
     "test-storybook:build": "cross-env STORYBOOK_NOT_MINIFIED=true storybook build --quiet --output-dir storybook",
     "test-storybook:server": "pnpm live-server --port=58414 storybook --no-browser",
-    "test-storybook:build-and-run": "pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414/?path=/story/some-path-to-a-real-story 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run'",
-    "test-storybook:generate-images": "cross-env GENERATE_IMAGES_LOCALLY=true pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414/?path=/story/some-path-to-a-real-story 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run -u'",
+    "test-storybook:build-and-run": "pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run'",
+    "test-storybook:generate-images": "cross-env GENERATE_IMAGES_LOCALLY=true pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run -u'",
     "start-playwright-server": "docker run -p 3000:3000 --rm --init -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/sh -c \"cd /home/pwuser && npx -y playwright@1.39.0 run-server --port 3000\"",
   }
 }
diff --git a/README.md b/README.md
@@ -1,4 +1,6 @@
-After [Loki.js](https://loki.js.org/) served us well for some years we finally converted our Visual Regression Testing logic to use the [Storybook Test Runner](https://github.com/storybookjs/test-runner).
+## Intro
+
+After [Loki.js](https://loki.js.org/) served us well for some years we finally converted our **Visual Regression Testing**** logic to use the [Storybook Test Runner](https://github.com/storybookjs/test-runner).
 
 The benefits:
 - it's **way** faster
@@ -10,4 +12,8 @@ But the setup might be non-trivial and there are some rough edges. The biggest d
 
 What I do now in order to generate images locally is a custom "production" build (without minification and optimization and just the stories that I actually want to test visually).
 
-The other tricky part was the Playwright setup. In order to get the same results across machines we want to run Playwright in a Docker container. For various reasons I decided to use the Network API of Playwright. That means I'll _only_ run Playwright in the Docker container, but I keep my running Storybook instance on the host. (At least in the local setup. Within the Gitlab CI we run everything inside Docker.)
+The other tricky part was the Playwright setup. In order to get the same results across machines we want to run Playwright in a Docker container. For various reasons I decided to use the Network API of Playwright. That means I'll _only_ run Playwright in the Docker container, but I keep my running Storybook instance on the host. (At least in the local setup. Within the Gitlab CI we run everything inside Docker.)
+
+## `package.json`
+
+...
diff --git a/package.json b/package.json
@@ -0,0 +1,10 @@
+{
+  "scripts": {
+    "test-storybook:run": "DEBUG_PRINT_LIMIT=0 test-storybook --index-json",
+    "test-storybook:build": "cross-env STORYBOOK_NOT_MINIFIED=true storybook build --quiet --output-dir storybook",
+    "test-storybook:server": "pnpm live-server --port=58414 storybook --no-browser",
+    "test-storybook:build-and-run": "pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414/?path=/story/some-path-to-a-real-story 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run'",
+    "test-storybook:generate-images": "cross-env GENERATE_IMAGES_LOCALLY=true pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414/?path=/story/some-path-to-a-real-story 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run -u'",
+    "start-playwright-server": "docker run -p 3000:3000 --rm --init -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/sh -c \"cd /home/pwuser && npx -y playwright@1.39.0 run-server --port 3000\"",
+  }
+}
diff --git a/README.md b/README.md
@@ -4,4 +4,10 @@ The benefits:
 - it's **way** faster
 - it has better official support
 - it does _more_ (component smoke tests, `play` tests, extensibility for more like a11y tests)
-- it's **way** more stable (actually it looks like there is not a _single_ flaky test right now 🤯)
+- it's **way** more stable (actually it looks like there is not a _single_ flaky test right now 🤯)
+
+But the setup might be non-trivial and there are some rough edges. The biggest downside: while it is way faster it could be even more faster, if we could just use generate images against the Storybook Dev Server. Sadly this was _very_ flaky and error boundaries would always fail (I assume this is because of Reacts unfortunate decision to re-throw errors in _Dev Mode_). See also [this issue](https://github.com/storybookjs/test-runner/issues/218).
+
+What I do now in order to generate images locally is a custom "production" build (without minification and optimization and just the stories that I actually want to test visually).
+
+The other tricky part was the Playwright setup. In order to get the same results across machines we want to run Playwright in a Docker container. For various reasons I decided to use the Network API of Playwright. That means I'll _only_ run Playwright in the Docker container, but I keep my running Storybook instance on the host. (At least in the local setup. Within the Gitlab CI we run everything inside Docker.)
diff --git a/README.md b/README.md
@@ -0,0 +1,7 @@
+After [Loki.js](https://loki.js.org/) served us well for some years we finally converted our Visual Regression Testing logic to use the [Storybook Test Runner](https://github.com/storybookjs/test-runner).
+
+The benefits:
+- it's **way** faster
+- it has better official support
+- it does _more_ (component smoke tests, `play` tests, extensibility for more like a11y tests)
+- it's **way** more stable (actually it looks like there is not a _single_ flaky test right now 🤯)
No results found