[Dynamic configuration] [Playwright] Can't reconfigure configuration per scenario

dynamic-configuration/override-config.e2e-test.js で再現を実行し、各 Scenario の .config() 呼び出しを、動作する Feature('Dynamic Configuration').config() のケースと比較します。Scenario の設定が Playwright ヘルパーに到達する経路を追跡し、その後、ブラウザーの override が各 Scenario で有効になることを確認します。同時に、コマンドラインの --override の動作は変更されないことも確認します。

What are you trying to achieve?

I'm trying to reconfigure browser configuration per scenario

What do you get instead?

Provide console output if related. Use --verbose mode for more details.

CodeceptJS v3.0.4
Helpers: Playwright
Plugins: screenshotOnFail, retryFailedStep, tryTo

Dynamic Configuration --
    [1]  Starting recording promises
 › [Session] Starting singleton browser session
  should override browser in config per scenario try#1 @dynamic-configuration
 › [Test Config] Playwright {"browser":"firefox"}
    I am on page "/"
    › [Browser:Warning] 
    I execute script () => !!window.chrome
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
  ✔ OK in 3224ms

 › [Test Config] Reverted for Playwright
 › [Session] cleaning cookies and localStorage
 › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
  should override browser in config per scenario try#2 @dynamic-configuration
 › [Test Config] Playwright {"Playwright":{"browser":"firefox"}}
    I am on page "/"
    › [Browser:Warning] 
    I execute script () => !!window.chrome
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
  ✔ OK in 611ms

 › [Test Config] Reverted for Playwright
 › [Session] cleaning cookies and localStorage
 › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
  should override browser in config per scenario try#3 @dynamic-configuration
 › [Test Config] Playwright {"helpers":{"Playwright":{"browser":"webkit"}}}
    I am on page "/"
    › [Browser:Warning] 
    I execute script () => !!window.chrome
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
    › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@
  ✔ OK in 552ms

 › [Test Config] Reverted for Playwright
 › [Session] cleaning cookies and localStorage
 › [Browser:Warning] JSHandle@THREE.WebGLProgram: gl.getProgramInfoLog() JSHandle@

  OK  | 3 passed   // 5s

Provide test source code if related

const assert = require('assert');

Feature('Dynamic Configuration');

Scenario('should override browser in config per scenario try#1 @dynamic-configuration', async ({ I }) => {
    I.amOnPage('/');
    const isChrome = await I.executeScript(() => !!window.chrome);

    assert(isChrome, 'should check that browser is chrome');
}).config({"browser": "firefox"});

Scenario('should override browser in config per scenario try#2 @dynamic-configuration', async ({ I }) => {
    I.amOnPage('/');
    const isChrome = await I.executeScript(() => !!window.chrome);

    assert(isChrome, 'should check that browser is chrome');
}).config({"Playwright": {"browser": "firefox"}});

Scenario('should override browser in config per scenario try#3 @dynamic-configuration', async ({ I }) => {
    I.amOnPage('/');
    const isChrome = await I.executeScript(() => !!window.chrome);

    assert(isChrome, 'should check that browser is chrome');
}).config({ "helpers": {"Playwright": {"browser": "webkit"}}});

Details

• CodeceptJS version: v3.0.4
• NodeJS Version: v12.18.3
• Operating System: macOS Catalina 10.15.7
• playwright
• Configuration file:

const { setHeadlessWhen } = require('@codeceptjs/configure');

// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

exports.config = {
  tests: '*/*.e2e-test.js',
  output: './output',
  helpers: {
    Playwright: {
      url: 'http://github.com',
      show: true,
      browser: 'chromium'
    }
  },
  include: {
    I: './steps_file.js'
  },
  bootstrap: null,
  mocha: {},
  name: 'experiments-with-codeceptjs',
  plugins: {
    pauseOnFail: {},
    retryFailedStep: {
      enabled: true
    },
    tryTo: {
      enabled: true
    },
    screenshotOnFail: {
      enabled: true
    }
  }
}

NOTE:

But it works for the Feature

Feature('Dynamic Configuration').config({"browser": "firefox"})

and I can override the configuration in runtime by using --override or -o option in the command line, passing valid JSON as a value:

codeceptjs run -o '{ "helpers": {"Playwright": {"browser": "firefox"}}}'

Also here is the link to reproduce the issue:
https://github.com/elaichenkov/experiments-with-codeceptjs/blob/main/dynamic-configuration/override-config.e2e-test.js