Recipe call arguments that name a data-driven column are not expanded, and the result depends on map order

Open
#886 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
testing-qa

Research direction

Start in internal/runner/executor.go at runRecipeCall and resolve, then read the Data-driven page mentioned in the issue. Reproduce the issue with an Executor unit test using row values in e.vars and recipe arguments containing <email> and <password>. Done when recipe arguments are either expanded consistently before binding or the lint and documentation rule is made explicit, with behavior independent of map order.

Written by the indexing model from the issue text.

Description

What happens

A recipe called from a data-driven test cannot reliably receive the row's columns as call arguments.

recipe "sign in as" (email, password)
  type "<email>" into #email
  type "<password>" into #password

test "role-based access"
  sign in as "<email>" with "<password>"

with examples:
  email               password
  "[email protected]"    "admin123"

runRecipeCall binds each argument's text as it is (internal/runner/executor.go, the loop that calls setVar(param, args[i], ...)), so the parameter email is set to the text <email>. That replaces the row's email, and resolve then finds a variable whose value is its own placeholder and makes no progress: the step types <email> literally.

With parameter names that differ from the columns (recipe "sign in as" (address, secret)), the call binds address to the text <email>. resolve makes one pass over the variables in map order, so whether <address> becomes <email> and then the row's value depends on the order the pass visits address and email.

Measured

In a unit test on Executor (newTestExecutor, the row values set in e.vars, runRecipeCall with the arguments "<email>" and "<password>", then e.resolve("type <address> then <secret>")), 300 runs gave four different results:

With the parameters named like the columns, the result is type <email> every time.

Expected

Either the arguments of a recipe call are expanded (against the row and the other variables) before they are bound, or probe lint and the docs say that a column must not be passed as an argument. The Data-driven page now says the latter, and shows a recipe without parameters that reads the columns directly; that form does not depend on the order.

Found while checking the recipe "log in as" example, which also had a keyword problem (fixed separately).

Dominant language
Go
Stars
6
Forks
3
Avg merge
6h 28m
Merged PRs (30d)
362

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AlphaWaveSystems/flutter-probe

All issues in AlphaWaveSystems/flutter-probe

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.