Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Testing Center custom `string_comparison` evaluation errors on the JSONPath the server itself generates from `$.generatedData.outcome`

Open
  • Is something wrong?
#3,636 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Domain
cli, testing-qa

Research direction

No source files or tests are named. Start by reproducing the custom string_comparison evaluation with the documented $.generatedData.outcome reference and inspect how the server rewrites and parses that path. Done means the reference evaluates successfully, maps to a parseable path, or is rejected at creation with a clear error.

Written by the indexing model from the issue text.

Description

area:afdx investigating owned by another team validated

CLI Version:
@salesforce/cli/2.149.9

Architecture:
win32-arm64

Node Version:
node-v24.14.1

Plugin Version:
@oclif/plugin-autocomplete 3.2.56 (core)
@oclif/plugin-commands 4.1.63 (core)
@oclif/plugin-help 6.2.58 (core)
@oclif/plugin-not-found 3.2.93 (core)
@oclif/plugin-plugins 5.4.87 (core)
@oclif/plugin-search 1.2.54 (core)
@oclif/plugin-update 4.7.59 (core)
@oclif/plugin-version 2.2.57 (core)
@oclif/plugin-warn-if-update-available 3.1.73 (core)
@oclif/plugin-which 3.2.61 (core)
@salesforce/cli 2.149.9 (core)
agent 2.0.5 (user)
apex 4.1.0 (core)
api 2.0.9 (core)
auth 5.0.6 (core)
code-analyzer 5.12.0 (user)
data 5.1.5 (core)
deploy-retrieve 4.1.2 (core)
info 4.0.9 (core)
limits 4.0.3 (core)
marketplace 2.0.5 (core)
org 6.0.9 (core)
packaging 3.0.5 (core)
schema 4.0.5 (core)
settings 3.0.5 (core)
sobject 2.0.5 (core)
telemetry 4.0.5 (core)
templates 57.0.9 (core)
trust 4.0.9 (core)
user 5.0.1 (core)

OS and Version:
Windows_NT 10.0.26200

Shell:
powershell

Summary

A custom evaluation asserting on the agent's response text is unusable: the
server rewrites the documented $.generatedData.outcome reference into a
filter-expression JSONPath and then fails to parse its own rewrite.

Test case YAML (per the documented shape):

customEvaluations:
  - label: "response contains estimat"
    name: string_comparison
    parameters:
      - name: operator
        value: contains
        isReference: false
      - name: actual
        value: "$.generatedData.outcome"
        isReference: true
      - name: expected
        value: "estimat"
        isReference: false
Actual result

The evaluation returns status: ERROR on every case:

"errorMessage": "Error parsing JSONPath
'$.outputs[?(@.type == 'general.echo')].payload.planner_response.lastExecution.outcome'"

i.e. the server maps $.generatedData.outcome to a path containing a
filter expression [?(@.type == '...')], and the JSONPath implementation
evaluating it rejects filter syntax. The author's input is well-formed; the
failing path is generated internally.

Expected result

Either the generated path parses, or $.generatedData.outcome maps to a
filter-free path, or the reference is rejected at test-create time with a
clear message. An ERROR status at run time on the documented reference makes
deterministic response-text assertions impossible in Testing Center.

Why it matters

The LLM-judged expectedOutcome is lenient (see reproduction context in the
related issues: an agent whose instruction clause was removed still passed a
judge rubric that named the missing behavior). string_comparison is the
platform's only deterministic assertion over response text — precisely the
tool needed when the judge is too soft — and it errors as documented.

Environment
  • sf CLI: <paste sf version --verbose>
  • Org: Agentforce Developer Edition (orgfarm-*.develop.my.salesforce.com)
Dominant language
No language data
Stars
571
Forks
80
Avg merge
2d 21h
Merged PRs (30d)
3

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from forcedotcom/cli

All issues in forcedotcom/cli

Similar issues

More CLI issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.