sierra-research

Issues somebody may already be working on are listed last. List everything by date

• docs: evaluation.md reward_basis claims contradict the shipped v1.0.1 task data (retail/telecom), and ACTION is used in telecom

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 82/100

  sierra-research/tau2-bench#635 · Opened Oct 18, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] task_008: clarify unavailable-offer vs unverified-flyer transfer reasons; fixed reference rejects a defensible handoff

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 15/100

  sierra-research/tau2-bench#628 · Opened Oct 16, 2026

  Maintainers usually reply within 1 day

• License for τ²-bench leaderboard trajectories

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 15/100

  sierra-research/tau2-bench#612 · Opened Oct 13, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] task_086: gold liability amounts need a statement date that the task never gives

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 53/100

  sierra-research/tau2-bench#592 · 1 comment · Opened Oct 3, 2026

  Maintainers usually reply within 1 day

• banking_knowledge: with TMPDIR under /dev/shm, the shell tool runs outside the knowledge base

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 55/100

  sierra-research/tau2-bench#587 · Opened Oct 2, 2026

  Maintainers usually reply within 1 day

• Proposal: `loan_servicing` domain with English and French task variants

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 25/100

  sierra-research/tau2-bench#579 · Opened Sep 28, 2026

  Maintainers usually reply within 1 day

• banking_knowledge: `list_discoverable_agent_tools` reveals reference-derived tool membership

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 49/100

  sierra-research/tau2-bench#574 · Opened Sep 27, 2026

  Maintainers usually reply within 1 day

• Committed uv.lock is stale at tag v1.0.1 (records tau2 1.0.0 vs pyproject's 1.0.1), so any uv run dirties a fresh clone

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 75/100

  sierra-research/tau2-bench#560 · Opened Sep 22, 2026

  Maintainers usually reply within 1 day

• ToolCall has no provider_specific_fields: Gemini thought signatures survive only because LiteLLM packs them into the id (observed 4,572-char ids), so a transport change breaks multi-turn tool calling silently

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 54/100

  sierra-research/tau2-bench#559 · Opened Sep 22, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] BM25 and grep ignore document titles

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 54/100

  sierra-research/tau2-bench#545 · Opened Sep 16, 2026

  Maintainers usually reply within 1 day

• Data/provenance request: historical Pass⁴ and missing trials for GLM-5 banking

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 35/100

  sierra-research/tau2-bench#543 · Opened Sep 14, 2026

  Maintainers usually reply within 1 day

• Two telecom/airline tool bodies don't enforce what their docstrings promise, plus a question about `suspend_line`'s `reason` argument

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 35/100

  sierra-research/tau2-bench#541 · Opened Sep 12, 2026

  Maintainers usually reply within 1 day

• Run-to-run noise floor of the published baselines and leaderboard submissions, computed from the released trajectory files

  May be free again @Universeyi claimed this 25 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 35/100

  sierra-research/tau2-bench#540 · 5 comments · Opened Sep 12, 2026

  Maintainers usually reply within 1 day

• banking_knowledge: business-savings opening procedure contradicts account_type contract

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 42/100

  sierra-research/tau2-bench#530 · Opened Sep 10, 2026

  Maintainers usually reply within 1 day

• Pre-registered audit: two documented conventions for pass^k, and a question about which governs published figures

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  sierra-research/tau2-bench#497 · Opened Aug 28, 2026

  Maintainers usually reply within 1 day

• [retail/task 5] Ambiguous confirmation triggers couple Agent reward to hidden User state

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 48/100

  sierra-research/tau2-bench#496 · Opened Aug 28, 2026

  Maintainers usually reply within 1 day

• [airline/task 7] “Other upcoming flights” conflicts with the $1,628 Gold total

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 55/100

  sierra-research/tau2-bench#495 · Opened Aug 28, 2026

  Maintainers usually reply within 1 day

• [airline/task 6] Missing exit condition can induce duplicate bookings

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 49/100

  sierra-research/tau2-bench#494 · Opened Aug 28, 2026

  Maintainers usually reply within 1 day

• [telecom] Fixed persona ages conflict with customer DOB and can induce fabricated identity data

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 75/100

  sierra-research/tau2-bench#492 · Opened Aug 28, 2026

  Maintainers usually reply within 1 day

• EvalPort interop: mapping reward_basis-gated Task/EvaluationCriteria onto a portable TestCase/Grader/Result schema

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  sierra-research/tau2-bench#490 · Opened Aug 25, 2026

  Maintainers usually reply within 1 day

• [Bug] Solo agent cannot run against Gemini: forced function call with no preceding user turn

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 72/100

  sierra-research/tau2-bench#487 · Opened Aug 20, 2026

  Maintainers usually reply within 1 day

• telecom: the tech-support manual tells the agent to call tools it does not have (~92% of conversations); clarifying ownership is worth +34pp

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 55/100

  sierra-research/tau2-bench#472 · 1 comment · 1 reaction · Opened Aug 14, 2026

  Maintainers usually reply within 1 day

• Telecom scores 1.0 for a run that suspends an uninvolved customer's line

  May be free again A pull request for this issue was closed without being merged. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 35/100

  sierra-research/tau2-bench#459 · Opened Aug 7, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] Cash-back dispute correction (doc_004) requires a cash_back_disputes read that no tool provides — task_026/027 undecidable

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 35/100

  sierra-research/tau2-bench#439 · Opened Jul 27, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] Cash-back fixtures violate the documented floor rounding policy (task_017, task_026)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 62/100

  sierra-research/tau2-bench#438 · 1 comment · Opened Jul 27, 2026

  Maintainers usually reply within 1 day

• NL-assertion evaluator returns met: true on non-compliant trajectories (airline tasks 10 & 13)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 48/100

  sierra-research/tau2-bench#436 · Opened Jul 26, 2026

  Maintainers usually reply within 1 day

• EvalScope integration: one-command evaluation support for tau2-bench and tau3-bench

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 1-3 hours Newbie friendliness 72/100

  sierra-research/tau2-bench#435 · Opened Jul 24, 2026

  Maintainers usually reply within 1 day

• [banking_knowledge] task_102: how do we verify the age of the "Ember Analytics" startup?

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 48/100

  sierra-research/tau2-bench#432 · Opened Jul 22, 2026

  Maintainers usually reply within 1 day

• Fix Mei Kovacs known_info to use name instead of user_id in retail tasks

  May be free again A pull request for this issue was closed without being merged. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 35/100

  sierra-research/tau2-bench#337 · 1 comment · Opened May 28, 2026

  Maintainers usually reply within 1 day

• Several retail tasks reward success even when the listed calculate(...) action is absent

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 45/100

  sierra-research/tau2-bench#327 · 1 comment · Opened May 19, 2026

  Maintainers usually reply within 1 day

• Several retail tasks may have an unclear contract around helper read actions and rewarded success

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 1-2 days Newbie friendliness 38/100

  sierra-research/tau2-bench#326 · 1 comment · Opened May 19, 2026

  Maintainers usually reply within 1 day

• Retail task 100 / modify_pending_order_items may be order-sensitive in evaluation

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 1-2 days Newbie friendliness 46/100

  sierra-research/tau2-bench#325 · 1 comment · Opened May 19, 2026

  Maintainers usually reply within 1 day

• Retail task 105 / T105: infeasible gold exchange is still rewarded as success

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 48/100

  sierra-research/tau2-bench#321 · 1 comment · Opened May 17, 2026

  Maintainers usually reply within 1 day

• Retail task 50 / T50: missing required human handoff is still rewarded as success

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 1-2 days Newbie friendliness 48/100

  sierra-research/tau2-bench#320 · 1 comment · Opened May 17, 2026

  Maintainers usually reply within 1 day

• Retail task 41 / T41: DB oracle hardcodes PayPal even though the user selected credit_card_9503061

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 47/100

  sierra-research/tau2-bench#319 · 1 comment · Opened May 17, 2026

  Maintainers usually reply within 1 day

• Retail Task 10 DB oracle appears to omit required return side effects

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 55/100

  sierra-research/tau2-bench#316 · 2 comments · Opened May 17, 2026

  Maintainers usually reply within 1 day

• Procedural compliance gap: outcome-based reward passes trajectories that violate the documented confirm-before-mutation policy (proposal: src/experiments/llmcontract/)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 1-2 days Newbie friendliness 48/100

  sierra-research/tau2-bench#298 · Opened May 5, 2026

  Maintainers usually reply within 1 day

• Full duplex audio saved to both.wav is clipped at the end

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 54/100

  sierra-research/tau2-bench#241 · Opened Apr 16, 2026

  Maintainers usually reply within 1 day

• Telecom MMS ticket text missing data refuel willingness hint

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 70/100

  sierra-research/tau2-bench#237 · 2 comments · Opened Apr 15, 2026

  Maintainers usually reply within 1 day

• [IMPORTANT] Airline domain may NOT be a reliable benchmark for transactional agent capability

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 35/100

  sierra-research/tau2-bench#224 · 5 comments · Opened Apr 9, 2026

  Maintainers usually reply within 1 day

• dev/tau3 Adding NL_ASSERTIONS during retail evaluation resulted in a large number of retry attempts and exceeding the retry limit.

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 1-2 days Newbie friendliness 35/100

  sierra-research/tau2-bench#211 · 1 comment · Opened Apr 2, 2026

  Maintainers usually reply within 1 day

• [Bug] User simulator ends conversation too early when the agent is requesting confirmation (domain-retail)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 1-3 hours Newbie friendliness 55/100

  sierra-research/tau2-bench#182 · 1 reaction · Opened Mar 20, 2026

  Maintainers usually reply within 1 day

• [Bug Report] Task 35 airline: User simulator introduces extra passenger not in task instructions

  May be free again A pull request for this issue was closed without being merged. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 35/100

  sierra-research/tau2-bench#172 · 1 comment · Opened Mar 9, 2026

  Maintainers usually reply within 1 day

• Measuring agent robustness to corrupted tool data — AVER framework

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  sierra-research/tau2-bench#167 · 1 comment · 1 reaction · Opened Feb 21, 2026

  Maintainers usually reply within 1 day

• Tau2 Task 9 Rewards Success for Scenario That Should Be Impossible per Policy

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 48/100

  sierra-research/tau2-bench#156 · 2 reactions · Opened Feb 5, 2026

  Maintainers usually reply within 1 day

• Evaluate standalone agents

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 25/100

  sierra-research/tau2-bench#150 · 2 reactions · Opened Jan 28, 2026

  Maintainers usually reply within 1 day

• RFC #2: Task Semantics & Evaluation for Tau3

  May be free again @victorbarres claimed this 285 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  design rfc

  sierra-research/tau2-bench#129 · 1 assignee · Opened Jan 5, 2026

  Maintainers usually reply within 1 day

• Add a new Healthcare domain for medical patient support interactions

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  enhancement good first issue

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  sierra-research/tau2-bench#127 · Opened Dec 30, 2025

  Maintainers usually reply within 1 day

• Addition of a structured issue template and internal issue tracking

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 55/100

  sierra-research/tau2-bench#123 · Opened Dec 26, 2025

  Maintainers usually reply within 1 day

• Retail domain evaluation crash

  May be free again A pull request for this issue was closed without being merged. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  enhancement question

  Difficulty 4/5 3-5 days Newbie friendliness 35/100

  sierra-research/tau2-bench#115 · Opened Dec 16, 2025

  Maintainers usually reply within 1 day