Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Cascade routing mode: user-selected primary model with verifier-gated frontier escalation

Đang mở
  • Có gì không ổn?
#1,134 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
typescript
Lĩnh vực
ai-infra-agents, cli

Hướng nghiên cứu

Bắt đầu với các subtasks #1138, #1135, #1136 và #1137, sau đó lần theo các điểm vào hiện có của config, flags và lựa chọn model tương tác. Kiểm tra cách verifier stack báo cáo các kết quả của build, data-test và schema-contract. Hoàn tất có nghĩa là người dùng có thể chọn cả hai model, việc xác minh thất bại được chuyển cấp kèm ngữ cảnh, và các kết quả định tuyến session và aggregate được báo cáo.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

Add a cascade routing mode: sessions run on a user-chosen primary model (typically a local/self-hosted or inexpensive model) and automatically escalate to a user-chosen frontier model when deterministic verification fails. Users pick BOTH models — this is a first-class routing option, not a hardcoded config.

Motivation

  • Local/inexpensive models complete a meaningful share of data-engineering tasks correctly; the problem is knowing which ones. altimate-code's deterministic verifier stack (build success, data tests, schema-contract checks) can answer that without ground truth.
  • A verifier-gated cascade delivers near-frontier effective quality at a fraction of the cost — while making the cost/latency trade visible to the user instead of implicit.
  • Human time matters more than tokens: the design must escalate FAST (default: one primary attempt → gate → escalate) and route around the primary entirely for task classes it historically fails.

Approach

  1. Model selection is user-driven: primary and escalation are both picked by the user (any configured provider/model pair) via config, flags, and interactive selection. No baked-in model names.
  2. Escalation trigger is deterministic verification, not model self-assessment: the gate runs the same checks in both lanes.
  3. Fast defaults: 1 primary attempt in interactive sessions; batch/unattended mode may raise attempts.
  4. Measurement built in from day one: per-session routing summary + aggregate reporting (escalation rate, gate outcomes, latency and cost per lane).
  5. Learn from mistakes: outcomes feed a local routing memory so task classes that consistently escalate skip the primary next time.

Subtasks

  • #1138 Cascade routing core: two-model selection + escalate-on-gate-failure flow
  • #1135 Verification gate: pluggable deterministic checks as the escalation trigger
  • #1136 Measurement: routing telemetry, per-session summary, aggregate report
  • #1137 Adaptive routing memory: learn task-class priors from outcomes

Non-goals (v1)

  • No learned/ML router — priors are simple aggregates.
  • No mid-task model switching within a single attempt; escalation restarts the task on the escalation model with context.

Prior art & evidence (researched 2026-08-24)

A four-lane research pass (papers, vendor first-party writing, harness survey, verifier-gating literature) backs this design. Full 18-system prior-art map lives in the internal brief; key facts:

Novelty claim, stated precisely: execution-based candidate selection is mature (CodeT, AlphaCode, CHASE-SQL), and pre-generation model routing is mature (FrugalGPT, RouteLLM, Cursor Router, GPT-5's router, OpenRouter Auto). But a survey of 10 shipped harnesses/routers (Aider, Cline, Roo-Code, Cursor, OpenRouter, LiteLLM, Copilot, Continue.dev, RA-Aid, gptme/goose) found zero that escalate across model tiers on a deterministic execution signal (build/tests/contract). Every shipped "smart" router classifies before generation. Verifier-gated cross-tier escalation is unshipped territory.

Numbers that justify the pattern:

  • FrugalGPT: try-then-escalate cascade matches best-single-LLM accuracy at up to 98% cost reduction (arxiv.org/abs/2305.05176)
  • RouteLLM: >2× cost cut, no quality loss; only 13.4% of queries need the strong model for half the quality gap (arxiv.org/abs/2406.18665)
  • CodeT: execution-agreement selection lifts HumanEval pass@1 47.0%→65.8% (arxiv.org/abs/2207.10397)
  • AlphaCode: execution-test filtering rejects ~99% of samples and is what makes cheap over-sampling viable at all (deepmind AlphaCode paper)
  • Query and Conquer: execution-guided self-consistency helps weakest models most (+15.9pts for an 8B vs +3.5pts for a 70B) — directly supports gating a cheap/local primary (arxiv.org/html/2503.24364)
  • Cursor Router A/B: 60% cost saving vs always-frontier (cursor.com/blog/router) — but gated on a classifier, not execution
  • Anthropic (Building Effective Agents; Agent SDK subagents) and OpenAI (GPT-5 router, model-selection guide) both document tiered-model routing as the recommended pattern; both trigger pre-generation

Evidence-driven design adjustments (applied to subtasks):

  1. Primary-model attempts default: NOT a strict single shot. Execution-checked multi-sample (2-3 attempts, agreement-gated) disproportionately benefits small/local primaries (Query and Conquer) — use n=2-3 in unattended mode; n=1 remains the interactive default for latency.
  2. Gate quality is an ongoing investment, not a checkbox: CHASE-SQL's execution selector still leaves ~40-70% of oracle headroom, and TestPrune shows better tests add +2.4 to +12.9pts on an already-gated pipeline. The gate interface must make adding/refining checks cheap.
  3. Manual model-pair selection is the right v1 (matches Aider/Cline/Continue UX norms); design the config so a learned pre-classifier can be layered later without breaking it — but do NOT ship a learned router early: RouteLLM shows routers are near-random when undertrained, which is exactly the trap for our low-volume early data.

Risks from the literature: router overconfidence on thin data (RouteLLM); verifier gaming by weak self-generated checks (CodeT caveats) — our gate uses project-declared tests, not model-generated ones, which sidesteps the worst of this; latency stacking on serial escalation (mitigated by fast-fail gates and the adaptive skip-primary memory).

Ngôn ngữ chính
TypeScript
Star
805
Fork
122
Merge trung bình
1 ngày 21 giờ
Pull request đã merge (30 ngày)
56

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của AltimateAI/altimate-code

Tất cả issue của AltimateAI/altimate-code

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.