Round 3 control (c): a window-4 observer does not lose 0.05 BA on converged on the calibration data (#1758 R3.0-I)

Aberta
  • Algo errado?
#2,327 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
8/100
Tipo de issue
Bug
Clareza
Precisa de esclarecimento
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
testing-qa

Direção de pesquisa

Start with tools/observer_oracle/judge.py on branch r30i and the docs/PREDICATES.md section "How Round 3 is judged", then read control_c_hcal.log and judge_calibration.log. Row f01_0 at step 10 shows the window-4 build certifying convergence earlier than INC, which is the mechanism behind the lost BA margin. Work cannot start until a maintainer rules on #1758 whether control (c) is restated or kept as registered; done means that ruling is recorded as a Deviation and control (c) is re-run and posted on #1758.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

area:observer kind:decision

The pre-registration's positive control (c) on #1758 says a planted build with OBSERVER_WINDOW_N = 4 must lose at least 0.05 BA on converged against INC. Measured with the R3.0-I instrument (tools/observer_oracle/judge.py, branch r30i), the window-4 build gains BA instead:

  • on H-cal, BA(INC) − BA(W4) = −0.0591 [−0.1354, +0.0074];
  • on corpus.json ∪ H-cal, −0.0305 [−0.0847, +0.0173];
  • on corpus.json alone, +0.0081 [−0.0600, +0.0722].

The instrument reads correctly. A direct run of H-cal row f01_0 @10 prints CK 10 false on INC and CK 10 true on W4. The label is true: x_10 is 5e-5 from the limit, eps is 5.7e-3, and INC's 11-value window still holds x_0 = 8.7. A shorter window certifies a settled geometric tail sooner, so it trades FPR for TPR: converged FPR is 0.0217 against 0 on H-cal. Rules 2 and 3 still FAIL W4 on H-cal, but the 0.05 BA margin named by (c) does not hold.

The logs are control_c_hcal.log and judge_calibration.log in the R3.0-I outputs.

Done when

  • A ruling on #1758 decides whether (c) is re-stated, for example as an FPR or decision-table criterion, or kept as registered and judged on H1 only.
  • The ruling is recorded as a Deviation in docs/PREDICATES.md "How Round 3 is judged" before the reveal, if it changes the text.
  • R3.0-I's control (c) is re-run under the ruling and its output is posted on #1758.

— Claude (builder, Claude Code)

Linguagem predominante
C
Estrelas
3
Forks
7
Merge médio
6h 32min
PRs com merge (30d)
285

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de InauguralSystems/EigenScript

Todas as issues de InauguralSystems/EigenScript

Issues semelhantes

Mais issues de C

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.