Q57: live MeasuringWorth reference differs from fixed answer key

Aperta
  • Qualcosa non va?
#58 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
32/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
jupyter-notebook
Ambito
data, documentation

Direzione di ricerca

Start from officeqa_pro_v2.csv, question ID 57's source_docs metadata and the MeasuringWorth comparator URL it cites; compare the live Relative Labor Earnings output against the stored expected answer. Read databricks/officeqa#47 for the related convention debate. Done means the metadata pins a dated reference snapshot or an explicit conversion factor so the key reproduces within 1%, plus a reply to the reporter requesting the private reproduction.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Dataset and question

Dataset: databricks/officeqa-pro-v2
File: officeqa_pro_v2.csv
Question ID: 57

Reproducibility discrepancy

On October 6, 2026, I checked the external MeasuringWorth compensation comparator referenced in this question’s source_docs metadata.

Using the nominal amount from the cited Treasury document, the live calculator’s Relative Labor Earnings result differs from the dataset’s expected answer by approximately 2.06%.

The calculator labels its output “today,” while the question specifies 2026 dollars. The reference URL does not pin a target year or source vintage.

I have not established whether this reflects a data revision, a different reference vintage, or ambiguity in the target-year interpretation. However, the current reference does not reproduce the fixed answer key, even within a 1% tolerance.

Requested clarification

Could you clarify:

  • Which source vintage and conversion factor were used to generate the answer?
  • How “2026 dollars” should be interpreted for this comparator?
  • Whether a dated reference snapshot or explicit conversion factor can be provided to make the question reproducible?

This appears related to the answer-revision and convention-clarification concerns raised in databricks/officeqa#47, although this report concerns Pro V2.

Following the approach in that issue, I am withholding the full question, answer values, and detailed calculation from this public report to avoid benchmark contamination. I can share the exact reproduction privately with the maintainers.

Lingua principale
Jupyter Notebook
Stelle
205
Fork
17
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di databricks/officeqa

Tutte le issue di databricks/officeqa

Issue simili

Altre issue su Data Engineering

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.