Translation fidelity along the traceability map: compare each COBOL paragraph with the Java method it became

Open
  • Is something wrong?
#7,532 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java, python
Domain
tooling

Research direction

Start from traceability.json produced by the forge/det-port, which maps each generated Java method to its COBOL source line and fact, and from the existing structural fingerprint tooling (sfs) the issue contrasts against. The new tool goes under tests/tools/; first check how traceability entries link paragraphs to methods. Done means a per-paragraph structural comparison with shuffled and unrelated controls, plus a run on the burned-estate ports in both styles.

Written by the indexing model from the issue text.

Description

enhancement legacy-modernization

Why

A structural-fidelity analysis of the det-port on the burned estates (2026-11-02; ~/gitgalaxy/cics-fidelity/ on nyx) found two things:

  • The translations are complete: 184 cases, 47,717 statements, 0 holes.
  • They keep each estate's relative structure: the between-program distance geometry is preserved (Mantel ρ 0.45-0.47, p = 0.001), and per-program function count and size correlate 0.82-0.86.

But a global fingerprint (sfs, portable or structure lens) cannot pick out a port's own COBOL source: 1-3 of 124 programs, against 0.8 % chance. A plain size matcher does better. The reason is that det-port Java is mechanical by design: byte storage, a runtime, one call per statement, dispatch methods. Every port has the same shape, a median of 32-36 Java methods against 7 COBOL paragraphs, and that shared style swamps each program's individual signal.

So whole-program fingerprints are the wrong instrument for translation fidelity. The forge and det-port already know the mapping: every generated class and method is traced to its source line and fact (traceability.json). Fidelity should be measured along that map.

Proposal

For each translated program, pair every COBOL paragraph or section with the Java method(s) traced to it, and compare their structure:

  • branch outcomes (IF/EVALUATE/PERFORM UNTIL against if/switch/loops);
  • calls out (PERFORM / CALL / LINK against invocations);
  • size, and the reads/writes of data items against field access.

Report per program, per paragraph and per estate:

  • the share of paragraphs whose structure survives within a tolerance;
  • the outliers;
  • an estate-level fidelity score.

Controls: shuffled paragraph-method pairs, and pairs from unrelated programs, so a fidelity score comes with its chance level.

Where it fits

  • The architecture roadmap's "cross-language blueprint fidelity" item.
  • A structural layer beside the equivalence proofs. Proofs say the Java does what the COBOL did; this would say the Java is shaped like the COBOL, paragraph by paragraph, which is what a reviewer reads.
  • It is measurement only: no new cases or proofs on the burned estates (that evidence work is melek's lane).

Acceptance

  • A tool (tests/tools/…) that reads a port plus its traceability.json and writes a per-paragraph structural comparison.
  • Shuffled and unrelated controls, with the score reported against them.
  • A run over the burned-estate det-ports in both styles (default, --style structured --typed --groups), with the headline numbers posted here.
Dominant language
Python
Stars
57
Forks
11
Avg merge
1h 35m
Merged PRs (30d)
975

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from squid-protocol/gitgalaxy

All issues in squid-protocol/gitgalaxy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.