Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

A NUL byte inside a CSV cell is dropped by pandas and kept by the web engine

Closed
  • Is something wrong?
#1,025 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
65/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, python
Domain
data

Research direction

Start with parseCSV() in assets/profiler-engine.js and the CSV loading path in faircode/loaders.py, then reproduce with the NUL-byte CSV from the issue using both python3 -m faircode profile and node scripts/engine-js.js profile. Both engines should reject text containing NUL with the suggested --encoding utf-16 message. Done means the new test in tests/test_js_parity_profile_parsing.py passes and the two engines give the same outcome on the repro file.

Written by the indexing model from the issue text.

Description

bug good first issue

Where: assets/profiler-engine.js parseCSV(), faircode/loaders.py, tests/test_js_parity_profile_parsing.py.

Current gap: pandas' C parser truncates a field at an embedded NUL, so b\x00c becomes b; the JS parser keeps the whole string, so the label differs between engines. NULs are rare in real exports but common in corrupted or mis-decoded UTF-16 files, which is the situation where a clear error would help most.

Repro:

$ python3 -c "open('nul.csv','wb').write(b'sex,race\nm,a\nf,b\x00c\nm,a\n')"
$ python3 -m faircode profile nul.csv --json | python3 -c "import json,sys; print([g['label'] for g in json.load(sys.stdin)['dimensions'][1]['groups']])"
['a', 'b']           # node scripts/engine-js.js profile nul.csv gives ['a', 'b\x00c']

Suggested approach: reject text containing NUL in both engines with "the file contains NUL bytes - is it UTF-16? try --encoding utf-16" (which also feeds the mis-decoded-UTF-16 case), and test it.

Dominant language
HTML
Stars
47
Forks
45
Avg merge
3h 15m
Merged PRs (30d)
92

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from yakew7/Fair-Code

All issues in yakew7/Fair-Code

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.