Parquet captions: skip null or empty rows instead of refusing to start, and never train on "nan"
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- backend, data, machine-learning
Research direction
Read simpletuner/helpers/metadata/backends/parquet.py around check_column_values and _extract_captions_to_fast_list, then inspect simpletuner/helpers/prompts.py around _caption_payload_values and the existing tests for those paths. Verify the proposed null, empty, NaN, and fallback cases, including that rows without usable captions are dropped; update the Parquet section of DATALOADER.md and its translations.
Written by the indexing model from the issue text.
Description
Summary
A Parquet caption column with null or empty cells should skip those rows rather than stop training at startup. A NaN must never become the caption text "nan".
Checked on refactor/provider-transform-cache (931460c57).
Current behaviour
-
Startup refusal.
check_column_values(simpletuner/helpers/metadata/backends/parquet.py:35-67, called at:170) raises when any caption cell is null or empty, unlessfallback_caption_columnis set:- "contains null values in the 'caption' column"
- "contains empty strings …"
- "contains null arrays …"
- "null values within arrays …"
One bad row in a large caption file stops the run. This is the same on
main. -
Extraction already skips. Past that check,
_extract_captions_to_fast_list(parquet.py:212,clean()) turns null and NaN into no caption, and the sample is dropped with the other missing-caption samples (f718362d7). So the refusal is the only thing blocking skip behaviour. -
NaN text.
PromptHandler._caption_payload_values(simpletuner/helpers/prompts.py:272-288) stringifies anything it doesn't recognise.Probe:
float("nan")→["nan"]np.float64("nan")→["nan"]["a", None, np.nan]→["a", "nan"]pd.NA→["<NA>"]
On a Webshart Parquet dataset, null string cells come back as
Noneand are skipped (probe on a real file). A float-typed caption column with NaN still came back fromget_captionas'nan'.
Proposed fix
check_column_valueskeeps the refusal for a column with no usable value at all. It stops refusing individual null, empty or array-with-null rows, and logs how many rows have no caption instead._caption_payload_valuestreats NaN andpd.NAscalars as no value.- Tests:
- a Parquet file with a null caption row, a NaN row, an empty row and a list with a null element trains on the remaining rows and skips the rest;
- a float NaN through the Webshart path yields no caption;
fallback_caption_columnstill applies when the primary cell is null, and a row where both are null is skipped.
- Docs: the
DATALOADER.mdParquet section and its translations.
Related commits on refactor/fable-5.2-datasets
These need adapting to the provider branch's clean() and missing-caption removal:
- a534a1bcf: treat a null Parquet or JSON caption cell as no caption (
_is_nullinparquet.py). The provider'sclean()already covers this; only the tests may be worth porting. - e0aa9609d: remove indexed Parquet samples whose caption row is gone (provider equivalent:
f718362d7). - f0d250784: remove samples without a caption from every metadata backend.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 295
- Avg merge
- 3h 28m
- Merged PRs (30d)
- 42
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bghira/SimpleTuner
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bghira/SimpleTuner#3345 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 57/100
bghira/SimpleTuner#3346 ·
Maintainers usually reply within 1 day
-
Epoch checkpoints can't be resumed once the next epoch has recropped (provider-transform-cache)Openbug
Difficulty 5/5 Over a week Newbie friendliness 35/100
bghira/SimpleTuner#3343 ·
Maintainers usually reply within 1 day
-
bug webui
Difficulty 3/5 1-2 days Newbie friendliness 72/100
bghira/SimpleTuner#3342 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
bghira/SimpleTuner#3340 ·
Maintainers usually reply within 1 day
All issues in bghira/SimpleTuner
Similar issues
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
cubrid-lab/sqlalchemy-cubrid#1397 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
cubrid-lab/pycubrid#1307 ·
Maintainers usually reply within 1 day
-
[Bug]: Installer hides SSL certificate errors as bare URLError, and the suggested fix can't workOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ShaftHQ/chaosengine-harness#143 ·
Maintainers usually reply within 1 day
-
LLMContextNode tool builders fail on nested params like `params:retrieval.max_matches`Possibly taken @Shizoqua claimed this today. OpenCommunity
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
btclib-org/portanode#985 ·
Maintainers usually reply within 1 day