Parquet captions: skip null or empty rows instead of refusing to start, and never train on "nan"

Open
#3,341 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
numpy, pandas, python

Research direction

Read simpletuner/helpers/metadata/backends/parquet.py around check_column_values and _extract_captions_to_fast_list, then inspect simpletuner/helpers/prompts.py around _caption_payload_values and the existing tests for those paths. Verify the proposed null, empty, NaN, and fallback cases, including that rows without usable captions are dropped; update the Parquet section of DATALOADER.md and its translations.

Written by the indexing model from the issue text.

Description

bug

Summary

A Parquet caption column with null or empty cells should skip those rows rather than stop training at startup. A NaN must never become the caption text "nan".

Checked on refactor/provider-transform-cache (931460c57).

Current behaviour

  1. Startup refusal. check_column_values (simpletuner/helpers/metadata/backends/parquet.py:35-67, called at :170) raises when any caption cell is null or empty, unless fallback_caption_column is set:

    • "contains null values in the 'caption' column"
    • "contains empty strings …"
    • "contains null arrays …"
    • "null values within arrays …"

    One bad row in a large caption file stops the run. This is the same on main.

  2. Extraction already skips. Past that check, _extract_captions_to_fast_list (parquet.py:212, clean()) turns null and NaN into no caption, and the sample is dropped with the other missing-caption samples (f718362d7). So the refusal is the only thing blocking skip behaviour.

  3. NaN text. PromptHandler._caption_payload_values (simpletuner/helpers/prompts.py:272-288) stringifies anything it doesn't recognise.

    Probe:

    • float("nan") → ["nan"]
    • np.float64("nan") → ["nan"]
    • ["a", None, np.nan] → ["a", "nan"]
    • pd.NA → ["<NA>"]

    On a Webshart Parquet dataset, null string cells come back as None and are skipped (probe on a real file). A float-typed caption column with NaN still came back from get_caption as 'nan'.

Proposed fix

  • check_column_values keeps the refusal for a column with no usable value at all. It stops refusing individual null, empty or array-with-null rows, and logs how many rows have no caption instead.
  • _caption_payload_values treats NaN and pd.NA scalars as no value.
  • Tests:
    • a Parquet file with a null caption row, a NaN row, an empty row and a list with a null element trains on the remaining rows and skips the rest;
    • a float NaN through the Webshart path yields no caption;
    • fallback_caption_column still applies when the primary cell is null, and a row where both are null is skipped.
  • Docs: the DATALOADER.md Parquet section and its translations.

Related commits on refactor/fable-5.2-datasets

These need adapting to the provider branch's clean() and missing-caption removal:

  • a534a1bcf: treat a null Parquet or JSON caption cell as no caption (_is_null in parquet.py). The provider's clean() already covers this; only the tests may be worth porting.
  • e0aa9609d: remove indexed Parquet samples whose caption row is gone (provider equivalent: f718362d7).
  • f0d250784: remove samples without a caption from every metadata backend.
Dominant language
Python
Stars
2.9k
Forks
295
Avg merge
3h 28m
Merged PRs (30d)
42

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bghira/SimpleTuner

All issues in bghira/SimpleTuner

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.