Phenotype node genes grid returns NaN in JSON (invalid) when a gene matches HPO but not MONDO or vice versa - pandas 3 iterrows re-infers None to NaN

Open Beginner friendly
  • Is something wrong?
#2,684 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
pandas, python
Domain
backend, testing

Research direction

Read snpdb/views/datatable_dataframe.py and inspect DataFrameTableView._rows(); the issue identifies iterrows() as the source of the NaN values under pandas 3. First check how rows are serialized, then add a regression test that serializes a frame with a missing cell and verifies strict JSON parsing with json.loads and parse_constant raising. Done when missing cells remain valid JSON nulls in the response.

Written by the indexing model from the issue text.

Description

🤖 Written by Claude

Reported from testing; root cause reproduced on a developer box.

Problem

The Phenotype node (and anything else served by DataFrameTableView) returns invalid JSON when the DataFrame has holes - a gene matched by an HPO term but not a MONDO term, or vice versa. The response contains bare NaN tokens, which JSON.parse rejects, so the node's genes grid fails with a JS error:

{"c0": "ELP4", "c1": "HP:0000526 Aniridia", "c2": NaN}, {"c0": "FOXC1", "c1": NaN, "c2": "MONDO:0019172 aniridia"}

Seen on analysis/<id>/node/ontology/genes/datatable/<node_id>/<version>/ for a Phenotype node reading its terms from a Sample node's patient, but any ontology genes grid with mixed HPO/OMIM/MONDO coverage will do it.

Cause

DataFrameTableView._dataframe() runs df_nan_to_none() (library/pandas_utils.py), which does work - the cells are None afterwards. But _rows() then walks the frame with df.iterrows(), and under pandas 3 each row Series is re-inferred as the new str dtype, which has no None: the missing values come back as NaN. itertuples() and to_dict(orient="records") both keep None, so the fix is to iterate one of those (or to_dict) in _rows() instead of iterrows(), in snpdb/views/datatable_dataframe.py.

Minimal reproduction:

df = pd.DataFrame.from_dict({'ELP4': {'HPO': 'x'}, 'FOXC1': {'MONDO': 'y'}}, orient='index')
a = df.astype(object).where(pd.notna(df), None)
a.iloc[0, 1]                              # None
next(a.iterrows())[1]['MONDO']            # nan  <- str-dtype row Series
next(a.itertuples())[2]                   # None

Also worth a test on DataFrameTableView that serialises a frame with a missing cell and asserts the response parses as strict JSON (json.loads with parse_constant raising).

Dominant language
Python
Stars
31
Forks
4
Avg merge
7h 20m
Merged PRs (30d)
347

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SACGF/variantgrid

All issues in SACGF/variantgrid

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.