QuiverInsiderTrading readers throw (fatal) on rows in a legacy column layout; validate column count and return null
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- csharp
- Domain
- data-engineering
Research direction
Start with QuiverInsiderTradingUniverse.cs:150 and QuiverInsiderTrading.cs, then review the cited SubscriptionDataReader.cs handling of BaseData.Reader returning null. Reproduce the malformed rows over the 2026-07-27 to 2026-08-05 window; done means unexpected column counts or unparseable fields are skipped without a reader exception, while the current layouts still produce data points.
Written by the indexing model from the issue text.
Description
Summary
Both QuiverInsiderTradingUniverse.Reader and QuiverInsiderTrading.Reader assume every CSV line is in the current 16-column (universe) / 15-column (per-ticker) layout. A line in any other shape throws out of the reader, and LEAN treats a reader exception as a fatal runtime error (SubscriptionDataReader.cs:551 -> ReaderErrorDetected -> IResultHandler.RuntimeError), so one stray row in one daily file stops every algorithm that subscribes to the universe. The readers should validate the column count and return null for a line they cannot parse, so a bad row costs one data point instead of the whole backtest.
This is a sibling of #9 (legacy 14-column rows after the Name field was added). The rows here are older still: the pre-a50ad5f1 layout, and they fail on the date parse before any index is out of range, so the hasName fallback proposed in #9 does not cover them.
Evidence
Universe file alternative/quiver/insidertrading/universe/20260731.csv on the cloud data store (read from a backtest on LEAN 2.5.0.0.18114, 2026-09-21): 663 rows, 497 in the 16-column layout and 166 in the pre-refactor 6-column layout {sid},{ticker},{name},{shares},{price},{sharesOwnedFollowing}, spanning 37 tickers. Example:
ASPS UF02B1SL6QG5,ASPS,winkler matthew t.,1000.0,5.045,39989.0
The matching per-ticker file asps.csv carries the same trades twice: once in the 15-column layout and once as 5-column {date},{name},{shares},{price},{sharesOwnedFollowing} rows:
20260731,,20260730,P,5.045,1000.0,39989.0,A,D,Winkler Matthew T.,,T,F,F,F
20260731,winkler matthew t.,1000.0,5.045,39989.0
Every other universe file from 2026-05-01 to 2026-09-19 (99 files) is 16-column only, so the stray rows come from a single run of a pre-refactor build (the redundancy job for this dataset is still pinned to 6d71e402, which predates a50ad5f1) for upload date 2026-07-31; ProcessUniverse() then copied them into the universe file because it keys rows on the first column only (QuiverInsiderTradingDataDownloader.cs:219-251).
Reader failure, QuiverInsiderTradingUniverse.cs:150: csv[2] is winkler matthew t. on a 6-column row, and Parse.DateTimeExact(s, "yyyyMMddHHmmss") throws:
Error invoking QUIVERINSIDERTRADINGUNIVERSE-USA-... .QuiverInsiderTradingUniverse 2S data reader. Line: ASPS UF02B1SL6QG5,ASPS,winkler matthew t.,1000.0,5.045,39989.0 Error: String 'winkler matthew t.' was not recognized as a valid DateTime.
Same for the per-ticker reader, QuiverInsiderTrading.cs (csv[1] parsed as yyyyMMddHHmmss):
Error invoking ASPS.QuiverInsiderTrading data reader. Line: 20260731,winkler matthew t.,701.0,5.1,38989.0 Error: String 'winkler matthew t.' was not recognized as a valid DateTime.
Reproduced in a cloud backtest with AddUniverse<QuiverInsiderTradingUniverse>(_ => Enumerable.Empty<Symbol>()) over 2026-07-27 to 2026-08-05, and with AddData<QuiverInsiderTrading>("ASPS") over the same window. Note that the read-ahead worker (SubscriptionUtils.CreateAndScheduleWorker) stamps the runtime error with the algorithm clock, so a reporter sees the failure "on" an earlier date than the file that holds the row (here 2026-07-07, because there are no universe files between 2026-07-09 and 2026-07-26).
Proposed change
- In both readers, check
csv.Lengthagainst the expected column count first and returnnull(optionallyLog.Erroronce per file) when it does not match, instead of letting the parse throw.BaseData.Readerreturningnullis already handled byTextSubscriptionDataSourceReaderas "no data point". Combined with the #9 fallback this makes the readers tolerant of every layout the store has carried. - Wrap the
Parse.DateTimeExact/decimal.Parsecalls so a single unparseable field also yieldsnullrather than an exception. - Separately (data side, not this repo): strip the 5-column rows from the 37 per-ticker files and let
ProcessUniverse()regenerate20260731.csv; repoint the redundancy job at the current commit so a pre-refactor build cannot write again.
Open questions
- Should
ProcessUniverse()itself skip lines whose column count does not match the current layout, so a per-ticker file with stray rows cannot poison the universe file? It already skips lines whose first column is not ayyyyMMdddate.
Reference
Intercom conversation: 215476021685114.
- Dominant language
- C#
- Stars
- 0
- Forks
- 3
- Avg merge
- 39m
- Merged PRs (30d)
- 4
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from QuantConnect/Lean.DataSource.QuiverQuant
-
priority:P2
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Insider trading readers subtract a day from explicit FileDate timestampsPossibly taken @quantconnect-kronos claimed this today. Openkronos:skip priority:P2
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
QuantConnect/Lean.DataSource.QuiverQuant#17 · 1 comment ·
-
priority:P2
Difficulty 4/5 3-5 days Newbie friendliness 49/100
All issues in QuantConnect/Lean.DataSource.QuiverQuant
Similar issues
-
clawsweeper:fix-shape-clear clawsweeper:queueable-fix clawsweeper:source-repro impact:ux-friction issue-rating: 🦞 diamond lobster no-stale P2
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
openclaw/openclaw-windows-node#2287 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
area:host area:operations type:bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
kind:ux needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
elsa-workflows/elsa-foundation#4742 ·
Maintainers usually reply within 1 day
-
enhancement good first issue help wanted new-language
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
approved area: ui bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day