DocumentInserter drops whole task batches when a payload key contains a dot
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Start at mongodb_dao.insert_and_update_many_tasks, where payload keys become $set upserts in a bulk_write; reproduce the failure with a key such as yr.doy and inspect the resulting BulkWriteError. Establish the expected key encoding for MongoDB, then verify that a batch containing dotted or leading-$ keys completes without dropping tasks.
Written by the indexing model from the issue text.
Description
Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:
pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
Consider using $getField or $setField."
Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.
- Dominant language
- Python
- Stars
- 43
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ORNL/flowcept
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Dependency-mgmt devops priority:low
Difficulty 3/5 1-2 days Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
-
code-enhancement
Difficulty 3/5 1-2 days Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Netcracker/qubership-ai-packages#266 ·
Maintainers usually reply within 1 day
-
PROCESSED_DATA_FOLDER: strip whitespace and treat empty as unsetPossibly taken @TorecLuik claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NL-BioImaging/BIOMERO.importer#63 ·
Maintainers usually reply within 1 day
-
MLX training-complete chat/merge suggestions omit --backend mlxPossibly taken @vaputa claimed this today. Openneeds-triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
MakazhanAlpamys/Soup#2913 · 1 comment ·
Maintainers usually reply within 1 day
-
area:security effort:S priority:P3 status:needs-triage type:chore
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
randomparity/kdive#3657 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aws/mcp-proxy-for-aws#476 ·
Maintainers usually reply within 1 day