DocumentInserter drops whole task batches when a payload key contains a dot

Open
#352 0 comments 0 reactions 0 assignees View on GitHub

@JaimeCernuda is already working on this.

Since Sep 8, 2026.

  • #353 by @JaimeCernuda — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
mongodb, python
Domain
databases

Research direction

Start at mongodb_dao.insert_and_update_many_tasks, where payload keys become $set upserts in a bulk_write; reproduce the failure with a key such as yr.doy and inspect the resulting BulkWriteError. Establish the expected key encoding for MongoDB, then verify that a batch containing dotted or leading-$ keys completes without dropping tasks.

Written by the indexing model from the issue text.

Description

Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:

pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
           Consider using $getField or $setField."

Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.

Dominant language
Python
Stars
43
Forks
21
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ORNL/flowcept

All issues in ORNL/flowcept

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.