Faster perspective flattening
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Refactor
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering, performance, stream-processing
Research direction
Start by tracing the current GatewayStruct → to_json → orjson → flatten → JSONL → pyarrow → perspective ingestion path described in the issue. Compare the three proposed directions—enhancing csp, emitting record batches directly, or having perspective flatten—and establish a measurable performance target; the issue currently names no files, tests, or acceptance criterion.
Written by the indexing model from the issue text.
Description
Right now to ingest into perspective we go GatewayStruct to JSON via to_json, then from json to plain python objects via orjson, then we flatten, then we dump back to jsonl for ingestion into pyarrow and finally from pyarrow into perspective. This is done to balance performance (pyarrow json loading and perspective ingestion of pyarrow are very fast, gateway struct to json is in C++ so also faster than flattening as structured objects, orjson very fast) with the fact that we want to flatten things. In increasing order of preference, we can:
- enhance
cspto flatten - try to emit record batches directly instead of flattening into json and moving back and forth between python objects
- have perspective flatten for us
- Dominant language
- Python
- Stars
- 32
- Forks
- 9
- Avg merge
- 14h 58m
- Merged PRs (30d)
- 17
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Point72/csp-gateway
-
Better mobile UIOpencomponent: UI
Difficulty 5/5 Over a week Newbie friendliness 15/100
Point72/csp-gateway#152 ·
Maintainers usually reply within 1 day
-
good first issue type: enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
Point72/csp-gateway#127 ·
Maintainers usually reply within 1 day
-
component: API type: enhancement
Difficulty 5/5 Over a week Newbie friendliness 15/100
Point72/csp-gateway#90 ·
Maintainers usually reply within 1 day
-
Add documentation on customizing fastapi instancePossibly taken @russlan23 claimed this 100 days ago. Openpart: documentation type: enhancement
Difficulty 3/5 1-2 days Newbie friendliness 25/100
Point72/csp-gateway#89 ·
Maintainers usually reply within 1 day
-
component: UI type: feature
Difficulty 5/5 Over a week Newbie friendliness 15/100
Point72/csp-gateway#20 ·
Maintainers usually reply within 1 day
All issues in Point72/csp-gateway
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
Maintainers usually reply within 1 day
-
comp/desktop P3 type/feature
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#154285 ·
Maintainers usually reply within 1 day
-
docs: qualify copied and custom kubectl debug policy notesPossibly taken @nym01 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
kubescape/cel-admission-library#166 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
SACGF/variantgrid#2804 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] HTTP lifecycle dispatch treats a NO_PROXY-only environment as corporate egress and skips the resolve-and-pinned sessionPossibly taken @bodapatisaikrishna claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day