Data Limit/Inefficient data transfer of chart data (options) to the browser
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- javascript, nodejs
- Domain
- backend
Research direction
Reproduce the failure with a chart containing 3 million data points, then investigate the exporter crash and the duplicated JSON transfer described in the report. Done means large chart data no longer crashes at the transfer limit and the service logs the specific reason when the limit is exceeded.
Written by the indexing model from the issue text.
Description
Expected behavior
The export server should handle large amounts of data. The technical limitation is likely due to the Chrome DevTools protocol implementation, which is capped at 100 MB. This allows for approximately 12,979,200 data points to be transferred (assuming double precision at 8 bytes per point). However, some bytes should be reserved for other settings within the options object.
Additionally, the exporter crashes without a proper error message when the 100 MB limit is exceeded.
Actual behavior
Data is transferred as a string representation of the JSON object, which consumes more than twice the amount of data per data point. We are currently observing a limit of around 2,550,000 data points, resulting in a JSON string of roughly 100 MB. I expected it to be around 50 MB, but the data appears redundant in the transferred message (data.params.arguments[1].value.export.options and data.params.arguments[1].value.export.strInj).
The exporter should not crash when the limit is exceeded. Instead, it should log the specific reason.
One potential solution could be to serialize the JSON to BSON, transfer it as binary data, and then deserialize it back to JSON on the client side.
Reproduction steps
Render a chart with 3 million data points and observe the service crashing.
- Dominant language
- JavaScript
- Stars
- 368
- Forks
- 265
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from highcharts/node-export-server
-
Update Puppeteer to version 25.11.0+ to allow ARM64 linux images to run node-export-serverPossibly taken @daniatic claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
highcharts/node-export-server#686 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
highcharts/node-export-server#680 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
highcharts/node-export-server#679 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
highcharts/node-export-server#678 ·
-
Not working on s390xOpen
Difficulty 4/5 3-5 days Newbie friendliness 30/100
highcharts/node-export-server#676 ·
All issues in highcharts/node-export-server
Similar issues
-
bug confirmed perf
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
videojs/video.js#9400 · 1 comment ·
Maintainers usually reply within 1 day
-
agent/scanner bug hive/hosted-available-lke648397-260827-5n31
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
rescript-lang/rescript#8765 ·
Maintainers usually reply within 1 day
-
feedback simulation workshop
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
githubnext/gh-aw-workshop#4417 ·
Maintainers usually reply within 1 day
-
[Good First Issue]: Add unit tests for NetworkVersionInfoPossibly taken @attilayener claimed this today. OpenGood First Issue hacktoberfest
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
hiero-ledger/hiero-sdk-js#4489 ·
Maintainers usually reply within 1 day