Issues with run times of data pulling
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- r
- Domain
- data-engineering, performance
Research direction
Start by reproducing the issue with bcdata::bcdc_get_data(), comparing a single run with the parallel workload described for the shinywqg app. Profile the seven data pulls under concurrent access, then use the results to identify a supported way to reduce runtime and verify that the pulls complete reliably under multiple simultaneous users.
Written by the indexing model from the issue text.
Description
The shinywqg shiny app uses the bcdata::bcdc_get_data() function to pull seven data sets from the BC data catalogue when the app launches.
Normally it takes about 14 seconds for the data to pull using the bcdata::bcdc_get_data() function. Users have reported that sometimes it can take over 90 seconds or even longer and this is often during periods where you would expect high volume use ie during Monday morning.
This issue has been difficult to diagnose because it is intermittent. I was able to simulate the issue by running the function in parallel and the more functions that were running at the same time I started to see the pull times for the seven data sets increase. When the function was run a single time it took 14 seconds to pull the seven data sets and when I had the function running 10 times simultaneously the pull times for the seven data sets increased to 35 seconds.
App users have been reporting these issues and so we are looking for solutions for how to decrease this run time when multiple users are accessing the app.
- Dominant language
- R
- Stars
- 87
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bcgov/bcdata
-
running air in CI Open
Difficulty 3/5 1-2 days Newbie friendliness 25/100
-
considering
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
considering enhancement wfs
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
Similar issues
-
documentation pkg infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
epiverse-trace/epiparameter#511 ·
-
function:write_dwc
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Urgent request: Due to vulnerabilities move to API version 12.6.1 (12.6.2 eventually) or 13.1.1 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
jbkunst/highcharter#849 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100