documentationgood first issue
Repository metrics
- Stars
- (4 stars)
- PR merge metrics
- (PR metrics pending)
Description
There is a recurring but intermittent error that appears when ingesting a shapefile into R from the database. This error seems to be somehow related to connect/disconnect pattern, as it occurs more frequently if connections are not well managed. However, it does appear after multiple consecutive spatial table imports.
More context:
- spatial table imports are done in batches of 1000 rows at a time. This was initially put in place to avoid memory problems from spatial imports of ~3000 rows.
- even with the batch import, mutliple consecutive imports (pulling the same 3k row table in 3-4 times) will cause the error.
- Since the problem is still occurring after a total of ~10000 rows are imported during a single session, it is worrisome to imagine a larger table failing to load even once.
- The problem always goes away/ memory resets when the R session is restarted. Obviously, any imported data is cleared and all packages are unloaded.