H3Core.nonZeroLongArrayToList takes too much memory

Open
#68 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
java
Domain
backend

Research direction

Start in src/main/java/com/uber/h3core/H3Core.java at the polyfill result preparation around line 691 and nonZeroLongArrayToList around lines 1242-1254. Rework the conversion so the prepared non-zero result does not require an oversized intermediate ArrayList, then verify that polyfillAddress preserves the generated hexes without the reported out-of-memory failure.

Written by the indexing model from the issue text.

Description

Trying to polyfill big area with resolution 11. I can see that internally algorithm prepared result https://github.com/uber/h3-java/blob/3a1e9bc9bcd1d62e9e8d4361230fe47bb5395242/src/main/java/com/uber/h3core/H3Core.java#L691 but when it tries to copy it to List it gets out of memory on this line:
https://github.com/uber/h3-java/blob/a500880eef493dca44903bf7ad0e249aed5b85b1/src/main/java/com/uber/h3core/H3Core.java#L1242-L1254

As you can see in the stacktrace, result is ready, but it just tries to copy it to ArrayList:
image

Memory, the arrow shows to the place when nonZeroLongArrayToList started :
image

Scala code:

import scala.collection.JavaConverters._
import java.util.{Collections => JCollections}

import beam.utils.ProfilingUtils
import com.uber.h3core.AreaUnit
import com.uber.h3core.util.GeoCoord

def main(args: Array[String]): Unit = {

  val h3Core = com.uber.h3core.H3Core.newInstance

  val xMin = -106.645646
  val xMax = -93.508292
  val yMin = 25.837377
  val yMax = 36.500704

  val rectangle = java.util.Arrays.asList(
    new GeoCoord(yMin, xMin),
    new GeoCoord(yMax, xMin),
    new GeoCoord(yMax, xMax),
    new GeoCoord(yMin, xMax),
  )

  val holes = java.util.Collections.emptyList[java.util.List[GeoCoord]]()
  val resolution = 11
  val hexes = h3Core.polyfillAddress(rectangle, holes, resolution)
  println(s"Generated ${hexes.size()}")
}

The easiest fix is just to get the total number of non-zero elements and allocate an array with that size and copy elements over. Other solution can be moving all zero elements in the original array to the end of the array and wrap it by ArrayList via Arrays.asList(array).subList(index, IDX_OF_FIRST_ZERO);. I can create PR if this sounds good.

Thanks.

Dominant language
Java
Stars
333
Forks
68
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from uber/h3-java

All issues in uber/h3-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.