Experimental catalyst conversions for low-dimension tensors
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- data, distributed-systems, performance
Research direction
Start by investigating how Catalyst logical plans and Spark's Tungsten representation handle scalar, vector, and higher-order tensor columns. Compare the current covariance-matrix computation with the proposed buffer-copy and flattening approaches, and determine how the four basic types could share generated code. Done means an experimental conversion path, a higher-order-tensor fallback, and a benchmark result.
Written by the indexing model from the issue text.
Description
Directly integrate with the logical plan of Catalyst when working on small dimensions.
For columns containing scalars or vectors, perform an unsafe memory copy to a buffer instead of using scala collections, and provide a fallback for higher-order tensors. It is recommended to flatten the operations for higher order tensors because the Tungsten representation is not very efficient from a memory perspective.
Implementation for the 4 basic types will duplicate some code, so investigate a templating library or scala macros to generate the code for each type?
The benchmark for this task will be the computation of the covariance matrix of a dataset (hopefully, it can get faster than the current computation in Spark).
- Dominant language
- Scala
- Stars
- 743
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/tensorframes
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
databricks/tensorframes#185 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
databricks/tensorframes#181 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
databricks/tensorframes#179 ·
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
databricks/tensorframes#176 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
databricks/tensorframes#165 ·
All issues in databricks/tensorframes
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
scala-native/scala-native#5238 ·
Maintainers usually reply within 1 day
-
CI/CD ⚒️ Flaky-tests 🐦 Rust ⚙️
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7658 ·
Maintainers usually reply within 1 day
-
[kunminghu-v3] Debug SBA 32/64-bit accesses hang: missing TLWidthWidget(1) on error_xbar pathPossibly taken A pull request linked to this issue is open or already merged. Openmodule: backend type: bug/reported
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OpenXiangShan/XiangShan#6938 ·
Maintainers usually reply within 3 days
-
documentation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Babali42/DrumBeatRepo#639 ·
Maintainers usually reply within 1 day
-
module: core presentation conversion
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bigbluebutton/bigbluebutton#26508 ·
Maintainers usually reply within 1 day