`rand` and `randn` with an integer seed fall back to Spark without a fallback reason

Open Beginner friendly
  • Is something wrong?
#7,479 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
scala, spark
Domain
backend, testing

Research direction

Start in spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala and trace how seeded rand and randn support checks relate to extractSeedFromExpr. Compare the analyzed IntegerType and LongType seeds described in the issue, then check the relevant conversion tests. Done means integer seeds convert like long seeds, or produce a specific fallback reason, with the strict fallback check passing.

Written by the indexing model from the issue text.

Description

requires-triage
Describe the bug

A Project or Filter that contains rand(<int>) or randn(<int>), for example rand(42), is
not converted, and Comet records no fallback reason on the operator or on any of its expressions.
In SQL, an integer literal seed such as rand(42) has type IntegerType. Only a LongType
seed, such as rand(42L), is converted.

With spark.comet.explain.fallback.strict.enabled=true, which CometTestBase turns on for
Comet's own suites, the query fails with:

java.lang.IllegalStateException: Comet did not convert Filter but recorded no fallback reason on
the operator or any of its expressions. Add a withFallbackReason call stating why conversion
failed.

(or ... did not convert Project ... for a projection). Without the strict check, the only
reason shown is the generic "<operator> is not supported" from
CometExecRule.reportUnexplainedFallback, which does not say what to change.

Steps to reproduce

Run on main (0f7f8e453) with the default profile (Spark 4.1.3), in a CometTestBase suite:

spark.range(0, 200, 1, numPartitions = 1)
  .selectExpr("CAST(id AS INT) AS id", "IF(id % 3 = 0, NULL, CAST(id AS INT)) AS y")
  .write.parquet(path)
spark.read.parquet(path).createOrReplaceTempView("t")

These four queries each fail with the error above:

SELECT id FROM t WHERE y IS NOT NULL AND rand(42) < 0.5;
SELECT id FROM t WHERE abs(rand(42)) < 0.5;
SELECT id, rand(1) AS r FROM t;
SELECT id, randn(1) AS r FROM t;

The same queries with a Long seed are converted:

Query Executed plan
SELECT id FROM t WHERE y IS NOT NULL AND rand(42L) < 0.5 CometProject > CometFilter > CometNativeScan
SELECT id FROM t WHERE abs(rand(42L)) < 0.5 contains CometFilter
SELECT id, rand(1L) AS r FROM t CometProject
SELECT id, randn(1L) AS r FROM t CometProject

In the analyzed plans, the seed of rand(1) and rand(42) is IntegerType, and the seed of
rand(1L) is LongType.

Cause

In spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala, the seeded serde's
getSupportLevel returns Compatible() for any Literal seed. But CometRand.convert and
CometRandn.convert build the proto from extractSeedFromExpr:

protected def extractSeedFromExpr(expr: Expression): Option[Long] = {
  expr match {
    case Literal(seed: Long, _) => Some(seed)
    case Literal(null, _) => Some(0L)
    case _ => None
  }
}

An IntegerType literal matches neither case, so convert returns None without recording a
reason.

Expected behavior

rand and randn with an integer literal seed are converted, as they are with a Long seed.
Spark accepts an IntegerType or LongType seed and widens an integer seed to a long
(RDG.seed in Spark's randomExpressions.scala). If an integer seed is not supported,
the conversion should record a fallback reason that says so.

Additional context

A possible fix: accept an Int literal in extractSeedFromExpr and widen it to a Long, or have
getSupportLevel reject what convert cannot take, with a reason.

🤖 Generated with Claude Code

Dominant language
Scala
Stars
1.3k
Forks
405
Avg merge
3d 6h
Merged PRs (30d)
239

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-comet

All issues in apache/datafusion-comet

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.