`rand` and `randn` with an integer seed fall back to Spark without a fallback reason

Aperta Adatta ai principianti
  • Qualcosa non va?
#7,479 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
scala, spark
Ambito
backend, testing

Direzione di ricerca

Inizia da spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala e segui la relazione tra i controlli di supporto per seeded rand e randn ed extractSeedFromExpr. Confronta i seed analizzati di tipo IntegerType e LongType descritti nell’issue, quindi controlla i test di conversione pertinenti. Il lavoro è completato quando i seed integer vengono convertiti come i seed long oppure producono un motivo di fallback specifico, e il controllo rigoroso del fallback ha esito positivo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

requires-triage
Describe the bug

A Project or Filter that contains rand(<int>) or randn(<int>), for example rand(42), is
not converted, and Comet records no fallback reason on the operator or on any of its expressions.
In SQL, an integer literal seed such as rand(42) has type IntegerType. Only a LongType
seed, such as rand(42L), is converted.

With spark.comet.explain.fallback.strict.enabled=true, which CometTestBase turns on for
Comet's own suites, the query fails with:

java.lang.IllegalStateException: Comet did not convert Filter but recorded no fallback reason on
the operator or any of its expressions. Add a withFallbackReason call stating why conversion
failed.

(or ... did not convert Project ... for a projection). Without the strict check, the only
reason shown is the generic "<operator> is not supported" from
CometExecRule.reportUnexplainedFallback, which does not say what to change.

Steps to reproduce

Run on main (0f7f8e453) with the default profile (Spark 4.1.3), in a CometTestBase suite:

spark.range(0, 200, 1, numPartitions = 1)
  .selectExpr("CAST(id AS INT) AS id", "IF(id % 3 = 0, NULL, CAST(id AS INT)) AS y")
  .write.parquet(path)
spark.read.parquet(path).createOrReplaceTempView("t")

These four queries each fail with the error above:

SELECT id FROM t WHERE y IS NOT NULL AND rand(42) < 0.5;
SELECT id FROM t WHERE abs(rand(42)) < 0.5;
SELECT id, rand(1) AS r FROM t;
SELECT id, randn(1) AS r FROM t;

The same queries with a Long seed are converted:

Query Executed plan
SELECT id FROM t WHERE y IS NOT NULL AND rand(42L) < 0.5 CometProject > CometFilter > CometNativeScan
SELECT id FROM t WHERE abs(rand(42L)) < 0.5 contains CometFilter
SELECT id, rand(1L) AS r FROM t CometProject
SELECT id, randn(1L) AS r FROM t CometProject

In the analyzed plans, the seed of rand(1) and rand(42) is IntegerType, and the seed of
rand(1L) is LongType.

Cause

In spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala, the seeded serde's
getSupportLevel returns Compatible() for any Literal seed. But CometRand.convert and
CometRandn.convert build the proto from extractSeedFromExpr:

protected def extractSeedFromExpr(expr: Expression): Option[Long] = {
  expr match {
    case Literal(seed: Long, _) => Some(seed)
    case Literal(null, _) => Some(0L)
    case _ => None
  }
}

An IntegerType literal matches neither case, so convert returns None without recording a
reason.

Expected behavior

rand and randn with an integer literal seed are converted, as they are with a Long seed.
Spark accepts an IntegerType or LongType seed and widens an integer seed to a long
(RDG.seed in Spark's randomExpressions.scala). If an integer seed is not supported,
the conversion should record a fallback reason that says so.

Additional context

A possible fix: accept an Int literal in extractSeedFromExpr and widen it to a Long, or have
getSupportLevel reject what convert cannot take, with a reason.

🤖 Generated with Claude Code

Lingua principale
Scala
Stelle
1.3k
Fork
405
Merge medio
3g 6h
PR unite (30g)
239

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.