Implement Spark-compatible `log1p`

Aperta Adatta ai principianti
  • Qualcosa non va?
#266 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
80/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
database

Direzione di ricerca

Inizia leggendo crates/spark/src/function/math/expm1.rs per il pattern Signature::exact(vec![Float64]), l’annotazione #[user_doc] e la gestione dei null, poi registra la nuova funzione allo stesso modo. Implementa ln_1p e mappa qualsiasi input <= -1.0 su NULL, in modo coerente con UnaryLogExpression di Spark. Abilita ed estendi le query segnaposto in crates/sqllogictest/test_files/math/log1p.slt, quindi esegui quel file slt. Fatto significa che i casi slt per NULL, -1.0, -Infinity, NaN, Infinity, -0.0 e valori minuscoli passano e i risultati corrispondono a StrictMath.log1p.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement good first issue

Part of #174 and apache/datafusion-comet#7300.

What the function does

log1p(expr) returns ln(1 + expr), computed accurately for expr near 0. The argument is implicitly cast to DOUBLE (UnaryMathExpression, inputTypes = Seq(DoubleType)) and the result is DOUBLE. Like the other Hive-derived log functions (UnaryLogExpression), any input <= -1.0 returns NULL instead of -Infinity or NaN. The result is always nullable. Available since 1.4.0.

Spark evaluates java.lang.StrictMath.log1p (fdlibm) in both the interpreted and code-generated paths.

Examples
SELECT log1p(0);
-- 0.0
SELECT log1p(-1);
-- NULL
SELECT log1p(-2);
-- NULL
SELECT log1p(CAST('NaN' AS DOUBLE));
-- NaN
SELECT log1p(CAST('Infinity' AS DOUBLE));
-- Infinity

MathExpressionsSuite checks log1p against StrictMath.log1p for 0.0, 0.1, ..., 2.0 and expects NULL for -10.0, -9.0, ..., -1.0.

Edge cases to test
  • NULL input.
  • Exactly -1.0 returns NULL (the boundary is <= yAsymptote with yAsymptote = -1.0), and values just above it (e.g. -0.9999999999999999) return a large negative finite value.
  • Values below -1 (including -Infinity) return NULL, not NaN.
  • NaN returns NaN (NaN <= -1.0 is false). +Infinity returns Infinity.
  • -0.0 returns -0.0 and 0.0 returns 0.0 (check the sign).
  • Tiny values (1e-300, 1e-20) return the input value, which is where ln(1 + x) would lose precision.
  • Bit-exact results: Spark uses fdlibm via StrictMath. Rust's f64::ln_1p calls the platform log1p, which can differ in the last ulp on some inputs and platforms. Compare against StrictMath.log1p values for a range of inputs (e.g. 0.1 to 2.0 and large values like 1e300), or use a port of fdlibm s_log1p.c if they differ.
  • Integer, DECIMAL, FLOAT and STRING arguments (cast to DOUBLE first; with ANSI on an invalid string fails in the cast).
Differences across Spark versions

None found (3.4 through 4.2; only contextIndependentFoldable was added).

Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala, Log1p, base classes UnaryLogExpression and UnaryMathExpression.
  • DataFusion core has no log1p (it has ln, log, log2, log10).
  • Follow crates/spark/src/function/math/expm1.rs (Signature::exact(vec![Float64]), exp_m1), using ln_1p and mapping x <= -1.0 to NULL.
  • Tests: the placeholder crates/sqllogictest/test_files/math/log1p.slt exists with a commented-out query; enable and extend it. Document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.