Implement Spark-compatible `schema_of_csv`
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 40/100
Direzione di ricerca
Start by reading SchemaOfCsv in sql/catalyst/.../csvExpressions.scala and SchemaOfCsvEvaluator in CsvExpressionEvalUtils.scala, then CSVInferSchema.inferRowType/inferField/toStructFields and CSVOptions — those define the inference ladder and option parsing to mirror. The target is crates/spark/src/function/csv/, reusing the tokenizer and CsvOptions already shared with from_csv/to_csv, and reusing schema_of_json's DDL formatting if present. Done means a foldable expression returning the STRUCT<...> string with all listed error codes, verified by the new crates/sqllogictest/test_files/csv/schema_of_csv.slt cases and annotated with #[user_doc].
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Part of #272 and apache/datafusion-comet#7300.
What the function does
schema_of_csv(csv[, options]) parses one CSV record and returns the inferred schema as a DDL string such as
STRUCT<_c0: INT, _c1: STRING>. Fields are named _c0, _c1, ... by position. csv must be foldable
(DATATYPE_MISMATCH.NON_FOLDABLE_INPUT) and not NULL (DATATYPE_MISMATCH.UNEXPECTED_NULL). From 4.1, a non-string
argument fails with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE. options is a map(...) of string literals, read
with the same CSVOptions as from_csv (sep/delimiter, quote, escape, nullValue, nanValue,
positiveInf, negativeInf, dateFormat, timestampFormat, timestampNTZFormat, preferDate, locale,
ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, ...). The time zone is always UTC. lineSep is forced to
\uFFFF, so a newline is part of the field.
Returns a non-nullable STRING. Since 3.0.0. Because the input is foldable, Spark always constant-folds it
(in 4.x it is RuntimeReplaceable over an Invoke on a literal). Its main use is as the schema argument of
from_csv, so DataFusion can evaluate it at planning time (for example in simplify).
Examples
SELECT schema_of_csv('1,abc');
-- STRUCT<_c0: INT, _c1: STRING>
SELECT schema_of_csv('1|abc', map('delimiter', '|'));
-- STRUCT<_c0: INT, _c1: STRING>
SELECT schema_of_csv('0.1,1');
-- STRUCT<_c0: DOUBLE, _c1: INT>
SELECT schema_of_csv('0.1 1', map('sep', ' '));
-- STRUCT<_c0: DOUBLE, _c1: INT>
SELECT schema_of_csv('14:30:45');
-- STRUCT<_c0: TIMESTAMP>
SELECT schema_of_csv('14:30:45.123456');
-- STRUCT<_c0: TIMESTAMP>
SELECT schema_of_csv(null);
-- error: DATATYPE_MISMATCH.UNEXPECTED_NULL
SELECT schema_of_csv(42);
-- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (4.1+)
SELECT schema_of_csv(csvField) FROM csvTable;
-- error: DATATYPE_MISMATCH.NON_FOLDABLE_INPUT
These come from the ExpressionDescription, csv-functions.sql.out and CsvFunctionsSuite. Others, by reading
CSVInferSchema: '1,,abc' gives _c1: STRING; '12345678901234567890' gives DECIMAL(20,0);
'2015-08-26' gives DATE; 'TRUE' gives BOOLEAN; 'NaN' and 'Inf' give DOUBLE.
Edge cases to test
Inference (CSVInferSchema.inferField, starting from NullType for each field, single row so no merging):
- Empty field or a field equal to
nullValue(default"") staysNullType, which becomesSTRING. - The type ladder is tried in this order, each falling through to the next on failure:
INT(Integer.parseInt),BIGINT(Long.parseLong),DECIMAL(p, s)only when theBigDecimalhas
scale <= 0 (so integers too big for BIGINT; precision > 38 or a negative scale such as1e5throws and falls
through),DOUBLE(Double.parseDouble, or thenanValue/positiveInf/negativeInfstrings),DATE
(only whenpreferDate, default true; strict parse withdateFormat, defaultyyyy-MM-dd),TIMESTAMP_NTZ(only
whenspark.sql.timestampType=TIMESTAMP_NTZ),TIMESTAMP(withtimestampFormat, or the lenient default parser
that also accepts time-only strings, see above),BOOLEAN(case-insensitivetrue/false), elseSTRING. 1.5has scale 1, so it skips DECIMAL and becomesDOUBLE.TIMEis never inferred, even in 4.2.- Whitespace:
' 0.8'fails INT, BIGINT and BigDecimal butDouble.parseDoubletrims, so it isDOUBLE.
' 1'isDOUBLEtoo, notINT(by code reading). WithignoreLeadingWhiteSpaceit becomesINT. - Java
Double.parseDoublealso accepts1d,1f,Infinityand hex floats, all givingDOUBLE. - With a non-US
localethe decimal step usesDecimalFormatfor that locale (grouping and decimal separators).
The US default uses plainnew BigDecimal(s)(unlikefrom_csv, commas are not stripped). spark.sql.legacy.timeParserPolicy=LEGACYturnspreferDateoff and sets the default timestamp pattern to
yyyy-MM-dd'T'HH:mm:ss.SSSXXX, which changes date and timestamp inference.- A quoted field with the delimiter inside (
'"a,b",1') is one field. The number of fields equals the number of
tokens univocity returns, including a trailing empty one ('1,'gives two fields). - Output formatting is
StructType.sql:STRUCT<name: TYPE, ...>with:and,separators, upper-case types,
and field names quoted with backticks only when needed. - Bad option values give the same errors as
from_csv(OPTION_VALUE_EXCEEDS_ONE_CHARACTER,
INVALID_DELIMITER_VALUE.*,INVALID_OPTIONS.NON_MAP_FUNCTION,INVALID_OPTIONS.NON_STRING_TYPE).
Differences across Spark versions
- 3.4 / 3.5:
CodegenFallbackexpression. No explicit string type check on the argument. - 4.0: becomes
RuntimeReplaceable(SchemaOfCsvEvaluator).spark.sql.legacy.timeParserPolicydefault moves
fromEXCEPTIONtoCORRECTED. - 4.1: adds the
UNEXPECTED_INPUT_TYPEcheck (child.dataType != StringType). - 4.2:
csv-functions.sql.outadds the time-only cases, which inferTIMESTAMP.
Implementation notes
- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
(SchemaOfCsv),expressions/csv/CsvExpressionEvalUtils.scala(SchemaOfCsvEvaluator),
catalyst/csv/CSVInferSchema.scala(inferRowType,inferField,toStructFields),catalyst/csv/CSVOptions.scala. - DataFusion core has no equivalent. Arrow's CSV schema inference (
arrow-csvinfer_schema) uses different
rules (regex based, Int64/Float64/Boolean/Date/Timestamp) and can't be reused. - Share the CSV tokenizer and
CsvOptionswithfrom_csvandto_csv(apache/datafusion-spark#138; Comet's
nativeto_csvinnative/spark-expr/src/csv_funcs/already has write options) in
crates/spark/src/function/csv/.schema_of_jsonshould share the DDL type-string formatting if it exists. - Tests: add
crates/sqllogictest/test_files/csv/schema_of_csv.slt; document with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
- Lingua principale
- Rust
- Stelle
- 7
- Fork
- 15
- Merge medio
- 16h 43m
- PR unite (30g)
- 26
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di apache/datafusion-spark
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
apache/datafusion-spark#287 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
apache/datafusion-spark#266 ·
I maintainer di solito rispondono entro 1 giorno
-
[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the intervalForse già presa @Subodh-17 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
apache/datafusion-spark#109 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 18/100
apache/datafusion-spark#299 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
apache/datafusion-spark#298 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di apache/datafusion-spark
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
suhina-daw/suhina-daw#1477 ·
I maintainer di solito rispondono entro 1 giorno
-
parity priority: low ui
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
storytold/pdfcraft#7483 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
IngvarConsulting/unica#3097 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement in-progress
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 64/100