Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

DuckDB Spark API is incompatible with the PySpark API's `spark.createDataFrame(list of dict)`

Open
  • Is something wrong?
#183 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
data

Research direction

Start at duckdb.experimental.spark.sql.SparkSession.createDataFrame and run the provided Python reproduction against the DuckDB Spark API and PySpark. Compare the resulting temporary-view output, and consider the issue complete when DuckDB produces the same column name and row value as PySpark for a list of dictionaries.

Written by the indexing model from the issue text.

Description

needs triage
What happens?

The DuckDB Spark API is incompatible with the PySpark API's spark.createDataFrame(list of dict) method.

To Reproduce
from duckdb.experimental.spark.sql import SparkSession as DuckdbSparkSession
from pyspark.sql import SparkSession

sql_text = "SELECT * FROM t0"
data = [
    {"c0": "1969-12-21"}
]
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame(data)
df.createOrReplaceTempView("t0")

print("PySpark SQL result:")
pyspark_result = spark.sql(sql_text)
pyspark_result.show()

duckdb_spark = DuckdbSparkSession.builder.getOrCreate()
df = duckdb_spark.createDataFrame(data)
df.createOrReplaceTempView("t0")

print("Duckdb Spark SQL result: ")
duckdb_spark_result = duckdb_spark.sql(sql_text)
duckdb_spark_result.show()

PySpark SQL result:
+----------+                                                                    
|        c0|
+----------+
|1969-12-21|
+----------+

Duckdb Spark SQL result: 
┌─────────┐
│  col0   │
│ varchar │
├─────────┤
│ c0      │
└─────────┘
OS:

x86_64 Ubuntu 24.04 Linux-6.14.0-35-generic-x86_64-with-glibc2.39

DuckDB Version:

1.4.2

DuckDB Client:

Python

Hardware:

No response

Full Name:

asddfl

Affiliation:

xxx

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?
  • Yes, I have
Did you include all code required to reproduce the issue?
  • Yes, I have
Did you include all relevant data sets for reproducing the issue?

Yes

Dominant language
Python
Stars
189
Forks
117
Avg merge
1d 58m
Merged PRs (30d)
15

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from duckdb/duckdb-python

All issues in duckdb/duckdb-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.