Parallel COPY to Postgres ?
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- postgresql
- Domain
- databases
Research direction
Start by reproducing the ATTACH and COPY commands from the issue with PostgreSQL 16 and DuckDB v0.10.0, then inspect the postgres_scanner path handling COPY. Done means the documented multi-file import can use parallel PostgreSQL COPY processes, or the supported limitation is clearly established.
Written by the indexing model from the issue text.
Description
What happens?
Hello,
I tried to do a parallel import in Postgres.
I wonder if postgres_scanner allows this ?
To Reproduce
I tried :
ATTACH 'user=postgres host=/var/run/postgresql port=5440' AS db (TYPE postgres);
COPY db.votes2 FROM 'data.zstd/*' (FORMAT parquet);
I tried to change file format (csv, parquet), compression, multiples files...
I also tried to first import in duckdb, then, copy to Postgres.
I only see one COPY process on Postgres.
OS:
Linux
PostgreSQL Version:
16
DuckDB Version:
v0.10.0 20b1486d11
DuckDB Client:
CLI
Full Name:
Adrien Nayrat
Affiliation:
Self Employed
Have you tried this on the latest main branch?
- I agree
Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
- I agree
- Dominant language
- C++
- Stars
- 375
- Forks
- 113
- Avg merge
- 5h 29m
- Merged PRs (30d)
- 37
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from duckdb/duckdb-postgres
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
duckdb/duckdb-postgres#638 ·
Maintainers usually reply within 1 day
-
v2.0: generate_subscripts(list, 1) fails with 'positional argument following named argument' after attaching PostgresPossibly taken @staticlibs claimed this 2 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
duckdb/duckdb-postgres#630 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
duckdb/duckdb-postgres#628 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 62/100
duckdb/duckdb-postgres#623 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 62/100
duckdb/duckdb-postgres#618 ·
Maintainers usually reply within 1 day
All issues in duckdb/duckdb-postgres
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
notepad-plus-plus/notepad-plus-plus#18553 ·
Maintainers usually reply within 1 day
-
bug docs performance
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 3 days
-
good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
good first issue neural P2
Difficulty 2/5 1-3 hours Newbie friendliness 65/100