move postgres connections from psycopg2 to psycopg 3
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- postgresql, python
- Domain
- build-system, databases
Research direction
Start by finding every AsIs and copy_expert use, beginning with dcpy.utils.postgres and products/zap-opendata/src/pg.py, then the utils.py files in the developments, cpdb, knownprojects, colp and ceqr products. Dependencies are in python/utils/pyproject.toml. Done means psycopg2 is gone from that file, the sqlalchemy pin is removed, and the product builds pass. Those builds need a maintainer's CI run, so check with them first.
Written by the indexing model from the issue text.
Description
sqlalchemy 2.1 loads the psycopg 3 driver for postgresql:// URIs, and we only install psycopg2, so #2722 pins sqlalchemy<2.1 in python/utils/pyproject.toml. On #2681 this broke 11 of 12 product builds with ModuleNotFoundError: No module named 'psycopg' at create_engine. Moving to psycopg 3 lets us drop the pin.
The psycopg2-specific code:
AsIsfor schema and table names throughoutdcpy.utils.postgresand inproducts/zap-opendata/src/pg.py. psycopg 3 has noAsIs, so these probably becomepsycopg.sql.Identifieror similarcursor.copy_expertindcpy.utils.postgresand theutils.pyof developments, cpdb, knownprojects, colp, and ceqr. psycopg 3 usescursor.copy()insteadpsycopg2-binaryandtypes-psycopg2inpython/utils/pyproject.toml
Needs product builds to verify, so probably worth running it from compile_python_reqs or dispatching builds on the branch.
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 13h 8m
- Merged PRs (30d)
- 138
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NYCPlanning/data-engineering
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
NYCPlanning/data-engineering#2996 ·
Maintainers usually reply within 1 day
-
db-facilities
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
NYCPlanning/data-engineering#2847 · 1 assignee ·
Maintainers usually reply within 1 day
-
db-knownprojects
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
NYCPlanning/data-engineering#2629 ·
Maintainers usually reply within 1 day
-
CSCL - coincident_seg_count double-counts subway/rail crossingsPossibly taken @alexrichey claimed this today. Opendb-cscl
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NYCPlanning/data-engineering#2616 · 1 assignee ·
Maintainers usually reply within 1 day
-
db-gft
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
NYCPlanning/data-engineering#2532 ·
Maintainers usually reply within 1 day
All issues in NYCPlanning/data-engineering
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pearu/agent-sandbox#451 ·
Maintainers usually reply within 1 day
-
run_gate loads a custom tasks file from outside the working directoryPossibly taken @Ru0k3 claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
MakazhanAlpamys/Soup#3189 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day