move postgres connections from psycopg2 to psycopg 3

Open
#2,723 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
postgresql, python

Research direction

Start by finding every AsIs and copy_expert use, beginning with dcpy.utils.postgres and products/zap-opendata/src/pg.py, then the utils.py files in the developments, cpdb, knownprojects, colp and ceqr products. Dependencies are in python/utils/pyproject.toml. Done means psycopg2 is gone from that file, the sqlalchemy pin is removed, and the product builds pass. Those builds need a maintainer's CI run, so check with them first.

Written by the indexing model from the issue text.

Description

sqlalchemy 2.1 loads the psycopg 3 driver for postgresql:// URIs, and we only install psycopg2, so #2722 pins sqlalchemy<2.1 in python/utils/pyproject.toml. On #2681 this broke 11 of 12 product builds with ModuleNotFoundError: No module named 'psycopg' at create_engine. Moving to psycopg 3 lets us drop the pin.

The psycopg2-specific code:

  • AsIs for schema and table names throughout dcpy.utils.postgres and in products/zap-opendata/src/pg.py. psycopg 3 has no AsIs, so these probably become psycopg.sql.Identifier or similar
  • cursor.copy_expert in dcpy.utils.postgres and the utils.py of developments, cpdb, knownprojects, colp, and ceqr. psycopg 3 uses cursor.copy() instead
  • psycopg2-binary and types-psycopg2 in python/utils/pyproject.toml

Needs product builds to verify, so probably worth running it from compile_python_reqs or dispatching builds on the branch.

Dominant language
Python
Stars
2
Forks
0
Avg merge
13h 8m
Merged PRs (30d)
138

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NYCPlanning/data-engineering

All issues in NYCPlanning/data-engineering

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.