[Bug] BrokenPipe: server socket closed error when we execute any dbt model in dbt core with redshift adapter
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 25/100
Línea de trabajo
No se nombran archivos del repositorio, pruebas ni puntos de entrada. Empieza reproduciendo una ejecución multi-threaded de un modelo de dbt con el Redshift adapter y compara el ciclo de vida de la conexión alrededor del periodo de inactividad mostrado en los logs; se considera terminado cuando se haya identificado y verificado la causa del fallo BrokenPipe.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
We are facing an issue if we execute a dbt model in dbt core with redshift adapter in multi nodes or multi threads. We did some analysis to find the reason for a failure by checking the dbt logs and we found that at some random point one of the threads was going to idle state for a certain period of time and it was not executing any tasks sent to it and eventually it got failed with the below statement.
BrokenPipe: server socket closed. Please check that client side networking configurations such as Proxies, firewalls, VPN, etc. are not affecting your network connection.
We suspect this could be because of introduction of redshift_connector in dbt-redshift adapter by replacing psycopg2. We weren't facing any such issues in dbt older versions which used psycopg2.
Attaching the context from dbt log file
13:28:59.394786 [debug] [Thread-1 (]: Elementary: [dbt_tests] Artifacts changed.
.
. <LOT OF OTHER THREADS GO HERE USING ANOTHER CONNECTIONS>
.
13:53:01.835859 [debug] [Thread-1 (]: Elementary: Deleting from and inserting to: "elementary"."dbt_tests"
13:53:01.859812 [debug] [Thread-1 (]: Using redshift connection "model.elementary.dbt_tests"
13:53:01.860853 [debug] [Thread-1 (]: On model.elementary.dbt_tests: /* {"app": "dbt", "dbt_version": "1.8.9", "profile_name": "", "target_name": "", "node_id": "model.elementary.dbt_tests"} */
<SQL GOES HERE>
13:53:01.863164 [debug] [Thread-1 (]: Redshift adapter: Rolling back transaction.
13:53:01.864028 [debug] [Thread-1 (]: On model.elementary.dbt_tests: ROLLBACK
13:53:01.866800 [debug] [Thread-1 (]: Failed to rollback 'model.elementary.dbt_tests'
13:53:01.873315 [debug] [Thread-1 (]: On model.elementary.dbt_tests: Close
13:53:01.879586 [debug] [Thread-1 (]: Runtime Error in model dbt_tests (models/edr/dbt_artifacts/dbt_tests.sql)
BrokenPipe: server socket closed. Please check that client side networking configurations such as Proxies, firewalls, VPN, etc. are not affecting your network connection.
13:53:01.881652 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '382ae431-f242-4924-a625-8b4dd918ff58', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb52f6a2a10>]}
13:53:01.882856 [error] [Thread-1 (]: 1 of 1 ERROR creating sql incremental model elementary.dbt_tests .......... [ERROR in 1646.81s]
As you can see last time when connection was used in Thread-1 was at 13:28:59.394786 (first line)
After almost 25mins dbt got back to this connection to sent the next statement but the connection was no longer open
There is no keep_alive option in redshift_connector - this is something we had specified in the old redshift adapter.
- Lenguaje dominante
- Python
- Estrellas
- 220
- Forks
- 86
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de aws/amazon-redshift-python-driver
-
Dificultad 1/5 1-3 horas Aptitud para principiantes 74/100
-
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
aws/amazon-redshift-python-driver#268 · 2 reacciones ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
aws/amazon-redshift-python-driver#265 · 1 comentario · 4 reacciones ·
-
VARBYTE is decoded into utf-8Abierto
Dificultad 3/5 1-2 días Aptitud para principiantes 45/100
Todos los issues de aws/amazon-redshift-python-driver
Issues similares
-
Broken links found in docsAbiertodocs pydanty:is-working
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
pydantic/pydantic-ai#8863 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
run-llama/llama_index#23278 ·
Los mantenedores suelen responder en 2 días
-
documentation from-review-extraction github-actions priority: low severity:nit
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
LearningCircuit/local-deep-research#6946 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
oracle/langchain-oracle#323 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 88/100
tenstorrent/tt-metal#58057 · 1 comentario ·
Los mantenedores suelen responder en 1 día