cmu-delphi/delphi-epidata

Remove footgun from meta cache update

オープン

#776 opened on 2021/12/02

 (2 件のコメント) (0 件のリアクション) (0 人の担当者)Python (67 件のフォーク)auto 404
acquisitionbugdata qualityhelp wanted

Repository metrics

Stars
 (104 個のスター)
PR merge metrics
 (PR metrics pending)

説明

So this happened:

Traceback (most recent call last):
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 216, in _open_connection
   self._cmysql.connect(**cnx_kwargs)
_mysql_connector.MySQLInterfaceError: Too many connections
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 932, in _bootstrap_inner
   self.run()
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 870, in run
   self._target(*self._args, **self._kwargs)
 File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 327, in worker
   worker_dbc.connect(connector_impl=self._connector_impl)
 File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 89, in connect
   self._connection = self._connector_impl.connect(
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/__init__.py", line 264, in connect
   return CMySQLConnection(*args, **kwargs)
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 80, in __init__
   self.connect(**kwargs)
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/abstracts.py", line 966, in connect
   self._open_connection()
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 218, in _open_connection
   raise errors.get_mysql_exception(msg=exc.msg, errno=exc.errno,
mysql.connector.errors.OperationalError: 1040 (08004): Too many connections

It turns out all 7 worker threads dropped out with this exception, and while in theory srcsigs.join() blocks until all items have been retrieved and completed, we do straight-up overwrite meta with the results no matter what they are, which makes us nervous.

Let's try:

  • finite retry to open connections
  • some basic checks to make sure we don't accidentally overwrite meta with a blank list

コントリビューターガイド