quantopian/zipline

Better error message when QUANDL_API_KEY is incorrect

Open

#2093 opened on Jan 27, 2018

View on GitHub
 (7 comments) (0 reactions) (0 assignees)Python (17,019 stars) (4,633 forks)batch import
Data BundleHelp Wanted

Description

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

  • Operating System: (Windows Version or $ uname --all) Linux raffaele-GL753VD 4.14.13-041413-generic #201801101001 SMP Wed Jan 10 10:02:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Python Version: $ python --version Python 3.5.4 :: Anaconda, Inc.

  • Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))' 64

  • How did you install Zipline: (pip, conda, or other (please explain))

As suggested in #2083

conda create -n py35 python=3.5 source activate py35 conda install -c quantopian/label/ci -c quantopian zipline

  • Python packages: $ pip freeze or $ conda list

packages in environment at /home/raffaele/anaconda3/envs/py35:

alembic 0.7.7 py35_0 quantopian asn1crypto 0.24.0 py35_0
bcolz 0.12.1 np111py35_0 quantopian bottleneck 1.2.1 py35he1b16f1_0
bzip2 1.0.6 h6d464ef_2
ca-certificates 2017.08.26 h1d4fec5_0
certifi 2018.1.18 py35_0
cffi 1.11.4 py35h9745a5d_0
chardet 3.0.4 py35hb6e9ddf_1
click 6.7 py35h353a69f_0
contextlib2 0.5.5 py35h6690dba_0
cryptography 2.1.4 py35hbeb2da1_0
cyordereddict 0.2.2 py35_0 quantopian cython 0.27.3 py35h6cdc64b_0
decorator 4.2.1 py35_0
empyrical 0.3.3 py35_0 quantopian hdf5 1.10.1 h9caa474_1
idna 2.6 py35h8605a33_1
intel-openmp 2018.0.0 hc7b2577_8
intervaltree 2.1.0 py35_0 quantopian libedit 3.1 heed3624_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 7.2.0 h7cc24e2_2
libgfortran 3.0.0 1
libgfortran-ng 7.2.0 h9f7466a_2
libstdcxx-ng 7.2.0 h7a57d05_2
logbook 0.12.5 py35_0 quantopian lru-dict 1.1.4 py35_0 quantopian lzo 2.10 h49e0be7_2
mako 1.0.7 py35h69899ea_0
markupsafe 1.0 py35h4f4fcf6_1
mkl 2017.0.4 h4c4d0af_0
multipledispatch 0.4.9 py35h2ff591a_0
ncurses 6.0 h9df7e31_2
networkx 1.11 py35hf333c83_0
numexpr 2.6.2 np111py35_0
numpy 1.11.3 py35_0
openssl 1.0.2n hb7f436b_0
pandas 0.18.1 np111py35_0
pandas-datareader 0.5.0 py35_0
patsy 0.5.0 py35_0
pip 9.0.1 py35h7e7da9d_4
pycparser 2.18 py35h61b3040_1
pyopenssl 17.5.0 py35h4f8b8c8_0
pysocks 1.6.7 py35h6aefbb0_1
pytables 3.4.2 py35hfa98db7_2
python 3.5.4 h417fded_24
python-dateutil 2.6.1 py35h90d5b31_1
pytz 2017.3 py35hb13c558_0
readline 7.0 ha6073c6_4
requests 2.18.4 py35hb9e6ad1_1
requests-file 1.4.1 py35_0
requests-ftp 0.3.1 py35_0
scipy 0.19.0 np111py35_0
setuptools 38.4.0 py35_0
six 1.11.0 py35h423b573_1
sortedcontainers 1.4.4 py35_0 quantopian/label/ci sqlalchemy 1.2.1 py35h14c3975_0
sqlite 3.21.0 h1bed415_0
statsmodels 0.8.0 py35haa9d50b_0
tk 8.6.7 hc745277_3
toolz 0.9.0 py35_0
urllib3 1.22 py35h2ab6e29_0
wheel 0.30.0 py35hd3883cf_1
xz 5.2.3 h55aa19d_2
zipline 1.1.1 np111py35_230_gfb735e76 quantopian/label/ci zlib 1.2.11 ha838bed_2

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

  • What did you expect to happen? To ingest data

  • What happened instead? Exception

Here is how you can reproduce this issue on your machine: type "zipline ingest" after installing

Reproduction Steps

  1. zipline ingest

...

What steps have you taken to resolve this already?

...

Anything else?

ERROR LOG:
[2018-01-27 13:04:51.382275] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
[2018-01-27 13:04:51.535455] ERROR: zipline.data.bundles.quandl: Exception raised reading Quandl data. Retrying.
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 89, in fetch_data_table
    format_metadata_url(api_key)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read
    compression=kwds.get('compression', None))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
[2018-01-27 13:04:51.537257] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
[2018-01-27 13:04:52.221225] ERROR: zipline.data.bundles.quandl: Exception raised reading Quandl data. Retrying.
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 89, in fetch_data_table
    format_metadata_url(api_key)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read
    compression=kwds.get('compression', None))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
[2018-01-27 13:04:52.225160] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
[2018-01-27 13:04:52.943029] ERROR: zipline.data.bundles.quandl: Exception raised reading Quandl data. Retrying.
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 89, in fetch_data_table
    format_metadata_url(api_key)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read
    compression=kwds.get('compression', None))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
[2018-01-27 13:04:52.946730] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
[2018-01-27 13:04:53.675329] ERROR: zipline.data.bundles.quandl: Exception raised reading Quandl data. Retrying.
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 89, in fetch_data_table
    format_metadata_url(api_key)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read
    compression=kwds.get('compression', None))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
[2018-01-27 13:04:53.678803] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
[2018-01-27 13:04:54.075195] ERROR: zipline.data.bundles.quandl: Exception raised reading Quandl data. Retrying.
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 89, in fetch_data_table
    format_metadata_url(api_key)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read
    compression=kwds.get('compression', None))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Traceback (most recent call last):
  File "/home/raffaele/anaconda3/envs/py35/bin/zipline", line 11, in <module>
    load_entry_point('zipline==1.1.1+230.gfb735e76', 'console_scripts', 'zipline')()
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/__main__.py", line 323, in ingest
    show_progress,
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/core.py", line 451, in ingest
    pth.data_path([name, timestr], environ=environ),
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 209, in quandl_bundle
    environ.get('QUANDL_DOWNLOAD_ATTEMPTS', 5)
  File "/home/raffaele/anaconda3/envs/py35/lib/python3.5/site-packages/zipline/data/bundles/quandl.py", line 113, in fetch_data_table
    "Failed to download Quandl data after %d attempts." % (retries)
ValueError: Failed to download Quandl data after 5 attempts.

Sincerely, $ whoami $whoamI Dunno, try to search on Google

Contributor guide

Tech stack
python
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
basic Pythonfamiliarity with exception handling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Investigate the file zipline/data/bundles/quandl.py, specifically the fetch data table function around line 89 where the HTTPError is raised. The current error just shows a generic HTTP 400; improve it to detect when the error is due to an invalid API key and display a user friendly message like 'Invalid Quandl API key. Please check your QUANDL API KEY environment variable.' Consider checking the URL or response content to confirm it's an API key issue. Ensure the fix handles the error gracefully without retrying unnecessarily.