Keep the market-hours database when a refresh returns malformed JSON

Ouverte
  • Quelque chose ne va pas ?
#709 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 2 jours

@pcbeingused333 y travaille déjà.

Depuis le 4/11/2026.

  • #710 par @pcbeingused333 — ouverte

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
À l'abandon
Stack technique
python
Domaine
cli, databases, testing

Piste de recherche

Read lean/components/cloud/data_downloader.py and lean/components/util/market_hours_database.py to trace the download, file replacement, refresh timestamp, and JSON reader behavior. Check the existing tests for these components, then add the described regression case with a valid cached database and malformed response. Done means invalid content leaves the prior database usable and refresh is recorded only after a successful update. An open linked pull request (#710) is already against this issue.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Problem

On master (d812d26), a successful HTTP response containing malformed JSON can overwrite a usable local market-hours database. Subsequent database reads fail instead of retaining the last valid data.

The path is:

  • DataDownloader.update_database_files() (lean/components/cloud/data_downloader.py:83-92) downloads the market-hours database and passes its bytes directly to _store_local_file.
  • DataClient.download_public_file() (lean/components/api/data_client.py:89-95) returns raw response content, without JSON validation.
  • _store_local_file() (lean/components/cloud/data_downloader.py:24-27) opens the existing file with wb+ and writes those bytes.
  • MarketHoursDatabase._get_all_entries() (lean/components/util/market_hours_database.py:63-69) only parses the JSON when it is later read.

The refresh timestamp is also recorded before the download/write (data_downloader.py:85), so a malformed successful response can leave the bad file in place until the next configured refresh interval.

Synthetic reproduction

With the CLI installed and initialized, this uses only a temporary file, not an actual data directory:

import json
import tempfile
from pathlib import Path
from lean.components.cloud.data_downloader import _store_local_file

with tempfile.TemporaryDirectory() as directory:
    cache = Path(directory) / "market-hours-database.json"
    previous = '{"entries": {}}'
    cache.write_text(previous, encoding="utf-8")

    # Bytes returned by a malformed HTTP 200 response.
    _store_local_file(b"not json", cache)
    print(cache.read_text(encoding="utf-8"))  # not json; valid data is lost
    json.loads(cache.read_text(encoding="utf-8"))  # JSONDecodeError

I verified the same result using the isolated _store_local_file function from master: the file becomes not json, and the next decode raises JSONDecodeError. This is a synthetic malformed response, not a report that the hosted database is currently malformed.

Suggested fix

Validate the downloaded market-hours content using the format accepted by its reader (including supported comments and the required entries structure) before replacing the local file. Write through a temporary file, preserve the existing file's access metadata, and replace only after validation and writing succeed. Record a successful refresh only after it completes. Add a regression test with a pre-existing valid database and a malformed successful response.

Langage dominant
Python
Étoiles
330
Forks
171
Merge moyen
2 j 21 h
PR mergées (30 j)
18

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de QuantConnect/lean-cli

Toutes les issues de QuantConnect/lean-cli

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.