Keep the market-hours database when a refresh returns malformed JSON

Abierto
  • ¿Algo no está bien?
#709 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@pcbeingused333 ya está trabajando en esto.

Desde el 4/11/2026.

  • #710 de @pcbeingused333 — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
cli, databases, testing

Línea de trabajo

Read lean/components/cloud/data_downloader.py and lean/components/util/market_hours_database.py to trace the download, file replacement, refresh timestamp, and JSON reader behavior. Check the existing tests for these components, then add the described regression case with a valid cached database and malformed response. Done means invalid content leaves the prior database usable and refresh is recorded only after a successful update. An open linked pull request (#710) is already against this issue.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

On master (d812d26), a successful HTTP response containing malformed JSON can overwrite a usable local market-hours database. Subsequent database reads fail instead of retaining the last valid data.

The path is:

  • DataDownloader.update_database_files() (lean/components/cloud/data_downloader.py:83-92) downloads the market-hours database and passes its bytes directly to _store_local_file.
  • DataClient.download_public_file() (lean/components/api/data_client.py:89-95) returns raw response content, without JSON validation.
  • _store_local_file() (lean/components/cloud/data_downloader.py:24-27) opens the existing file with wb+ and writes those bytes.
  • MarketHoursDatabase._get_all_entries() (lean/components/util/market_hours_database.py:63-69) only parses the JSON when it is later read.

The refresh timestamp is also recorded before the download/write (data_downloader.py:85), so a malformed successful response can leave the bad file in place until the next configured refresh interval.

Synthetic reproduction

With the CLI installed and initialized, this uses only a temporary file, not an actual data directory:

import json
import tempfile
from pathlib import Path
from lean.components.cloud.data_downloader import _store_local_file

with tempfile.TemporaryDirectory() as directory:
    cache = Path(directory) / "market-hours-database.json"
    previous = '{"entries": {}}'
    cache.write_text(previous, encoding="utf-8")

    # Bytes returned by a malformed HTTP 200 response.
    _store_local_file(b"not json", cache)
    print(cache.read_text(encoding="utf-8"))  # not json; valid data is lost
    json.loads(cache.read_text(encoding="utf-8"))  # JSONDecodeError

I verified the same result using the isolated _store_local_file function from master: the file becomes not json, and the next decode raises JSONDecodeError. This is a synthetic malformed response, not a report that the hosted database is currently malformed.

Suggested fix

Validate the downloaded market-hours content using the format accepted by its reader (including supported comments and the required entries structure) before replacing the local file. Write through a temporary file, preserve the existing file's access metadata, and replace only after validation and writing succeed. Record a successful refresh only after it completes. Add a regression test with a pre-existing valid database and a malformed successful response.

Lenguaje dominante
Python
Estrellas
330
Forks
171
Merge medio
2 d 21 h
PR fusionados (30 d)
18

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de QuantConnect/lean-cli

Todos los issues de QuantConnect/lean-cli

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.