Python DataFrame column lookup is nondeterministic when a Symbol column and its ticker string column coexist

Abierto
  • ¿Algo no está bien?
#10,093 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
pandas, python
Área
data

Línea de trabajo

Start by tracing how Lean’s Python Symbol objects compare and hash when used as pandas DataFrame column labels; the report provides a reproduction using AlgorithmImports, history, and pd.DataFrame. Check existing tests for Symbol equality, hashing, and DataFrame column lookup. Done when df['open'] consistently selects the intended column when a Symbol ticker and matching string column coexist, with a regression test covering the reported case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Expected Behavior

In Python, a DataFrame that has both a Symbol column and a string column named like that Symbol's ticker returns the same result for df['open'] in every run.

Actual Behavior

With the ticker cached, Symbol == 'open' is True while hash(symbol) != hash('open'). A frame with columns [OPEN Symbol, 'open'] then returns different results from run to run for df['open']: usually a Series, but in some processes a DataFrame with both columns. Python randomizes string hashes per process, which is likely why the outcome changes between processes. Code that builds a mask from it, such as history.loc[lambda d: other['open'] > 0, 'close'], then fails intermittently with ValueError: Cannot index with multidimensional key.

Potential Solution

A suggestion only: Python requires that objects that compare equal have the same hash, which Symbol vs. ticker string doesn't meet. Either the equality or the hash would need to change for these cases.

Reproducing the Problem
from AlgorithmImports import *

class SymbolTickerColumnAmbiguity(QCAlgorithm):
    def initialize(self):
        spy = self.add_equity("SPY", Resolution.DAILY).symbol
        opened = self.add_equity("OPEN", Resolution.DAILY).symbol
        history = self.history(spy, 10, Resolution.DAILY)
        other = pd.DataFrame({opened: 1.0, 'open': -1.0}, index=history.index)
        self.log(f"{opened == 'open'} {hash(opened) == hash('open')} {type(other['open']).__name__}")

The first two values are always True False. The type varies between processes: in ten runs of the same steps as a unit test, other['open'] was a DataFrame once and a Series nine times. The same steps with LOW/low and HIGH/high gave a Series in all ten runs.

System Information

pandas 2.3.3, Python 3.11, current master.

Checklist
  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues
Lenguaje dominante
C#
Estrellas
22.5k
Forks
5.4k
Merge medio
1 d 2 min
PR fusionados (30 d)
71

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de QuantConnect/Lean

Todos los issues de QuantConnect/Lean

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.