Python DataFrame column lookup is nondeterministic when a Symbol column and its ticker string column coexist

Aperta
  • Qualcosa non va?
#10,093 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
pandas, python
Ambito
data

Direzione di ricerca

Start by tracing how Lean’s Python Symbol objects compare and hash when used as pandas DataFrame column labels; the report provides a reproduction using AlgorithmImports, history, and pd.DataFrame. Check existing tests for Symbol equality, hashing, and DataFrame column lookup. Done when df['open'] consistently selects the intended column when a Symbol ticker and matching string column coexist, with a regression test covering the reported case.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Expected Behavior

In Python, a DataFrame that has both a Symbol column and a string column named like that Symbol's ticker returns the same result for df['open'] in every run.

Actual Behavior

With the ticker cached, Symbol == 'open' is True while hash(symbol) != hash('open'). A frame with columns [OPEN Symbol, 'open'] then returns different results from run to run for df['open']: usually a Series, but in some processes a DataFrame with both columns. Python randomizes string hashes per process, which is likely why the outcome changes between processes. Code that builds a mask from it, such as history.loc[lambda d: other['open'] > 0, 'close'], then fails intermittently with ValueError: Cannot index with multidimensional key.

Potential Solution

A suggestion only: Python requires that objects that compare equal have the same hash, which Symbol vs. ticker string doesn't meet. Either the equality or the hash would need to change for these cases.

Reproducing the Problem
from AlgorithmImports import *

class SymbolTickerColumnAmbiguity(QCAlgorithm):
    def initialize(self):
        spy = self.add_equity("SPY", Resolution.DAILY).symbol
        opened = self.add_equity("OPEN", Resolution.DAILY).symbol
        history = self.history(spy, 10, Resolution.DAILY)
        other = pd.DataFrame({opened: 1.0, 'open': -1.0}, index=history.index)
        self.log(f"{opened == 'open'} {hash(opened) == hash('open')} {type(other['open']).__name__}")

The first two values are always True False. The type varies between processes: in ten runs of the same steps as a unit test, other['open'] was a DataFrame once and a Series nine times. The same steps with LOW/low and HIGH/high gave a Series in all ten runs.

System Information

pandas 2.3.3, Python 3.11, current master.

Checklist
  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues
Lingua principale
C#
Stelle
22.5k
Fork
5.4k
Merge medio
22h 6m
PR unite (30g)
81

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di QuantConnect/Lean

Tutte le issue di QuantConnect/Lean

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.