Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Dagster comparison is not correct

Aperta
  • Qualcosa non va?
#1,289 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
pandas, python
Ambito
data-engineering, documentation

Direzione di ricerca

Inizia dalla pagina di confronto di Dagster collegata nell’issue, in particolare dalle sezioni Dataflow Definition e Dataflow Execution e dalla tabella di confronto principale. Verifica gli esempi e le affermazioni rispetto ai riferimenti di Dagster a IOManager, Declarative Materialization, Pipes, data versioning e dagster-dbt; il lavoro è completato quando il confronto è accurato, utilizza funzionalità analoghe e copre le omissioni elencate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

triage

Hi!

I was curious about Hamilton because I was looking for a lightweight DAG library.

Coming from Dagster, I got naturally interested in the Hamilton vs Dagster comparison and found this page in the docs: https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/

I noticed it does not provide accurate information about Dagster and the code examples are not using some of Dagster's main features. More concrete:

  • issues with the first example:
    • It does not utilize the IOManager to decouple I/O from computations
    • It incorrectly states that asset descriptions have to be defined via metadata, which is not correct (they can be defined in native function docstrings or via the @asset(description=... argument).
  • issues with the second example:
    • it incorrectly states that the Dagster job can't be executed in a local Python process
    • it incorrectly states that I/O and computations are coupled (duplicate)
    • the comparison between loading environment variables at runtime and providing configuration time references like dagster.EnvVar does not make much sense. Dagster's configuration purposely enables deferring the setting of the exact configuration parameters (since Dagster runs can be executed remotely, e.g. in a Kubernetes pod, and the env var might not be available outside of the remote system). But nothing is preventing the user from setting the value with os.getenv directly if needed.

Minor (in the main comparison table):

  • important and unique Dagster features such as Declarative Materialization and Pipes are not mentioned
  • data versioning comparison is a bit strange: it's not very clear how does Hamilton automatically identify code versions (e.g. how does it distinguish between refactoring-like changes and changes in the actual business logic). Dagster's data versioning system enforces explicit code version management to avoid unwanted expensive materializations of the entire asset graph (see: declarative automation).
  • important Dagster integrations such as dagster-dbt are not mentioned

Current behavior

The Dagster example is not using relevant Dagster features and provides inaccurate information.

Expected behavior

The comparison between Hamilton and Dagster should use analogous features in both frameworks to be fair. In particular, it should use the IOManager as it's one of the main selling points of Dagster:

import dagster as dg
import pandas as pd

@dg.asset
def topstory_ids() -> pd.DataFrame: ...

@dg.asset
def topstories(topstory_ids: pd.DataFrame): ...

Note that some of the popular IOManagers for Pandas and Polars also support loading a subset of the dataframe columns: @asset(metadata={"columns": ["title"]}).

It should also provide accurate information on other topics mentioned above.

Additional context

Technically, this is not a bug, but I couldn't find a better label for this issue.

I am willing to help with improving these docs if my help is considered welcome!

Lingua principale
Jupyter Notebook
Stelle
2.6k
Fork
218
Merge medio
2g 13h
PR unite (30g)
18

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/hamilton

Tutte le issue di apache/hamilton

Issue simili

Altre issue su Data Engineering

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.