Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

unifying the ecosystem around observations-as-row convention

Aperta
  • Qualcosa non va?
#114 9 commenti 11 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Refactoring
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
julia
Ambito
data

Direzione di ricerca

Inizia facendo un inventario degli entry point di MultivariateStats che attualmente richiedono input trasposto, incluso l’esempio PCA nella documentazione, e confronta le loro convenzioni con le funzioni di Statistics come cov e cor. Definisci l’ambito della migrazione attorno a una parola chiave dims esplicita e determina quali API devono cambiare prima che le osservazioni come righe diventino il comportamento predefinito.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

In statistical analysis, e.g. with DataFrames.jl, it is a convention that columns equal variables, rows equal observations. The same is true for the entire R analytical ecosystem (statistics + machine learning) + the entire Python analytical ecosystem (stats + machine learning) + matlab. I believe it is also the case in most cases in the rest of the Julia ecosystem. For DataFrames this is the only obvious choice, as the same observation (row) can contain multiple types; and given that different columns can have different types, observations-are-rows is the only feasible way. Not true for Matrixes, which are the same type in both directions, so in principle columns can be interpreted as observations for Matrices.

In some parts of the Julia ecosystem that are built only for Matrix{<:Number}, that is actually done - the convention is flipped. This is true for the kmeans algorithm in Clustering (https://github.com/JuliaStats/Clustering.jl/issues/79), and for NearestNeighbours. It also used to be the case for standardize in StatsBase and for Distances, but a deprecation has just been added requiring an explicit dims argument (e.g. https://github.com/JuliaStats/StatsBase.jl/pull/490/files), which is the first step towards remedying this. And, it is also the case here in MultivariateStats - e.g. you'll need to transpose the data if you get them from a DataFrame, in order to do a PCA on them.

On the other hand, the stdlib Statistics (e.g. functions cov and cor), StatsModels, GLM, MixedModels, Plots, the MLJ machine learning library, and Turing all use the more widely accepted standard of observations-as-rows.

I think we should standardize on using the same convention across the ecosystem, and preferably the same standard as everyone else. The main arguments against has been 1) that Machine-learning often uses columns-as-observations, and 2) that because Julia is column-major, analyses of all variables for an observation should be faster.

Ad 1), I'd like to question that assertion - most machine-learning frameworks I know of uses rows-are-observations (or indeed DataFrames). A notable exception appears to be Flux, which looks like it is columns-are-observations, AFAICS.

Ad 2) I have some other concerns:

  1. Column-major is an implementation detail, that shouldn't define the API. Especially as the fastest implementation will differ depending on what type of analysis you're using, but a mixture of rows or columns as observations is a really dangerous footgun for introducing bugs.
  2. Often rows-as-observations is likely to be fastest, as many analyses focus on each variable individually.
  3. If most users get their data from DataFrames anyway, like in the very example used for PCA in the docs for MultivariateStats, the user is likely to transpose, and the algorithm will work on a lazy adjoint - losing the entire value of column-major memory layout. So most users will only experience the inconvenience but not harvest the performance benefits. Instead, with observations-as-rows, algorithms can be optimised for that; and in cases where they can't be, superusers that want max performance can make their object with observations-as-columns, then lazy transpose before passing in; if I'm not much mistaken these users would then still reap the performance gains.

I believe an added advantage is to improve the JuliaStats ecosystem combatability with Tables. The approach I'd suggest would be to require a dims keyword across all functions for at least 1 julia minor version, then flip the default, to make all code calling these functions deprecate noisily.

cc @nalimilan @andreasnoack @ararslan @tsela

Lingua principale
Julia
Stelle
387
Fork
83
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di JuliaStats/MultivariateStats.jl

Tutte le issue di JuliaStats/MultivariateStats.jl

Issue simili

Altre issue su Julia

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.