User Stories for Interface / Feature Design and Documentation

No se nombran archivos, pruebas ni puntos de entrada. Empieza revisando los ejemplos propuestos y las capacidades actuales de DataFusion Python; después, organiza los casos de uso de los contribuidores y los requisitos de la interfaz. La tarea estará terminada cuando se haya definido una dirección aclarada y priorizada para las operaciones de DataFrame y la documentación.

As discussed in the rust arrow chat, this is a place to chart a way forward by collecting examples of both what people do in other libraries and what they want to do but can't do easily with current tools. In addition to clarifying Python interface requirements, I hope it provides fodder for lower-level functions, encourages knowledgeable folks to explain how to do things (which can then get documented), and clarifies what is important to what types of people and why. There's a daunting amount of work, but the opportunity and potential are tremendous.

For contributors, I'd like to structure this roughly as follows:

• If it's your first post in this thread, a brief description of your background and the types of work you've done/do.
• Examples of operations on data you think are important. Broadly of the form "I can do X in library Y with code sample Z". If it's something that you like, consider saying why. If you don't like it, explain how it could be easier or clearer.
• Examples of things you frequently do but have had to implement yourself and think should be considered for core operations.
• OPTIONALLY: Thoughts you have on what's important to a good data frame API in a dynamic language.

OK, now it's my turn to start.
My background is in statistics, machine learning, and data science. Most of what I do is focused on modeling and analyzing data, though I've done a good bit of pipeline and data processing/cleaning too. As such, I place a premium on in-memory interactive work (notebooks, rmarkdown, etc.). Partly because of this, I think a lot of tools mistake verboseness for clarity, and striving for conciseness often helps readability rather than hurting it if done correctly.

I have the most experience with R's data.table but I've also used dplyr, polars, pandas, and Julia. So here's sampling of a few things I would want in a dataframe library in no particular order.

1. Here's an example in data.table that shows features I think are both good and bad.

> dt1 = data.table(t = 1:5, v = 5:1)
> dt2 = data.table(start = c(1, 4), end = c(3, 10), x = c("a", "b"))
> dt1[dt2, x := i.x, on = .(t >= start, t < end)]
> dt1
   t v    x
1: 1 5    a
2: 2 4    a
3: 3 3 <NA>
4: 4 2    b
5: 5 1    b

First, I find non-equi joins, especially range joins incredibly useful. They're common in SQL but a lot of dataframe libraries don't have them. data.table also makes it easy to update in place with the := operator, which can be used to create new columns as well as update existing ones. As I understand it, arrow strives for immutability, but at the same time, it won't make copies of the whole frame if it doesn't have to, so maybe this is less of an issue. However, I do like the idea of using a join like this to explicitly tag/annotate another table.

2. Reshaping data. These functions transform data between "wide" to "long" (sometimes known as "tidy") formats. Sometimes they go by cast/melt or pivot and there's even a simple transpose function in a lot of packages. It doesn't seem common in database-world, but to me reshaping in-memory data is important for a lot of use cases.

3. Rolling groupbys. Both Pandas and Polars have pretty good support for creating overlapping groups and aggregating over them. These are commonly used for time series analysis. I also think the ability to define groups not just by a number of rows, but by a potentially variable-width lookback (like, at most 1 month before the current date) is useful. Polars does a pretty good job at this, and I think Pandas might too.

Alright, that's a few to get started and this is long enough as it is. I'm looking forward to seeing what everyone thinks is important, their thoughts on good DataFrame API design, and what is and isn't currently possible in DataFusion.