Real tail of stream (last n elements)

Abierto
#415 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
32/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
pandas, python
Área
stream-processing

Línea de trabajo

Start with the Stream.from_periodic example and trace the accumulate(...).to_dataframe path used to retain DataFrame batches. The issue names no source file or test, so first determine where stream accumulation and window behavior are implemented. Done means agreeing on the API and semantics for exposing the last n elements, then verifying that behavior for the shown DataFrame workflow.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

To my knowledge there is currently no way to show (or plot) the last n elements of a Streamz DataFrame.

I think this is a very useful function for debugging, but also for slower Streamz, for example CFD results from OpenFOAM.

Here is a naive implementation:

def tail(length):
    def tail_func(x, y):
        x = x.append(y, ignore_index=True)
        return x.iloc[-length:].reset_index(drop=True)
    return tail_func

Here is an example:
(thanks for the feedback @martindurant)

import pandas as pd
from streamz import Stream
import numpy as np

names = ["Mike", "Tim", "Anna", "Kim", "Andy"]

def emitter():
    n = np.random.randint(1, 5)
    return {"name": np.random.choice(names, n), "age": np.random.randint(18, 32, n)}

stream = Stream.from_periodic(emitter, 1)
df1 = pd.DataFrame({"name": ["test"], "age": [40]})
out = stream.map(pd.DataFrame).accumulate(tail(6), start=pd.DataFrame()).to_dataframe(df1)
# out.stream.sink(print)  # optional, or some other output

stream.start()

# if using a Jupyter Lab
out

This could either be the bare representation of window or could replace the tail function. I think the name tail suits this function better than the current one. What do you think?

Implementation wise one could be much more efficient by using .loc and round robin replacing values and saving the order somewhere else, but for a first go, I would stick by the naive version.

Lenguaje dominante
Python
Estrellas
1.3k
Forks
149
Sin PR fusionados en 30 d

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de python-streamz/streamz

Todos los issues de python-streamz/streamz

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.