Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Running preprocessing steps as one pass over each chunk

未关闭
  • 哪里不对？
#4,859 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
8/100
Issue 类型
功能
描述清晰度
需要澄清
活跃度
活跃
技术栈
python
领域
data, performance

调研方向

This is a design discussion, not a task to pick up: the author wants views on three questions before any PRs. Read the proposed fuse() API and the filter steps it would combine, phase_shift, bandpass_filter with method="fir" and common_reference, and the prototype on branch proto/fused-v2. Done means the maintainers have answered the questions about explicit versus automatic fusion, dtype conversion and saving provenance.

由索引模型根据 Issue 内容生成。

描述

Each preprocessing step currently computes a whole chunk before the next step starts. I prototyped phase_shift → bandpass → median common_reference as a single pass over small tiles of each chunk. Its output matches the steps run one after another, and it runs 1.4 times faster on 1 thread and 1.5 times faster on 8, with a third less peak RAM.

Before opening any PRs, I'd like your views on three design questions.

Image

1 s chunks of a 384-channel Neuropixels recording read from RAM, on an i9-12900K. "FIR steps" are phase_shift(method="fir") (#4563), bandpass_filter(method="fir") (#4856) and common_reference with the median from #4854, run one after another. On 8 threads each step splits its work across the threads, as proposed in #4857.

1 thread 8 threads peak RAM, multiples of the chunk (1 / 8 threads)
FIR steps, 1 s chunks 83 MS/s 422 MS/s 5.7 / 6.3
fused, 1 s chunks 113 MS/s 644 MS/s 3.8 / 4.0
FIR steps, 10 s chunks 90 MS/s 475 MS/s 4.2 / 4.2
fused, 10 s chunks 115 MS/s 704 MS/s 2.2 / 2.2

MS/s counts channel-samples, so one probe in real time is 11.5 MS/s. Best of 7; peak RAM is above the loaded chunk.

The single pass removes the phase shift's pass and the memory traffic between steps

  • As @oliche suggested on #4563, each channel's phase-shift kernel is convolved into the bandpass kernel, which grows from 503 to 518 taps. The phase shift then costs about 1% of the filter.
  • The bandpass runs on tiles small enough to stay in L2, instead of streaming the whole chunk through memory once per step.

The median gains little, since compute rather than memory limits it: on 1 thread it takes 35 ms per 1 s chunk fused and 39 ms as a separate step.

On 10 s of the recording, saved in 1 s chunks, the fused output differs from the separate steps by 6.2e-6 ADC counts rms, a millionth of the signal's 6.5 counts rms, and by at most 1.3e-4. The error is float32 rounding: it is the same within 32 samples of a chunk or tile edge as elsewhere, so the seams leave no artefacts.

Proposal

A user would opt in by calling fuse on the end of a chain:

recording = phase_shift(raw, method="fir")
recording = bandpass_filter(recording, method="fir")
recording = common_reference(recording)
recording = fuse(recording)  # one FusedFIRReferenceRecording on top of raw

fuse walks back from the last step. If the steps form a chain it knows, it replaces them with one fused step on the same parent; otherwise it returns the recording unchanged. The fused step keeps the last step's dtype and metadata, so code downstream sees the same recording.

The fused step's get_traces reads the chunk with the filter's margin, then works through tiles of about 1,600 samples by a slab of channels:

  1. Filter the tile with the combined phase-shift and bandpass kernel, in one FFT and one inverse.
  2. Once every slab of a time block is filtered, subtract the median or average across channels from those rows.

The arithmetic is in float32, converted to the last step's dtype once, at the end.

The prototype recognises one chain by its classes: an optional phase_shift(method="fir"), then fir_filter or a highpass_filter or bandpass_filter with method="fir", then an optional global common_reference. To fuse other chains without writing a fused step for each, steps could declare how they read their input:

access example what a fused step must do
per sample scale nothing; apply it anywhere in a tile
per channel, over a window filters, phase_shift read a margin around each tile
across channels, per sample common_reference wait until every channel of the rows is ready

A step that declares nothing is never fused and works as it does now.

Threads

The fused step splits its tiles across the threads budgeted by #4857. That budget alone makes the separate FIR steps 5.1 times faster on 8 threads, and fusion adds another 1.5 times.

Questions

  1. Should fuse() be called explicitly, or applied automatically when a chain matches?
  2. Is converting the dtype once, at the end, acceptable? An int16 chain currently rounds to int16 after every step, so its output would change by that rounding noise, about 0.3 ADC counts rms per step.
  3. Should a fused recording save as its original chain, so saved files and provenance stay the same?

The prototype is on branch proto/fused-v2, on top of #4563, #4853, #4854, #4855 and #4856.

主要语言
Python
星标
866
派生
285
平均合并
2 天 7 小时
30 天内合并 PR
43

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

SpikeInterface/spikeinterface 的其他 Issue

查看 SpikeInterface/spikeinterface 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。