Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Running preprocessing steps as one pass over each chunk

オープン
  • 何か問題がありますか？
#4,859 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
8/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
活発
技術スタック
python
領域
data, performance

調査の方向性

This is a design discussion, not a task to pick up: the author wants views on three questions before any PRs. Read the proposed fuse() API and the filter steps it would combine, phase_shift, bandpass_filter with method="fir" and common_reference, and the prototype on branch proto/fused-v2. Done means the maintainers have answered the questions about explicit versus automatic fusion, dtype conversion and saving provenance.

索引モデルが issue の本文から書いたものです。

説明

Each preprocessing step currently computes a whole chunk before the next step starts. I prototyped phase_shift → bandpass → median common_reference as a single pass over small tiles of each chunk. Its output matches the steps run one after another, and it runs 1.4 times faster on 1 thread and 1.5 times faster on 8, with a third less peak RAM.

Before opening any PRs, I'd like your views on three design questions.

Image

1 s chunks of a 384-channel Neuropixels recording read from RAM, on an i9-12900K. "FIR steps" are phase_shift(method="fir") (#4563), bandpass_filter(method="fir") (#4856) and common_reference with the median from #4854, run one after another. On 8 threads each step splits its work across the threads, as proposed in #4857.

1 thread 8 threads peak RAM, multiples of the chunk (1 / 8 threads)
FIR steps, 1 s chunks 83 MS/s 422 MS/s 5.7 / 6.3
fused, 1 s chunks 113 MS/s 644 MS/s 3.8 / 4.0
FIR steps, 10 s chunks 90 MS/s 475 MS/s 4.2 / 4.2
fused, 10 s chunks 115 MS/s 704 MS/s 2.2 / 2.2

MS/s counts channel-samples, so one probe in real time is 11.5 MS/s. Best of 7; peak RAM is above the loaded chunk.

The single pass removes the phase shift's pass and the memory traffic between steps

  • As @oliche suggested on #4563, each channel's phase-shift kernel is convolved into the bandpass kernel, which grows from 503 to 518 taps. The phase shift then costs about 1% of the filter.
  • The bandpass runs on tiles small enough to stay in L2, instead of streaming the whole chunk through memory once per step.

The median gains little, since compute rather than memory limits it: on 1 thread it takes 35 ms per 1 s chunk fused and 39 ms as a separate step.

On 10 s of the recording, saved in 1 s chunks, the fused output differs from the separate steps by 6.2e-6 ADC counts rms, a millionth of the signal's 6.5 counts rms, and by at most 1.3e-4. The error is float32 rounding: it is the same within 32 samples of a chunk or tile edge as elsewhere, so the seams leave no artefacts.

Proposal

A user would opt in by calling fuse on the end of a chain:

recording = phase_shift(raw, method="fir")
recording = bandpass_filter(recording, method="fir")
recording = common_reference(recording)
recording = fuse(recording)  # one FusedFIRReferenceRecording on top of raw

fuse walks back from the last step. If the steps form a chain it knows, it replaces them with one fused step on the same parent; otherwise it returns the recording unchanged. The fused step keeps the last step's dtype and metadata, so code downstream sees the same recording.

The fused step's get_traces reads the chunk with the filter's margin, then works through tiles of about 1,600 samples by a slab of channels:

  1. Filter the tile with the combined phase-shift and bandpass kernel, in one FFT and one inverse.
  2. Once every slab of a time block is filtered, subtract the median or average across channels from those rows.

The arithmetic is in float32, converted to the last step's dtype once, at the end.

The prototype recognises one chain by its classes: an optional phase_shift(method="fir"), then fir_filter or a highpass_filter or bandpass_filter with method="fir", then an optional global common_reference. To fuse other chains without writing a fused step for each, steps could declare how they read their input:

access example what a fused step must do
per sample scale nothing; apply it anywhere in a tile
per channel, over a window filters, phase_shift read a margin around each tile
across channels, per sample common_reference wait until every channel of the rows is ready

A step that declares nothing is never fused and works as it does now.

Threads

The fused step splits its tiles across the threads budgeted by #4857. That budget alone makes the separate FIR steps 5.1 times faster on 8 threads, and fusion adds another 1.5 times.

Questions

  1. Should fuse() be called explicitly, or applied automatically when a chain matches?
  2. Is converting the dtype once, at the end, acceptable? An int16 chain currently rounds to int16 after every step, so its output would change by that rounding noise, about 0.3 ADC counts rms per step.
  3. Should a fused recording save as its original chain, so saved files and provenance stay the same?

The prototype is on branch proto/fused-v2, on top of #4563, #4853, #4854, #4855 and #4856.

主要言語
Python
スター
859
フォーク
286
平均マージ
3日 16時間
マージ済み PR（30日）
52

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SpikeInterface/spikeinterface のほかの issue

SpikeInterface/spikeinterface の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。