quantopian/zipline

Add option to filter data through Winsorisation

Open

#127 opened on Apr 5, 2013

View on GitHub
 (8 comments) (0 reactions) (0 assignees)Python (17,019 stars) (4,633 forks)batch import
Help Wanted

Description

Filter out extreme values which are assumed to be spurious because of their extremity.

As requested by Jessica Stauth on Quantopian forums, https://www.quantopian.com/posts/feature-requests-what-changes-would-you-like-to-see

Quoted from that post:

  1. add an option to 'winsorise' returns for outlier handling - a notorious issue with backtests is hidden outliers in returns data - sometimes they are obvious, you trade a stock and it makes 10,000% in 1 day (oops pricing error, currency issue etc) - but sometimes these errors can be hidden. Winsorizing your returns data allows you to set sanity bounds on what returns you think a stock can achieve, so you might say, clip my returns data at -99% and + 2 standard deviations from the mean returns for that time period. Better explained here: http://en.wikipedia.org/wiki/Winsorising

Contributor guide

Tech stack
pythonpandasnumpy
Domain
databackend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythonpandasbasic statisticszipline internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Explore the existing data filtering and preprocessing modules in the Zipline codebase, such as `zipline/data/filter.py` or `zipline/utils/events.py`. Review the 8 comments on the issue for any additional context or suggestions from maintainers. Implement Winsorisation by adding a function that clips return values based on specified percentiles or standard deviation bounds, and integrate it as an option in the pipeline API or backtest configuration.