ENH: VaR backtests (Kupiec, Christoffersen, Basel traffic light) and a "maximum undetectable understatement" diagnostic

Open
  • Is something wrong?
#904 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
analytics

Research direction

This is a new feature proposal awaiting maintainer decisions on placement (arch/univariate/backtest.py beside diagnostics.py) and whether a method wrapper is wanted. Before any code, read the residual diagnostics added in #850 and the WaldTestStatistic return type they reuse. Done would mean the five standard VaR backtests plus max_undetectable_understatement, each tested against published Kupiec and Basel tables, with the maintainers' scope decisions settled first.

Written by the indexing model from the issue text.

Description

arch has tools for fitting models that produce VaR forecasts and for forecasting them, but nothing that backtests a VaR series against realized returns. I searched the current main branch for "backtest", "kupiec" and "christoffersen" and found no matches.

I would like to add a small set of standalone functions in arch.univariate, in the same shape as the residual diagnostics added in #850:

from arch.univariate import (
    var_exceptions,                       # bool array: -returns > var
    kupiec_pof,                           # LR test, WaldTestStatistic (df=1)
    christoffersen_independence,          # LR test, WaldTestStatistic (df=1)
    christoffersen_conditional_coverage,  # LR test, WaldTestStatistic (df=2)
    traffic_light,                        # Basel zone for the last 250 days
    max_undetectable_understatement,
)

The first five are textbook. The three likelihood-ratio tests return the existing WaldTestStatistic, so .stat, .pval and .critical_values work as they do elsewhere in the package; var_exceptions returns a boolean array and traffic_light returns a small result object with the zone, the count and the cumulative probability. The Kupiec non-rejection regions match the ones usually quoted from Kupiec (1995) for 99% VaR, and the traffic light cumulative probabilities reproduce Table 2 of the Basel Committee's 1996 backtesting framework.

The sixth function is the part I think is new. A passing backtest says the reported VaR was not rejected. It does not say how close it was. max_undetectable_understatement(returns, var) answers a concrete question: by what fraction could the whole VaR series have been scaled down, with the same returns, and still pass every selected test? The result also reports the extra exceptions that scaling would produce and which test binds first. A pass with a limit of 2% reads very differently from a pass with a limit of 25%.

Because the exception set only changes at the points 1 - loss/var, the search over proportional scalings is exact and needs one pass over the breakpoints, no optimizer. It is a lower bound on what a non-proportional series could hide, and the docstring says so. The tests include a brute-force grid check of the limit, and it also agrees with a bisection solver I wrote earlier on 34 simulated series to about 1e-16.

Three design points I would like your view on:

  1. Placement. I put it in arch/univariate/backtest.py next to diagnostics.py, with docs under the utility page. If you would rather have it elsewhere, moving it is trivial.

  2. Kupiec is two-sided, so a bank that reports zero exceptions fails it for being too conservative. For the understatement question that is not informative, so max_undetectable_understatement ignores rejections caused by too few exceptions by default (ignore_too_few=True).

  3. Wrapper. On #845 you said you would rather have standalone functions with a lightweight method wrapper. These are standalone, and the first five would wrap onto ARCHModelResult cleanly once a VaR series is supplied. I left the wrapper out of the sketch so the functions can be reviewed on their own, and will add it if you want it.

I have a working branch with tests (hand-computed Christoffersen values, the published Kupiec and Basel tables, property tests and a brute-force grid check of the understatement limit) and docs. I am happy to open a PR with the whole thing, or with only the five standard tests first if you prefer to review the new diagnostic separately. If this is out of scope for arch, I will publish it as a small standalone package instead.

Dominant language
Python
Stars
1.6k
Forks
298
Avg merge
14h 46m
Merged PRs (30d)
106

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bashtage/arch

All issues in bashtage/arch

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.