sktime/sktime

[ENH] implementing forecasting accuracy metrics from Chen & Yang 2004 "Assessing Forecast Accuracy Measures"

Open

#8681 aperta il 10 ago 2025

Vedi su GitHub
 (5 commenti) (0 reazioni) (0 assegnatari)Python (1192 fork)batch import
enhancementgood first issuemodule:metrics&benchmarking

Metriche repository

Star
 (7162 star)
Metriche merge PR
 (Merge medio 26g 10h) (86 PR mergiate in 30 g)

Descrizione

Is your feature request related to a problem? Please describe.

This is about a specific variant of the performance metric sMAPE in the context of time series forecasting.

It is also very much related to a documentation problem that can be quite misleading.

The documentation advertises that the user of the library can set eps.

See: https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.performance_metrics.forecasting.MeanAbsolutePercentageError.html

Now of course it is not totally clear what this epsilon would be used for.

The user might think that this epsilon enables using a variant of sMAPE similar to the variant msMAPE described by Chen and Yang in their 2004 paper "Assessing Forecast Accuracy Measures".

However, analysis of the sktime code bears the impression that eps in fact cannot be passed to the sktime sMAPE implementation.

Describe the solution you'd like

Instead of now dropping the eps parameter from the auto-generated API documentation somehow, it would probably be better and more useful to instead consider supporting one or two more variants of sMAPE, namely one variant where the problem of inflated errors caused by low (near-zero or somewhat near-zero) actual values is tackled by having an epsilon, and one variant that is exactly the msMAPE as described by Chen and Yang in their 2004 paper "Assessing Forecast Accuracy Measures".

An aspect to clarify first

That said, for considering the first variant with the eps parameter actually being supported (as currently advertised somehow in the API documentation), it would first need to be clarified whether the already existing library-internal EPS parameter is already sort of used in the literary sense of "an epsilon when talking about more robust sMAPE", or if these are in fact two different applications of an epsilon, such that this circumstance could even justify setting both epsilons separately.

Additional notes

This issue description talks about sMAPE where s stands for symmetric. However, an even more thorough investigation could be considering variants of MAPE without the s, too, although this could be considered out-of-scope for now, as sMAPE with epsilon (in the most common sense) and msMAPE (as described in the above mentioned paper) are the ones of interest.

Guida contributor