Edit - Cross Validated

1. Home
2. Questions
3. Unanswered
4. AI Assist
5. Tags
7. Chat
8. Users
Stack Internal

Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work.
Try for free Learn more
Stack Internal
Bring the best of human thought and AI automation together at your work. Learn more

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

Best way to optimize MAPE

The MAPE is a metric that can be used for regression problems :

$$\mbox{MAPE} = \frac{1}{n}\sum_{t=1}^n \left|\frac{A_t-F_t}{A_t}\right|$$

Where $A$ represents the actual value and $F$ the the forecast.

I have to optimize my models with respect to this metric. However, I am not sure of the best way to proceed. I could rewrite the objective function of my models (but most of the common libraries do not support custom objective functions) but this requires a lot of efforts.

Alternatively, I could use a transformation $f$ of the target, run the learning on the image $f(target)$ and return the prediction $f^{-1}(predicted)$.

I noticed that training the model, keeping the sum of squares metric, on $\log(target)$ and returning $\exp(predicted)$ resulted in a significant improvement.

Is there a way to know the best transform to use? Or should I cross validate over various transformations of the target?

Answer*

> I noticed that training the model, keeping the sum of squares metric, on log(target) and returning exp(predicted) resulted in a significant improvement.

This does not surprise me. Look at things probabilistically. Your out-of-sample targets follow a certain unknown distribution. You are calculating a point forecast, which is a one-point summary of this unknown distribution, using the expected MAPE as a loss function.

The *first* key question is the following: *which functional of the unknown future distribution minimizes the expected MAPE?* Minimizing the MAPE will automatically draw your predictions to this functional. It turns out that the expected MAPE is minimized by a somewhat exotic functional, the (-1)-median of the future distribution ([Gneiting, 2011, *JASA*](http://www.tandfonline.com/doi/abs/10.1198/jasa.2011.r10138), p. 752 with $\beta = -1$).

However, and this is the *second* key question, I strongly suspect that your *estimation* procedure does not aim at this functional, but uses a loss function that is minimized by the *expected value* of the residuals. This discrepancy between the loss function used by your model estimation algorithm and the loss function you use to evaluate predictions leads to strange results.

[The MAPE tag wiki contains pointers to literature.](https://stats.stackexchange.com/tags/mape/info)

Here is an example. Suppose you have [lognormally distributed data](https://en.wikipedia.org/wiki/Log-normal_distribution), with mean 0 and variance 1 on the log scale, and you have no useful predictors. Without predictors, the fit should be the same for all observations.

 * Minimizing the in-sample squared error will lead to a fit which is simply the mean of your data, or $\sqrt{e}\approx 1.65$. A simulation tells us that this fit yields an expected MAPE of about 197% and an expected MSE of about 4.70.
 * However, minimizing the MAPE will lead to the (-1)-median of your data, which for the lognormal distribution turns out to be its mode at $\frac{1}{e}\approx 0.37$. The minimal expected MAPE is 68%, while the expected MSE for this fit is 6.34.

The MSE-optimal fit is five times as large as the MAPE-optimal fit, because of the asymmetry of the MAPE - fits that are too large can incur APEs larger than 100%, while the APE is bounded by 100% for fits that are too small. This pulls the MAPE-optimal fit towards zero.

 * If you take logarithms of your data and then estimate an MSE-minimizing fit on the logged data, we of course get the mean of the logged data, which is zero. Backtransforming this yields a fit of 1, which is optimal for neither the MAPE nor the MSE for original data, but is closer to the MAPE-optimal value than the original MSE-optimal fit. It yields a MAPE of 113% and an MSE of 5.12.

You probably have asymmetrically distributed data. I would take a *very* critical look at whether the MAPE is really a useful error measure in such a situation. If you do decide to minimize the MAPE, the best solution would quite probably indeed be to change the objective function. If this is not possible, cross-validation and checking various parameters for (say) Box-Cox transformations may be your best bet. (But it would still feel like hammering a nail into a board using a screwdriver instead of a hammer.) In addition, I would very much recommend that you look at your fits' and forecasts' bias and think about whether large biases are a source of concern.

Here is R code for the simulations:

    set.seed(1)
    xx <- rlnorm(1e7)
    yy <- sqrt(exp(1))
    yy
    mean(abs(xx-yy)/xx)
    mean((xx-yy)^2)
    
    yy <- 1/exp(1)
    yy
    mean(abs(xx-yy)/xx)
    mean((xx-yy)^2)
    
    yy <- 1
    mean(abs(xx-yy)/xx)
    mean((xx-yy)^2)

Edit Summary*

Cancel

$\begingroup$ Thank you so much for the detailed answers ! What is the -1-median ? I could not find informations about it... $\endgroup$

RUser4512
– RUser4512

2016-05-27 08:49:26 +00:00
Commented May 27, 2016 at 8:49
$\begingroup$ That's quite understandable, it's not very common. I have edited my post to include a link to Gneiting (2011, JASA). It's discussed on page 752. $\endgroup$

Stephan Kolassa
– Stephan Kolassa

2016-05-27 08:58:04 +00:00
Commented May 27, 2016 at 8:58

Add a comment |

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="/api/flow.js?q=https%3A%2F%2Fexample.com">example</a>
MathJax equations $\sin^2 \theta$

formatting help »
answering help »

MathJax help »