2
$\begingroup$

Hello I am trying to calculate the MASE of a store at hourly level. My questions are below:

  1. If I sum up the different values, it sums to 24 (number of hours) and the average comes to 1. What am I doing wrong?

Numerator = Absolute Scaled Error Denominator: Mean of Absolute Scaled Error

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ I took the liberty of changing your title, because I think this may be a common confusion, and I think this title will be easier to google for future generations. Feel free to roll it back if you disagree. $\endgroup$ Commented Jul 28, 2022 at 12:10

1 Answer 1

6
$\begingroup$

Let $y_t$ be the actual and $\hat{y}_t$ the forecast at time $t$, for times $t=1, \dots, T$.

You calculate the denominator used in the MASE as $$ d := \frac{1}{T}\sum_{t=1}^T|\hat{y}_t-y_t|.$$ For a time point $\theta$, you calculate an Absolute Scaled Error as $$ \text{ASE}_\theta = \frac{|\hat{y}_\theta-y_\theta|}{d} = \frac{|\hat{y}_\theta-y_\theta|}{\frac{1}{T}\sum_{t=1}^T|\hat{y}_t-y_t|}.$$ Finally, you calculate the MASE as $$ \text{MASE} = \frac{1}{T}\sum_{\theta=1}^T\text{ASE}_\theta= \frac{1}{T}\sum_{\theta=1}^T\frac{|\hat{y}_\theta-y_\theta|}{\frac{1}{T}\sum_{t=1}^T|\hat{y}_t-y_t|} = \frac{\frac{1}{T}\sum_{\theta=1}^T|\hat{y}_\theta-y_\theta|}{\frac{1}{T}\sum_{t=1}^T|\hat{y}_t-y_t|},$$ which is equal to $1$, because the numerator and the denominator are the exact same sum.

Your error lies in your calculation of $d$. You need to use the MAE of some specific benchmark, which you want to compare your forecast to. In the original formulation by Hyndman & Koehler (2006), they used the MAE of the naive one-step-ahead forecast in-sample for $d$, but other people use a seasonal naive method, or some other benchmark, evaluated in-sample or out-of sample.

Essentially, what you did was to use the exact forecast you were evaluating as the benchmark that yielded $d$. So it is not surprising that when you compare your benchmark to itself, you get $1$.

Finally, note that because $d$ is a fixed number, minimizing the MASE amounts to minimizing the MAE, which elicits the conditional median, which may or may not be what you want (Kolassa, 2020).

$\endgroup$
5
  • $\begingroup$ Thanks a lot for your response I have a follow up response. I saw that in some cases for example below MASE is calculated using this method, in this case why is actuals for Day 1 being subtracted from actuals of Day 2 (link here as I'm unable to use picture while commenting: aws.amazon.com/blogs/machine-learning/… $\endgroup$ Commented Jul 28, 2022 at 12:28
  • $\begingroup$ Per above, the MASE is the ratio of the MAE of your focal method, divided by some benchmark error. On that page, they are using the MAE of the naive forecast as a benchmark. That is, as a benchmark they always use today's observation as the forecast for tomorrow, and they then calculate the MAE of this forecast, and this MAE is used as the denominator in the MASE. Of course, if you have only two data points, this reduces to the absolute difference between the two observations. $\endgroup$ Commented Jul 28, 2022 at 12:37
  • $\begingroup$ So in case of me having 24 observations, what you would suggest I should keep as the denominator in this case. I have a similar scenario where I am using today's observation as a benchmark for the forecast for tomorrow $\endgroup$ Commented Jul 28, 2022 at 12:40
  • $\begingroup$ Do you think I could possibly using standard deviation of the actual demand as a denominator? $\endgroup$ Commented Jul 28, 2022 at 12:45
  • $\begingroup$ The MAE of the naive forecast (as on the Amazon page) and the SD are valid possibilities. The choice is not really all that important. It is just a scaling factor to make the MAE of a forecasting method comparable across multiple series on different scales, so you can then compare the MASE of different forecasting methods applied to these different series. There are pointers to literature in the tag wiki. $\endgroup$ Commented Jul 28, 2022 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.