Edit - Cross Validated

1. Home
2. Questions
3. Unanswered
4. AI Assist
5. Tags
7. Chat
8. Users
Stack Internal

Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work.
Try for free Learn more
Stack Internal
Bring the best of human thought and AI automation together at your work. Learn more

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.

Accuracy, the proportion of correct classifications among all classifications, is very simple and very "intuitive" measure, yet it may be a poor measure for imbalanced data. Why does our intuition misguide us here and are there any other problems with this measure?

Answer*

After reading through all the answers above, here is an appeal to common sense.  Optimality is a flexible term and always needs to be qualified; in other words, saying a model or algorithm is "optimal" is meaningless, especially in a scientific sense.

Whenever anyone says they are scientifically optimizing something, I recommend asking a question like: "In what sense do you define optimality?"  This is because in science, unless you can measure something, you cannot optimize (maximize, minimize, etc.) it.

As an example, the OP asks the following:

"*Why is accuracy not the best measure for assessing classification models?*"

There is an embedded reference to optimization in the word "best" from the question above.  "Best" is meaningless in science because "goodness" cannot be measured scientifically.

The scientifically correct response to this question is that the OP needed to define what "good" means.  In the real world (outside of academic exercises and Kaggle competitions) there is always a cost/benefit structure to consider when using a machine to suggest or make decisions to or on behalf of/instead of people.

For classification tasks, that information can be embedded in a cost/benefit matrix with entries corresponding to those of the confusion matrix.  Finally, since cost/benefit information is a function of the people who are considering using mechanistic help for their decision-making, it is subject to change with the circumstances, and therefore, there is never going to be one fixed measure of optimality which will work for all time in even one problem, let alone all problems (i.e., "models") involving classification.

Any measure of optimality for classification which ignores costs does so at its own risk.  Even the ROC AUC fails to be cost-invariant, as shown in this [figure][1].


  [1]: https://www.researchgate.net/profile/Mark_Oxley/publication/258366133/figure/fig2/AS:670019147018253@1536756421516/Two-ROC-Curves-with-Equal-AUC.png

Edit Summary*

Cancel

3

$\begingroup$ One of my favorite quotes from a statistician (Youden): "It is, in fact, not a statistical matter to decide what weights should be attached to these two types of diagnostic error." First page of acsjournals.onlinelibrary.wiley.com/doi/abs/10.1002/… $\endgroup$

tdMJN6B2JtUe
– tdMJN6B2JtUe

2020-03-28 19:00:21 +00:00
Commented Mar 28, 2020 at 19:00
$\begingroup$ Link to my full master's thesis (from which the above linked image was taken): researchgate.net/publication/… $\endgroup$

tdMJN6B2JtUe
– tdMJN6B2JtUe

2020-10-14 21:18:56 +00:00
Commented Oct 14, 2020 at 21:18
$\begingroup$ After reading the answers above and discussions of "imbalanced classes," I am reminded of N. N. Taleb's writings (e.g., "The Black Swan"). Just sayin... $\endgroup$

tdMJN6B2JtUe
– tdMJN6B2JtUe

2020-11-09 14:38:07 +00:00
Commented Nov 9, 2020 at 14:38

Add a comment |

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="/api/flow.js?q=https%3A%2F%2Fexample.com">example</a>
MathJax equations $\sin^2 \theta$

formatting help »
answering help »

MathJax help »