[Repost] Comprehensive Analysis of Large Model Evaluation Metrics: How to Accurately Measure AI Model Performance

[doggie]

This article is transcoded by SimpRead 简悦, original article at zhuanlan.zhihu.com

Starting with a chart

This is part of the multi-modal large model leaderboard updated by OpenCompass yesterday. We can see the scores of different multi-modal large models under the same evaluation benchmark. Have you ever wondered which metric or metrics these scores represent, and how these scores are obtained?

Today, let’s break down the large model evaluation metrics.

Large model evaluation metrics include Accuracy, Precision, Recall, F1 Score, ROUGE, Perplexity, BLEU, etc. We will explain what each metric is, what tasks they are suitable for, and how these metrics are applied in evaluation tasks one by one.

Accuracy

Accuracy is an important metric used to assess the performance of classification models. It represents the proportion of samples that the model correctly predicts out of the total samples.

The formula for accuracy is:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Where:

• TP (True Positives): The number of samples correctly predicted as positive by the model.
• TN (True Negatives): The number of samples correctly predicted as negative by the model.
• FP (False Positives): The number of samples incorrectly predicted as positive by the model.
• FN (False Negatives): The number of samples incorrectly predicted as negative by the model.

To illustrate with an example: suppose a doctor uses a certain testing method to diagnose a disease. We can map this example to classification model terms:

• True Positive (TP): Patients who actually have the disease and test positive.
• True Negative (TN): People who are healthy and test negative.
• False Positive (FP): Healthy people incorrectly tested as positive.
• False Negative (FN): Patients who actually have the disease but are tested negative.

Now suppose 100 people undergo the test, with 50 actually having the disease and 50 being healthy. The test results are:

• 40 diseased patients tested positive (TP)
• 10 diseased patients incorrectly tested negative (FN)
• 45 healthy people tested negative (TN)
• 5 healthy people incorrectly tested positive (FP)

Plugging the numbers into the formula:

$$ Accuracy = (40 + 45) / (40 + 45 + 5 + 10) = 85 / 100 = 0.85 $$

Therefore, this testing method has an accuracy of 85%, meaning 85% of all tested individuals were correctly diagnosed. Accuracy reflects the proportion of correct predictions made by the model or testing method across all samples. It is an intuitive performance metric but can be biased in certain situations, so it usually needs to be evaluated alongside other metrics.

Precision

Precision is an important performance metric used to measure the accuracy of a classification model when it predicts the positive class. It represents the proportion of actual positives among all samples predicted as positive by the model.

The formula for precision is:

Precision = TP / (TP + FP)

Using the previous example:

• 40 diseased patients tested positive — True Positives (TP)
• 5 healthy people incorrectly tested positive — False Positives (FP)

According to the precision formula:

$$ Precision = 40 / (40 + 5) \approx 0.889 $$

Thus, this testing method’s precision is about 88.9%. This means among all samples predicted positive, 88.9% are actually diseased. Precision is an important metric to evaluate model performance, especially in cases of class imbalance. It helps us understand the model’s accuracy in predicting the positive class and reflects the model’s reliability effectively.

Recall

Recall is another important performance metric used to measure the proportion of correctly predicted positive samples out of all actual positive samples.

The formula for recall is:

Recall = TP / (TP + FN)

Using the previous example:

• 40 diseased patients tested positive — True Positives (TP)
• 10 diseased patients incorrectly tested negative — False Negatives (FN)

According to the recall formula:

$$ Recall = 40 / (40 + 10) = 0.8 $$

Thus, this testing method’s recall is 80%. This means 80% of all actually diseased samples were correctly detected as positive. Recall reflects the model’s coverage of actual positive samples. It differs from precision by focusing on the model’s performance on true positive samples. In some scenarios, such as medical diagnosis, recall might be more important than precision because the goal is to identify as many actual diseased individuals as possible. A higher recall score is better.

F1 Score

The F1 score is an important metric used to comprehensively evaluate the precision and recall of a classification model. It is especially suitable for imbalanced datasets because it balances the effect of these two metrics.

The formula for the F1 score is:

$$ F1 = 2 \times (Precision \times Recall) / (Precision + Recall) $$

In the above example, Precision = 40/(40+5); Recall = 40/(40+10). Using the F1 score formula, we get:

$$ F1 = \frac{2 \times (40 \div 45) \times (40 \div 50)}{(40 \div 45 + 40 \div 50)} = 84.21% $$

The F1 score is a comprehensive metric that effectively reflects model performance in handling class imbalance. It is very important in many fields such as medical diagnosis and fraud detection, where the impact of false positives and false negatives differs. A higher F1 score is better.

ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a metric used for evaluating the quality of automatic text summarization and machine translation. It evaluates by comparing the similarity between the automatically generated summary and the reference summary (usually human-generated).

Main types of ROUGE:

• ROUGE-N: Calculates the n-gram overlap between the generated summary and the reference summary.
• ROUGE-L: Evaluates the matching based on the longest common subsequence (LCS) between the generated summary and the reference summary.
• ROUGE-W: N-gram evaluation considering word order weights.
• ROUGE-S: Evaluates the matching degree between generated sentences and the reference summary.

The formula for ROUGE-N is:

ROUGE - N = \frac{\sum_{S \in \text{Reference Summaries}} \sum_{n \text{ gram} \in S} Count_{\text{match}}(n \text{ gram})}{\sum_{S \in \text{Reference Summaries}} \sum_{n \text{gram} \in S} Count(n \text{ gram})}

For example, suppose we have the following automatic and reference summaries:

• Automatic summary: the cat was found under the bed
• Reference summary: the cat was under the bed

Calculating ROUGE-1:

• 1-gram extraction:

  • Automatic summary 1-grams: the, cat, was, found, under, the, bed
  • Reference summary 1-grams: the, cat, was, under, the, bed

• Number of overlapping 1-grams:

  • Overlaps: the, cat, was, under, the, bed (6 total)

• Calculating ROUGE-1:

  • Total 1-grams in reference summary: 6
  • ROUGE-1 = 6 / 6 = 1.0

This indicates the 1-gram overlap rate between the automatic and reference summaries is 100%.

Calculating ROUGE-2:

• 2-gram extraction:

  • Automatic summary 2-grams: the cat, cat was, was found, found under, under the, the bed
  • Reference summary 2-grams: the cat, cat was, was under, under the, the bed

• Number of overlapping 2-grams:

  • Overlaps: the cat, cat was, under the, the bed (4 total)

• Calculating ROUGE-2:

  • Total 2-grams in reference summary: 5
  • ROUGE-2 = 4 / 5 = 0.8

This indicates the 2-gram overlap rate between the automatic and reference summaries is 80%. ROUGE-1 and ROUGE-2 are the most commonly used evaluation metrics.

Perplexity

Perplexity is a commonly used metric in natural language processing (NLP) to evaluate the performance of language models. It measures the “confusion” degree of a language model when predicting a text sequence, reflecting the model’s ability to predict the next word.

Perplexity formula:

Perplexity can be defined as the exponentiated average negative log likelihood of a sequence. Specifically, for a given word sequence (w_1, w_2, \dots, w_N), the formula is:

PPL = \exp \left(-\frac{1}{N} \sum_{i=1}^N \log P(w_i|w_1, w_2, \dots, w_{i-1}) \right)

Where:

• (N) is the number of words in the sequence.
• (P(w_i|w_1, w_2, …, w_{i-1})) is the probability predicted by the model for the (i^{th}) word given the previous words.

Intuitively:

Perplexity can be viewed as the model’s “surprise” level when predicting the next word. A lower perplexity indicates the model is more confident in its predictions, whereas a higher perplexity means the model is more uncertain. For example, if a language model has a perplexity of 10, it means on average the model is choosing the next word from 10 possible options.

Suppose we have a simple language model predicting the next word given a context. For the sentence “the cat sat on the”, the model might give the following probabilities for the next word:

• (P(\text{mat} | \text{the cat sat on the}) = 0.6)
• (P(\text{floor} | \text{the cat sat on the}) = 0.3)
• (P(\text{table} | \text{the cat sat on the}) = 0.1)

If we calculate the perplexity for just the last word “mat”:

PPL = \exp(-\log(0.6)) \approx 1.51

This perplexity value indicates the confidence level of the model’s prediction. The smaller the perplexity, the more confident the model is in its word predictions.

BLEU

BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine translation. It scores by comparing the n-gram overlap between machine translation output and one or more reference translations.

Definition

The core idea of BLEU is to determine how similar the machine translation output is to human translations. The metric is computed following these steps:

• N-gram matching: Calculate the n-gram overlaps between candidate translation (machine output) and reference translations. Common n-grams include:

  • BLEU-1: unigram (single-word) overlap.
  • BLEU-2: bigram (two-word phrase) overlap.
  • BLEU-3 and BLEU-4: tri-gram and four-gram overlaps respectively.

• Modified precision: BLEU uses modified n-gram precision to avoid counting repeated matches of the same word multiple times. Specifically, the count of each n-gram in the candidate is capped at the maximum number it appears in any reference.

• Length penalty: To prevent machine translation systems from producing translations that are too short, BLEU incorporates a Brevity Penalty (BP), which ensures the length of the translation is relatively consistent with the references.

Formula:

BLEU = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n \right)

Where:

• (BP) is the brevity penalty factor.
• (N) is the maximum n-gram length (usually 4).
• (w_n) are the weighting coefficients (usually set to (1/N)).
• (p_n) is the precision of n-grams.

Intuitive explanation:

Suppose we have the following reference and machine translations:

• Reference translation: The cat is on the mat.
• Machine translation: The cat sat on the mat.

Extract the n-grams:

• Reference unigrams: [The, cat, is, on, the, mat]
• Machine unigrams: [The, cat, sat, on, the, mat]

Calculate matching unigrams:

• Matches: [The, cat, on, the, mat] (5 matches)

Calculate unigram precision:

• Machine unigrams total: 6
• (p_1 = 5 / 6 \approx 0.8333)

Calculate brevity penalty:

• Suppose both translations have length 6, so (BP = 1.0).

Calculate BLEU score:

• Final BLEU score:

BLEU = 1.0 \times \exp(\log(0.8333)) \approx 0.8333

A higher BLEU score indicates better translation quality.

Back to the initial question: Which metrics correspond to the different benchmark scores on the leaderboard? This depends on the specific implementation.

Generally speaking, a benchmark can correspond to multiple tasks and multiple metrics. The next section will provide detailed introductions.