# 交叉熵损失cross entropy loss **URL:** \ **Category:** 🛠工具与编程\ **Tags:** 基本概念, 损失函数, 交叉熵, 分类任务\ **Created:** [2025 年11 月 25 日 10:06 UTC](https://forum.beginner.center/t/topic/2432 "2025-11-25T10:06:24Z")\ **Posts on this page:** 2\ **Page:** 1
作者： ![doggie](https://forum.beginner.center/user_avatar/forum.beginner.center/doggie/32/550_2.png) [@doggie](https://forum.beginner.center/u/doggie)\ 发布日期： [2025 年8 月 6 日 03:15 UTC](https://forum.beginner.center/t/topic/2432/1 "2025-08-06T03:15:40Z")
## 原文链接 [损失函数｜交叉熵损失函数](https://www.zhihu.com/tardis/zm/art/35709485) ## 正文 这篇文章中，讨论的Cross Entropy损失函数常用于分类问题中，但是为什么它会在分类问题中这么有效呢？我们先从一个简单的分类例子来入手。 ## **1. 图像分类任务** 我们希望根据图片动物的轮廓、颜色等特征，来预测动物的类别，有三种可预测类别：猫、狗、猪。假设我们当前有两个模型（参数不同），这两个模型都是通过sigmoid/softmax的方式得到对于每个预测结果的概率值： **模型1** ： | 预测 | 真实 | 是否正确 | | --- | --- | --- | | 0.3 0.3 0.4 | 0 0 1 (猪) | 正确 | | 0.3 0.4 0.3 | 0 1 0 (狗) | 正确 | | 0.1 0.2 0.7 | 1 0 0 (猫) | 错误 | **模型1** 对于样本1和样本2以非常微弱的优势判断正确，对于样本3的判断则彻底错误。 **模型2** ： | 预测 | 真实 | 是否正确 | | --- | --- | --- | | 0.1 0.2 0.7 | 0 0 1 (猪) | 正确 | | undefined | -— | -— | | 0.1 0.7 0.2 | 0 1 0 (狗) | 正确 | | undefined | -— | -— | | 0.3 0.4 0.3 | 1 0 0 (猫) | 错误 | | undefined | -— | -— | **模型2** 对于样本1和样本2判断非常准确，对于样本3判断错误，但是相对来说没有错得太离谱。 好了，有了模型之后，我们需要通过定义损失函数来判断模型在样本上的表现了，那么我们可以定义哪些损失函数呢？ ## **1.1 Classification Error（分类错误率）** 最为直接的损失函数定义为： ![classification error=frac{count of error items}{count of all items}](https://www.zhihu.com/equation?tex=classification%5C+error%3D%5Cfrac%7Bcount%5C+of%5C+error%5C+items%7D%7Bcount%5C+of+%5C+all%5C+items%7D&consumer=ZHI_MENG) **模型1：**![classification error=frac{1}{3}](https://www.zhihu.com/equation?tex=classification%5C+error%3D%5Cfrac%7B1%7D%7B3%7D&consumer=ZHI_MENG) **模型2：**![classification error=frac{1}{3}](https://www.zhihu.com/equation?tex=classification%5C+error%3D%5Cfrac%7B1%7D%7B3%7D&consumer=ZHI_MENG) 我们知道， **模型1** 和 **模型2** 虽然都是预测错了1个，但是相对来说 **模型2** 表现得更好，损失函数值照理来说应该更小，但是，很遗憾的是， ![classification error](https://www.zhihu.com/equation?tex=classification%5C+error&consumer=ZHI_MENG) 并不能判断出来，所以这种损失函数虽然好理解，但表现不太好。 ## **1.2 Mean Squared Error (均方误差)** 均方误差损失也是一种比较常见的损失函数，其定义为： ![MSE=frac{1}{n}sum_{i}^n(hat{y_i}-y_i)^2](https://www.zhihu.com/equation?tex=MSE%3D%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%7D%5En%28%5Chat%7By_i%7D-y_i%29%5E2&consumer=ZHI_MENG) **模型1：** ![begin{aligned} text{sample 1 loss=}(0.3-0)^2 + (0.3-0)^2 + (0.4-1)^2 = 0.54 text{sample 2 loss=}(0.3-0)^2 + (0.4-1)^2 + (0.3-0)^2 = 0.54 text{sample 3 loss=}(0.1-1)^2 + (0.2-0)^2 + (0.7-0)^2 = 1.34 end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++++%5Ctext%7Bsample+1+loss%3D%7D%280.3-0%29%5E2+%2B+%280.3-0%29%5E2+%2B+%280.4-1%29%5E2+%3D+0.54+%5C%5C++++%5Ctext%7Bsample+2+loss%3D%7D%280.3-0%29%5E2+%2B+%280.4-1%29%5E2+%2B+%280.3-0%29%5E2+%3D+0.54+%5C%5C++++%5Ctext%7Bsample+3+loss%3D%7D%280.1-1%29%5E2+%2B+%280.2-0%29%5E2+%2B+%280.7-0%29%5E2+%3D+1.34+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 对所有样本的loss求平均： ![MSE=frac{0.54+0.54+1.34}{3}=0.81](https://www.zhihu.com/equation?tex=MSE%3D%5Cfrac%7B0.54%2B0.54%2B1.34%7D%7B3%7D%3D0.81+%5C%5C&consumer=ZHI_MENG) **模型2：** ![begin{aligned} & text{sample 1 loss=}(0.1-0)^2 + (0.2-0)^2 + (0.7-1)^2 = 0.14 &text{sample 2 loss=}(0.1-0)^2 + (0.7-1)^2 + (0.2-0)^2 = 0.14 &text{sample 3 loss=}(0.3-1)^2 + (0.4-0)^2 + (0.3-0)^2 = 0.74 end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+++%26+%5Ctext%7Bsample+1+loss%3D%7D%280.1-0%29%5E2+%2B+%280.2-0%29%5E2+%2B+%280.7-1%29%5E2+%3D+0.14%5C%5C++++%26%5Ctext%7Bsample+2+loss%3D%7D%280.1-0%29%5E2+%2B+%280.7-1%29%5E2+%2B+%280.2-0%29%5E2+%3D+0.14%5C%5C++++%26%5Ctext%7Bsample+3+loss%3D%7D%280.3-1%29%5E2+%2B+%280.4-0%29%5E2+%2B+%280.3-0%29%5E2+%3D+0.74%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 对所有样本的loss求平均： ![MSE=frac{0.14+0.14+0.74}{3}=0.34](https://www.zhihu.com/equation?tex=MSE%3D%5Cfrac%7B0.14%2B0.14%2B0.74%7D%7B3%7D%3D0.34+%5C%5C&consumer=ZHI_MENG) 我们发现，MSE能够判断出来 **模型2** 优于 **模型1** ，那为什么不采样这种损失函数呢？主要原因是在分类问题中，使用sigmoid/softmx得到概率，配合MSE损失函数时，采用梯度下降法进行学习时，会出现模型一开始训练时，学习速率非常慢的情况（[MSE损失函数](https://zhuanlan.zhihu.com/p/35707643)）。 有了上面的直观分析，我们可以清楚的看到，对于分类问题的损失函数来说，分类错误率和均方误差损失都不是很好的损失函数，下面我们来看一下交叉熵损失函数的表现情况。 ## **1.3 Cross Entropy Loss Function（交叉熵损失函数）** ## **1.3.1 表达式** ## **(1) 二分类** 在二分的情况下，模型最后需要预测的结果只有两种情况，对于每个类别我们的预测得到的概率为 ![p](https://www.zhihu.com/equation?tex=p&consumer=ZHI_MENG) 和 ![1-p](https://www.zhihu.com/equation?tex=1-p&consumer=ZHI_MENG) ，此时表达式为（ ![log](https://www.zhihu.com/equation?tex=log&consumer=ZHI_MENG) 的底数是 ![e](https://www.zhihu.com/equation?tex=e&consumer=ZHI_MENG)）： ![L = frac{1}{N}sum_{i} L_i = frac{1}{N}sum_{i}-y_icdot log(p_i) + (1-y_i)cdot log(1-p_i)](https://www.zhihu.com/equation?tex=L+%3D+%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bi%7D+L_i+%3D+%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bi%7D-%5By_i%5Ccdot+log%28p_i%29+%2B+%281-y_i%29%5Ccdot+log%281-p_i%29%5D+%5C%5C&consumer=ZHI_MENG) 其中： - ![y_i](https://www.zhihu.com/equation?tex=y_i&consumer=ZHI_MENG) —— 表示样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 的label，正类为 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，负类为 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) - ![p_i](https://www.zhihu.com/equation?tex=p_i&consumer=ZHI_MENG) —— 表示样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 预测为正类的概率 ## **(2) 多分类** 多分类的情况实际上就是对二分类的扩展： ![L = frac{1}{N}sum_{i} L_i = - frac{1}{N}sum_{i} sum_{c=1}^My_{ic}log(p_{ic})](https://www.zhihu.com/equation?tex=L+%3D+%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bi%7D+L_i+%3D+-+%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bi%7D+%5Csum_%7Bc%3D1%7D%5EMy_%7Bic%7D%5Clog%28p_%7Bic%7D%29+%5C%5C&consumer=ZHI_MENG) 其中： - ![M](https://www.zhihu.com/equation?tex=M&consumer=ZHI_MENG) ——类别的数量 - ![y_{ic}](https://www.zhihu.com/equation?tex=y_%7Bic%7D&consumer=ZHI_MENG) ——符号函数（ ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) 或 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ），如果样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 的真实类别等于 ![c](https://www.zhihu.com/equation?tex=c&consumer=ZHI_MENG) 取 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，否则取 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) - ![p_{ic}](https://www.zhihu.com/equation?tex=p_%7Bic%7D&consumer=ZHI_MENG) ——观测样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 属于类别 ![c](https://www.zhihu.com/equation?tex=c&consumer=ZHI_MENG) 的预测概率 现在我们利用这个表达式计算上面例子中的损失函数值： **模型1** ： ![begin{aligned} text{sample 1 loss} = - (0times log0.3 + 0times log0.3 + 1times log0.4) = 0.91 text{sample 2 loss} = - (0times log0.3 + 1times log0.4 + 0times log0.3) = 0.91 text{sample 3 loss} = - (1times log0.1 + 0times log0.2 + 0times log0.7) = 2.30 end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++++%5Ctext%7Bsample+1+loss%7D+%3D+-+%280%5Ctimes+log0.3+%2B+0%5Ctimes+log0.3+%2B+1%5Ctimes+log0.4%29+%3D+0.91+%5C%5C++++%5Ctext%7Bsample+2+loss%7D+%3D+-+%280%5Ctimes+log0.3+%2B+1%5Ctimes+log0.4+%2B+0%5Ctimes+log0.3%29+%3D+0.91+%5C%5C++++%5Ctext%7Bsample+3+loss%7D+%3D+-+%281%5Ctimes+log0.1+%2B+0%5Ctimes+log0.2+%2B+0%5Ctimes+log0.7%29+%3D+2.30+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 对所有样本的loss求平均： ![L=frac{0.91+0.91+2.3}{3}=1.37](https://www.zhihu.com/equation?tex=L%3D%5Cfrac%7B0.91%2B0.91%2B2.3%7D%7B3%7D%3D1.37+%5C%5C&consumer=ZHI_MENG) **模型2：** ![begin{aligned} text{sample 1 loss} = - (0times log0.1 + 0times log0.2 + 1times log0.7) = 0.35 text{sample 2 loss} = - (0times log0.1 + 1times log0.7 + 0times log0.2) = 0.35 text{sample 3 loss} = - (1times log0.3 + 0times log0.4 + 0times log0.4) = 1.20 end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++++%5Ctext%7Bsample+1+loss%7D+%3D+-+%280%5Ctimes+log0.1+%2B+0%5Ctimes+log0.2+%2B+1%5Ctimes+log0.7%29+%3D+0.35+%5C%5C++++%5Ctext%7Bsample+2+loss%7D+%3D+-+%280%5Ctimes+log0.1+%2B+1%5Ctimes+log0.7+%2B+0%5Ctimes+log0.2%29+%3D+0.35+%5C%5C++++%5Ctext%7Bsample+3+loss%7D+%3D+-+%281%5Ctimes+log0.3+%2B+0%5Ctimes+log0.4+%2B+0%5Ctimes+log0.4%29+%3D+1.20+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 对所有样本的loss求平均： ![L=frac{0.35+0.35+1.2}{3}=0.63](https://www.zhihu.com/equation?tex=L%3D%5Cfrac%7B0.35%2B0.35%2B1.2%7D%7B3%7D%3D0.63+%5C%5C&consumer=ZHI_MENG) 上述计算可以使用python的sklearn库 ```auto from sklearn.metrics import log_loss y_true = [[0, 0, 1], [0, 1, 0], [1, 0, 0]] y_pred_1 = [[0.3, 0.3, 0.4], [0.3, 0.4, 0.3], [0.1, 0.2, 0.7]] y_pred_2 = [[0.1, 0.2, 0.7], [0.1, 0.7, 0.2], [0.3, 0.4, 0.3]] print(log_loss(y_true, y_pred_1)) print(log_loss(y_true, y_pred_2)) ____________ 1.3783888522474517 0.6391075640678003 ``` 可以发现，交叉熵损失函数可以捕捉到 **模型1** 和 **模型2** 预测效果的差异。 ## **2. 函数性质** ![](https://discoursecdn.beginner.center/original/2X/e/e21e08d37445fd425a36e082af533468cb95023d.webp) 可以看出，该函数是凸函数，求导时能够得到全局最优值。 ## **3. 学习过程** 交叉熵损失函数经常用于分类问题中，特别是在神经网络做分类问题时，也经常使用交叉熵作为损失函数，此外，由于交叉熵涉及到计算每个类别的概率，所以交叉熵几乎每次都和**sigmoid(或softmax)函数**一起出现。 我们用神经网络最后一层输出的情况，来看一眼整个模型预测、获得损失和学习的流程： 1. 神经网络最后一层得到每个类别的得分 **scores（也叫logits）** ； 2. 该得分经过**sigmoid(或softmax)函数**获得概率输出； 3. 模型预测的类别概率输出与真实类别的one hot形式进行交叉熵损失函数的计算。 学习任务分为二分类和多分类情况，我们分别讨论这两种情况的学习过程。 ## **3.1 二分类情况** ![](https://discoursecdn.beginner.center/original/2X/b/b8823be065eb1fc13b01ee555a6933429c373520.webp) 二分类交叉熵损失函数学习过程 如上图所示，求导过程可分成三个子过程，即拆成三项偏导的乘积： ![frac{partial L_i}{partial w_i}=frac{partial L_i}{partial p_i}cdot frac{partial p_i}{partial s_i}cdot frac{partial s_i}{partial w_i}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+w_i%7D%3D%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_i%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_i%7D%7B%5Cpartial+s_i%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_i%7D%7B%5Cpartial+w_i%7D%5C%5C&consumer=ZHI_MENG) ## **3.1.1 计算第一项： ![frac{partial L_i}{partial p_i}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_i%7D&consumer=ZHI_MENG)** ![L_i = -y_icdot log(p_i) + (1-y_i)cdot log(1-p_i)](https://www.zhihu.com/equation?tex=L_i+%3D+-%5By_i%5Ccdot+log%28p_i%29+%2B+%281-y_i%29%5Ccdot+log%281-p_i%29%5D+%5C%5C&consumer=ZHI_MENG) - ![p_i](https://www.zhihu.com/equation?tex=p_i&consumer=ZHI_MENG) 表示样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 预测为正类的概率 - ![y_i](https://www.zhihu.com/equation?tex=y_i&consumer=ZHI_MENG) 为符号函数，样本 ![i](https://www.zhihu.com/equation?tex=i&consumer=ZHI_MENG) 为正类时取 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，否则取 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) ![begin{aligned} frac{partial L_i}{partial p_i} &=frac{partial -y_icdot log(p_i) + (1-y_i)cdot log(1-p_i)}{partial p_i} &= -frac{y_i}{p_i}-(1-y_i)cdot frac{1}{1-p_i}cdot (-1) &= -frac{y_i}{p_i}+frac{1-y_i}{1-p_i} &= -frac{y_i}{sigma(s_i)}+frac{1-y_i}{1-sigma(s_i)} end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_i%7D+%26%3D%5Cfrac%7B%5Cpartial+-%5By_i%5Ccdot+log%28p_i%29+%2B+%281-y_i%29%5Ccdot+log%281-p_i%29%5D%7D%7B%5Cpartial+p_i%7D%5C%5C++%26%3D+-%5Cfrac%7By_i%7D%7Bp_i%7D-%5B%281-y_i%29%5Ccdot+%5Cfrac%7B1%7D%7B1-p_i%7D%5Ccdot+%28-1%29%5D+%5C%5C+++%26%3D+-%5Cfrac%7By_i%7D%7Bp_i%7D%2B%5Cfrac%7B1-y_i%7D%7B1-p_i%7D+%5C%5C++%26%3D+-%5Cfrac%7By_i%7D%7B%5Csigma%28s_i%29%7D%2B%5Cfrac%7B1-y_i%7D%7B1-%5Csigma%28s_i%29%7D+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) ## **3.1.2 计算第二项： ![frac{partial p_i}{partial s_i}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p_i%7D%7B%5Cpartial+s_i%7D+&consumer=ZHI_MENG)** 这一项要计算的是sigmoid函数对于score的导数，我们先回顾一下sigmoid函数和分数求导的公式： > ![p = sigma(s) = frac{e^{s}}{1+e^{s}}](https://www.zhihu.com/equation?tex=p+%3D+%5Csigma%28s%29+%3D+%5Cfrac%7Be%5E%7Bs%7D%7D%7B1%2Be%5E%7Bs%7D%7D++%5C%5C&consumer=ZHI_MENG) > ![f'(x) = frac{g(x)}{h(x)}=frac{g'(x)h(x)-g(x){h}'(x)}{h^2(x)}](https://www.zhihu.com/equation?tex=f%27%28x%29+%3D+%5Cfrac%7Bg%28x%29%7D%7Bh%28x%29%7D%3D%5Cfrac%7Bg%27%28x%29h%28x%29-g%28x%29%7Bh%7D%27%28x%29%7D%7Bh%5E2%28x%29%7D+%5C%5C&consumer=ZHI_MENG) ![begin{aligned} frac{partial p_i}{partial s_i} &= frac{(e^{s_i})'cdot (1+e^{s_i})-e^{s_i}cdot (1+e^{s_i})'}{(1+e^{s_i})^2} &= frac{e^{s_i}cdot (1+e^{s_i})-e^{s_i}cdot e^{s_i}}{(1+e^{s_i})^2} &= frac{e^{s_i}}{(1+e^{s_i})^2} &= frac{e^{s_i}}{1+e^{s_i}}cdot frac{1}{1+e^{s_i}} &= sigma(s_i)cdot 1-sigma(s_i) end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++%5Cfrac%7B%5Cpartial+p_i%7D%7B%5Cpartial+s_i%7D+%26%3D+%5Cfrac%7B%28e%5E%7Bs_i%7D%29%27%5Ccdot+%281%2Be%5E%7Bs_i%7D%29-e%5E%7Bs_i%7D%5Ccdot+%281%2Be%5E%7Bs_i%7D%29%27%7D%7B%281%2Be%5E%7Bs_i%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_i%7D%5Ccdot+%281%2Be%5E%7Bs_i%7D%29-e%5E%7Bs_i%7D%5Ccdot+e%5E%7Bs_i%7D%7D%7B%281%2Be%5E%7Bs_i%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_i%7D%7D%7B%281%2Be%5E%7Bs_i%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_i%7D%7D%7B1%2Be%5E%7Bs_i%7D%7D%5Ccdot+%5Cfrac%7B1%7D%7B1%2Be%5E%7Bs_i%7D%7D+%5C%5C++%26%3D+%5Csigma%28s_i%29%5Ccdot+%5B1-%5Csigma%28s_i%29%5D+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) ## **3.1.3 计算第三项： ![frac{partial s_i}{partial w_i }](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+s_i%7D%7B%5Cpartial+w_i+%5C%5C%7D&consumer=ZHI_MENG)** 一般来说，scores是输入的线性函数作用的结果，所以有： ![frac{partial s_i}{partial w_i}=x_i](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+s_i%7D%7B%5Cpartial+w_i%7D%3Dx_i+%5C%5C&consumer=ZHI_MENG) ## **3.1.4 计算结果 ![frac{partial L_i}{partial w_i}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+w_i%7D&consumer=ZHI_MENG)** ![begin{aligned} frac{partial L_i}{partial w_i} &= frac{partial L_i}{partial p_i}cdot frac{partial p_i}{partial s_i}cdot frac{partial s_i}{partial w_i} &= -frac{y_i}{sigma(s_i)}+frac{1-y_i}{1-sigma(s_i)} cdot sigma(s_i)cdot 1-sigma(s_i)cdot x_i &= -frac{y_i}{sigma(s_i)}cdot sigma(s_i)cdot (1-sigma(s_i))+frac{1-y_i}{1-sigma(s_i)}cdot sigma(s_i)cdot (1-sigma(s_i))cdot x_i &= -y_i+y_icdot sigma(s_i)+sigma(s_i)-y_icdot sigma(s_i)cdot x_i &= sigma(s_i)-y_icdot x_i end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+w_i%7D+%26%3D+%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_i%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_i%7D%7B%5Cpartial+s_i%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_i%7D%7B%5Cpartial+w_i%7D+%5C%5C+++%26%3D+%5B-%5Cfrac%7By_i%7D%7B%5Csigma%28s_i%29%7D%2B%5Cfrac%7B1-y_i%7D%7B1-%5Csigma%28s_i%29%7D%5D+%5Ccdot+%5Csigma%28s_i%29%5Ccdot+%5B1-%5Csigma%28s_i%29%5D%5Ccdot+x_i+%5C%5C+++%26%3D+%5B-%5Cfrac%7By_i%7D%7B%5Csigma%28s_i%29%7D%5Ccdot+%5Csigma%28s_i%29%5Ccdot+%281-%5Csigma%28s_i%29%29%2B%5Cfrac%7B1-y_i%7D%7B1-%5Csigma%28s_i%29%7D%5Ccdot+%5Csigma%28s_i%29%5Ccdot+%281-%5Csigma%28s_i%29%29%5D%5Ccdot+x_i+%5C%5C+++%26%3D+%5B-y_i%2By_i%5Ccdot+%5Csigma%28s_i%29%2B%5Csigma%28s_i%29-y_i%5Ccdot+%5Csigma%28s_i%29%5D%5Ccdot+x_i+%5C%5C++%26%3D+%5B%5Csigma%28s_i%29-y_i%5D%5Ccdot+x_i+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 可以看到，我们得到了一个非常漂亮的结果，所以，使用交叉熵损失函数，不仅可以很好的衡量模型的效果，又可以很容易的的进行求导计算。 ## **3.2 多分类情况** ![](https://discoursecdn.beginner.center/original/2X/8/8160f080b5a8d6983d8547744b4171fe55220d82.webp) 多分类交叉熵损失函数学习过程 如上图所示，求导过程可以分为三个子过程： ![frac{partial L_i}{partial w_{ic}}=frac{partial L_{i}}{partial p_{ik}}cdot frac{partial p_{ik}}{partial s_{ic}}cdot frac{partial s_{ic}}{partial w_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+w_%7Bic%7D%7D%3D%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+p_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%5C%5C&consumer=ZHI_MENG) 和二分类区别在于： - 因为多分类只有一个类别为 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，其他为 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) ，不失一般性，我们可以假设 ![y_{ik}](https://www.zhihu.com/equation?tex=y_%7Bik%7D&consumer=ZHI_MENG) 为 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，其他为 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) ，所以损失函数求和式子中只有 ![y_{ik}](https://www.zhihu.com/equation?tex=y_%7Bik%7D&consumer=ZHI_MENG) 这项不为 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) ，即 ![L_i = -log(p_{ik})](https://www.zhihu.com/equation?tex=L_i+%3D+-%5Clog%28p_%7Bik%7D%29&consumer=ZHI_MENG) - ![frac{partial p_{ik}}{partial s_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D&consumer=ZHI_MENG) 这一项求导时，需要针对 ![c](https://www.zhihu.com/equation?tex=c&consumer=ZHI_MENG) 是否等于 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) 进行分类讨论（这里 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) 表示的是样本真实类别为 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) ， ![c](https://www.zhihu.com/equation?tex=c&consumer=ZHI_MENG) 表示的是对我们想对输入到 ![s_{ic}](https://www.zhihu.com/equation?tex=s_%7Bic%7D&consumer=ZHI_MENG) 的参数 ![w_{ic}](https://www.zhihu.com/equation?tex=w_%7Bic%7D&consumer=ZHI_MENG) 求导） ### **3.2.1 计算第一项： ![frac{partial L_i}{partial p_{ik}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_%7Bik%7D%7D&consumer=ZHI_MENG)** 不失一般性，我们可以假设 ![y_{ik}](https://www.zhihu.com/equation?tex=y_%7Bik%7D&consumer=ZHI_MENG) 为 ![1](https://www.zhihu.com/equation?tex=1&consumer=ZHI_MENG) ，其他为 ![0](https://www.zhihu.com/equation?tex=0&consumer=ZHI_MENG) ，则 ![L_i = -log(p_{ik})](https://www.zhihu.com/equation?tex=L_i+%3D+-%5Clog%28p_%7Bik%7D%29+%5C%5C&consumer=ZHI_MENG) 求导： ![begin{aligned} frac{partial L_i}{partial p_{ik}} &=frac{partial -log(p_{ik})}{partial p_{ik}} &= -frac{1}{p_{ik}} end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%5Cfrac%7B%5Cpartial+L_i%7D%7B%5Cpartial+p_%7Bik%7D%7D+%26%3D%5Cfrac%7B%5Cpartial+-%5Clog%28p_%7Bik%7D%29%7D%7B%5Cpartial+p_%7Bik%7D%7D%5C%5C+%26%3D+-%5Cfrac%7B1%7D%7Bp_%7Bik%7D%7D%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) ### **3.2.2 计算第二项： ![frac{partial p_{ik}}{partial s_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D&consumer=ZHI_MENG)** 这一项要计算的是softmax函数对于得分的导数，我们先回顾一下softmax函数和分数求导的公式： > ![p_{ik} = sigma(s_{ik}) = frac{e^{s_{ik}}}{sum e^{s_{ij}}}](https://www.zhihu.com/equation?tex=p_%7Bik%7D+%3D+%5Csigma%28s_%7Bik%7D%29+%3D+%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D+%5C%5C&consumer=ZHI_MENG) > ![f'(x) = frac{g(x)}{h(x)}=frac{g'(x)h(x)-g(x){h}'(x)}{h^2(x)}](https://www.zhihu.com/equation?tex=f%27%28x%29+%3D+%5Cfrac%7Bg%28x%29%7D%7Bh%28x%29%7D%3D%5Cfrac%7Bg%27%28x%29h%28x%29-g%28x%29%7Bh%7D%27%28x%29%7D%7Bh%5E2%28x%29%7D+%5C%5C&consumer=ZHI_MENG) 这里 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) 表示的是样本真实类别为 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) ， ![c](https://www.zhihu.com/equation?tex=c&consumer=ZHI_MENG) 表示的是对输入到 ![s_{ic}](https://www.zhihu.com/equation?tex=s_%7Bic%7D&consumer=ZHI_MENG) 的参数 ![w_{ic}](https://www.zhihu.com/equation?tex=w_%7Bic%7D&consumer=ZHI_MENG) 求导，这时候存在两种情况： **情况1:**![c=k](https://www.zhihu.com/equation?tex=c%3Dk&consumer=ZHI_MENG) 则第二项的求导式子，可以写成： ![frac{partial p_{ik}}{partial s_{ic}} = frac{partial p_{ik}}{partial s_{ik}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D+%3D+%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bik%7D%7D+%5C%5C&consumer=ZHI_MENG) 求导后得 ![begin{aligned} frac{partial p_{ik}}{partial s_{ik}} &= frac{frac{partial e^{s_{ik}}}{partial s_{ik}}cdot sum e^{s_{ij}} -e^{s_{ik}}cdot frac{partial sum e^{s_{ij}}}{partial s_{ik}}}{(sum e^{s_{ij}})^2} &= frac{e^{s_{ik}}cdot sum e^{s_{ij}}-e^{s_{ik}}cdot e^{s_{ik}}}{(sum e^{s_{ij}})^2} &= frac{e^{s_{ik}}}{sum e^{s_{ij}}} - (frac{e^{s_{ik}}}{sum e^{s_{ij}}})^2 &= frac{e^{s_{ik}}}{sum e^{s_{ij}}}cdot (1-frac{e^{s_{ik}}}{sum e^{s_{ij}}}) &= p_{ik}cdot (1-p_{ik}) end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bik%7D%7D+%26%3D+%5Cfrac%7B%5Cfrac%7B%5Cpartial+e%5E%7Bs_%7Bik%7D%7D%7D%7B%5Cpartial+s_%7Bik%7D%7D%5Ccdot+%5Csum+e%5E%7Bs_%7Bij%7D%7D+-e%5E%7Bs_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%7B%5Cpartial+s_%7Bik%7D%7D%7D%7B%28%5Csum+e%5E%7Bs_%7Bij%7D%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%5Ccdot+%5Csum+e%5E%7Bs_%7Bij%7D%7D-e%5E%7Bs_%7Bik%7D%7D%5Ccdot+e%5E%7Bs_%7Bik%7D%7D%7D%7B%28%5Csum+e%5E%7Bs_%7Bij%7D%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D+-+%28%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%29%5E2+%5C%5C++%26%3D+%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%5Ccdot+%281-%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%29+%5C%5C++%26%3D+p_%7Bik%7D%5Ccdot+%281-p_%7Bik%7D%29+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) **情况2:**![cneq k](https://www.zhihu.com/equation?tex=c%5Cneq+k&consumer=ZHI_MENG) 此时 ![s_{ic}](https://www.zhihu.com/equation?tex=s_%7Bic%7D&consumer=ZHI_MENG) 这一项只在分母中存在，求导后得： ![begin{aligned} frac{partial p_{ik}}{partial s_{ic}} &= frac{frac{partial e^{s_{ik}}}{partial s_{ic}}cdot sum e^{s_{ij}} -e^{s_{ik}}cdot frac{partial sum e^{s_{ij}}}{partial s_{ic}}}{(sum e^{s_{ij}})^2} &= frac{0cdot sum e^{s_{ij}}-e^{s_{ik}}cdot e^{s_{ic}}}{(sum e^{s_{ij}})^2} &= -frac{e^{s_{ik}}cdot e^{s_{ic}}}{(sum e^{s_{ij}})^2} &= -frac{e^{s_{ik}}}{sum e^{s_{ij}}}cdot frac{e^{s_{ic}}}{sum e^{s_{ij}}} &= -p_{ik}cdot p_{ic} end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D++%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D+%26%3D+%5Cfrac%7B%5Cfrac%7B%5Cpartial+e%5E%7Bs_%7Bik%7D%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%5Ccdot+%5Csum+e%5E%7Bs_%7Bij%7D%7D+-e%5E%7Bs_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%7D%7B%28%5Csum+e%5E%7Bs_%7Bij%7D%7D%29%5E2%7D+%5C%5C++%26%3D+%5Cfrac%7B0%5Ccdot+%5Csum+e%5E%7Bs_%7Bij%7D%7D-e%5E%7Bs_%7Bik%7D%7D%5Ccdot+e%5E%7Bs_%7Bic%7D%7D%7D%7B%28%5Csum+e%5E%7Bs_%7Bij%7D%7D%29%5E2%7D+%5C%5C++%26%3D+-%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%5Ccdot+e%5E%7Bs_%7Bic%7D%7D%7D%7B%28%5Csum+e%5E%7Bs_%7Bij%7D%7D%29%5E2%7D++%5C%5C++%26%3D+-%5Cfrac%7Be%5E%7Bs_%7Bik%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D%5Ccdot+%5Cfrac%7Be%5E%7Bs_%7Bic%7D%7D%7D%7B%5Csum+e%5E%7Bs_%7Bij%7D%7D%7D+%5C%5C++%26%3D+-p_%7Bik%7D%5Ccdot+p_%7Bic%7D+%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) ### **3.2.3 计算第三项： ![frac{partial s_{ic}}{partial w_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D&consumer=ZHI_MENG)** 一般来说，scores是输入的线性函数作用的结果，所以有： ![frac{partial s_{ic}}{partial w_{ic}}=x_{ic}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D%3Dx_%7Bic%7D+%5C%5C&consumer=ZHI_MENG) ### **3.2.4 计算结果 ![frac{partial L_{i}}{partial w_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D&consumer=ZHI_MENG)** ![frac{partial L_{i}}{partial w_{ic}}=frac{partial L_{i}}{partial p_{ik}}cdot frac{partial p_{ik}}{partial s_{ic}}cdot frac{partial s_{ic}}{partial w_{ic}}](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D%3D%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+p_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%5C%5C&consumer=ZHI_MENG) **情况1:**![c=k](https://www.zhihu.com/equation?tex=c%3Dk&consumer=ZHI_MENG) ![begin{aligned} frac{partial L_{i}}{partial w_{ic}} &= frac{partial L_{i}}{partial p_{ik}}cdot frac{partial p_{ik}}{partial s_{ic}}cdot frac{partial s_{ic}}{partial w_{ic}} &= (-frac{1}{p_{ik}})cdot p_{ik}cdot (1-p_{ik})cdot x_{ik} &= (p_{ik} - 1)cdot x_{ik} &= (p_{ik} - y_{ik})cdot x_{ik} &= sigma(s_{ik}) - y_{ik}cdot x_{ik} end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%26%3D+%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+p_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%5C%5C+%26%3D+%28-%5Cfrac%7B1%7D%7Bp_%7Bik%7D%7D%29%5Ccdot+%5Bp_%7Bik%7D%5Ccdot+%281-p_%7Bik%7D%29%5D%5Ccdot+x_%7Bik%7D%5C%5C+%26%3D+%28p_%7Bik%7D+-+1%29%5Ccdot+x_%7Bik%7D%5C%5C+%26%3D+%28p_%7Bik%7D+-+y_%7Bik%7D%29%5Ccdot+x_%7Bik%7D%5C%5C+%26%3D+%5B%5Csigma%28s_%7Bik%7D%29+-+y_%7Bik%7D%5D%5Ccdot+x_%7Bik%7D%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) **情况2:**![cneq k](https://www.zhihu.com/equation?tex=c%5Cneq+k&consumer=ZHI_MENG) ![begin{aligned} frac{partial L_{i}}{partial w_{ic}} &= frac{partial L_{i}}{partial p_{ik}}cdot frac{partial p_{ik}}{partial s_{ic}}cdot frac{partial s_{ic}}{partial w_{ic}} &= (-frac{1}{p_{ik}})cdot -p_{ik}cdot p_{ic}cdot x_{ic} &= p_{ic}cdot x_{ic} &= (p_{ic}-0)cdot x_{ic} &= (p_{ic}-y_{ic})cdot x_{ic} &= sigma(s_{ic})-y_{ic}cdot x_{ic} end{aligned}](https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%26%3D+%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+p_%7Bik%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+p_%7Bik%7D%7D%7B%5Cpartial+s_%7Bic%7D%7D%5Ccdot+%5Cfrac%7B%5Cpartial+s_%7Bic%7D%7D%7B%5Cpartial+w_%7Bic%7D%7D+%5C%5C+%26%3D+%28-%5Cfrac%7B1%7D%7Bp_%7Bik%7D%7D%29%5Ccdot+%5B-p_%7Bik%7D%5Ccdot+p_%7Bic%7D%5D%5Ccdot+x_%7Bic%7D%5C%5C+%26%3D+p_%7Bic%7D%5Ccdot+x_%7Bic%7D%5C%5C+%26%3D+%28p_%7Bic%7D-0%29%5Ccdot+x_%7Bic%7D%5C%5C+%26%3D+%28p_%7Bic%7D-y_%7Bic%7D%29%5Ccdot+x_%7Bic%7D%5C%5C+%26%3D+%5B%5Csigma%28s_%7Bic%7D%29-y_%7Bic%7D%5D%5Ccdot+x_%7Bic%7D%5C%5C+%5Cend%7Baligned%7D+%5C%5C&consumer=ZHI_MENG) 不失一般性，我们上述假设样本的真实类别为 ![k](https://www.zhihu.com/equation?tex=k&consumer=ZHI_MENG) ，则有: ![y_{ik}=1](https://www.zhihu.com/equation?tex=y_%7Bik%7D%3D1+%5C%5C&consumer=ZHI_MENG) ![y_{ic}=0, cneq k](https://www.zhihu.com/equation?tex=y_%7Bic%7D%3D0%2C+c%5Cneq+k+%5C%5C&consumer=ZHI_MENG) 我们求导时，对不同情况带入 ![y](https://www.zhihu.com/equation?tex=y&consumer=ZHI_MENG) 的值后，得到了一致的表达式，如果采用向量化的形式，那么导数就不用再分情况写了，统一成： ![frac{partial L_{i}}{partial w_{i}} = sigma(s_i)-y_icdot x_i](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+L_%7Bi%7D%7D%7B%5Cpartial+w_%7Bi%7D%7D+%3D+%5B%5Csigma%28s_i%29-y_i%5D%5Ccdot+x_i+%5C%5C&consumer=ZHI_MENG) 可以看出，交叉熵损失函数对于二分类和多分类求导时，采用向量化的形式后，求导结果的形式是一致的。 ## **4. 优缺点** ## **4.1 优点** 在用梯度下降法做参数更新的时候，模型学习的速度取决于两个值：一、 **学习率** ；二、 **偏导值** 。其中，学习率是我们需要设置的超参数，所以我们重点关注偏导值。从上面的式子中，我们发现，偏导值的大小取决于 ![x_i](https://www.zhihu.com/equation?tex=x_i&consumer=ZHI_MENG) 和 ![sigma(s)-y](https://www.zhihu.com/equation?tex=%5B%5Csigma%28s%29-y%5D&consumer=ZHI_MENG) ，我们重点关注后者，后者的大小值反映了我们模型的错误程度，该值越大，说明模型效果越差，但是该值越大同时也会使得偏导值越大，从而模型学习速度更快。所以，使用逻辑函数得到概率，并结合交叉熵当损失函数时，在模型效果差的时候学习速度比较快，在模型效果好的时候学习速度变慢。 ## **4.2 缺点** Deng [4]在2019年提出了ArcFace Loss，并在论文里说了Softmax Loss的两个缺点：1、随着分类数目的增大，分类层的线性变化矩阵参数也随着增大；2、对于封闭集分类问题，学习到的特征是可分离的，但对于开放集人脸识别问题，所学特征却没有足够的区分性。对于人脸识别问题，首先人脸数目(对应分类数目)是很多的，而且会不断有新的人脸进来，不是一个封闭集分类问题。 另外，sigmoid(softmax)+cross-entropy loss 擅长于学习类间的信息，因为它采用了类间竞争机制，它只关心对于正确标签预测概率的准确性，忽略了其他非正确标签的差异，导致学习到的特征比较散。基于这个问题的优化有很多，比如对softmax进行改进，如L-Softmax、SM-Softmax、AM-Softmax等。 ## **5. 参考** [1]. [博客 - 神经网络的分类模型 LOSS 函数为什么要用 CROSS ENTROPY](http://jackon.me/posts/why-use-cross-entropy-error-for-loss-function/) [2]. [博客 - Softmax as a Neural Networks Activation Function](http://sefiks.com/2017/11/08/softmax-as-a-neural-networks-activation-function/) [3]. [博客 - A Gentle Introduction to Cross-Entropy Loss Function](https://sefiks.com/2017/12/17/a-gentle-introduction-to-cross-entropy-loss-function/) [4]. Deng, Jiankang, et al. “Arcface: Additive angular margin loss for deep face recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. * * * ---
作者： ![doggie](https://forum.beginner.center/user_avatar/forum.beginner.center/doggie/32/550_2.png) [@doggie](https://forum.beginner.center/u/doggie)\ 发布日期： [2025 年11 月 25 日 10:06 UTC](https://forum.beginner.center/t/topic/2432/2 "2025-11-25T10:06:25Z")
交叉熵损失cross entropy loss是分类任务中常见的主要损失函数 ![image](https://discoursecdn.beginner.center/original/2X/e/eb40bb00b33c1a49208e95173c54c26a6d9498a9.jpeg) ![image](https://discoursecdn.beginner.center/original/2X/7/7f663d2683db33c5401bb1ec531031c6dbf5f59b.jpeg)