# 对数几率回归（逻辑回归）原理与Python实现

#### 一、对数几率和对数几率回归

$logit=\log\frac{y^*}{1-y^*}$

$condition1:w^Tx=\log\frac{y^*}{1-y^*}$

#### 二、Sigmoid函数

$y^*=\frac{e^{w^Tx}}{1+e^{w^Tx}}=\frac{1}{1+e^{-w^Tx}}$

![](https://img2020.cnblogs.com/blog/1740641/202101/1740641-20210110191640063-801988894.png)

$h'(z)=\frac{-e^{-z}}{(1+e^{-z})^2}=\frac{1}{1+e^{-z}}(1-\frac{1}{1+e^{-z}})=h(z)(1-h(z))$

#### 三、极大似然法

$P(y=1|x)=y^*=h(w^Tx)\,\,\,\,\,\,\,\,P(y=0|x)=1-y^*=1-h(w^Tx)$

$P(y|x)=h(w^Tx)^y(1-h(w^Tx))^{1-y}$

$L(y_i|x_i,w)=\prod^{m}_{i=1}h(w^Tx_i)^{y_i}(1-h(w^Tx_i))^{1-{y_i}}$

$l(y_i|x_i,w)=log(L)=\sum^{m}_{i=1}{(y_i\log h(w^Tx_i)+(1-y_i)log(1-h(w^Tx_i)))}$

$J(w)=-\frac{1}{m}\sum^{m}_{i=1}{(y_i\log h(w^Tx_i)+(1-y_i)log(1-h(w^Tx_i)))}$

#### 四、梯度下降法

$w=\arg\min J(w)=\arg\min -\frac{1}{m}\sum^{m}_{i=1}{(y_i\log h(w^Tx_i)+(1-y_i)log(1-h(w^Tx_i)))}$

《线性回归：梯度下降法优化》 中，我已经详细介绍了梯度下降法的数学原理，这里直接使用梯度下降法来对对数几率回归模型进行优化。

$$J(w)$$

$\frac{\partial J}{\partial w}=-\frac{1}{m}\sum^{m}_{i=1}(y_i(1-h(w^Tx_i))x_i+(y_i-1)h(w^Tx_i)x_i)=-\frac{1}{m}\sum^{m}_{i=1}{(y_i-h(w^Tx_i))x_i}$

$$\frac{\partial J}{\partial w}$$ 带入参数 $$w$$ 的更新公式 $$w^*=w-\eta\frac{\partial J}{\partial w}$$ ，最终得到 $$w$$ 的更新公式如下：

$w^*=w+\frac{\eta}{m}\sum^{m}_{i=1}{(y_i-h(w^Tx_i))x_i}$

#### 四、Python实现

def fit(self, X, y):
self.W = np.zeros(X.shape[1] + 1)
for i in range(self.max_iter):
delta = self._activation(self._linear_func(X)) - y
self.W[0] -= self.eta * delta.sum()
self.W[1:] -= self.eta * (delta @ X)
return self

if __name__ == "__main__":
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X = irirs["data"][:100]
y = irirs["target"][:100]
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3)
classifier = LRClassifier().fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classification_report(y_test, y_pred))