.. _sec_autograd:
自动微分
========
正如
:numref:`sec_calculus`\ 中所说，求导是几乎所有深度学习优化算法的关键步骤。
虽然求导的计算很简单，只需要一些基本的微积分。
但对于复杂的模型，手工进行更新是一件很痛苦的事情（而且经常容易出错）。
深度学习框架通过自动计算导数，即\ *自动微分*\ （automatic
differentiation）来加快求导。
实际中，根据设计好的模型，系统会构建一个\ *计算图*\ （computational
graph）， 来跟踪计算是哪些数据通过哪些操作组合起来产生输出。
自动微分使系统能够随后反向传播梯度。
这里，\ *反向传播*\ （backpropagate）意味着跟踪整个计算图，填充关于每个参数的偏导数。
一个简单的例子
--------------
作为一个演示例子，假设我们想对函数\ :math:`y=2\mathbf{x}^{\top}\mathbf{x}`\ 关于列向量\ :math:`\mathbf{x}`\ 求导。
首先，我们创建变量\ ``x``\ 并为其分配一个初始值。
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import autograd, np, npx
npx.set_np()
x = np.arange(4.0)
x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[07:14:47] ../src/storage/storage.cc:196: Using Pooled (Naive) StorageManager for CPU
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([0., 1., 2., 3.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
x = torch.arange(4.0)
x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([0., 1., 2., 3.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
x = tf.range(4, dtype=tf.float32)
x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import warnings
warnings.filterwarnings(action='ignore')
import paddle
x = paddle.arange(4, dtype='float32')
x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=float32, place=Place(cpu), stop_gradient=True,
[0., 1., 2., 3.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 通过调用attach_grad来为一个张量的梯度分配内存
x.attach_grad()
# 在计算关于x的梯度后，将能够通过'grad'属性访问它，它的值被初始化为0
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([0., 0., 0., 0.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.requires_grad_(True) # 等价于x=torch.arange(4.0,requires_grad=True)
x.grad # 默认值是None
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x = tf.Variable(x)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x = paddle.to_tensor(x, stop_gradient=False)
x.grad # 默认值是None
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 把代码放到autograd.record内，以建立计算图
with autograd.record():
y = 2 * np.dot(x, x)
y
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array(28.)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y = 2 * torch.dot(x, x)
y
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor(28., grad_fn=)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 把所有计算记录在磁带上
with tf.GradientTape() as t:
y = 2 * tf.tensordot(x, x, axes=1)
y
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y = 2 * paddle.dot(x, x)
y
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[1], dtype=float32, place=Place(cpu), stop_gradient=False,
[28.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y.backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[07:14:47] ../src/base.cc:48: GPU context requested, but no GPUs found.
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([ 0., 4., 8., 12.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y.backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([ 0., 4., 8., 12.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x_grad = t.gradient(y, x)
x_grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y.backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=float32, place=Place(cpu), stop_gradient=False,
[0. , 4. , 8. , 12.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.grad == 4 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([ True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.grad == 4 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x_grad == 4 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.grad == 4 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=bool, place=Place(cpu), stop_gradient=False,
[True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
with autograd.record():
y = x.sum()
y.backward()
x.grad # 被新计算的梯度覆盖
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([1., 1., 1., 1.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 在默认情况下，PyTorch会累积梯度，我们需要清除之前的值
x.grad.zero_()
y = x.sum()
y.backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([1., 1., 1., 1.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
with tf.GradientTape() as t:
y = tf.reduce_sum(x)
t.gradient(y, x) # 被新计算的梯度覆盖
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 在默认情况下，PaddlePaddle会累积梯度，我们需要清除之前的值
x.clear_gradient()
y = paddle.sum(x)
y.backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=float32, place=Place(cpu), stop_gradient=False,
[1., 1., 1., 1.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 当对向量值变量y（关于x的函数）调用backward时，将通过对y中的元素求和来创建
# 一个新的标量变量。然后计算这个标量变量相对于x的梯度
with autograd.record():
y = x * x # y是一个向量
y.backward()
x.grad # 等价于y=sum(x*x)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([0., 2., 4., 6.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 对非标量调用backward需要传入一个gradient参数，该参数指定微分函数关于self的梯度。
# 本例只想求偏导数的和，所以传递一个1的梯度是合适的
x.grad.zero_()
y = x * x
# 等价于y.backward(torch.ones(len(x)))
y.sum().backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([0., 2., 4., 6.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
with tf.GradientTape() as t:
y = x * x
t.gradient(y, x) # 等价于y=tf.reduce_sum(x*x)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.clear_gradient()
y = x * x
paddle.sum(y).backward()
x.grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=float32, place=Place(cpu), stop_gradient=False,
[0., 2., 4., 6.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
with autograd.record():
y = x * x
u = y.detach()
z = u * x
z.backward()
x.grad == u
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([ True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.grad.zero_()
y = x * x
u = y.detach()
z = u * x
z.sum().backward()
x.grad == u
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 设置persistent=True来运行t.gradient多次
with tf.GradientTape(persistent=True) as t:
y = x * x
u = tf.stop_gradient(y)
z = u * x
x_grad = t.gradient(z, x)
x_grad == u
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.clear_gradient()
y = x * x
u = y.detach()
z = u * x
paddle.sum(z).backward()
x.grad == u
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=bool, place=Place(cpu), stop_gradient=False,
[True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y.backward()
x.grad == 2 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([ True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.grad.zero_()
y.sum().backward()
x.grad == 2 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
t.gradient(y, x) == 2 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
x.clear_gradient()
paddle.sum(y).backward()
x.grad == 2 * x
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=bool, place=Place(cpu), stop_gradient=False,
[True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def f(a):
b = a * 2
while np.linalg.norm(b) < 1000:
b = b * 2
if b.sum() > 0:
c = b
else:
c = 100 * b
return c
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def f(a):
b = a * 2
while b.norm() < 1000:
b = b * 2
if b.sum() > 0:
c = b
else:
c = 100 * b
return c
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def f(a):
b = a * 2
while tf.norm(b) < 1000:
b = b * 2
if tf.reduce_sum(b) > 0:
c = b
else:
c = 100 * b
return c
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def f(a):
b = a * 2
while paddle.norm(b) < 1000:
b = b * 2
if paddle.sum(b) > 0:
c = b
else:
c = 100 * b
return c
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a = np.random.normal()
a.attach_grad()
with autograd.record():
d = f(a)
d.backward()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a = torch.randn(size=(), requires_grad=True)
d = f(a)
d.backward()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a = tf.Variable(tf.random.normal(shape=()))
with tf.GradientTape() as t:
d = f(a)
d_grad = t.gradient(d, a)
d_grad
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a = paddle.to_tensor(paddle.randn(shape=[1]), stop_gradient=False)
d = f(a)
d.backward()
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a.grad == d / a
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array(True)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a.grad == d / a
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor(True)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
d_grad == d / a
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
a.grad == d / a
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[1], dtype=bool, place=Place(cpu), stop_gradient=False,
[True])
.. raw:: html
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html