Pytorch2

7. 神经网络

7.1 神经网络基本骨架-nn.Module的使用

官方说明

代码如下所示，必须要重写forward函数。

Python

import torch
from torch import nn

class Howjul(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, input):
        output = input + 1
        return output

howjul = Howjul()
x = torch.tensor(1.0)
output = howjul(x)
print(output)

7-1-nn_Module.py

7.2 卷积操作

官方说明

torch.nn和torch.nn.functional的区别：

• torch.nn：平常使用这个就足够了，已经封装好了；
• torch.nn.functional：更加精细的操作，用于了解神经网络函数内部的操作，其实和torch.nn里面的函数是一一对应的，这里为了讲解，选用torch.nn.functional。

本次以torch.nn.functional.conv2d为例，参数如下：

• input：输入，注意是四维张量
• weight：卷积核，注意也是四维张量
• stride：步长，卷积层每次移动的距离，(1,2)表示横向步长和纵向步长；
• padding：边缘填充

Python

import torch
import torch.nn.functional as F


input = torch.tensor([[1, 2, 0, 3, 1],
                      [0, 1, 2, 3, 1],
                      [1, 2, 1, 0, 0],
                      [5, 2, 3, 1, 1],
                      [2, 1, 0, 1, 1]])

kernel = torch.tensor([[1, 2, 1],
                       [0, 1, 0],
                       [2, 1, 0]])

input = torch.reshape(input, (1, 1, 5, 5))
kernel = torch.reshape(kernel, (1, 1, 3, 3))

output = F.conv2d(input, kernel, stride=1)
print(output)

output2 = F.conv2d(input, kernel, stride=2)
print(output2)

output3 = F.conv2d(input, kernel, stride=1, padding=1)
print(output3)

7-2-Conv_operation.py

7.3 卷积层

以torch.nn.Conv2d为例，这是处理二维图像最常用的函数，参数如下：

• in_channels (int) – Number of channels in the input image
• out_channels (int) – Number of channels produced by the convolution
• 输出通道数大于1但是输入通道数为1，那么会生成多个卷积核对输入进行卷积运算
• kernel_size (int or tuple) – Size of the convolving kernel
• stride (int or tuple, optional) – Stride of the convolution. Default: 1
• padding (int, tuple or str, optional) – Padding added to all four sides of the input. Default: 0
• padding_mode (str, optional) – 'zeros', 'reflect', 'replicate' or 'circular'. Default: 'zeros'
• dilation (int or tuple, optional) – Spacing between kernel elements. Default: 1
• groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1
• bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

Python

import torch
import torchvision
from torch import nn
from torch.nn import Conv2d
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

dataset = torchvision.datasets.CIFAR10("./Dataset",
                                       train=False,
                                       transform=torchvision.transforms.ToTensor(),
                                       download=True)
dataloader = DataLoader(dataset, batch_size=64)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.conv1 = Conv2d(in_channels=3, out_channels=6, kernel_size=3, stride=1, padding=0)

    def forward(self, x):
        x = self.conv1(x)
        return x


howjul = Howjul()
print(howjul)

writer = SummaryWriter("logs")
step = 0
for data in dataloader:
    imgs, targets = data
    output = howjul(imgs)
    print(imgs.shape)
    print(output.shape)
    # torch.Size([64, 3, 32, 32])
<div markdown="1" style="margin-top: -30px; font-size: 0.75em; opacity: 0.7;">
:material-circle-edit-outline: 约 1435 个字 :fontawesome-solid-code: 406 行代码 :material-image-multiple-outline: 12 张图片 :material-clock-time-two-outline: 预计阅读时间 10 分钟
</div>
    writer.add_images("input", imgs, step)
    # torch.Size([64, 6, 30, 30]) -> [xxx, 3, 30, 30]
    # tensorboard无法显示六通道，所以要进行reshape
    # 输出shape的第一个值可以使用-1，函数自动进行计算
    output = torch.reshape(output, (-1, 3, 30, 30))
    writer.add_images("output", output, step)
    step = step + 1

7-3-Conv.py

7.4 最大池化的使用

以torch.nn.MaxPool2d为例，参数为：

• kernel_size (Union[int, Tuple[int, int]]) – the size of the window to take a max over
• stride (Union[int, Tuple[int, int]]) – the stride of the window. Default value is kernel_size
• padding (Union[int, Tuple[int, int]]) – Implicit negative infinity padding to be added on both sides
• dilation (Union[int, Tuple[int, int]]) – a parameter that controls the stride of elements in the window
• return_indices (bool) – if True, will return the max indices along with the outputs. Useful for torch.nn.MaxUnpool2d later
• ceil_mode (bool) – when True, will use ceil instead of floor to compute the output shape

如下图为一个最大池化的例子，就是在原来的图像中，每3x3的像素选最大的那个像素，如果Ceil_model为False，那么如果不足3x3的尺寸则会舍弃。

Python

import torch
from torch import nn
from torch.nn import MaxPool2d

input = torch.tensor([[1, 2, 0, 3, 1],
                      [0, 1, 2, 3, 1],
                      [1, 2, 1, 0, 0],
                      [5, 2, 3, 1, 1],
                      [2, 1, 0, 1, 1]], dtype=torch.float32)

input = torch.reshape(input, (-1, 1, 5, 5))
print(input.shape)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.maxpool1 = MaxPool2d(kernel_size=3, ceil_mode=True)

    def forward(self, input):
        output = self.maxpool1(input)
        return output


howjul = Howjul()
output = howjul(input)
print(output)

最大池化的作用：保留数据的特征的同时把数据量减小。

与之前的结合，对数据集进行处理

Python

import torch
import torchvision
from torch import nn
from torch.nn import MaxPool2d
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

dataset = torchvision.datasets.CIFAR10("./Dataset",
                                       train=False,
                                       transform=torchvision.transforms.ToTensor(),
                                       download=True)
dataloader = DataLoader(dataset, batch_size=64)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.maxpool1 = MaxPool2d(kernel_size=3, ceil_mode=True)

    def forward(self, input):
        output = self.maxpool1(input)
        return output


howjul = Howjul()

writer = SummaryWriter("logs_maxpool")
step = 0

for data in dataloader:
    imgs, labels = data
    writer.add_images("Input", imgs, step)
    output = howjul(imgs)
    writer.add_images("Output", output, step)
    step = step + 1

writer.close()

运行效果如下，可以看出池化层产生了类似马赛克的效果

7-4-nn_Pooling.py

7-4-nn_Pooling_dataset.py

7.5 非线性激活

比较常见的是 torch.nn.ReLU 和 torch.nn.Sigmoid，代码演示以ReLU为例

Python

import torch
from torch import nn
from torch.nn import ReLU

input = torch.tensor([[1, -0.5],
                      [-1, 3]])

output = torch.reshape(input, (-1, 1, 2, 2))
print(output.shape)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.relu1 = ReLU(inplace=False) # 不进行原地替换

    def forward(self, x):
        output = self.relu1(x)
        return output


howjul = Howjul()
output = howjul(input)
print(output)

输出结果为：

Text Only

torch.Size([1, 1, 2, 2])
tensor([[1., 0.],
        [0., 3.]])

我们再使用sigmoid函数对数据集进行变换

Python

import torch
import torchvision
from torch import nn
from torch.nn import ReLU, Sigmoid
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

dataset = torchvision.datasets.CIFAR10("./Dataset",
                                       train=False,
                                       transform=torchvision.transforms.ToTensor(),
                                       download=True)
dataloader = DataLoader(dataset, batch_size=64)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.relu1 = ReLU(inplace=False)  # 不进行原地替换
        self.sigmoid1 = Sigmoid()

    def forward(self, x):
        output = self.sigmoid1(x)
        return output


howjul = Howjul()

writer = SummaryWriter("logs_relu")
step = 0
for data in dataloader:
    imgs, labels = data
    writer.add_images("Input", imgs, step)
    output = howjul(imgs)
    writer.add_images("Output", output, step)
    step = step + 1
writer.close()

输出结果为：

7-5-nn_Relu.py

7-5-nn_Relu_dataset.py

7.6 线性层及其他层介绍

其他层：

• Padding Layers：进行填充的层

• Normalization Layers：对输入采用正则化，采用正则化可以加快神经网络的训练速度

• Dropout Layers：训练过程中，随机地把一些input变成0，防止过拟合

特定网络结构：

• Recurrent Layers：有RNN、LSTM等，是一些特定的网络结构
• Transform Layers：也是特定的网络结构
• Sparse Layers：主要用于自然语言处理

---

线性层 Linear Layers

Python

torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)

$$g_1 = (k_1 \times x_1 + b_1)+(k_2 \times x_2 + b_2) + ...$$

$b$ 就是如上函数参数中的 bias ，$g$ 就是输出，$x$ 就是输入。

对于每个 $x$ 的权重 $k$ 和偏移 $b$ ，会按照如下的方式进行确定

我们以 vgg16 为例，在最后的时候4096变成了1000，就是采用了线性层进行变换

Python

import torch
import torchvision
from torch import nn
from torch.nn import Linear
from torch.utils.data import DataLoader

dataset = torchvision.datasets.CIFAR10("./Dataset", train=False, transform=torchvision.transforms.ToTensor(), download=True)
dataloader = DataLoader(dataset, batch_size=64)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul,self).__init__()
        self.linear1 = Linear(196608, 10)

    def forward(self, x):
        x = self.linear1(x)
        return x


howjul = Howjul()

for imgs, labels in dataloader:
    print(imgs.shape)
    # output = torch.reshape(imgs, (1, 1, 1, -1))
    output = torch.flatten(imgs)  # 把输入展成一行
    print(output.shape)
    output = howjul(output)
    print(output.shape)

7-6-nn_Linear.py

7.7 搭建小实战和Sequential的使用

torch.nn.Sequential ，是一类容器，比 torch.nn.Module 更加简单，直接看如下例子即可，

Python

# Using Sequential to create a small model. 
model = nn.Sequential(
          nn.Conv2d(1,20,5),
          nn.ReLU(),
          nn.Conv2d(20,64,5),
          nn.ReLU()
        )

# Using Sequential with OrderedDict.
model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))

接下来我们实现如下所示的网络结构

对于第一个卷积层，输入通道为3，输出通道为32，卷积核为5，要保持尺寸仍然为32x32，那么我们需要进行如下计算，因为不进行空洞卷积，所以 dilation 为1，我们要求的是 padding 和 stride ，我们把 stride 设置为1，那么 padding 就是2。

最后的神经网络定义代码如下所示：

Python

class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.conv1 = Conv2d(3, 32, 5, padding=2)
        self.maxpool1 = MaxPool2d(2)
        self.conv2 = Conv2d(32, 32, 5, padding=2)
        self.maxpool2 = MaxPool2d(2)
        self.conv3 = Conv2d(32, 64, 5, padding=2)
        self.maxpool3 = MaxPool2d(2)
        self.flatten = Flatten()
        self.linear1 = Linear(1024, 64)
        self.linear2 = Linear(64, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.maxpool1(x)
        x = self.conv2(x)
        x = self.maxpool2(x)
        x = self.conv3(x)
        x = self.maxpool3(x)
        x = self.flatten(x)
        x = self.linear1(x)
        x = self.linear2(x)
        return x

也可以是使用 Sequential 函数，使代码更加简洁

Python

class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.model1 = Sequential(
            Conv2d(3, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 64, 5, padding=2),
            MaxPool2d(2),
            Flatten(),
            Linear(1024, 64),
            Linear(64, 10)
        )

    def forward(self, x):
        x = self.model1(x)
        return x

然后可以使用 tensorboard 来使得过程可视化

Python

howjul = Howjul()
input = torch.ones((64, 3, 32, 32))
output = howjul(input)
print(output.shape)

writer = SummaryWriter("logs_seq")
writer.add_graph(howjul, input)
writer.close()

结果如下

7-7-nn_Sequential.py

7.8 损失函数与反向传播

损失函数一方面可以计算实际输出和目标之间的差距，另一方面可以为我们更新输出提供一定的依据（反向传播）比如计算出梯度。

Loss Functions 官方文档，在使用损失函数的时候，只需要注意自己的需求和输入输出即可。

L1Loss：直接对应的绝对值差进行相加或者取平均

Python

inputs = torch.tensor([1, 2, 3], dtype=torch.float32)
targets = torch.tensor([1, 2, 5], dtype=torch.float32)

inputs = torch.reshape(inputs, (1, 1, 1, 3))
targets = torch.reshape(targets, (1, 1, 1, 3))

loss = L1Loss(reduction='mean')
result = loss(inputs, targets) # 结果为0.6667

loss2 = L1Loss(reduction='sum')
result = loss(inputs, targets) # 结果为2.

MSELoss：对应的绝对值差的平方

Python

loss3 = MSELoss(reduction='mean')
result = loss3(inputs, targets) # 结果为1.3333

CrossEntropyLoss：在训练分类问题中比较有用，实际计算如下图所示

Python

x = torch.tensor([0.1, 0.2, 0.3])
y = torch.tensor([1])
x = torch.reshape(x, (1, 3))
loss4 = CrossEntropyLoss()
result = loss4(x, y) # 结果为1.1019

以下为实际的例子，读入 CIFAR10 数据集，输出每个类别的可能性，然后与实际的 label 进行比对。

Python

import torchvision
from torch import nn
from torch.nn import Sequential, Conv2d, MaxPool2d, Flatten, Linear
from torch.utils.data import DataLoader

dataset = torchvision.datasets.CIFAR10("./Dataset", train=False, transform=torchvision.transforms.ToTensor(), download=True)
dataloader = DataLoader(dataset, batch_size=1)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.model1 = Sequential(
            Conv2d(3, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 64, 5, padding=2),
            MaxPool2d(2),
            Flatten(),
            Linear(1024, 64),
            Linear(64, 10)
        )

    def forward(self, x):
        x = self.model1(x)
        return x


loss = nn.CrossEntropyLoss()
howjul = Howjul()
for data in dataloader:
    imgs, labels = data
    output = howjul(imgs)
    result_loss = loss(output, labels)
    print(result_loss)
    result_loss.backward()  # 可以计算出梯度

结果如下：

7-8-LossFunc.py

7-8-LossFunc_network.py

7.9 优化器

pytorch 的优化器主要集中在 torch.optim 中，官方文档点此。

优化器使用方法如下，先对梯度进行清零，然后使输入经过神经网络模型进行输出，然后计算输出和目标之间的误差，然后调用 backward 来计算梯度，然后调用优化器的 step 函数来更新参数。

Python

for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

接下来就是具体的优化算法了，但是具体优化算法需要具体学习，一般需要给算法需要优化的参数的列表和学习速率（ learning rate ）。

如下是代码样例：

Python

import torch
import torchvision
from torch import nn
from torch.nn import Sequential, Conv2d, MaxPool2d, Flatten, Linear
from torch.utils.data import DataLoader

dataset = torchvision.datasets.CIFAR10("./Dataset", train=False, transform=torchvision.transforms.ToTensor(), download=True)
dataloader = DataLoader(dataset, batch_size=1)


class Howjul(nn.Module):
    def __init__(self):
        super(Howjul, self).__init__()
        self.model1 = Sequential(
            Conv2d(3, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 32, 5, padding=2),
            MaxPool2d(2),
            Conv2d(32, 64, 5, padding=2),
            MaxPool2d(2),
            Flatten(),
            Linear(1024, 64),
            Linear(64, 10)
        )

    def forward(self, x):
        x = self.model1(x)
        return x


loss = nn.CrossEntropyLoss()
howjul = Howjul()
optim = torch.optim.SGD(howjul.parameters(), lr=0.01)
for epoch in range(20):
    running_loss = 0.0
    for data in dataloader:
        imgs, labels = data
        output = howjul(imgs)
        result_loss = loss(output, labels)
        optim.zero_grad()  # 清除梯度信息
        result_loss.backward()  # 计算出梯度
        optim.step()  # 对每个参数进行优化
        running_loss += result_loss
    print("Epoch: {}/{} Loss: {}".format(epoch+1, 20, running_loss))

结果如下，可以看到 Loss 之和在不断变小

Text Only

Files already downloaded and verified
Epoch: 1/20 Loss: 18687.427734375
Epoch: 2/20 Loss: 16134.6591796875
Epoch: 3/20 Loss: 15481.9697265625

7-9-optim.py

参考：Enl_Z，小土堆教程

本文总阅读量18次