Pytorch torch.optim优化器个性化使用
一、简化前馈网络LeNet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
|
二、优化器基本使用方法
- 建立优化器实例
- 循环:
- 清空梯度
- 向前传播
- 计算Loss
- 反向传播
- 更新参数
1 2 3 4 5 6 7 8 9 10 11 |
|
三、网络模块参数定制
为不同的子网络参数不同的学习率,finetune常用,使分类器学习率参数更高,学习速度更快(理论上)。
1.经由构建网络时划分好的模组进行学习率设定,
1 2 3 |
|
2.以网络层对象为单位进行分组,并设定学习率
1 2 3 4 5 6 7 8 9 10 |
|
四、在训练中动态的调整学习率
1 2 3 4 5 6 7 8 9 |
|
可以看到optimizer.param_groups结构,[{'params','lr', 'momentum', 'dampening', 'weight_decay', 'nesterov'},{……}],集合了优化器的各项参数。
torch.optim的灵活使用
- 重写sgd优化器
import torch from torch.optim.optimizer import Optimizer, required class SGD(Optimizer): def __init__(self, params, lr=required, momentum=0, dampening=0, weight_decay1=0, weight_decay2=0, nesterov=False): defaults = dict(lr=lr, momentum=momentum, dampening=dampening, weight_decay1=weight_decay1, weight_decay2=weight_decay2, nesterov=nesterov) if nesterov and (momentum <= 0 or dampening != 0): raise ValueError("Nesterov momentum requires a momentum and zero dampening") super(SGD, self).__init__(params, defaults) def __setstate__(self, state): super(SGD, self).__setstate__(state) for group in self.param_groups: group.setdefault('nesterov', False) def step(self, closure=None): """Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. """ loss = None if closure is not None: loss = closure() for group in self.param_groups: weight_decay1 = group['weight_decay1'] weight_decay2 = group['weight_decay2'] momentum = group['momentum'] dampening = group['dampening'] nesterov = group['nesterov'] for p in group['params']: if p.grad is None: continue d_p = p.grad.data if weight_decay1 != 0: d_p.add_(weight_decay1, torch.sign(p.data)) if weight_decay2 != 0: d_p.add_(weight_decay2, p.data) if momentum != 0: param_state = self.state[p] if 'momentum_buffer' not in param_state: buf = param_state['momentum_buffer'] = torch.zeros_like(p.data) buf.mul_(momentum).add_(d_p) else: buf = param_state['momentum_buffer'] buf.mul_(momentum).add_(1 - dampening, d_p) if nesterov: d_p = d_p.add(momentum, buf) else: d_p = buf p.data.add_(-group['lr'], d_p) return loss
- 上一篇 »基于html5 Canvas图表库 : ECharts
- 下一篇 »CSS自定义多个字体引用