Adding L1/L2 Regularization In PyTorch?
Answer : Following should help for L2 regularization: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5) This is presented in the documentation for PyTorch. Have a look at http://pytorch.org/docs/optim.html#torch.optim.Adagrad. You can add L2 loss using the weight decay parameter to the Optimization function. For L2 regularization, l2_lambda = 0.01 l2_reg = torch.tensor(0.) for param in model.parameters(): l2_reg += torch.norm(param) loss += l2_lambda * l2_reg References: https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951. http://pytorch.org/docs/master/torch.html?highlight=norm#torch.norm.