Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions beginner_source/basics/optimization_tutorial.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,10 @@ def forward(self, x):
# (`read more <https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html>`__ about hyperparameter tuning)
#
# We define the following hyperparameters for training:
# - **Number of Epochs** - the number of times to iterate over the dataset
# - **Batch Size** - the number of data samples propagated through the network before the parameters are updated
# - **Learning Rate** - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.
#
# - **Number of Epochs** - the number of times to iterate over the dataset
# - **Batch Size** - the number of data samples propagated through the network before the parameters are updated
# - **Learning Rate** - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.
#

learning_rate = 1e-3
Expand All @@ -95,8 +96,9 @@ def forward(self, x):
# iteration of the optimization loop is called an **epoch**.
#
# Each epoch consists of two main parts:
# - **The Train Loop** - iterate over the training dataset and try to converge to optimal parameters.
# - **The Validation/Test Loop** - iterate over the test dataset to check if model performance is improving.
#
# - **The Train Loop** - iterate over the training dataset and try to converge to optimal parameters.
# - **The Validation/Test Loop** - iterate over the test dataset to check if model performance is improving.
#
# Let's briefly familiarize ourselves with some of the concepts used in the training loop. Jump ahead to
# see the :ref:`full-impl-label` of the optimization loop.
Expand Down Expand Up @@ -134,9 +136,10 @@ def forward(self, x):

#####################################
# Inside the training loop, optimization happens in three steps:
# * Call ``optimizer.zero_grad()`` to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.
# * Backpropagate the prediction loss with a call to ``loss.backward()``. PyTorch deposits the gradients of the loss w.r.t. each parameter.
# * Once we have our gradients, we call ``optimizer.step()`` to adjust the parameters by the gradients collected in the backward pass.
#
# - Call ``optimizer.zero_grad()`` to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.
# - Backpropagate the prediction loss with a call to ``loss.backward()``. PyTorch deposits the gradients of the loss w.r.t. each parameter.
# - Once we have our gradients, we call ``optimizer.step()`` to adjust the parameters by the gradients collected in the backward pass.


########################################
Expand Down