RMSPropStep¶
The RMSProp optimizer step.
Based on Tieleman and Hinton, 2012 and Graves et al. 2013, this optimizer was one of the first successful deep learning optimizers, and remains popular today. This implementation has several options that it is sometimes used with, including momentum and Nesterov acceleration. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
learning_rate¶
Learning rate. A typical choice may be 0.001 here, but this is problem dependent. If a learning rate schedule is provided, this value should be left unspecified.
decay¶
Exponential decay rate for the first moment estimates.
epsilon¶
Small value applied to the denominator outside the square root to avoid dividing by zero when rescaling.
initial_scale¶
Initial scale. Initial value of accumulators tracking the magnitude of previous updates. Note that PyTorch uses 0.0 here while TensorFlow 1 uses 1. When reproducing results from a paper, verify the value used by the authors.
centered¶
If True, use the centered version of RMSProp.
momentum¶
Optional exponential decay rate for momentum.
nesterov¶
Whether to use Nesterov acceleration.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.