AdamStep¶
The Adam optimizer step.
Based on Kingma and Ba, 2014, Adam is one of the most popular optimizers for deep learning due to its effectiveness given a wide variety of network topologies and training regimes, making it a good initial choice. Note that for best accuracy, the learning rate is often adapted using a schedule, which all optimizers support via the learning_rate_schedule port. Adam can suffer from failure to converge in some cases, which is addressed by some close relatives like AMSGrad and Yogi. Another potential issue is instability or large variance during early training, which can be addressed by using a warmup schedule for the learning rate, or by using a different optimizer, such as Novograd and RAdam. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
learning_rate¶
Learning rate. A typical choice may be 0.001 here, but this is problem dependent. If a learning rate schedule is provided, this value should be left unspecified.
beta1¶
Exponential decay rate for the first moment estimates.
beta2¶
Exponential decay rate for the second moment estimates.
epsilon¶
Small value applied to the denominator outside the square root to avoid dividing by zero when rescaling. Note that larger epsilon values have been explored in the literature.
epsilon_inroot¶
Small value applied to the denominator inside the square root to avoid dividing by zero when rescaling. A case where this is needed is when differentiating the optimizer itself, eg for bilevel optimization.
mu_precision¶
Numeric precision for the first-order accumulator. Keep resolves to the precision of the inputs.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.