LARSStep¶
The LARS optimizer step.
Based on You et al, 2017, LARS is a layer-wise adaptive optimizer that enables the use of very large batch sizes (e.g., 16k examples) with SGD. This is a precursor to the more recent LAMB optimizer. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. The weight decay can be used in conjunction with a mask data structure that has the same nested structure as the weights being optimized, but which contains booleans indicating which weights should be decayed. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
weight_decay_mask¶
Mask structure for the weight decay.
trust_ratio_mask¶
Mask structure for the where to apply the trust ratio step.
learning_rate¶
Learning rate. A typical choice may be 0.001 here, but this is problem dependent. If a learning rate schedule is provided, this value should be left unspecified.
weight_decay¶
Strength of the weight decay. This is multiplied by the learning rate as in e.g., PyTorch and Optax, but differs from the paper, where it is only multiplied by the schedule multiplier but not the base learning rate.
trust_coefficient¶
Multiplier for the trust ratio.
epsilon¶
Optional additive constant in the trust ratio denominator.
momentum¶
Optional exponential decay rate for momentum.
nesterov¶
Whether to use Nesterov acceleration.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.