AdaFactorStep¶
The AdaFactor optimizer step.
Based on Shazeer and Stern, 2018, this is an adaptive optimizer that is designed for fast training of large-scale networks (might be overkill the small networks usually used with biosignals). The approach is to saves memory by using a factored representation of the second moment gradient estimates, and only applies to matrix/tensor-shaped parameters that meet a minimum axis size (see the min_size_to_factor setting). Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. The weight decay can be used in conjunction with a mask data structure that has the same nested structure as the weights being optimized, but which contains booleans indicating which weights should be decayed. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
weight_decay_mask¶
Mask structure for the weight decay (optional).
learning_rate¶
Learning rate. Must be provided. If a learning rate schedule is provided, this value should be left unspecified.
min_size_to_factor¶
Only factor statistics if two dimensions of some weight are larger than this value.
decay_rate¶
Controls second-moment exponential decay schedule.
decay_offset¶
Starting step when the fine-tuning phase begins.
multiply_by_parameter_scale¶
Scale learning rate by parameter norm. If False, provided learning rate is absolute step size.
clipping_threshold¶
Optional gradient clipping threshold (norm). If set to None, this is disabled. This is per parameter vector/matrix.
momentum¶
Optional exponential decay rate for momentum.
momentum_precision¶
Numeric precision for the momentum buffer. Keep resolves to the precision of the inputs.
weight_decay¶
Optional rate at which to decay weights. This is usually a small number, like e.g., 1e-4.
epsilon¶
Regularization constant for RMS gradient.
factored¶
Whether to use factored second-moment estimates. This can be turned off to disable the factorization (e.g., to mimick a simpler optimizer).
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.