OptimisticGDStep¶
The Optimistic gradient descent optimizer step.
Based on Mokhtari et al, 2019, this is an advanced optimizer that was originally proposed in the context of saddle-point problems, and has strong convergence for min-max games, where standard gradient descent can oscillate or diverge. Note that this optimizer can be used with schedulers for not only the learning rate but also the alpha and beta parameters, by wiring in the appropriate schedule nodes into the respective ports, without having to use a CustomStep node. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
alpha_schedule¶
Optional alpha schedule.
beta_schedule¶
Optional beta schedule.
learning_rate¶
Learning rate. A typical choice may be 0.001 here, but this is problem dependent. If a learning rate schedule is provided, this value should be left unspecified.
alpha¶
Alpha coefficient for generalized OGD.
beta¶
Beta coefficient for generalized OGD negative momentum.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.