ChainedStep¶
Compose a sequence of gradient processing steps into a single update step.
This can be used to build either custom optimizers or to augment existing optimizer steps with additional steps (e.g. gradient clipping). IMPORTANT: note that, by convention, a full end-to-end optimizer step (e.g., as implemented by AdamStep) negates the gradient before returning it, yielding an additive weight update, so that it can be applied to weights using the Add node. Therefore, when you build a custom optimizer from raw gradient processing steps, you need to include a negation step (scale by -1) near or at the end of the chain so as to obtain a full end-to-end optimizer step that is swappable with other optimizers. If your chain includes one of the end-to-end steps (which already do the negation), however, this will not be necessary. The scaling typically happens last (usually implemented using ScalingStep, which additionally can also take a learning rate), unless you are also using a ConstraintStep, in which case that must be the last step in the chain. Tip: Almost all canonical end-to-end optimizers (those that are not described as chainable, e.g., AdamStep, RMSPropStep, etc.), consist of a core gradient scaling rule followed by optional additional steps such as weight decay, momentum, scaling by a learning rate (schedule), etc. and generally a negation. For some optimizers, the core rule is available as a separate node, but for others, it is not -- for these optimizers, you can generally obtain a step analogous to the core scaling rule by setting the learning rate to -1.0 and disabling all optional features of the optimizer (like momentum, weight decay, etc). This rule can then be combined with other steps in a sensible order and a final negation to form a customized optimizer. Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
step1¶
Step 1.
step2¶
Step 2.
step3¶
Step 3.
step4¶
Step 4.
step5¶
Step 5.
step6¶
Step 6.
step7¶
Step 7.
step8¶
Step 8.
step9¶
Step 9.
step0¶
Step 0.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.