AggregateStep¶
A modifier for the step node that is wired into it which accumulates k successive gradient evaluations and passes them to the optimizer for use in one summed (or averaged) update.
This can be used for things like multi-task learning (if the k successive gradients stem from multiple tasks that are visited round-robin), or for varying the batch size over the course of training (since k can be controlled by a schedule), or for simulating large-batch training with batch sizes that otherwise would not fit in memory. Note this is not simply chained after another step using the ChainedStep node, but rather it is a modifier of the step node that is wired into it. Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
modify_step¶
Step to modify.
num_ministeps_schedule¶
Optional schedule for the k parameter.
num_ministeps¶
Number of successive updates to accumulate. If not provided, a schedule must be wired in. This is the k parameter in the documentation.
reduction¶
Whether to sum or mean the updates.
skip_ministep_if¶
Optional criteria under which to skip mini-steps as if they did not happen. Note that using this on multi-task problems may cause the stepping to go out of sync between tasks.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.