PiecewiseInterpolatedSchedule¶
A piecewise interpolated parameter schedule.
This schedule has a series of boundary steps and associated scale factors. The parameter starts with the initial value and whenever the step count reaches a boundary step, the parameter becomes multiplied by the respective scale factor times all prior scale factors. In between boundaries, the parameter is interpolated according to the given interpolation type. Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
value¶
Schedule value at current step count.
init_value¶
Initial parameter value. This is the value at the beginning of the schedule. The parameter interpolates from the prior value to the prior value times the first boundary scale factor by the time the first boundary step is reached, and so forth, where successively reached scale factors multiply on top of each other.
step_boundaries¶
Step boundaries associated with scale factors. When the step reaches the boundary, the parameter becomes the initial value times all scale factors whose boundaries have so far been reached.
scale_factors¶
Scale factors to multiply the parameter by when the respective step boundaries are reached.
interpolation¶
Interpolation type. Cosine follows the shape of the cosine function from its peak to its successive trough.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.