SequenceSchedule¶
A composite schedule that is a sequence of multiple provided schedules, each with a given starting step count.
This node is used by wiring one or more base schedule nodes, which can be any of the nodes ending in Schedule, into the schedule0, schedule1, etc. inputs (using the "this" output of the respective schedule node). The first provided schedule starts immediately (at step 0), and each subsequent schedule starts at the step count specified by the respective start1, start2, etc. inputs, and its timing is relative to that start point. However, note that the transition durations of the individual steps are not adjusted to match the difference in successive start points, so you need to make sure that the timing of each step is reasonable given the start point of the next step. The most common use cases of this node is to chain a specific warmup curve (e.g., linear) with a plateau and/or falloff step (e.g., cosine decay), although note that there are also ready-made nodes for some of the most common scenarios of this type. Another use case is to chain multiple decaying schedule to get a cyclical behavior, which is sometimes used to prevent the optimization from getting stuck in a suboptimal local minimum. Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
value¶
Schedule value at current step count.
schedule1¶
Schedule 1.
schedule2¶
Schedule 2.
schedule3¶
Schedule 3.
schedule4¶
Schedule 4.
schedule5¶
Schedule 5.
scheduleN¶
Additional schedules.
start1¶
Starting step for 2nd schedule.
start2¶
Starting step for 3rd schedule.
start3¶
Starting step for 4th schedule.
start4¶
Starting step for 5th schedule.
start5¶
Starting step for 6th schedule.
startN¶
Starting steps for additional schedules.
schedule0¶
Schedule 0.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.