ExponentialDecaySchedule¶
An exponential decay (or growth) schedule.
This schedule holds the parameter at the initial value until the current step count reaches the value set via decay_begin, and then applies exponential decay until the given final value is reached, at which point the parameter is held at the final value. The falloff is further parameterized by the decay_steps parameter, which determines over how many steps the parameter decays by the specified decay rate (e.g., if this is set to 100 and the decay rate is 0.9, the it will take 100 steps (after transition begin) for the parameter to reach 0.9initial_value, and another 100 steps to reach 0.81initial_value, and so forth). The node can also operate in "staircase" mode, where the transition is not smooth but is constant for each decay_steps steps, and then changes abruptly by the given decay rate. Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
value¶
Schedule value at current step count.
init_value¶
Initial parameter value. This is the value at the beginning of the schedule. The parameter is held at this value until the current step count reaches the value set via decay_begin.
final_value¶
Final parameter value. Once the schedule reaches this value, it will remain at this value for the remainder of the optimization process. (if the decay rate is < 1, this is effectively a lower bound on the parameter value, and if the decay rate is > 1, this is an upper bound)
decay_rate¶
Decay rate. The parameter value decays by this factor for every decay_steps. This can be between 0 and 1 for a regular decay schedule, or greater than 1 for an exponential growth schedule.
decay_begin¶
Step count at which to begin the transition from the initial value to the final value. The parameter is held at the initial value until this step count is reached.
decay_steps¶
The number of steps over which the parameter decays by decay_rate. The basic formula is value = initial_value * decay_rate ^ (count_since_decay_begin / decay_steps), followed by clipping according to the final_value.
staircase¶
If True, the parameter value is decayed in a staircase fashion, i.e ., the parameter changed by exactly decay_rate every decay_steps steps. If False, the parameter value is decayed in a continuous fashion according to the formula given in the docs for decay_steps.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.