NoisySGDStep¶
The NoisySGD optimizer step.
Based on Neelakantan et al, 2014, noisy SGD is a variant of stochastic gradient descent (SGD) that adds additional Gaussian-distributed noise. This can help generalization in particularly deep networks. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
learning_rate_schedule¶
Optional learning rate schedule.
eta¶
Initial variance for the Gaussian noise added to gradients.
gamma¶
A parameter controlling the annealing of noise over time, the variance decays according to (1+t)^-gamma.
seed¶
A seed for the pseudo-random number generation.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.