Skip to content

← deep_learning package

WeightDecayStep

Chainable step that applies weight decay (analogous to l2 regularization) to the parameters.

This is typically applied after gradients have been scaled by criteria such as past gradient updates, but is applied before scaling by the learning rate, so that the learning rate does not change the ratio of the weight decay to the gradient update. The weight decay can be used in conjunction with a mask data structure that has the same nested structure as the weights being optimized, but which contains booleans indicating which weights should be decayed. Version 0.2.0

Ports/Properties

gradients

Gradients to be transformed.

verbose name
Gradients
default value
None
port type
DataPort
value type
object (can be None)
data direction
INOUT

weights

Optional current weights.

verbose name
Weights
default value
None
port type
DataPort
value type
object (can be None)
data direction
IN

state

Explicit state of the node.

verbose name
State
default value
None
port type
DataPort
value type
object (can be None)
data direction
INOUT

weight_decay_mask

Mask structure for the weight decay.

verbose name
Weight Decay Mask
default value
None
port type
DataPort
value type
object (can be None)
data direction
IN

decay_rate

Weight decay rate. This is typically a small value, such as 1e-4.

verbose name
Decay Rate
default value
0.0001
port type
FloatPort
value type
float (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

verbose name
Set Breakpoint (Debug Only)
default value
False
port type
BoolPort
value type
bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

verbose name
Metadata
default value
{}
port type
DictPort
value type
dict (can be None)