Skip to content

← deep_learning package

VarianceScalingInitializer

Initialize weights from a distribution whose scale is adapted to the shape of the initialized array.

This can be configured to obtain a variety of standard initializers, including Glorot (scale=1, mode=fan_avg), LeCun (scale=1, mode=fan_in), and He (scale=2, mode=fan_in), in combination with distribution=uniform or truncated_normal. Version 0.5.0

Ports/Properties

scale

Scale to multiply the variance by. This is most commonly 1 (e.g., Glorot and LeCun initialization), but can also be 2 (e.g., for He initialization).

verbose name
Scale Multiplier
default value
1
port type
FloatPort
value type
float (can be None)

mode

Scale the variance based on one of: the number of input units (fan_in, used in LeCun or He initialization), number of output units (fan_out), or the average of the number of input and output units (fan_avg, used in Glorot initialization).

verbose name
Scale By
default value
fan_in
port type
EnumPort
value type
str (can be None)

distribution

Distribution from which to draw the initial weights. Typically, this is either truncated normal or uniform. The parameters of the distribution are computed as follows. First, the scale s is computed as: s = sqrt(scale / n) where n is the number of input units for fan_in, the number of output units for fan_out, or the average of the number of input and output units for fan_avg. Then the mean is 0 and the standard deviation is sqrt(s) for normal, and sqrt(adjs) for truncated normal, where adj is an adjustment factor to compensate for the truncation. For uniform, the bounds are [-sqrt(3s), sqrt(3*s)].

verbose name
Distribution Type
default value
truncated_normal
port type
EnumPort
value type
str (can be None)

fan_in_axes

Axes to use for computing the number of input units. If None, all but the last dimension are used (default for e.g., convolutional kernels). Otherwise this may need to be set to a list of axis indices (counting from 0). The number of output units is always the remaining axes.

verbose name
Fan-In Axis Indices
default value
None
port type
ListPort
value type
list (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

verbose name
Set Breakpoint (Debug Only)
default value
False
port type
BoolPort
value type
bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

verbose name
Metadata
default value
{}
port type
DictPort
value type
dict (can be None)