BatchNorm¶
Apply batch normalization to the given data.
Like most normalization nodes, batch norm is usually applied right before the activation function in a neural network, and may follow for example a convolutional layer or a dense layer. This will z-score the data using statistics aggregated over the selected axes, which notably includes the instance (aka "batch") axis and usually all other axes except the feature axis. Consequently this operation will normalize each feature independently by statistics that go across all instances in the batch. Like most normalizations, batch normalization typically includes a learned scale and bias parameter, separately per feature, and these can be optionally overridden with externally generated values (note that, in the rare case that there is more than one feature axis in the input, these leared parameters are only per each of the elements in the trailing feature axis; you can use the Fold Into Axis node beforehand to combine multiple feature axes into one to avoid this). When this node is used at test time, the node will by default use statistics that were accumulated during training using an exponential moving average (controlled by the decay factor parameter); this is the recommended default, but one may optionally set the local_stats_on_test option, in which case the statistics from the current test batch are used. Note that the effect of batch norm varies with the batch size, and the normalization can become noisy if the batch size is small. In such cases, one may consider using layer or group normalization, which instead aggregates over the feature axes but not over the instance axes in a batch. These alternative norms are particularly common in conjunction with RNNs or when processing large-format image or spatio-temporal data. If packet data is given, this node ensures that the instance axes come first and the feature axes come last. More Info... Version 0.2.1
Ports/Properties¶
data¶
Data to process.
is_training¶
Whether the node is used in training mode.
scale_init¶
Initializer for the trainable scale.
bias_init¶
Initializer for the trainable bias.
scale_prior¶
Optional prior distribution for the scale.
bias_prior¶
Optional prior distribution for the bias.
axes¶
Optional comma-separated list of axis names or indices over which to accumulate the normalization statistics. If unspecified, the statistics will be accumulated over all except the feature axis ("channels" axis in classic deep learning nomenclature). If an axis shall occur more than once, one may list it multiple times, or prefix the axis name with an asterisk to apply to all axes of this type. This parameter is not limited to the predefined options.
decay_rate¶
Decay rate / momentum across subsequent mini-batches, for accumulating test-time statistics. Values close to 1 will result in slower decay and thus more stable statistics across the whole training set.
epsilon¶
Small value to add to the variance to avoid division by zero.
learnable_scale¶
Whether to learn a trainable scale parameter. Batch normalization typically includes such a parameter in order to drive the subsequent activation function in a regime that is desirable for downstream computations (e.g., saturating or linear).
learnable_bias¶
Whether to learn a trainable bias parameter. See the learnable scale for more details.
data_format¶
Format of the input data. This is only respected when working with plain arrays and is ignored for packet data, which always normalizes the data to 'channels_last' layout. If 'channels_last', the data is assumed to be in the format ({batch}, ..., channels). If 'channels_first', the data is assumed to be in the format ({batch}, channels, ...).
scale_initializer¶
Choice of scale initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
bias_initializer¶
Choice of bias initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
local_stats_on_test¶
At test time, whether to scale the data by stats of the test batch only, or using the global statistics accumulated during training. Classic batch normalization will use training statistics during testing, and this is the recommended default; local test statistics may be used when performing predictions in sufficiently large batches (e.g., same size as the training batches) and when there is specific concern of covariate shift between training and test data. The value 'default' translates to 'false'.
layername¶
Name of the layer. Used for naming of the trainable parameters.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.