MultiHeadAttentionLayer¶
A multi-head attention layer.
This layer enables information retrieval using an array of query patterns that are matched against a memory of key patterns and their associated value patterns, using a "soft" (0 to 1) matching score. This can be interpreted as a content-based memory lookup, or as a "soft" dictionary lookup where keys and values are patterns, and where each query pattern is matched against all keys to retrieve (a blend of) the respectively matching values. What represents a key, value, or query in the input data is learned by the layer in the form of learnable projections of the input data into a key, value, or query space. A triplet of these projections is called an "attention head", and the layer is usually used with multiple heads (e.g., 8-32). Due to the learnable nature, all parts of the attention process are ultimately neurally driven and thus learned, including the input data to the layer. Attention is the core building block of the Transformer architecture where it is combined with a few other layers (LayerNorm and Dense) and then repeated, but is not specific to it. The layer can either be used with three input packets (or arrays) wired into query, key and value, respectively, or with a single packet/array wired into just the query port, which is then used for all three inputs (this is called self-attention). All three inputs are expected to have an axis along which the data is interpreted as a sequence of items across which attention acts (e.g., time for temporal data or space for image data), and another axis along which the data is interpreted as patterns. These axes generally need not have the same length among the three inputs, although key and value need to have the same sequence length. The inputs may have additional axes (e.g., an instance axis holding a batch of data), and the operation proceeds independently in parallel across these axes; however, the extra axes must be the same for all inputs. It is possible to omit either the key or value (in which case the other of the two will be used for both key and value); if key and value are omitted, the query data is used for all three (self-attention). Self-attention can be used as a drop-in replacement for either recurrent or convolutional layers for processing fixed-length or variable-length sequences. For data with detailed time structure, often the input data is augmented with synthetic features representing a "positional" encoding, e.g., sine/cosine waves of different frequencies, to enable the attention to learn to attend to specific positions in the sequence. The basic operation of the layer is, separately for each attention head, as follows: for each entry of the query data (along the sequence axis), the pattern of the k'th query along the pattern axis is passed through a linear layer (a learnable projection corresponding to the current query head) to produce a pattern in the "query space" (of dimensionality key_dim). Likewise, for each entry in the key data along its sequence axis, the pattern of the n'th key (along the respective pattern axis) is passed through another linear projection (also learnable) to produce a pattern in the "key space" (also of dimensionality key_dim). The query and key patterns are then matched against each other using a dot product (i.e., the sum of elementwise products), which yields a sort of matching score. After possibly some normalization, the score is then passed through a sigmoid (softmax) to produce a 0 to 1 "weight" for that query pattern compared to each of the key patterns. The weight is then applied to the corresponding n'th value pattern, and the lookup result for the k'th query is simply the weighted sum of all the value patterns. This is an adaptive selection, which can act across long distances in the data (namely across the entire sequence length of the key/query data). The operation is performed in parallel for all attention heads (using their separate projections) and the retrieved value patterns are stacked. The resulting stacked patterns are then passed through a final linear projection (again learnable) to reduce dimensionality to the desired output dimensionality. More Info... Version 0.2.0
Ports/Properties¶
queries¶
Query data.
keys¶
Key data.
values¶
Value data.
mask¶
Optional mask data. If specified, this must amount to a boolean array with a shape that's compatible with the attention weights.
data¶
Retrieved data.
w_init¶
Initializer for the weights.
w_prior¶
Optional prior distribution for the weights.
num_heads¶
Number of parallel heads. Each head represents a different learnable projection of the input data into a projected query, key, and value space, and the operation is usually used with multiple parallel heads whose outputs are stacked before being reduced to the final output dimensionality by another learnable projection.
sequence_axis¶
The axis across which to attend. The lookup acts along this axis in the key and value data and treats each element along it as a different item that can be retrieved ("attended to"). The query data is also interpreted as a sequence of multiple queries along this axis, and therefore the output will inherit this axis from the query. Like pattern_axis, this is not limited to the predefined choices (see pattern_axis for an example).
pattern_axis¶
Axis along which data is interpreted as query, key, or value patterns. This can be a single axis or comma-separated list of axes in the query, key, and value data along which data will be treated as patterns that are mapped into the key/query or value space. This can also by the index of an axis. This parameter is not limited to the predefined choices; for example, you can refer to an axis by its custom label, e.g., feature.mylabel.
mask_axes¶
The trailing axes of the mask, if a mask is given (otherwise ignored). This should correspond to (optionally) an axis indexing the heads (whose length is either 1, or the number of heads), followed by an axis of the same length as the query's sequence axis, and lastly an axis of the same length as the key's sequence axis. The special value singletonaxis means that a dummy axis of length 1 will be inserted at the respective position (relative to the end of the mask array).
key_dim¶
Size of the key and query vectors. The key and query patterns are each projected into a space of this dimensionality before the matching is performed between them.
value_dim¶
Size of the value vectors. The value pattern retrived by each attention head is projected into a space of this dimensionality.
output_dim¶
Size of the output vectors. The stacked outputs produced by the attention heads are projected into a space of this dimensionality and represent the final output of the node. Note that the output axis will always be of type feature.
w_initializer¶
Choice of weight initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
layername¶
Name of the layer. Used for naming of weights.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.