EmbeddingLayer¶
A trainable layer for mapping categorical (integer) data to low-dimensional vectors.
This is used because neural networks are not inherently well adapted to mapping categorical data to continuous representations, so for small to mid-sized categorical data, one typically has to convert such data to a one-hot encoding, which can then be used with a Dense layer. However, in the case of many categories, this can be inefficient, and if categories inherently share separable features, this node provides a scalable solution that works for tens of thousands of categories (e.g., when the inputs are indices corresponding to a large tabularized vocabulary of words or other tokens, e.g., event markers) by mapping these to a learned lower-dimensional feature vectors. The maximum integer that can be encountered as well as the dimensionality of the vector embedding are pre-specified as parameters. The node has a few options for performance tuning (op_precision and internal_mapping), and can also be initialized with a pretrained embedding matrix (w_pretrained) as an alternative to random initialization. In this case the input data may only contain one stream. For Packet data, the output of this node has the same axes as the input, except for an additional feature axis (of length embedding_features) at the end. Note that this node does not automatically work on event marker data in packets, since this node, like all other DNN nodes, ignores marker streams. Instead, you will have to define a fixed tabulated vocabulary of event markers and convert any markers you want to process into indices into this vocabulary, which would go into the data array of a regular non-marker stream (which can and most likely should still be indexed by an instance axis etc.) More Info... Version 0.2.0
Ports/Properties¶
data¶
Data to process.
w_init¶
Initializer for the weights.
w_pretrained¶
Optionally a pretrained embedding matrix.
w_prior¶
Optional prior distribution for the weights.
num_categories¶
Number of discrete categories. This is equivalent to the largest integer index that the node may receive plus 1. This is also known as the vocabulary size if the inputs are indices into a dictionary when processing text data.
embedding_features¶
Number of features in the embedding.
internal_mapping¶
How the node internally implements the lookup operation. The external behavior of the node is the same in either case, but performance characteristics may differ depending on the implementation and the other settings. The default is currently indices, but this may change in the future.
w_initializer¶
Choice of weight initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
op_precision¶
Operation precision. This is a compute performance optimization. See jax documentation for details on these options. Note that this only applies to the operation, while the storage precision may be separately configurable depending on the node in question.
layername¶
Name of the layer. Used for naming of weights.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.