ClusteredPenalty¶
A penalty encouraging the patterns along individual rows of the solution matrix to condense into few similar stereotypical patterns; implemented as a proximal operator.
The assumption realized by this node is that a model may have sets of weights which are similar to each other, in the sense that only a few stereotypical weight patterns (cluster centroids) occur and each individual weight pattern deviates by a small amount from one of the centroids. This is most commonly used in a machine-learning context where a model is learned that has weights for multiple related tasks (e.g., for different users), and those weights are assumed to be similar to each other, also known as multi-task learning (MTL). The first axis parameter indicates the axis indexing the observations (here tasks) and the second axis specifies the features or variables across which the weight patterns are expressed. However, this node can also be used with tensor-shaped data, in which case each 2d slice through the data will independently satisfy the clustering assumption. The norm is convex and can be viewed as a convex relaxation of the k-means clustering problem; as such, the solution admits global optimality, but the solution is also not exactly classical clustering, but rather a convex surrogate. Note that there is a correspondence between this penalty and the low-rank penalty (LowRankProx), in that the low-rank penalty can be viewed as likewise concentrating the rows of the solution matrix towards latent profiles (cluster centroids), although with different characteristics. In practice, the low-rank penalty is the more commonly used one, while the cluster penalty could yield better performance when there are many observations (tasks) and relatively few features per cluster, and when the number of clusters is approximately known. The node may be used in conjunction with a technique called centered multi-task learning (CMTL), where a model consists of a shared component and a task-specific component that are summed together in the model formulation. In this case, the cluster penalty is applied to the task-specific component. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node. Version 1.0.0
Ports/Properties¶
data¶
Data to process.
step_size¶
Step size.
step_count¶
Current step count for outer solver.
axis1¶
Axis that represents the individual observations that are being clustered. Together with axis2, this specifies the 2d slices of the solution matrix along which the clustering penalty is applied. Additional dimensions in the data are treated independently. If the input is given as a packet, this can be an axis name,a comma-separated list of axis names (e.g., time,frequency), or an axis index (0, 1, etc.) or comma-separated list thereof. If the input is given as an array, only axis indices or lists thereof can be given. Lists mean that the given axes are first flattened, and then a profile is learned along that flattened axis, which is then combined with the profile along axis2. The effect on the original data is in all cases back-transformed to the original data shape.
axis2¶
Axis that represents the feature-space dimensions in which the clusters exist. See the description of axis1 for more details.
clustering_strength¶
Degree of clustering that is encouraged in the solution. Larger values will cause solution rows to be more similar to some cluster centroids.
num_clusters¶
Assumed number of clusters to learn. The norm performs best if this is reasonably close to the true number of clusters, but the behavior degrades gracefully if this is inaccurate.
within_cluster_variance¶
Degree of variance that is allowed within each cluster. Larger values will allow more variance within each cluster, while smaller values will cause tighther clusters.
between_cluster_variance¶
Degree of variance that is allowed between clusters. Larger values will allow more variance (separation) between clusters, while smaller values will cause the centroids for individual clusters (and thus all weights) to move closter towards a common centroid. Along with the former parameter, this may be set empirically, and choosing this as 10x as large as the former parameter is a good starting point.
sanity_checks¶
Enable sanity checks. Not supported when using the jax backend.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.