Module: optimization

Optimization nodes.

This module contains nodes related to optimization algorithms. The most important specialty nodes are Gradient, Jacobian, Performance Metric, and Value And Gradient. Besides these, the following types of nodes are found here: - Solve nodes: these solve a specific category of optimization problem and output the solution - Loss nodes: these measure the discrepancy between desired (target) and predicted values, and output that "loss" (per sample) - Penalty nodes: these implement a specific type of (usually non-differentiable) cost in a fashion that is usable with the Proximal Gradient Descent Solve and the Convex Model node (in the ML category) - Constraint nodes: these nodes accept a (usually vector) value and apply a constraint to it, such as bounding it to a specific range; these nodes are usable with the Proximal Gradient Descent Solve node.

BoxConstraint

A constraint projection forcing the given solution to lie inside an n-dimensional box.

The box is defined by a lower and upper limit of same dimensionality as the data itself, or alternatively two scalars. This corresponds to coordinatewise clipping to the box limits and will not maintain the same orientation of the solution vector, unlike the l-infinity norm constraint. Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• lower
  Lower bound for solution.

  • verbose name: Lower
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• upper
  Upper bound for solution.

  • verbose name: Upper
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

ClusteredPenalty

A penalty encouraging the patterns along individual rows of the solution matrix to condense into few similar stereotypical patterns; implemented as a proximal operator.

The assumption realized by this node is that a model may have sets of weights which are similar to each other, in the sense that only a few stereotypical weight patterns (cluster centroids) occur and each individual weight pattern deviates by a small amount from one of the centroids. This is most commonly used in a machine-learning context where a model is learned that has weights for multiple related tasks (e.g., for different users), and those weights are assumed to be similar to each other, also known as multi-task learning (MTL). The first axis parameter indicates the axis indexing the observations (here tasks) and the second axis specifies the features or variables across which the weight patterns are expressed. However, this node can also be used with tensor-shaped data, in which case each 2d slice through the data will independently satisfy the clustering assumption. The norm is convex and can be viewed as a convex relaxation of the k-means clustering problem; as such, the solution admits global optimality, but the solution is also not exactly classical clustering, but rather a convex surrogate. Note that there is a correspondence between this penalty and the low-rank penalty (LowRankProx), in that the low-rank penalty can be viewed as likewise concentrating the rows of the solution matrix towards latent profiles (cluster centroids), although with different characteristics. In practice, the low-rank penalty is the more commonly used one, while the cluster penalty could yield better performance when there are many observations (tasks) and relatively few features per cluster, and when the number of clusters is approximately known. The node may be used in conjunction with a technique called centered multi-task learning (CMTL), where a model consists of a shared component and a task-specific component that are summed together in the model formulation. In this case, the cluster penalty is applied to the task-specific component. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• axis1
  Axis that represents the individual observations that are being clustered. Together with axis2, this specifies the 2d slices of the solution matrix along which the clustering penalty is applied. Additional dimensions in the data are treated independently. If the input is given as a packet, this can be an axis name,a comma-separated list of axis names (e.g., time,frequency), or an axis index (0, 1, etc.) or comma-separated list thereof. If the input is given as an array, only axis indices or lists thereof can be given. Lists mean that the given axes are first flattened, and then a profile is learned along that flattened axis, which is then combined with the profile along axis2. The effect on the original data is in all cases back-transformed to the original data shape.

  • verbose name: Observations Axis
  • default value: instance
  • port type: ComboPort
  • value type: str (can be None)
    

• axis2
  Axis that represents the feature-space dimensions in which the clusters exist. See the description of axis1 for more details.

  • verbose name: Features Axis
  • default value: feature
  • port type: ComboPort
  • value type: str (can be None)
    

• clustering_strength
  Degree of clustering that is encouraged in the solution. Larger values will cause solution rows to be more similar to some cluster centroids.

  • verbose name: Clustering Strength
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• num_clusters
  Assumed number of clusters to learn. The norm performs best if this is reasonably close to the true number of clusters, but the behavior degrades gracefully if this is inaccurate.

  • verbose name: Num Clusters
  • default value: 3
  • port type: IntPort
  • value type: int (can be None)
    

• within_cluster_variance
  Degree of variance that is allowed within each cluster. Larger values will allow more variance within each cluster, while smaller values will cause tighther clusters.

  • verbose name: Within Cluster Variance
  • default value: 0.02
  • port type: FloatPort
  • value type: float (can be None)
    

• between_cluster_variance
  Degree of variance that is allowed between clusters. Larger values will allow more variance (separation) between clusters, while smaller values will cause the centroids for individual clusters (and thus all weights) to move closter towards a common centroid. Along with the former parameter, this may be set empirically, and choosing this as 10x as large as the former parameter is a good starting point.

  • verbose name: Between Cluster Variance
  • default value: 0.2
  • port type: FloatPort
  • value type: float (can be None)
    

• sanity_checks
  Enable sanity checks. Not supported when using the jax backend.

  • verbose name: Sanity Checks
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

CosineDistanceLoss

Calculate the cosine distance between vector predictions and vector targets.

This loss is suitable for pattern generation problems and is defined as 1-cos(theta) where theta is the angle between the two vectors. Note that this loss is scaled between 0 and 2, where 0 is a perfect match and 2 is a perfect mismatch (i.e., curve with opposite sign). The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape, and the last axis in the data should be the vector dimension (if you use two-dimensional packets where one axis is the instance axis, this is automatically taken care of). Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• epsilon
  Small value to avoid dividing by zero when normalizing terms. If either your targets or predictions can be all-zero, this must be specified (e.g., 1e-15) to avoid NaN's or infinities.

  • verbose name: Epsilon
  • default value: 0
  • port type: FloatPort
  • value type: float (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

FunctionTransform

Apply a library-based transformation to a graph and invoke the resulting graph with some arguments.

This node should not be used directly by the user; the node is a helper to implement other nodes that apply Python function transformations to NeuroPype graphs, for example the jax operations grad or jit. Note: for any tuples that the library-transformed function would normally emit, this will instead emit lists.

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• graph
  Graph to invoke.

  • verbose name: Graph
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• graph__signature
  Signature for the "graph" input. This represents the signature for the subgraph that is wired into the "graph" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "graph" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Graph [Signature]
  • default value: (params)
  • port type: Port
  • value type: object (can be None)
    

• result
  Result of the transformed graph invoked with the given arguments.

  • verbose name: Result
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: OUT
    

• operator
  Transform to apply to the graph.

  • verbose name: Operator
  • default value: jax.grad
  • port type: EnumPort
  • value type: str (can be None)
    

• config
  Optional configuration (keyword arguments) for the transform operator.

  • verbose name: Config
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• detect_changes
  How often to check for changes in the graph. The eager mode checks for changes before every run, which is safest during interactive editing of the graph being compiled, but this adds considerable overhead. The lax mode checks for changes only occasionally (every few calls), and logs a warning when a change is detected, and recompiles the graph then. It will run with the old graph until the change is detected. The never mode never checks for changes, and only compiles the graph at the beginning; this is useful for production runs where the graph being compiled is no longer modified. Note that downstream graph edits can be made freely in any mode as they do not affect the compiled portion of the graph.

  • verbose name: Detect Changes
  • default value: eager
  • port type: EnumPort
  • value type: str (can be None)
    

• arg1
  Argument 1.

  • verbose name: Arg1
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg2
  Argument 2.

  • verbose name: Arg2
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg3
  Argument 3.

  • verbose name: Arg3
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg4
  Argument 4.

  • verbose name: Arg4
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg5
  Argument 5.

  • verbose name: Arg5
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg6
  Argument 6.

  • verbose name: Arg6
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg7
  Argument 7.

  • verbose name: Arg7
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg8
  Argument 8.

  • verbose name: Arg8
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• arg9
  Argument 9.

  • verbose name: Arg9
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• argN
  Additional arguments.. .

  • verbose name: Argn
  • default value: None
  • port type: DataPort
  • value type: list (can be None)
  • data direction: IN
    

• name1
  Name 1.

  • verbose name: Name1
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val1
  Value 1.

  • verbose name: Val1
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name2
  Name 2.

  • verbose name: Name2
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val2
  Value 2.

  • verbose name: Val2
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name3
  Name 3.

  • verbose name: Name3
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val3
  Value 3.

  • verbose name: Val3
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name4
  Name 4.

  • verbose name: Name4
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val4
  Value 4.

  • verbose name: Val4
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name5
  Name 5.

  • verbose name: Name5
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val5
  Value 5.

  • verbose name: Val5
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name6
  Name 6.

  • verbose name: Name6
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val6
  Value 6.

  • verbose name: Val6
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name7
  Name 7.

  • verbose name: Name7
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val7
  Value 7.

  • verbose name: Val7
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name8
  Name 8.

  • verbose name: Name8
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val8
  Value 8.

  • verbose name: Val8
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name9
  Name 9.

  • verbose name: Name9
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val9
  Value 9.

  • verbose name: Val9
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• nameN
  Additional argument names.. .

  • verbose name: Namen
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• valN
  Additional named argument values.. .

  • verbose name: Valn
  • default value: None
  • port type: DataPort
  • value type: list (can be None)
  • data direction: IN
    

• arg0
  Argument 0.

  • verbose name: Arg0
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• name0
  Name 0.

  • verbose name: Name0
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• val0
  Value 0.

  • verbose name: Val0
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

Gradient

Calculate the gradient for a given graph with respect to one or more of its parameters.

This node accepts a graph (a function of some inputs) that calculates a single scalar value, and returns a modified graph that when called (via the Call node), will calculate, for a subset of inputs of interest, the vector (or matrix, etc.) that represents the direction of the greatest increase in the scalar value. If the extra_outputs option is checked, then the input function of which the gradient is taken may output a list whose first element is the scalar value of interest, and which may contain additional outputs that are not differentiated (these are passed through, and the gradient function will thus also output a list, whose first element is the gradient and whose other elements are what was generated by the function being differentiated. Consequently, the gradient can then be added to the initial input values to move them towards a local maximum of the original computational graph, i.e., the node can be used iteratively to perform gradient descent to find the optimal parameters for a given computation. Note that the gradient is different for different values of the inputs, including any extra parameters that are held fixed, and therefore the resulting function has the same number of inputs as the original graph, but returns not a scalar but a vector (or matrix, etc.) of the same shape as the subset of inputs of interest. This node will implicily configure any upstream nodes (those wired into this node's graph port) that support multiple backends to default to the selected backend when the gradient is being taken.

More Info...

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• function
  Function to process.

  • verbose name: Function
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• function__signature
  Signature for the "function" input. This represents the signature for the subgraph that is wired into the "function" port. This is formatted as in (a,b,c) ~> (b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes in, and out of, the "function" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c'], ['b','c'].

  • verbose name: Function [Signature]
  • default value: (params)~>(params)
  • port type: Port
  • value type: object (can be None)
    

• variables
  Variables with respect to which the gradient will be calculated. If not given, this defaults to the first positional argument of the function, i.e., the first name in the signature.

  • verbose name: Differentiate With Respect To
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• extra_outputs
  If True, the derivative will only be calculated for the first output of the graph, with additional outputs being passed through unchanged. In this case the graph is expected to return a list of two or more elements.

  • verbose name: Have Extra Outputs
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• backend
  Compute backend to use. This computation is currently only supported by the jax backend, and the data will be converted to this backend unless the backend is set to 'keep', which you can choose to reduce overhead if you know that the data is already in the jax backend.

  • verbose name: Backend
  • default value: jax
  • port type: EnumPort
  • value type: str (can be None)
    

• automove_data
  Automatically move data to the chosen backend. This adds a slight performance overhead if the data is already on the desired backend and is only included for convenience.

  • verbose name: Automove Data
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

GroupSparsePenalty

A penalty encouraging group-sparse solutions with groups corresponding to the entries along one or more axes; implemented as a proximal operator.

The general effect is that, given an axis along which to group (e.g., time), the values along that axis tend to be all-zero for most indices along the other axes (e.g., space or feature) that are not listed as a group axis. The idea is that only a few groups of elements are active in any given solution, but when a group is active, the elements in the group tend to be active or inactive together (e.g., all time points are active for a small subset of channels while no time points are active for most channels. This node can implement a number of different group norms, including the l1/l2 norm (the default), the more lax l1/linf norm (which leaves the magnitude of elements within an active group less constrained), and mixed norm that combine a group norm with a pointwise sparsity norm. The latter additionally encourage that even within active groups only a few elements are active at any given time. One can also give a comma-separated list of multiple axes (e.g., time,space) to group along both axes simultaneously. A less common use case is the non-convex lp,q norm, which can may work better in practice given a skilled choice of solver and problem setup, but which sacrifices the otherwise guaranteed global optimality. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• axis
  Axis along which the elements are in the same group. For example, if this is set to 'time', then all time points are active or inactive together, but the result is sparse along all other axes (e.g., channels), meaning that only a few channels will have active time points. You can also give a comma-separated list of multiple axes (e.g., time,space) to group along both axes simultaneously, which again results in a solution that is sparse along any other axes.

  • verbose name: Axis With Grouped Elements
  • default value: time
  • port type: ComboPort
  • value type: str (can be None)
    

• norm
  The norm to use for the group sparsity penalty. The default choice is l1/l2, which is the sum of l2 norms of each group. An alternative choice is l1/linf, which is the sum of the max norms of each group; this penalizes only the largest element in each group, which has the side effect of leaving the other elements in the group unpenalized; as such, the assumption is that if a group is active (nonzero), then any element in the group is equally likely to be active. The next two choices are the sum of a groupwise norm and a pointwise sparsity norm. This has the effect that only a few groups are active, and the active groups have only a few active nonzero elements; the pointwise sparsity is controlled via the point_sparsity parameter. The lp,q option is the only non-convex group- sparse penality, which jettisons the guaranteed optimality but which can in practice work better than the convex options given a skilled choice of solver and problem setup.

  • verbose name: Norm
  • default value: l1/l2
  • port type: EnumPort
  • value type: str (can be None)
    

• group_sparsity
  Degree of group-wise sparsity in the solution. Larger values will encourage fewer non-zero groups of elements to remain in the solution.

  • verbose name: Group Sparsity
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• point_sparsity
  Degree of point-wise sparsity in the solution. This can only be used when a mixed norm is chosen that has an additional +l1 term in it.

  • verbose name: Point Sparsity
  • default value: 0.0
  • port type: FloatPort
  • value type: float (can be None)
    

• norm_shape
  Shape of the non-convex p,q norm, if selected. This is only used if the norm is set to "lp,q". The shape is a list of two numbers, the first of which is the p value (within-group penality, must be one of 1,2) and between-group penalty q (must be between 0 and 1). Note that not all of these combinations are implemented; the most likely to be supported ones are [2,1/2], and [2,2/3].

  • verbose name: Norm Parameter (If Norm Is Lp,q)
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

HingeLoss

Calculate the Hinge loss between predictions and target values.

This is a robust loss for classification problems and is the underlying loss function for the Support Vector Machine (SVM) model. The labels are by default assumed to be {-1,+1}, but can be set to {0,1} by disabling the labels_are_signed option. Note that this loss is non-differentiable at 0, which can cause potentially problems for gradient-based optimization methods. The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• labels_are_signed
  Whether the labels are {-1,+1}. If disabled, the labels are assumed to be {0,1}.

  • verbose name: Labels Are Signed
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• sanity_checks
  Perform sanity checks on the inputs.

  • verbose name: Sanity Checks
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

HuberLoss

Calculate the robust Huber loss between predictions and regression target values.

This is a canonical robust substitute for the squared loss in regression problems. The Huber loss has the same shape as the squared loss within the given error threshold (delta), but continues at a fixed (linear) slope beyond that threshold. As a result, it does not over-penalize large deviations, and becomes robust to outliers. The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss. For a twice-differentiable alternative, see also the LogCoshLoss node. If target values are omitted, they default to 0. One may also specify a grouping structure where it is assumed that outliers may occur in clusters of observations, rather than in an i.i.d. manner; this is either an array of 0-based group ids (for non-overlapping groups), or a list of arrays of array indices (for overlapping groups), or an integer that represents the size of successive groups, which must evenly divide the number of observations (for uniformly-sized groups).

More Info...

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• delta
  Transition point between squared and linear (robust) regime, in data units. It is crucial to set this parameter correctly to obtain the desired robust behavior. The effect of setting this value is that your loss will behave like a standard least-squares loss for any prediction errors that are smaller than this value, and like a least absolute loss for any errors that are larger. Therefore, you should think of it as the threshold, in original data units, that separates outliers from inliers. A too-small value will result in reduced statistical efficiency (like the median vs mean) while a too-large value will be no more robust than the squared loss. In some applications one can also drive this value based on a robust estimate of the standard deviation of the data (or a multiple thereof) to get an adaptive robust threshold. For binary classification problems, a convenient (very rough) heuristic is to set this to the standard deviation of the labels (0.5) times the classic 1.35 factor for the normal distribution, i.e., 0.675.

  • verbose name: Delta
  • default value: 1
  • port type: FloatPort
  • value type: float (can be None)
    

• grouping
  Optional grouping structure of the observations. This can be an integer, in which case it is assumed that successive observations are uniformly grouped in blocks of this size. It can also be an array of group ids, where each observation is assigned to a group (allowing for non-uniform groups), or a list of arrays of indices, where each array contains the indices of the observations in a group (allowing for overlapping groups). Some restrictions apply when using this inside a node like ConvexModel, such as a fixed number of groups in the overlapping case.

  • verbose name: Grouping
  • default value: None
  • port type: Port
  • value type: object (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

Jacobian

Calculate the Jacobian matrix for a given graph with respect to one or more of its parameters.

This node accepts a graph (a function of some inputs) that calculates a vector-valued output, and returns a modified graph that will calculate, for a subset of inputs of interest, the matrix of partial derivatives for each of the vector components. This matrix represents a linear approximation to the function at the given point. For a function of n inputs and m outputs, the Jacobian matrix is an m x n matrix. Note that calculating the Jacobian can be computationally expensive; in some situations one may instead calculate Jacobian-vector products, which are more efficient.

More Info...

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• function
  Function to process.

  • verbose name: Function
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• function__signature
  Signature for the "function" input. This represents the signature for the subgraph that is wired into the "function" port. This is formatted as in (a,b,c) ~> (b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes in, and out of, the "function" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c'], ['b','c'].

  • verbose name: Function [Signature]
  • default value: (params)~>(params)
  • port type: Port
  • value type: object (can be None)
    

• variables
  Variables with respect to which the gradient will be calculated. If not given, this defaults to the first positional argument of the function, i.e., the first name in the signature.

  • verbose name: Differentiate With Respect To
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• extra_outputs
  If True, the derivative will only be calculated for the first output of the graph, with additional outputs being passed through unchanged. In this case the graph is expected to return a list of two or more elements.

  • verbose name: Have Extra Outputs
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• backend
  Compute backend to use. This computation is currently only supported by the jax backend, and the data will be converted to this backend unless the backend is set to 'keep', which you can choose to reduce overhead if you know that the data is already in the jax backend.

  • verbose name: Backend
  • default value: jax
  • port type: EnumPort
  • value type: str (can be None)
    

• automove_data
  Automatically move data to the chosen backend. This adds a slight performance overhead if the data is already on the desired backend and is only included for convenience.

  • verbose name: Automove Data
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• mode
  Whether to calculate the Jacobian using forward- or reverse-mode autodifferentiation. Which is more efficient depends on the shape of the Jacobian matrix (i.e., the number of inputs and outputs).

  • verbose name: Mode
  • default value: forward
  • port type: EnumPort
  • value type: str (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

KLDivergenceLoss

Calculates the KL divergence between a predicted/proposed discrete probability distribution (in log-space) and a target/known distribution (by default given as probabilities but optionally in log-space).

This node is a measure of mismatch between two probability distributions and is asymmetric, i.e., KL(p||t) != KL(t||p). Note that the measure is known to be potentially overly sensitive to areas with small probabilties, and there are alternative divergences that address this issue. The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape, and the last axis in the data should be the vector dimension. If you use a statistic axis to store the vector dimension, this is automatically taken care of, and likewise, if you use two-dimensional packets where one axis is the instance axis and the vector dimension is some other axis (e.g., feature), this will also work out of the box. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• convex_formulation
  Use convex formulation. If enabled, the distribution is guaranteed to be jointly convex in both parameters, which is useful when using this loss in a convex optimization context.

  • verbose name: Convex Formulation
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• targets_in_log_space
  Assume that the targets are in log-space, that is, are log probabilities. Note the predictions are always assumed to be in log-space.

  • verbose name: Targets In Log Space
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

LBFGSSolve

Iteratively solve a smooth nonlinear optimization problem using the L-BFGS method.

This node is suitable for solving optimization problems of intermediate dimensionality and data size efficiently and with relatively few evaluations of the cost function. Typically, this is used on convex problems, but it may also be used on non-convex problems at the risk of finding only a local optimum. For an alternative that supports non-smooth problems, see the ProximalGradientSolve node. The L-BFGS method is a quasi-Newton method that uses second-order (curvature) information of the cost function by maintaining a low-rank approximation of the inverse Hessian. This comes with additional tracking and memory overhead per iteration compared to first-order methods, but if the optimum is approximately quadratically shaped (e.g., due to a dominant quadratic term in the cost function), the method can reach the optimum in a smaller number of iterations. The node provides a wide range of tuning parameters to maximize the performance on specific types of data; however, the method tends to be sufficiently robust that these parameters often do not have to be tuned. In simple cases, the initial weights can be omitted, but in the general case, initial weights (e.g., an appropriately shaped all-zeros array or packet) must be provided since otherwise the shape is unknown. Like with all solve nodes, the solution is differentiable, meaning that the node can be used in a place where gradients are taken, for example in the network graph a DeepModel node.

More Info...

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• weights
  Initial and final weights. If not set, will be initialized to a packet equivalent to a single all-zeroes instance of the training-data.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: INOUT
    

• state
  Optional initial and final solver state.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• cost
  Smooth cost function given some data.

  • verbose name: Cost
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• cost__signature
  Signature for the "cost" input. This represents the signature for the subgraph that is wired into the "cost" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "cost" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Cost [Signature]
  • default value: (w,D)
  • port type: Port
  • value type: object (can be None)
    

• hyper_params
  Hyper-parameters for the cost function. This is a dictionary of arbitrary key-value pairs that can be used to configure the cost function. The graph may then declare and use placeholders named the same as of the dictionary keys. This is mainly such that the solver does not have to be recompiled when these values change; for constants, you may alternatively wire them directly into the cost function.

  • verbose name: Additional Cost Parameters
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• max_iter
  Maximum number of iterations.

  • verbose name: Max Iter
  • default value: 500
  • port type: IntPort
  • value type: int (can be None)
    

• abstol
  Absolute convergence tolerance. If weights change less than this (after normalization by step size), the optimization terminates. Note that this depends on the data scale.

  • verbose name: Abstol
  • default value: 0.001
  • port type: FloatPort
  • value type: float (can be None)
    

• memory_size
  Number of previous gradients to remember to approximate curvature. This configures the value of L in L-BFGS (i.e., the limited memory size). Typical values are 6 to 10.

  • verbose name: Memory Size
  • default value: 10
  • port type: IntPort
  • value type: int (can be None)
    

• stepsize
  Optional step size override. If unspecified, the step size is adapted automatically using back-tracking line search.

  • verbose name: Stepsize
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• max_stepsize
  Maximum step size to use. Only used if step size is dynamically adapted.

  • verbose name: Max Stepsize
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• min_stepsize
  Minimum step size to use. Only used if step size is dynamically adapted.

  • verbose name: Min Stepsize
  • default value: 1e-06
  • port type: FloatPort
  • value type: float (can be None)
    

• linesearch_type
  Line-search strategy to use to adapt step sizes.

  • verbose name: Linesearch Type
  • default value: zoom
  • port type: EnumPort
  • value type: str (can be None)
    

• linesearch_init
  Strategy to initialize the line-search step size for a new iteration. The least aggressive is current, which starts from the previous step size. Max sets it to the max_stepsize at each iteration, and increase increases it by a factor of increase_factor from the previous iteration (or min_stepsize if smaller than that).

  • verbose name: Linesearch Init
  • default value: increase
  • port type: EnumPort
  • value type: str (can be None)
    

• max_backtrack
  Maximum number of line search steps per iteration. Only used if step size is left to automatic.

  • verbose name: Max Backtrack
  • default value: 15
  • port type: IntPort
  • value type: int (can be None)
    

• backtrack_factor
  Backtracking line search factor. Only used if stepsize is 0 and if a backtracking linesearch type is used.

  • verbose name: Backtrack Factor
  • default value: 0.8
  • port type: FloatPort
  • value type: float (can be None)
    

• increase_factor
  Factor by which to increase the step size if the line search succeeds.

  • verbose name: Increase Factor
  • default value: 1.5
  • port type: FloatPort
  • value type: float (can be None)
    

• use_jit
  If enabled, attempt to use JIT compilation for the inner loop. This incurs a one-time compilation cost, but the actual solving will be greatly accelerated if using the GPU.

  • verbose name: Use Jit
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• unroll
  Whether to unroll the optimization loop. This can save some per-iteration overhead, but the benefit depends on the specific use case.

  • verbose name: Unroll
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• implicit_diff
  Whether to use implicit differentiation for computing gradients. The alternative is to unroll the solver iterations.

  • verbose name: Implicit Diff
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• implicit_diff_solver
  Solver to use for implicit differentiation. Auto is currently defaulting to normal-cg, which is a fast generic solver for high-dimensional problems. The options bicgstab and gmres can be more robust on badly conditioned problems, but are slower. The lu method is likely fastest on very low-dimensional problems, but note that the cost of implicit differentiation is likely marginal compared to the cost of the overall solve.

  • verbose name: Implicit Diff Solver
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• verbosity
  Verbosity level. 0: no output, 1: per-iteration summary. Note that JIT will be disabled if verbosity is used.

  • verbose name: Verbosity
  • default value: 0
  • port type: IntPort
  • value type: int (can be None)
    

• stop_if_linesearch_fails
  Whether to stop if the line search fails. This can be set to true to either debug bad settings of the min/max step-size values or line-search strategy, or to catch issues early with problematic data if one has strong assumptions about the cost function.

  • verbose name: Stop If Linesearch Fails
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• use_gamma
  Whether to use the gamma parameter in the initialization of the inverse Hessian.

  • verbose name: Init Hessian Using Gamma
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• incremental_updates
  Whether to use incremental updates from the previous state. If not set, the previous or wired-in state is ignored.

  • verbose name: Incremental Updates
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

LinearEqualityConstraint

A constraint projection forcing the given solution vector to satisfy a linear matrix equation Ax=b; the data will be flattened to a vector before applying the projection.

Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• matrix
  Matrix A.

  • verbose name: Matrix
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• vector
  Vector b.

  • verbose name: Vector
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

LinearSolve

Solve a linear system of equations of the form Ax=b.

This node currently always uses the jax backend, but will convert the solution back to the original backend of b if necessary. The matrix A can either be wired in directly (as an array) or alternatively as a small graph that applies A to a Placeholder named x, typically as a matrix-vector product, or using any of the specialized nodes that apply a linear operator (e.g. FFT, finite differences, etc.). The node implements the standard methods for this problem, which are largely equivalent to the SciPy or MATLAB(tm) implementations; note that the best choice depends on the desired accuracy, speed, and properties of the operator A (size, symmetry/positive definiteness, and conditioning); see the docstring of the method property for details. Like with all solve nodes, the solution is differentiable, meaning that the node can be used in a place where gradients are taken, for example in the network graph a DeepModel node.

More Info...

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• Ax
  Product of a matrix (or linear operator) A and vector x.

  • verbose name: Ax
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• Ax__signature
  Signature for the "Ax" input. This represents the signature for the subgraph that is wired into the "Ax" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "Ax" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Ax [Signature]
  • default value: (x)
  • port type: Port
  • value type: object (can be None)
    

• A
  Alternative direct specification of A.

  • verbose name: A
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• b
  Coefficient vector b.

  • verbose name: B
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• x
  Initial and final solution. If not provided, x will be initialized to a all-zeros. Ignored by the direct methods.

  • verbose name: X
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: INOUT
    

• P
  Optional preconditioning matrix P. Ignored by the direct methods.

  • verbose name: P
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• method
  Method to use for solving the system. The lu and cholesky methods are "direct" solvers, which materialize the matrix A in memory, and these are the fastest methods when A is small or when you need close to machine precision. The other methods are "indirect" solvers, which only need access to the matrix-vector product operation (the graph wired into the Ax port); this is more efficient if A is a large linear operator or has an more efficient implementation than dense matrix multiplication (examples are A being a Fourier transform, finite differences, or a large sparse matrix). These solvers may also not solve to the maximum precision. The solvers ending in (posdef) require A furthermore to be Hermitian and positive definite, for example this applies to covariance matrices. Of the available methods, cg (posdef) is the standard conjugate gradient method; note that for badly conditioned A, it may be necessary to precondition the linear system by pre-multiplying A and b by a skillfully chosen preconditioning matrix for best performance. The cg (normal) method instead solves the normal equation A'Ax = A'b, but note that this is not the most stable solver for badly conditioned matrices, and you may need to use gmres or bicgstab (also, you must specify an initial x if A is non-square since the shape is otherwise unknown). gmres is the generalized minimal residual method, which is robust but tends to be slow for near-symmetric or symmetric A. bicgstab is a hybrid method that alternates between an update equivalent to cg (normal) and an update equivalent to gmres, and is a good middle ground in terms of speed and robustness. lu is the LU decomposition method, which, is the most efficient choice for small A. Note that LU will ignore both the initial value provided for x, and the regularization parameter. cholesky (posdef) is the Cholesky decomposition method, which is the fastest method if A is small and positive definite.

  • verbose name: Method
  • default value: lu
  • port type: EnumPort
  • value type: str (can be None)
    

• regu
  Ridge regularization parameter. This is a scalar value that will be added to the diagonal of the matrix A before solving the system, which can help to stabilize the solution if A is ill-conditioned. This is ignored by the LU method.

  • verbose name: Regularization (Ridge)
  • default value: 0.0
  • port type: FloatPort
  • value type: float (can be None)
    

• reltol
  Relative convergence tolerance for the indirect solvers. The direct solvers lu and cholesky will ignore this parameter and always solve to approx. machine precision.

  • verbose name: Convergence Tolerance (Rel.)
  • default value: 1e-05
  • port type: FloatPort
  • value type: float (can be None)
    

• max_iter
  Maximum number of iterations. This need not be specified in most cases, since all solvers have a well-defined termination criterion. However, if reltol is too stringent, or if a bounded runtime is needed (e.g. for real-time use), this can be used to limit the number of iterations.

  • verbose name: Max. Iterations
  • default value: None
  • port type: IntPort
  • value type: int (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

LogCoshLoss

Calculate the robust log-cosh loss between predictions and regression targets.

This is a robust alternative to the squared loss in regression problems, and is similar to the Huber loss. Log-cosh is the logarithm of the hyperbolic cosine, defined by the formula log(cosh(p-t)), and is twice differentiable, which can be useful in certain types of optimization problems, while Huber is computationally more efficient and has a tunable parameter that gives more control over the robustness regime. When configured to have the same transition point (delta in the HuberLoss node), the two losses have the same robustness behavior. The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss. If target values are omitted, they default to 0.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

LowRankPenalty

A penalty encouraging solution matrices or tensors that are low rank with respect to two or more axes.

The operator penalizes the sum of singular values in the input (across one or more 2d slices through the data specified by the two axes), which has the side effect of encouraging solutions that are low rank with respect to those two axes. Low rank solutions can be interpreted as a sum of a small number of rank-1 matrices (along the given slices), where a rank-1 matrix can be interpreted as a profile along one axis multiplied by a profile along the other axis. Each of the two axes can also be given as a flattened list of axes, in which case the operator will learn a profile along the flattened list of axes and then combine that with the profile along the other axis (note that the flattened profile is full-rank with respect to the flattened axes however, which can be fairly high dimensional if these axes are long). A less common use case is to employ the alternative non- convex Schatten-q norm via the norm parameter and the associated shape parameter, which can yield better results in practice given a skilled choice of solver and problem setup, but which sacrifices the otherwise guaranteed global optimality. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• axis1
  First axis along which the data are low rank. Along with the second axis this specifies a slice through the data along which low rank structure is encouraged by the penalty. A low-rank structure can be interpreted as the solution being a sum of a small number of latent 2d patterns along these 2 axes, each of which is a rank-1 matrix; this can be interpreted as a profile along one axis multiplied by a profile along the other axis. Note that if the data has any additional axes, the solution will be sparse with respect to those other axes, meaning that only few of the elements have nonzero values (and those values correspond to the low-rank slices). If the input is given as a packet, this can be an axis name,a comma-separated list of axis names (e.g., time,frequency), or an axis index (0, 1, etc.) or comma-separated list thereof. If the input is given as an array, only axis indices or lists thereof can be given. Lists mean that the given axes are first flattened, and then a profile is learned along that flattened axis, which is then combined with the profile along axis2. This can also be interpreted as the profile extending across the given axes, but note that the profile will be full rank with respect to those flattened axes. The effect on the original data is in all cases back-transformed to the original data shape.

  • verbose name: First Axis
  • default value: time
  • port type: ComboPort
  • value type: str (can be None)
    

• axis2
  Second axis along which the data are low rank. See the description of the axis1 parameter for more details.

  • verbose name: Second Axis
  • default value: space
  • port type: ComboPort
  • value type: str (can be None)
    

• norm
  The norm to use to encourage a low-rank structure. The default choice is the trace norm, which penalizes the sum of singular values of each 2d slice of the solution (and if there are multiple slices then the sum of trace norms across those slices, which is equivalent to the trace norm of a block-diagonal matrix with the slices as blocks). The other choice is the Schatten-q norm, which is a non-convex alternative to the trace norm; as a result of the non-convexity, the solution is not guaranteed to be globally optimal; however, this norm can work better than the trace norm with a skilled choice of solver and problem structure for the same reasons as described in the SparseProx node.

  • verbose name: Norm
  • default value: trace
  • port type: EnumPort
  • value type: str (can be None)
    

• rank_sparsity
  Degree of rank-wise sparsity in the solution. Larger values will encourage the solution to have lower rank, meaning that the solution will be a sum of fewer rank-1 matrices (along some chosen slice directions).

  • verbose name: Rank Sparsity
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• norm_shape
  Shape of the non-convex Schatten-q norm, if selected (must be between 0 and 1). A good default is 0.75.

  • verbose name: Norm Parameter (If Norm Is Schatten-Q)
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

NonnegativeConstraint

A constraint projection forcing the given solution to have no negative elements.

Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

NonnegativeSumConstraint

A constraint projection forcing the given solution to sum to a constant value, with all values being non-negative.

The default choice is also known as the probability simplex, meaning that all elements add up to one while positive or zero. Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• value
  Value to which the elements of the solution must sum to. The default choice is also known as the probability simplex, meaning that all elements add up to one.

  • verbose name: Value
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

NonuniformGroupSparsePenalty

A penalty encouraging group-sparse (few nonzero groups of elements) solutions along one or more axes with non-uniform group sizes, implemented as a proximal operator.

The general use case is to specify an axis along which groups are defined (e.g., feature), and then to supply an array of group membership indices (e.g., [0,0,0,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]) that assignes different features to different groups. The effect is then that only a few groups of elements are active in any given solution. The node supports both the default l1/l2 norm and the more lax l1/linf norm (which leaves the magnitude of elements within an active group less constrained); currently, mixed norms that additionally feature a pointwise sparsity norm are not supported (but can be realized with solvers that support more than one proximal operator). One can also give a comma-separated list of multiple axes (e.g., time,space) to define groups that extend across both axes, but be careful about the order in which the elements appear in the flattened axis. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• axis
  Axis along which the elements appear in groups. For example, if this is set to 'feature', then different features can be assigned to different groups, but the result is sparse along all other axes (e.g., channels). You can also give a comma-separated list of multiple axes (e.g., time,space) which are flattened before group indices are taken into consideration.

  • verbose name: Axis With Groups
  • default value: feature
  • port type: ComboPort
  • value type: str (can be None)
    

• membership
  List of group membership indices. Each element is a 0-based integer that indicates the group to which the respective element along the given axis belongs.

  • verbose name: Membership
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• norm
  The norm to use for the group sparsity penalty. The default choice is l1/l2, which is the sum of l2 norms of each group. An alternative choice is l1/linf, which is the sum of the max norms of each group; this penalizes only the largest element in each group, which has the side effect of leaving the other elements in the group unpenalized; as such, the assumption is that if a group is active (nonzero), then any element in the group is equally likely to be active.

  • verbose name: Norm
  • default value: l1/l2
  • port type: EnumPort
  • value type: str (can be None)
    

• group_sparsity
  Degree of group-wise sparsity in the solution. Larger values will encourage fewer non-zero groups of elements to remain in the solution.

  • verbose name: Group Sparsity
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

NormConstraint

A constraint projection forcing the given solution to have a norm of at most or exactly the given scale.

This can be used to constrain the length of a vector (using the l2 norm), limit the values to a uniform box (using the l-infinity norm), or encourage sparsity (using the l1 norm). In all cases, the orientation of the solution vector remains unchanged, and only the norm is constrained. Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• scale
  Value of the norm to which the solution shall be constrained.

  • verbose name: Scale
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• constraint
  Whether the norm constraint is an upper bound (at most) or an equality (exactly). The constraint set is then also called the norm "sphere" (equality) or norm "ball" (upper bound).

  • verbose name: Constraint
  • default value: upper-bound
  • port type: EnumPort
  • value type: str (can be None)
    

• norm
  The type of norm to use. The default l2 norm is the Euclidean distance; the corresponding constraint set has a sphere shape centered at the coordinate origin whose radius is the configured scale. The L1 norm is the sum of absolute values, whose constraint set has a diamond shape, and the L-infinity norm is the maximum absolute value; the constraint set is then a cube.

  • verbose name: Norm
  • default value: l2
  • port type: EnumPort
  • value type: str (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

OverlappingGroupSparsePenalty

A penalty encouraging group-sparse (few nonzero groups of elements) solutions along one or more axes with overlapping and non-uniform group sizes.

Implemented as a proximal operator. The general use case is to specify an axis along which groups are defined (e.g., feature), and then to supply a list of group membership index arrays (e.g., [[0,1,2],[2,3,4],[5,6]]) which may overlap and which may differ in size. The effect is then that only a few groups of elements are active in any given solution, and the pattern of active elements will be a union of groups. This is also known as the latent group lasso. Note that an alternative way to realize this type of model is to duplicate all features that occur in multiple groups, to define a model in terms of those expanded features, and to use a non overlapping (but potentially still non-uniform) group sparse penalty, which can be much faster except for a very high overlap factor, but which may also use more memory. One can also give a comma-separated list of multiple axes (e.g., time,space) to define groups that extend across both axes, but be careful about the order in which the elements appear in the flattened axis. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• axis
  Axis along which the elements appear in groups. For example, if this is set to 'feature', then different features can be assigned to different groups, but the result is sparse along all other axes (e.g., channels). You can also give a comma-separated list of multiple axes (e.g., time,space) which are flattened before group indices are taken into consideration.

  • verbose name: Axis With Groups
  • default value: feature
  • port type: ComboPort
  • value type: str (can be None)
    

• groups
  List of group index arrays. The k'th list entry is an array or list indices of elements (along the axis of interest) that are in the k'th group.

  • verbose name: Groups
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• norm
  The norm to use for the group sparsity penalty. The default choice is l1/l2, which is the sum of l2 norms of each group. An alternative choice is l1/linf, which is the sum of the max norms of each group; this penalizes only the largest element in each group, which has the side effect of leaving the other elements in the group unpenalized; as such, the assumption is that if a group is active (nonzero), then any element in the group is equally likely to be active.

  • verbose name: Norm
  • default value: l1/l2
  • port type: EnumPort
  • value type: str (can be None)
    

• group_sparsity
  Degree of group-wise sparsity in the solution. Larger values will encourage fewer non-zero groups of elements to remain in the solution.

  • verbose name: Group Sparsity
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• eps0
  initial relative convergence tolerance. This value is exponentially decayed during the optimization according to the alpha parameter.

  • verbose name: Eps0
  • default value: 0.01
  • port type: FloatPort
  • value type: float (can be None)
    

• decay
  Exponential decay factor for eps0.

  • verbose name: Decay
  • default value: 2.0
  • port type: FloatPort
  • value type: float (can be None)
    

• abstol
  Absolute convergence tolerance.

  • verbose name: Abstol
  • default value: 1e-07
  • port type: FloatPort
  • value type: float (can be None)
    

• maxiter
  Maximum number of iterations (passes over the active groups).

  • verbose name: Maxiter
  • default value: 200
  • port type: IntPort
  • value type: int (can be None)
    

• verbose
  Whether to print verbose output.

  • verbose name: Verbose
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• use_copt
  Use the COPT implementation when using the linf norm. This is faster than the current built-in implementation, but can only be used in conjunction with custom solvers.

  • verbose name: Use Copt
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

ParameterOptimization

Optimize hyper-parameters of a pipeline given a dataset.

The main use case of this node is with machine-learning pipelines that have free parameters (e.g., regularization parameters). The node can auto-tune these parameters given a dataset and optionally an evaluation function (which defaults to an inner cross-validation on the dataset). This works as follows: the user specifies for each free parameter of the pipeline a search range, which can be discrete or continuous (see parameter documentation for the supported formats). The node then samples multiple combinations of parameter settings, and for each such setting it configures the pipeline with the setting and passes the pipeline to the evaluator, which assess the performance of the configured pipeline on the given dataset. Once training is completed, one can retrieve either the optimal parameter setting (via the optimal_params output), or the fully trained method graph (with parameters bound to optimal values) via the optimized output. Furthermore, the Parameter Optimization node can also be used as a proxy for the method and generate training-set predictions (via the data output) on the initial invocation or test-set predictions on subsequent invocations (if initialize_once is set). One can also save and re-load the node's trained model via the model input/output. The pipeline is typically a subgraph with a placeholder node with slotname "data" that represents the input data, and any number of other placeholder nodes representing the tunable parameters (e.g., "lambda", etc.). Note that the node represents a training strategy, and not a validation strategy. To obtain cross-validation scores, you need to pass the entire parameter search procedure with all its inputs, except for the data replaced by a "data" placeholder, into the "method" input of an outer cross-validation node, which then receives the dataset to use for validation. A key parameter is the compute budget, which specifies how many parameter settings to try; the default is a low value sufficient only for a single parameter. The node supports a number of different search strategies, including randomized grid searches and other gradient-free optimization strategies. The node also supports optional parallel computation via the num_procs argument, which can be useful for long-running optimization tasks. Note that the default evaluator is a parallel 5x-cross-validation, so it will by default already use multiple processes, which are further multiplied by the num_procs argument. You can override the evaluator by providing a subgraph with a data and pype argument placeholder, which typically, wire into a cross-validation node. The node will by default log into your NeuroPype logs folder to allow recovery of partial results in case of crashes or long-running tasks.

More Info...

Version 2.0.1

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• method
  Pipeline to optimize.

  • verbose name: Method
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• method__signature
  Arguments for the pipeline. Note that you do not need to list the placeholders corresponding to the hyper-parameters here, since these are best thought of as optional parameters that should have defaults. The method may have a second argument named is_training, which, if present, receives a True value when training and otherwise False.

  • verbose name: Method [Signature]
  • default value: (data)
  • port type: Port
  • value type: object (can be None)
    

• evaluator
  Optional evaluation function.

  • verbose name: Evaluator
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• evaluator__signature
  Arguments for the optional evaluation function. The job of this function is to take a dataset, a pipeline representing the method to evaluate, and to return a loss measure that shall be minimized. The default graph is simply a Placeholder named data and a GraphPlaceholder named method, which are wired into the data and method inputs of a Crossvalidation node. The default configuration of that crossvalidation is configured to run in serial using the configured settings below, and if the wired-in method graph has a Segmentation node, then it will have its trial_size parameter set to equal the time bounds of that segmentation. This function may be provided in order to have more complete control over how the nested cross-validation is performed (e.g., parallelism, repeats etc), or to replace the cross-validation process by a different evaluation strategy altogether.

  • verbose name: Evaluator [Signature]
  • default value: (data,method)
  • port type: Port
  • value type: object (can be None)
    

• optimized
  Optimally parameterized pipeline or trained model.

  • verbose name: Optimized
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• optimal_params
  Set of optimal parameters.

  • verbose name: Optimal Params
  • default value: None
  • port type: DataPort
  • value type: dict (can be None)
  • data direction: OUT
    

• searcher
  Search method. The following choices are available. For few (under 3) and relatively well-behaved regularization parameters, sampling strategies like Scrambled Hammersley go a long way, and most ML practitioners look no further. For higher- dimensional search spaces, it is typically not possible to exhaustively explore the space, and in such cases the ideal optimizer performs local search in reasonable directions with the abilty to escape local minima; the advanced optimizers are different ways of doing this given different assumptions about the objective function. Despite their sophistication, many of these optimizers require a hefty compute budget well beyond the default setting. Typically, practitioners will try a few different optimizers early on and pick the one that works best. It is not advisable to aggressively tune the choice or parameters of the optimizers in an attempt to boost the generalization error for a paper, since this will yield unreliable (artificially good) results. Choices are:

  • Hammersley Sampling: sample the search space in a regular grid; note that the grid is typically not aligned with the parameter axes, but it ensures regular spacing of sample points.
  • Scrambled Hammersley Sampling: this is essentially Hammersley with additional randomization of the grid points, which has a better chance of exploring values for individual parameters, since it will not repetitively sample the same values along a given axis.
  • Two-Points Crossover Differential Evolution: this is a variant of differential evolution that uses two points to generate a new candidate solution. It is recommended for continuous search spaces.
  • Test-Based Population Size Adaptation: performs well in noisy situations.
  • Particle-Swarm Optimization: a fairly robust optimization strategy.
  • OnePlusOne (generic): a simple and robust optimization strategy for continuous parameters. Has numerous options.
  • Covariance Matrix Adaptation (generic): a capable optimization strategy for low-noise siutations, which requires a high compute budget. Has a number of options.
  • Bayesian Optimization (generic): a classic Bayesian method, note the presence of options.
  • Differential Evolution (generic): a powerful evolutionary strategy; also features additional settings.
  • Estimation of Multivariate Normal Algorithm (generic): a fairly efficient optimizer in parallel settings (i.e., high num_procs).
  • Latin Hypercube Sampling: This is another pseudo-random sampling pattern with fairly good coverage, but which it is considerably less regular than Hammersley sampling.
  • verbose name: Search Strategy
  • default value: Hammersley Sampling
  • port type: EnumPort
  • value type: str (can be None)
    

• budget
  Number of parameter settings to evaluate. This is the main parameter that determines how long the optimization will take, and it governs how many parameter combinations will be passed to the given evaluator for evaluation. Note that the budget required to maintain a given sampling density increases exponentially with the number of parameters to optimize.

  • verbose name: Compute Budget
  • default value: 10
  • port type: IntPort
  • value type: int (can be None)
    

• param1
  Search range for parameter 1. The following formats are supported: name=[1,2,3] (list of values), name=(1...3) (continuous range), name=2**(1...3) (log-spaced continuous range).

  • verbose name: Parameter 1
  • default value: alpha=[1,2,3]
  • port type: StringPort
  • value type: str (can be None)
    

• param2
  Search range for parameter 2. See param1 for supported formats.

  • verbose name: Parameter 2
  • default value:
  • port type: StringPort
  • value type: str (can be None)
    

• param3
  Search range for parameter 3. See param1 for supported formats.

  • verbose name: Parameter 3
  • default value:
  • port type: StringPort
  • value type: str (can be None)
    

• param4
  Search range for parameter 4. See param1 for supported formats.

  • verbose name: Parameter 4
  • default value:
  • port type: StringPort
  • value type: str (can be None)
    

• param5
  Search range for parameter 5. See param1 for supported formats.

  • verbose name: Parameter 5
  • default value:
  • port type: StringPort
  • value type: str (can be None)
    

• searcher_config
  Additional configuration for the searcher. This can be used to pass additional parameters to the chosen searcher, but is rarely necessary. The searchers are part of the nevergrad package, and the documentation for the searchers can be found at https://facebookresearch.github.io/nevergrad/optimizers_ref.html.

  • verbose name: Searcher Configuration
  • default value: None
  • port type: DictPort
  • value type: dict (can be None)
    

• enabled
  Whether to enable parameter search. If disabled, the node will use the midpoint of the respective search ranges as the parameter value.

  • verbose name: Enable Search
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• default_settings
  Default settings when search is NOT enabled. Formatted like a Python dictionary, i.e., {'alpha': 10, 'beta': 0.1}. This is useful when rapidly prototyping parts of a method that requires sensible parameter values without incurring the time cost of a parameter search. Note however that this field should not be used to dial in the final parameter values and then report the resulting performance scores as if they represented the generalization error of the method, since this would be cherry-picking and falls under scientific malpractice -- unless the final parameters were determined on an entirely different dataset that one evaluates performance on, or were educated a priori guesses made well before any data was seen. Instead, when final scores (e.g., for a paper or study) are to be reported, one should leave the parameter search enabled and measure performance in this fashion.

  • verbose name: Default Settings
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• search_metric
  Scoring metric with regards to which to optimize hyper-parameters. Note that which metric is being optimized matters, and the resulting model is likely to perform better with regards to that metric than others, although typically other metrics will also generally improve as a side effect of optimizing one metric; good candidates are BAcc (balanced accuracy) or AUC (area under ROC curve) for classification methods, MSE (mean squared error) or the more robust MAE (mean absolute error) for regression pipelines. It is also possible to optimize for metrics like sensitivity or specificity in limited circumstances. See also the cross-validation node for more details.

  • verbose name: Metric To Optimize
  • default value: BAcc
  • port type: EnumPort
  • value type: str (can be None)
    

• num_folds
  Number of nested cross-validation folds for parameter search. Note that it is also possible to wire in a subgraph into the evaluator input that gives full control over the nested cross-validation setup. If so, it is recommended to set this parameter to 0 to avoid confusion over which setting is in effect. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. This happens on the respective training set (e.g., of an outer cross-validation). For each fold, the method is re-trained on all the other blocks, excluding the test block. This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training. It is also possible to set this to 1, which means leave-one-out CV; typically this is excessive, except in conjunction with the grouping field, which causes one group (of trials that have the same group id) to be held out at a time. If a number between 0 and 1 is used, this is p-holdout cross-validation, where the respective fixed fraction of trials is held out. See also the cross-validation node for additional details and options.

  • verbose name: Number Of Cross-Validation Folds
  • default value: 5
  • port type: FloatPort
  • value type: float (can be None)
    

• cv_cond_field
  The name of the instance data field that contains the values that the method should be optimized to predict. This parameter will be ignored if the input data packet has previously been processed by a BakeDesignMatrix node.

  • verbose name: Condition Field (Cross-Validaiton)
  • default value: TargetValue
  • port type: StringPort
  • value type: str (can be None)
    

• cv_group_field
  Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are Subject_id, Session_id, etc.

  • verbose name: Grouping Field (Cross-Validation)
  • default value:
  • port type: StringPort
  • value type: str (can be None)
    

• cv_stratified
  Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials with each label. Note that this requires labels to be quantized or binned to be meaningful.

  • verbose name: Stratified Cross-Validation
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• num_procs
  Number of processes to use for parallel computation. If None, the global setting NUM_PROC, which defaults to the number of CPUs on the system, will be used.

  • verbose name: Max Parallel Processes
  • default value: 1
  • port type: IntPort
  • value type: int (can be None)
    

• num_threads_per_proc
  Number of threads to use for each process. This can be used to limit the number of threads by each process to mitigate potential churn. This can be further overridden by the used evaluator.

  • verbose name: Threads Per Process
  • default value: 4
  • port type: IntPort
  • value type: int (can be None)
    

• multiprocess_backend
  Backend to use for farming out computation across multiple (CPU) processes. Multiprocessing is the simple Python default, which is not a bad start. Pebble can be more robust than multi-pprocessing and can be a useful alternative to multiprocessing if you are running out of memory after a while or if tasks otherwise appear to hang or crash unexpectedly. Serial means to not run things in parallel but instead in series (even if num_procs is >1), which can help with debugging. Threading uses Python threads in the same process, but this is not recommended for most use cases due to what is known as GIL contention.

  • verbose name: Multiprocess Backend
  • default value: multiprocessing
  • port type: EnumPort
  • value type: str (can be None)
    

• assign_gpus
  Assign GPUs for the parameter search. If enabled, the parameter search will make sure that each process gets a different GPU (potentially giving the same GPU to more than one process if there are more processes than GPUs on the system).

  • verbose name: Assign Gpus
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• serial_if_debugger
  If True, then if the Python debugger is detected, the node will run in serial mode, even if multiprocess_backend is set to something else. This is useful for debugging, since the debugger does not work well with parallel processes. This can be disabled if certain steps should nevertheless run in parallel (e.g., to reach a breakpoint more quickly).

  • verbose name: Serial If Debugger
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• loss_if_error
  Loss value to return if an error occurs during evaluation. If None, the error will be raised. This can be useful to prevent the optimization from crashing if a parameter setting causes an error, but it can also mask errors that would otherwise be caught. This value may be set to the worst-case loss according to the measure used by the evaluator (e.g., 1.0 for accuracy); potentially larger values can also be used, but note that their effect on the search process depends on the selected search method (some may be more likely to steer cleer of error-causing parameters while others might become unstable).

  • verbose name: Loss Value If Error
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• straggler_cutoff
  Cutoff time for stragglers in standard deviations. If a parameter setting takes more than this many standard deviations longer than the average time, it will be discarded. This helps mitigate hangs due to bad parameter settings hat take excessively long or hangs with individual runs. In rare cases this will trim off runs that would have been useful, in which case you may increase this parameter.

  • verbose name: Long-Running Task Cutoff
  • default value: 5
  • port type: FloatPort
  • value type: float (can be None)
    

• random_seed
  Random seed (int or None). Different values will give somewhat different outcomes.

  • verbose name: Random Seed
  • default value: 12345
  • port type: IntPort
  • value type: int (can be None)
    

• log_folder
  Folder to store optimization logs. This defaults to NeuroPype's log folder. This can be useful to recover partial results if your parameter search takes very long and has to be terminated (or crashes mid-way). If you set it to empty, no logs will be stored.

  • verbose name: Log Folder
  • default value: None
  • port type: StringPort
  • value type: str (can be None)
    

• initialize_once
  Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

  • verbose name: Calibrate Only Once
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• dont_reset_model
  Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

  • verbose name: Do Not Reset Model
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• verbose
  Print verbose output.

  • verbose name: Verbose
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

PerformanceMetric

Calculate one of a set of performance metrics between predictions and target values.

Performance metrics return a kind of performance score for a given model, and a fairly large number of metrics have been proposed across various statistical and machine-learning areas. This node supports a number of the most common metrics for classification and regression problems; see the documentation for the metric port for details on specific metrics. Beware that for classification metrics including accuracy, precision, recall, and f1, the predictions should be the class labels (not the class probabilities). The node will attempt to detect such cases and convert the predictions to classes if they are not already, but this may not be reliable in all cases, especially when there are few samples.

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• perf_metric
  Performance metric to evaluate. Note that, in constrast to loss functions (nodes ending in Loss), higher values are better for performance metrics, an in the case of metrics that are technically loss functions (e.g., the mean squared error), the negative of the underlying loss is returned. Generally some of the available metrics are suitable for classification problems (accuracy, precision/recall/f1, and roc metrics) while others are for use in regression problems (r2, explained_variance, max_error and the neg_error metrics). These metrics are all available in the scikit-learn package, and the documentation for them can be found on the web. In quick summary, balanced_accuracy is a good choice for generic classification problems on balanced or unbalanced data, precision/recall are sometimes asked for in detection problems of a relatively rare class (e.g., clinical diagnosis), and the roc metrics are useful both for measuring performance on unbalanced data (although on extremely unbalanced data, the measures can break down), and for measuring the performance in detection problems across a range of decision thresholds, e.g., when the relative importance of type-1 and type-2 errors is not fixed at the time the ML work is done. The F1 scores are a specific blend of precision and recall across classes that can be used as a general-purpose (albeit perhaps not very interpretable) performance measure on some signal detection tasks. For regression, the r2 score is a fair choice for generic signal regression and is relative to the scale of the target variable; explained variance is similar but specifically does not score systematic offset (i.e., bias) in the predictions (usually a deficiency of the metric but sometimes useful, e.g. while working on a model that will still receive a bias correction later). The neg_error metrics are the most common tools in regression problems; of those, the mean_absolute and median_absolute variants are perhaps the most interpretable as the errors are in the same units as the target variable (e.g., meters), and the latter is robust if there are outliers in either the targets or predictions. The mean_squared error is the default choice under a Gaussian noise assumption, and the mean_squared_log error is useful if the data are log-normally distributed (e.g., data from exponential growth processes). Somewhat similarly, the neg_mean_absolute_percentage_error is also adequate for measuring performance on data where the target variable ranges across several orders of magnitude in scale, and the error is relative to each individual target value (but note that the score is not measured in percent but is a fraction). Note that, unlike the loss measures, most of the metrics are not differentiable, so they cannot be directly optimized for using e.g., gradient descent. However, model hyper-parameters can be optimized with respect to these metrics when using, e.g., the Parameter Optimization node (although at significant computational cost).

  • verbose name: Perf Metric
  • default value: balanced_accuracy
  • port type: ComboPort
  • value type: str (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

PlaneConstraint

A constraint projection forcing the given solution to lie on a plane in N-dimensional space (hyper-plane) or in a half-space bounded by the plane (on one side of the plane).

The plane is defined by a normal vector of same dimensionality sa N, and can be shifted along the normal away from the origin. Like all constraint nodes, this is normally used as part of an optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• normal
  Normal vector of the plane.

  • verbose name: Normal
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• offset
  Offset or shift of the plane along the normal from the origin.

  • verbose name: Offset
  • default value: 0.0
  • port type: FloatPort
  • value type: float (can be None)
    

• constraint
  Whether the constraint is a half-space (upper bound) or a hyper-plane (equality). In the upper-bound case, values are constrained to lie on the negative side (opposite of the normal vector) of or on the plane.

  • verbose name: Constraint
  • default value: equality
  • port type: EnumPort
  • value type: str (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

ProximalGradientDescentSolve

Iteratively solve a convex optimization problems using the proximal gradient descent method.

This node can solve (find the minimum of) a wide range of convex optimization problems common in machine learning and engineering sciences, both smooth and non-smooth. If the problem is smooth, then a cost function to minimize, which is a function of some vector- or matrix-shaped weights to optimize (w), optionally along with a scalar bias (b) that is also optimized, and provided data (D), is wired into the "cost" input as a graph with placeholders of same name. This is then solved via gradient descent (or optionally accelerated gradient descent). Alternatively, if the cost is further subject to a hard constraint, a small graph that applies one of the constraint nodes to the weights w may be wired into the "constraint" input; this is then solved via projected gradient descent (optionally accelerated). Lastly, if the cost function has a non-smooth term (e.g., a regularization term), then a graph that uses one of the proximal operator nodes (or which manually implements a proximal operator corresponding to the desired non-smooth term) can be wired into the "prox" input; in this case the problem is solved via proximal gradient descent (this is also known as the ISTA algorithm or, if accelerated, FISTA). In simple cases, the initial weights can be omitted, but in the general case, initial weights (e.g., an appropriately shaped all-zeros array or packet) must be provided since otherwise the shape is unknown. Like with all solve nodes, the solution is differentiable, meaning that the node can be used in a place where gradients are taken, for example in the network graph a DeepModel node.

More Info...

Version 0.2.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• weights
  Initial and final weights. If not set, will be initialized to a packet equivalent to a single all-zeroes instance of the training-data.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: INOUT
    

• bias
  Optional estimated bias, if used as second argument in cost.

  • verbose name: Bias
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• state
  Optional initial and final solver state.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• cost
  Smooth part of cost function given some data.

  • verbose name: Cost
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• cost__signature
  Signature for the "cost" input. This represents the signature for the subgraph that is wired into the "cost" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "cost" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Cost [Signature]
  • default value: (w,b,D)
  • port type: Port
  • value type: object (can be None)
    

• prox
  Optional proximal operator of non-smooth term(s).

  • verbose name: Prox
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• prox__signature
  Signature for the "prox" input. This represents the signature for the subgraph that is wired into the "prox" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "prox" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Prox [Signature]
  • default value: (w,step)
  • port type: Port
  • value type: object (can be None)
    

• constraint
  Optional hard constraints.

  • verbose name: Constraint
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• constraint__signature
  Signature for the "constraint" input. This represents the signature for the subgraph that is wired into the "constraint" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "constraint" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

  • verbose name: Constraint [Signature]
  • default value: (w)
  • port type: Port
  • value type: object (can be None)
    

• hyper_params
  Hyper-parameters for the cost function. This is a dictionary of arbitrary key-value pairs that can be used to configure the cost function and proximal operators. The respective graphs may then declare and use placeholders named the same as of the dictionary keys.

  • verbose name: Hyper Params
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• max_iter
  Maximum number of iterations.

  • verbose name: Max Iter
  • default value: 500
  • port type: IntPort
  • value type: int (can be None)
    

• abstol
  Absolute convergence tolerance. If weights change less than this (after normalization by step size), the optimization terminates. Note that this depends on the data scale.

  • verbose name: Abstol
  • default value: 0.001
  • port type: FloatPort
  • value type: float (can be None)
    

• stepsize
  Optional step size. If unspecified, the step size is adapted automatically using back-tracking line search.

  • verbose name: Stepsize
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• max_backtrack
  Maximum number of line search steps per iteration. Only used if step size is left to automatic.

  • verbose name: Max Backtrack
  • default value: 15
  • port type: IntPort
  • value type: int (can be None)
    

• backtrack_factor
  Backtracking line search factor. Only used if stepsize is 0. The default depends on the chosen algorithm, and is 0.5 for PGD.

  • verbose name: Backtrack Factor
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• acceleration
  Whether to use Nesterov acceleration. This will usually lead to faster convergence at minimal per-iteration overhead.

  • verbose name: Acceleration
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• use_jit
  If enabled, attempt to use JIT compilation for the inner loop. This incurs a one-time compilation cost, but the actual solving will be greatly accelerated if using the GPU.

  • verbose name: Use Jit
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• unroll
  Whether to unroll the optimization loop. This can save some per-iteration overhead, but the benefit depends on the specific use case.

  • verbose name: Unroll
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• implicit_diff
  Whether to use implicit differentiation for computing gradients. The alternative is to unroll the solver iterations.

  • verbose name: Implicit Diff
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• verbosity
  Verbosity level. 0: no output, 1: per-iteration summary. Note that JIT will be disabled if verbosity is used.

  • verbose name: Verbosity
  • default value: 0
  • port type: IntPort
  • value type: int (can be None)
    

• incremental_updates
  Whether to use incremental updates from the previous state. If not set, the previous or wired-in state is ignored.

  • verbose name: Incremental Updates
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

SigmoidBinaryCrossEntropyLoss

Calculate a cross-entropy loss with either binary labels or multi-class (non-exclusive) labels and logit predictions (unnormalized log probabilities).

Note that, when you train a neural network with this loss, you would not use a sigmoid activation function on the output layer, since this loss already includes the sigmoid transform. This node can be used with two kinds of data, and in both cases the predictions and labels must have the same (or a broadcastable) shape, and predictions are in logits form (unnormalized log probabilities). Either the predictions are for the positive class and the labels are integers in {0,1}, or the predictions have a trailing statistic axis of length equal to the number of classes (and usually represent 1-vs-rest classifications), and the labels likewise have a trailing statistic axis, where entries do not necessarily have to sum to 1, i.e., labels are not necessarily mutually exclusive. For the regular multi-class case with exclusive classes, see the SoftmaxCrossEntropyLoss node. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss. This the canonical classification loss for the sigmoid activation function. For the softmax activation function, use the SoftmaxCrossEntropyLoss node.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• sanity_checks
  Perform sanity checks on the inputs.

  • verbose name: Sanity Checks
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

SoftmaxCrossEntropyLoss

Calculate a softmax cross-entropy loss between logit predictions (unnormalized log probabilities) and integer target labels or a label distribution.

Note that, when you train a neural network with this loss, you would not use a softmax activation function on the output layer, since this loss already includes the softmax transform. The predictions must have a (usually trailing) statistic axis of length equal to the number of classes. The labels can either be of the same shape as the predictions (or a broadcastable shape) if labels_are is set to 'distributions' (see labels_are parameter) or they can be encoded as integers (class indices) and in that case do not have a statistic axis. This is a multi-class loss; for a binary loss, see the SigmoidBinaryCrossEntropy node. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss. This the canonical classification loss for the softmax activation function. For the sigmoid activation function, use the SigmoidBinaryCrossEntropy node.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• labels_are
  Format of the labels. If 'integers', the labels are expected to be integer values between 0 and the number of classes minus 1. If 'distributions', the labels are expected to have a statistic axis of length equal to the number of classes, and the values are expected to be probabilities that sum to 1. In this case, labels can be a one-hot encoding, or a distribution corresponding to a soft labeling. If set to 'auto', then integers is inferred if the data is of integer type and distributions is inferred if the data is of same shape as the predictions or a broadcastable shape. Otherwise an error is raised.

  • verbose name: Labels Are
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• sanity_checks
  Perform sanity checks on the inputs.

  • verbose name: Sanity Checks
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

SparsePenalty

A penalty encouraging sparse (few nonzeros) solutions, as a proximal operator for the classic l1 norm.

The operator penalizes the sum of absolute values in the input, which has the side effect of reducing the number of non-zero elements in the input, depending on the given degree of sparsity. The operation applied by the operator is also known as soft thresholding. A less common use case is to employ an alternative non- convex norm via the norm parameter and the associated shape parameter, which can yield better results in practice (given a careful choice of solver and problem setup) which however sacrifices the otherwise guaranteed global optimality. Like all proximal operators, this is normally used as part of the optimization problem formulation given to a (typically convex) solver node.

Version 1.0.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• data
  Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT
    

• step_size
  Step size.

  • verbose name: Step Size
  • default value: None
  • port type: DataPort
  • value type: float (can be None)
  • data direction: IN
    

• step_count
  Current step count for outer solver.

  • verbose name: Step Count
  • default value: None
  • port type: DataPort
  • value type: int (can be None)
  • data direction: IN
    

• sparsity
  Degree of sparsity in the solution. Larger values will encourage a higher degree of sparsity in the input to this node.

  • verbose name: Sparsity
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)
    

• norm
  Alternative non-convex norm to use. Any other value than the default of l1 will result in a non-convex penalty, where global optimality guarantees are lost. However, in practice, and given the right solver, these norms can be more effective at encouraging sparsity than the convex l1 norm. Most importantly, the l1 norm is biased towards solutions with a small overall magnitude, whereas non-convex norms are closer to penalizing the number of non-zeros with a lower bias towards lower magnitude of those retained non-zeros. Successful usage of these norms requires skilled choice of problem structure and solver or extensive experimentation.

  • verbose name: Norm
  • default value: l1
  • port type: EnumPort
  • value type: str (can be None)
    

• norm_shape
  Shape parameter for the alternative norm. In case of the lq norm this is q (between 0 and 1), the power of the norm. for q- shrinkage, this is q (between -inf and 1), the shrinkage factor. For SCAD, this is the a parameter (a good default is 3.7). For MCP, this is the psi (also named gamma) parameter.

  • verbose name: Norm Parameter (If Norm Is Not L1)
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

SquaredLoss

Calculate a squared error between predictions and regression target values.

This is a canonical loss for regression or least-squares problems and is motivated by an assumption that the errors are normally (Gaussian) distributed. This loss is known to not be robust to outliers, and for that reason, it is sometimes replaced with the Huber loss (see HuberLoss node). The labels should have the same shape as the predictions or otherwise at least need to be of a broadcastable shape. Like all Loss nodes (except MeasureLoss), this node returns the per-example loss, which needs to be manually summed or averaged to get a total dataset/batch loss. If target values are omitted, they default to 0.

Version 1.1.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• preds
  Predictions.

  • verbose name: Preds
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• targs
  Target values.

  • verbose name: Targs
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: IN
    

• output
  Output.

  • verbose name: Output
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: OUT
    

• multiplier
  Optional multiplier that precedes the square error term. The default is the plain squared error with no multiplier. When set to 'l2 loss (0.5)', the value is multiplied by 0.5, which follows the definition in some (but not other) literature, e.g., the PRML book by Bishop. When replicating papers, make sure you check the definition of the loss. Also note that the l2 loss is not the same as the l2 norm, but rather it is the squared l2 norm (i.e., 0.5*||x||_2^2).

  • verbose name: Multiplier
  • default value: squared error (1.0)
  • port type: EnumPort
  • value type: str (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• axis_pairing
  How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.

  • verbose name: Axis Pairing
  • default value: default
  • port type: EnumPort
  • value type: str (can be None)
    

• label_pairing
  How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.

  • verbose name: Label Pairing
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)
    

• lists_as_arrays
  Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.

  • verbose name: Lists As Arrays
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

ValueAndGradient

Calculate the function value and gradient for a given graph with respect to one or more of its parameters.

This node accepts a graph (a function of some inputs) that calculates a single scalar value, and returns a modified graph that will calculate, for a subset of inputs of interest, the vector (or matrix, etc.) that represents the direction of the greatest increase in the scalar value. Note: there are several restrictions on the form of the graph that is wired into this node. Consequently, the gradient can then be added to the initial input values to move them towards a local maximum of the original computational graph, i.e., the node can be used iteratively to perform gradient descent to find the optimal parameters for a given computation. Note that the gradient is different for different values of the inputs, including any extra parameters that are held fixed, and therefore the resulting function has the same number of inputs as the original graph, but returns not a scalar but a vector (or matrix, etc.) of the same shape as the subset of inputs of interest.

More Info...

Version 0.5.0

Ports/Properties

• metadata
  User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)
    

• function
  Function to process.

  • verbose name: Function
  • default value: None
  • port type: GraphPort
  • value type: Graph
    

• function__signature
  Signature for the "function" input. This represents the signature for the subgraph that is wired into the "function" port. This is formatted as in (a,b,c) ~> (b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes in, and out of, the "function" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c'], ['b','c'].

  • verbose name: Function [Signature]
  • default value: (params)~>(params)
  • port type: Port
  • value type: object (can be None)
    

• variables
  Variables with respect to which the gradient will be calculated. If not given, this defaults to the first positional argument of the function, i.e., the first name in the signature.

  • verbose name: Differentiate With Respect To
  • default value: None
  • port type: ListPort
  • value type: list (can be None)
    

• extra_outputs
  If True, the derivative will only be calculated for the first output of the graph, with additional outputs being passed through unchanged. In this case the graph is expected to return a list of two or more elements.

  • verbose name: Have Extra Outputs
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)
    

• backend
  Compute backend to use. This computation is currently only supported by the jax backend, and the data will be converted to this backend unless the backend is set to 'keep', which you can choose to reduce overhead if you know that the data is already in the jax backend.

  • verbose name: Backend
  • default value: jax
  • port type: EnumPort
  • value type: str (can be None)
    

• automove_data
  Automatically move data to the chosen backend. This adds a slight performance overhead if the data is already on the desired backend and is only included for convenience.

  • verbose name: Automove Data
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)
    

• set_breakpoint
  Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)