CustomStep¶
A custom optimizer step that is defined by a graph applied to gradients and, optionally, current weights.
The graph must have either one or two placeholders (one for the gradients and one for the current weights), which must be listed in the graph's positional signature in that order. You can then implement any computation that processes and returns the updated gradients and, if you are building an end-to-end optimizer step, also converts the gradients into additive updates (by negating them), since the result of end-to-end steps is expected to be applied additively. Your graph may use any of the Step nodes (e.g., AdamStep, GradientClippinStep, etc.) or stateless operations (e.g., Add etc). If you are optimizing parameters that are in the form of a Packet (by setting prefer_packets to True in NetTransform), you can also use stateless nodes that operate on Packets such as ExtractStreams, MergeStreams, and so forth, which is useful when processing subsets of the parameters (e.g., corresponding to different layers) differently. There is a small set of stateful nodes that can also be used safely here; these are the nodes that have a "state" input and output port. Your graph may also expose additional hyper- parameters (e.g., the weight decay parameters), which can be defined using ParameterPort nodes and which need to have default values. These parameters can be overridden by passing name/value pairs into Custom Step. Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
weights¶
Optional current weights.
state¶
Explicit state of the node.
graph¶
Optimizer step returning additive updates.
graph__signature¶
Signature for the "graph" input. This represents the signature for the subgraph that is wired into the "graph" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "graph" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].
name1¶
Override 1.
val1¶
Value 1.
name2¶
Override 2.
val2¶
Value 2.
name3¶
Override 3.
val3¶
Value 3.
name4¶
Override 4.
val4¶
Value 4.
name5¶
Override 5.
val5¶
Value 5.
name6¶
Override 6.
val6¶
Value 6.
name7¶
Override 7.
val7¶
Value 7.
name8¶
Override 8.
val8¶
Value 8.
name9¶
Override 9.
val9¶
Value 9.
nameN¶
Additional overridden names.. .
valN¶
Additional overridden values.. .
name0¶
Override 0.
val0¶
Value 0.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.