LBFGSSolve¶
Iteratively solve a smooth nonlinear optimization problem using the L-BFGS method.
This node is suitable for solving optimization problems of intermediate dimensionality and data size efficiently and with relatively few evaluations of the cost function. Typically, this is used on convex problems, but it may also be used on non-convex problems at the risk of finding only a local optimum. For an alternative that supports non-smooth problems, see the ProximalGradientSolve node. The L-BFGS method is a quasi-Newton method that uses second-order (curvature) information of the cost function by maintaining a low-rank approximation of the inverse Hessian. This comes with additional tracking and memory overhead per iteration compared to first-order methods, but if the optimum is approximately quadratically shaped (e.g., due to a dominant quadratic term in the cost function), the method can reach the optimum in a smaller number of iterations. The node provides a wide range of tuning parameters to maximize the performance on specific types of data; however, the method tends to be sufficiently robust that these parameters often do not have to be tuned. In simple cases, the initial weights can be omitted, but in the general case, initial weights (e.g., an appropriately shaped all-zeros array or packet) must be provided since otherwise the shape is unknown. Like with all solve nodes, the solution is differentiable, meaning that the node can be used in a place where gradients are taken, for example in the network graph a DeepModel node. More Info... Version 0.5.0
Ports/Properties¶
data¶
Data to process.
weights¶
Initial and final weights. If not set, will be initialized to a packet equivalent to a single all-zeroes instance of the training-data.
state¶
Optional initial and final solver state.
cost¶
Smooth cost function given some data.
cost__signature¶
Signature for the "cost" input. This represents the signature for the subgraph that is wired into the "cost" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "cost" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].
hyper_params¶
Hyper-parameters for the cost function. This is a dictionary of arbitrary key-value pairs that can be used to configure the cost function. The graph may then declare and use placeholders named the same as of the dictionary keys. This is mainly such that the solver does not have to be recompiled when these values change; for constants, you may alternatively wire them directly into the cost function.
max_iter¶
Maximum number of iterations.
abstol¶
Absolute convergence tolerance. If weights change less than this (after normalization by step size), the optimization terminates. Note that this depends on the data scale.
memory_size¶
Number of previous gradients to remember to approximate curvature. This configures the value of L in L-BFGS (i.e., the limited memory size). Typical values are 6 to 10.
stepsize¶
Optional step size override. If unspecified, the step size is adapted automatically using back-tracking line search.
max_stepsize¶
Maximum step size to use. Only used if step size is dynamically adapted.
min_stepsize¶
Minimum step size to use. Only used if step size is dynamically adapted.
linesearch_type¶
Line-search strategy to use to adapt step sizes.
linesearch_init¶
Strategy to initialize the line-search step size for a new iteration. The least aggressive is current, which starts from the previous step size. Max sets it to the max_stepsize at each iteration, and increase increases it by a factor of increase_factor from the previous iteration (or min_stepsize if smaller than that).
max_backtrack¶
Maximum number of line search steps per iteration. Only used if step size is left to automatic.
backtrack_factor¶
Backtracking line search factor. Only used if stepsize is 0 and if a backtracking linesearch type is used.
increase_factor¶
Factor by which to increase the step size if the line search succeeds.
use_jit¶
If enabled, attempt to use JIT compilation for the inner loop. This incurs a one-time compilation cost, but the actual solving will be greatly accelerated if using the GPU.
unroll¶
Whether to unroll the optimization loop. This can save some per-iteration overhead, but the benefit depends on the specific use case.
implicit_diff¶
Whether to use implicit differentiation for computing gradients. The alternative is to unroll the solver iterations.
implicit_diff_solver¶
Solver to use for implicit differentiation. Auto is currently defaulting to normal-cg, which is a fast generic solver for high-dimensional problems. The options bicgstab and gmres can be more robust on badly conditioned problems, but are slower. The lu method is likely fastest on very low-dimensional problems, but note that the cost of implicit differentiation is likely marginal compared to the cost of the overall solve.
verbosity¶
Verbosity level. 0: no output, 1: per-iteration summary. Note that JIT will be disabled if verbosity is used.
stop_if_linesearch_fails¶
Whether to stop if the line search fails. This can be set to true to either debug bad settings of the min/max step-size values or line-search strategy, or to catch issues early with problematic data if one has strong assumptions about the cost function.
use_gamma¶
Whether to use the gamma parameter in the initialization of the inverse Hessian.
incremental_updates¶
Whether to use incremental updates from the previous state. If not set, the previous or wired-in state is ignored.
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.