Skip to content

← machine_learning package

EmpiricalBayesClassification

Sparse Bayesian logistic regression using the empirical Bayes (type-II maximum likelihood) framework.

The model places independent zero-mean Gaussian priors on each feature weight with per-feature precision hyperparameters alpha. Evidence maximization drives many alpha values to infinity, effectively pruning the corresponding features and yielding a sparse classifier with hard zeros.

Two algorithm variants are available. The fast method (Tipping & Faul, 2003) starts from an empty model and sequentially adds, deletes, or re-estimates features based on their marginal likelihood contribution. This is efficient for high-dimensional problems where only a few features are relevant. The full method uses the MacKay ARD update rule, starting with all features and iteratively pruning those whose precision exceeds a threshold. It is simpler and can be more thorough, but has higher per-iteration cost for large feature sets.

The underlying inference uses a Laplace approximation (iteratively reweighted least squares) to handle the logistic likelihood, combined with closed-form alpha updates in the outer evidence-maximization loop. This approach is deterministic (no sampling), fast, and produces exact zeros for irrelevant features.

More Info... Version 0.7.0

Ports/Properties

data

Data to process.

verbose name
Data
default value
None
port type
DataPort
value type
Packet (can be None)
data direction
INOUT

cond_field

Instance data field that contains the class labels to discriminate. The field must be binary (exactly two distinct values).

verbose name
Cond Field
default value
TargetValue
port type
StringPort
value type
str (can be None)

feature_scaling

Feature scaling applied before fitting. Standardization ('std') is recommended because the ARD prior is scale-sensitive. 'robust' uses the median and interquartile range instead of mean and standard deviation.

verbose name
Feature Scaling
default value
std
port type
EnumPort
value type
str (can be None)

method

Method variant.

'fast' uses the Tipping & Faul (2003) sequential add/delete algorithm that maintains a small active set of relevant features and is efficient for high-dimensional problems with sparse solutions. It starts from an empty model and greedily adds features that most increase the marginal likelihood.

'full' uses the MacKay ARD update rule starting from all features and progressively pruning irrelevant ones. It explores the full feature space more thoroughly but has higher per-iteration cost (O(p3) when p > n, mitigated via the Woodbury identity).

verbose name
Method
default value
fast
port type
EnumPort
value type
str (can be None)

include_bias

Include a bias (intercept) term in the logistic regression model. The bias receives a weak prior (small alpha) and is always retained.

verbose name
Include Bias Term
default value
True
port type
BoolPort
value type
bool (can be None)

pruning_threshold

Alpha threshold for feature pruning. Features whose ARD precision hyperparameter alpha exceeds this value are considered irrelevant and removed. Larger values are more permissive. Primarily used by the 'full' method; the 'fast' method uses its own marginal-likelihood-based deletion criterion.

verbose name
Pruning Threshold
default value
100000000.0
port type
FloatPort
value type
float (can be None)

max_iterations

Maximum outer iterations for the evidence-maximization loop. For 'fast', this is the maximum number of sequential add/delete/re-estimate steps. For 'full', it is the maximum number of alpha-update rounds. The 'fast' method typically converges in a few dozen steps; the 'full' method may use up to 200 for large feature sets.

verbose name
Maximum Iterations
default value
500
port type
IntPort
value type
int (can be None)

convergence_tolerance

Convergence tolerance on the change in log marginal likelihood between successive outer iterations. The default of 1e-4 is well suited for both method variants.

verbose name
Convergence Tolerance
default value
0.0001
port type
FloatPort
value type
float (can be None)

irls_max_iterations

Maximum IRLS (iteratively reweighted least squares) iterations per outer step. The IRLS loop finds the MAP estimate of the logistic regression coefficients given the current alpha values via the Laplace approximation.

verbose name
Irls Max Iterations
default value
25
port type
IntPort
value type
int (can be None)

irls_tolerance

Convergence tolerance for the IRLS inner loop on the maximum absolute weight change.

verbose name
Irls Tolerance
default value
1e-06
port type
FloatPort
value type
float (can be None)

initialize_once

Calibrate the model only once on the first data that arrives. Subsequent data chunks are only used for prediction. Uncheck to recalibrate whenever new labeled data arrives.

verbose name
Calibrate Only Once
default value
True
port type
BoolPort
value type
bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph structure is changed. Enable this to keep a previously calibrated model across graph edits.

verbose name
Do Not Reset Model
default value
False
port type
BoolPort
value type
bool (can be None)

probabilistic

Output class probabilities rather than hard class labels. When enabled, the output has two columns (one per class) with the predicted probabilities.

verbose name
Output Probabilities
default value
True
port type
BoolPort
value type
bool (can be None)

verbosity

Verbosity level. Set to 1 for summary output, 2 for per-iteration diagnostics.

verbose name
Verbosity Level
default value
0
port type
IntPort
value type
int (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

verbose name
Set Breakpoint (Debug Only)
default value
False
port type
BoolPort
value type
bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

verbose name
Metadata
default value
{}
port type
DictPort
value type
dict (can be None)