Bayesian Logistic Regression Software
Alex Genkin, David D. Lewis, and David Madigan
Last update:
This software implements Bayesian logistic regression with two choices of priors: Gaussian and Laplace. A general binary regression classifier takes the form:
where y is the class label, 1 or -1; x is the predictor vector; β is the vector of parameters; ψ is the link function. We consider logistic link function:
Our software finds the maximum a posteriori parameter estimates with two choices for prior: Gaussian or Laplace, the latter given by the formula:
where βj is a component of the vector of parameters. All prior components are assumed to be independent, equally scaled, and zero centered, though we plan to extend this in the future versions of the program. The Laplace prior corresponds to Tibshirani's LASSO algorithm.
To find the parameter estimates the software implements a coordinate descent algotrithm that draws on the ideas of Zhang and Oles (2001).
The user chooses one of two prior types: Laplace or Gaussian. Then the use has to specify the hyperparameter value λ defining the scale of the prior in the Laplace case and variance in the Gaussian case. There are three ways for the user to define the hyperparameter value.
The first way is to specify the hyperparameter value explicitly. The second way is to omit any specification and allow the program to set the value by default. The program sets the default prior variance equal to the inverse average squared value of all data elements in training. The third way is to select a value from a list using (“fractional”) cross-validation.
For the cross-validation option, the user specifies the list of hyperparameter values to select from. Since these are scale type parameters, it makes sense to use a part of geometric progression for that purpose, e.g. 0.001, 0.01, 0.1, 1, 10, 100, 1000. Each member of the list is tried in the cross-validation loop, the one which maximizes average log-likelihood on validation subsets is selected for final modeling on all training data. The user also specifies the number of folds for cross-validation. With a large data set it might be preferable to perform “fractional” cross-validation, i.e. do less training and validation runs than there are folds, so the user has the option to specify the number of runs. Default is 10 folds and 10 runs.
Logistic regression estimates the probability that a data vector belongs to the class with label 1. Classification requires a threshold: the model assigns a case to class 1 iff the probability estimate is greater or equal to the threshold value.
The software determines the threshold value after all the training samples have probability estimates assigned. Several criteria for threshold tuning are available. The confusion table below defines the necessary notation:
The program offers the following choices for threshold tuning criteria:
no tuning, threshold is equal to 0.5
sum of errors = b+c
balanced error rate = (b/(a+b) + c/(c+d))/2
T11U = 2*a - c
F1 = (2*a)/(2*a + b + c)
The two latter measures are popular in text classification.
The program allows the user to filter features that should be used in modeling by feature's utility value. Four choices for utility function are provided: Pearson's correlation coefficient, chi-square, Yule's Q, and bi-normal separation (BNS) (Forman, 2002). The user selects a utility function and the number of features (say, k) that should be used in modeling. The program then calculates the utility value for all features in the data and selects k best that will be used in modeling.
Optionally, all features may be standardized, i.e. transformed to have zero mean and unit standard deviation. This does not only have computational effect as is the case with regular regression. With Bayesian regression standardization also affects the relative amount of shrinkage applied to parameter values corresponding to different features.
This optional data transformation centrally projects each data vector onto the unit Euclidian sphere. After that the dot product of any two vectors is equal to the cosine of the angle betweeen those vectors, hence the name. Cosine normalization is popular in text classification because it helps to compensate for document length.
If both feature selection and cosine normalization are enabled, cosine normalization is applied first.
This software consists of two executable modules: the training module and the classification module. The training module uses a training data file as input and generates a model file. The classification module uses a data file with new data and the model file to generate a results file with predicted probabilities and labels.
Here is how to use the training module:
BBRtrain [options] training_data_file model_file
where the options are:
-p <[1,2]>, Type of prior, 1-Laplace 2-Gaussian (default is 2)
-H <float>, Hyperparameter, depends on the type of prior (optional)
-S <list of floats, comma-separated, no spaces> Search for hyperparameter value: list of values to try in cross-validation
-C <integer[,integer]> Cross-validation: number of folds, number of runs. If the number of runs is not given, it's assumed equal to the number of folds. Default is 10,10
-s <[0,1]>, Standardize variables (default is 0)
-c <[0,1]>, Cosine normalization (default is 0)
-u <[0..3]>, Feature selection utility, 0-Corr 1-Yule's Q 2-Chi-square 3-BNS (default is 0)
-f <integer>, Number of features to select (default is 0)
-t <[0..3]>, Threshold tuning, 0-no 1-sum error 2-T11U 3-F1 4-balanced error rate (default is 0)
-r <file_name>, Results file (optional)
-l <[0..2]>, Program log verbosity level (default is 0)
-v Displays version information and exits
-h Displays usage information and exits
The training data file format follows popular analogs for sparse data like SVMlight. Each row represents a case. Row format is:
<label> {<feature_id>:<value>}*
Here label may take value 1 or -1; feature id is positive integer; value is double float.
Result file rows also correspond to cases in the same order as in the training data file. Each row has two fields: score (any number) and label predicted by the model (1 or -1). Score is the model estimate of the probability for the case to have label 1. Unless threshold tuning is used, label is 1 if and only if score is greater than or equal to 0.
Here is how to use the classification module:
BBRclassify [options] new_data_file model_file result_file
where the options are:
-l <[0..2]>, Program log verbosity level (optional, default is 0)
-v Displays version information and exits.
-h Displays usage information and exits
New data file format is the same as training data file format described above. Result file format also has been described there. Note that result file is mandatory in this case.
Only a compiled Windows version is available right now. We have tested it on Windows XP, Windows 2000, Linux + Transgaming Winex 3.2 (thanks to Andrei Anghelescu).
You will need two executable modules: BBRtrain.exe and BBRclassify.exe
To complete the installation, just copy executables to the folder where you can execute them.
There are also sample data and results:
The work was partially supported under funds provided by the KD-D group for a project at DIMACS on Monitoring Message Streams, funded through National Science Foundation grant EIA-0087022 to Rutgers University. The NSF also partially supported the work through ITR grant DMS-0113236.
Source code used:
TNT and JAMA http://math.nist.gov/tnt/index.html
Peter J. Acklam's "An algorithm for computing the inverse normal cumulative distribution function" http://home.online.no/~pjacklam/notes/invnorm/
MATV package, (C) Mark Von Tress 1996
Infoscope, (C) Pavel Dubner
Software is free for non-commercial use.
Software is provided as is, without any guarantee. DIMACS or authors are not responsible for implications from the use of this software.
Questions, comments, bug reports welcome by email: agenkin@dimacs.rutgers.edu