B.4.3 lr

This is an implementation of logistic regression (LR). LR computes $\beta$ for which the model values $\mu_i$ best approximate the dataset outputs $y_i$ under the model

$$\mu_i = \ensuremath{ \frac{ \ensuremath{\exp\left({{\beta}_0 + {\... ...eft({{\beta}_0 + {\beta}_1 {x_i}_1 + \cdots {\beta}_{M} {x_i}_{M}}\right)} } }$$

where $x_i$ is one dataset row. Please see [13,25,20,10] for details about logistic regression. This implementation uses iterative re-weighted least squares (IRLS) [20,10] to maximize the LR log-likelihood

$$\sum_n{i=1}^R y_i \ln (u_i) + (1-y_i) \ln (1-u_i)$$

where $R$ is the number of rows in the dataset. For logistic regression, IRLS is equivalent to Newton-Raphson [25]. To improve the speed of IRLS, this implementation uses conjugate gradient [26,27,31,7] as an approximate linear solver [20]. This solver is applied to the linear regression

$$\left( X^T W X \right) \beta_{\mbox{new}} = X^T W z$$ (B.7)

where $W =$   diag$(\mu_i (1-\mu_i))$ and $z = X \beta_{\mbox{old}} + W^{-1} (y - \mu)$. The current estimate of $\beta$ is scored using the likelihood ratio $-2 log( L_{\mbox{sat}} / L_{\mbox{current}})$, where $L_{\mbox{sat})}$ is the likelihood of of a saturated model with $R$ parameters and $L_{\mbox{current}}$ is the likelihood of the current model. This ratio is called the ``deviance'', and the IRLS iterations are terminated when the relative difference of the deviance between iterations is sufficiently small. Other termination measures can be added, such as a maximum number of iterations.

Keyword	Arg Type	Arg Vals	Default
Common
cgdeveps	float	[1e-10, $\infty$)	0.0
cgeps	float	[1e-10, $\infty$)	0.001
lrmax	int	0,..., $\infty$	30
rrlambda	float	[0, $\infty$)	10.0
Rare
binitmean	none
cgbinit	none
cgwindow	int	0, ..., $\infty$	3
cgdecay	float	[1.0, $\infty$)	1000
cgmax	int	0, ..., $\infty$	200
holdout_size	float	[0.0, 1.0]	0.0
lrdevdone	float	[0.0, $\infty$]	0.0
lreps	float	[1e-10, $\infty$)	0.05
margin	float	[0.0, $\infty$)	0
modelmax	float	(modelmin, 1.0]	1.0
modelmin	float	[0.0, modelmax)	0.0
wmargin	float	[0.0, 0.5)	0.0

Common keywords and arguments:

• cgdeveps float: This parameter applies to conjugate gradient iterations while approximating the solution $\beta_{\mbox{new}}$ in Equation . If cgdeveps is positive, conjugate gradient is terminated if relative difference of the deviance falls below cgdeveps. The recommended value of cgdeveps is 0.005. Decrease cgdeveps for a tighter fit. Increase cgdeveps to save time or if numerical issues appear. Exactly one of cgdeveps and cgeps can be positive.

• cgeps float: This parameter applies to conjugate gradient iterations while approximating the solution $\beta_{\mbox{new}}$ in Equation . If cgeps is positive, conjugate gradient is terminated when the conjugate gradient residual falls below cgeps times the number of attributes in the data. Decrease cgeps for a tighter fit. Increase cgeps to save time or if numerical issues appear. Exactly one of cgeps and cgdeveps can be positive.

• lrmax float: This parameter sets an upper bound on the number of IRLS iterations. If performing a train/test experiment, you may want to decrease this value to 5, 2, or 1 to prevent over-fitting of the training data.

• rrlambda float: This is the ridge regression parameter $\lambda$. When using ridge regression, the large-coefficient penalty $\lambda \beta^T \beta$ is added to the sum-of-squares error for the linear regression represent by Equation .

Rare keywords and arguments:

• binitmean: If this keyword is used, the model offset parameter $\beta_0$ is initialized to the mean of the output values $y_i$. [26] reports that this eliminated some numerical problems in his implementation. We have not observed significant changes in our implementation when using this technique.

• cgbinit: If cgbinit is specified, each IRLS iteration will start the conjugate gradient solver at the current value of $\beta$. By default the conjugate gradient solver is started at the zero vector.

• cgwindow int: If the previous cgwindow conjugate gradient iterations have not produced a $\beta$ with smaller deviance than the best deviance previously found, then conjugate gradient iterations are terminated. As usual, the $\beta$ corresponding to the best observed deviance is returned to the IRLS iteration.

• cgdecay float: If the deviance ever exceeds cgdecay times the best deviance previously found, conjugate gradient iterations are terminated. As usual, the $\beta$ corresponding to the best observed deviance is returned to the IRLS iteration.

• cgmax int: This is the upper bound on the number of conjugate gradient iterations allowed. The final value of $\beta$ is returned.

• holdout_size float: If this parameter is positive IRLS iterations are terminated based on predictions made on a holdout set, rather then according to the relative difference of the deviance. This parameter specified the percentage, between 0.0 and 1.0, of the training data to be held out. In a k-fold cross-validation experiment, the training data is not the whole dataset but the data available for training during each fold. The deviance on the holdout set replaces the deviance on the prediction data.

• lrdevdone float: IRLS iterations will be terminated if the deviance becomes smaller than lrdevdone.

• lreps float: IRLS iterations are terminated If the relative difference of the deviance between IRLS iterations is less than lreps.

• margin float: The dataset outputs, which must be zero or one (c.f. Section ), are ``shrunk'' by this amount. That is, 1 is changed to 1 - margin and 0 is changed to margin. It is recommended that this parameter be left at its default value.

• modelmax float: This sets an upper bound on $\mu_i$ computed by the logistic regression. Although the computation of $\mu_i$ should assure it is positive, round-off error can create problems. It is recommended that this parameter be left at its default value.

• modelmin float: This sets a lower bound on $\mu_i$ computed by the logistic regression. Although the computation of $\mu_i$ should assure it is strictly less than one, round-off error can create problems. It is recommended that this parameter be left at its default value.

• wmargin float: wmargin is short for ``weight margin''. If nonzero, weights less than wmargin are are changed to wmargin and weights greater than 1 - wmargin are changed to 1 - margin. This is option is very helpful for controlling numerical problems, especially when rrlambda is zero. Values between 0.001 and 1e-15 are reasonable.