B.4.4 lr_cgmle

This is an implementation of logistic regression (LR). LR computes $\beta$ for which the model values $\mu_i$ best approximate the dataset outputs $y_i$ under the model

$$\mu_i = \ensuremath{ \frac{ \ensuremath{\exp\left({{\beta}_0 + {\... ...eft({{\beta}_0 + {\beta}_1 {x_i}_1 + \cdots {\beta}_{M} {x_i}_{M}}\right)} } }$$

where $x_i$ is one dataset row. Please see [13,25,20,10] for details about logistic regression. This implementation uses a conjugate gradient algorithm [26,27,31,7] to maximize the LR log-likelihood

$$\sum_{i=1}^R y_i \ln (u_i) + (1-y_i) \ln (1-u_i)$$

where $R$ is the number of rows in the dataset. The current estimate of $\beta$ is scored using the likelihood ratio $-2 log( L_{\mbox{sat}} / L_{\mbox{current}})$, where $L_{\mbox{sat})}$ is the likelihood of of a saturated model with $R$ parameters and $L_{\mbox{current}}$ is the likelihood of the current model. This ratio is called the ``deviance'', and the conjugate gradient iterations are terminated when the relative difference of the deviance between iterations is sufficiently small. Other termination measures can be added, such as a maximum number of iterations.

Keyword	Arg Type	Arg Vals	Default
Common
cgeps	float	[1e-10, $\infty$)	0.0005
rrlambda	float	[0.0, $\infty$)	10.0
Rare
binitmean	none
cgdecay	float	[1.0, $\infty$)	1000.0
cgmax	int	0,...,$\infty$	100
cgwindow	int	0,...,$\infty$	1
dufunc	int	1,..., 4	1
initalpha	float	(0, $\infty$)	1e-8
margin	float	[0, $\infty$)	0.0
modelmax	float	(modelmin, 1.0]	1.0
modelmin	float	[0.0, modelmax)	0.0

Common keywords and arguments:

• cgeps float: If the relative difference of the deviance is below cgeps, iterations are terminated. Decrease cgeps for a tighter fit. Increase cgeps to save time or if numerical issues appear.

• rrlambda float: This is the ridge regression parameter $\lambda$. When using ridge regression, the large-coefficient penalty $\lambda \beta^T \beta$ is subtracted from likelihood. This is equivalent to adding $2 \lambda \beta^T \beta$ to the logistic regression deviance. This is not really ridge regression, because this isn't a linear regression problem with a sum-of-squared error.

Rare keywords and arguments:

• binitmean: If this keyword is used, the model offset parameter $\beta_0$ is initialized to the mean of the output values $y_i$. [26] reports that this eliminated some numerical problems in his implementation. We have not observed significant changes in our implementation when using this technique.

• cgdecay float: If the deviance ever exceeds cgdecay times the best deviance previously found, conjugate gradient iterations are terminated. The $\beta$ corresponding to the best observed deviance among the iterations is used as the solution for the fold.

• cgmax int: This is the upper bound on the number of conjugate gradient iterations allowed. The $\beta$ corresponding to the best observed deviance among the iterations is used as the solution for the fold.

• cgwindow int: If the previous cgwindow conjugate gradient iterations have not produced a $\beta$ with smaller deviance than the best deviance previously found, then conjugate gradient iterations are terminated. The $\beta$ corresponding to the best observed deviance among the iterations is used as the solution for the fold.

• dufunc int: The ``direction update'' function chooses the next linesearch direction in conjugate gradient iterations. There are four options represented by the integers one through four.

  1	Polak-Ribiere (non-neg)	max$\left( 0.0, \frac{r_i^T z_i - r_i^T z_{i-1}} {r_{i-1} z_{i-1}} \right)$
  2	Polak-Ribiere	$\frac{r_i^T z_i - r_i^T z_{i-1}} {r_{i-1} z_{i-1}}$
  3	Hestenes-Stiefel	$\frac{r_i^T z_i - r_i^T z_{i-1}} {r_i^T p_{i-1} - r_{i-1}p_{i-1}}$
  4	Fletcher-Reeves	$\frac{r_i^T z_i}{r_{i-1} z_{i-1}}$

  The symbol $r_i$ represents the opposite of the gradient during the $i^{\mbox{th}}$ iteration, $z_i$ is related to the preconditioning matrix and is the same as $r_i$ at this time, and $p_{i-1}$ is the previous search direction. The default direction update function is Polak-Ribiere (non-neg). All four direction updates produce the same result for quadratic objective functions. However, the logistic regression likelihood function is not quadratic in it's parameters.

• initalpha float: This parameter is the initial line search step length for each conjugate gradient iteration. It is unlikely to affect the quality of results so long as it is sufficiently small. However, it may efficiency.

• margin float: The dataset outputs, which must be zero or one (c.f. Section ), are ``shrunk'' by this amount. That is, 1 is changed to 1 - margin and 0 is changed to margin. It is recommended that this parameter be left at its default value.

• modelmax float: This sets an upper bound on $\mu_i$ computed by the logistic regression. Although the computation of $\mu_i$ should assure it is positive, round-off error can create problems. It is recommended that this parameter be left at its default value.

• modelmin float: This sets a lower bound on $\mu_i$ computed by the logistic regression. Although the computation of $\mu_i$ should assure it is strictly less than one, round-off error can create problems. It is recommended that this parameter be left at its default value.