Logistic Regression and Information Theory: Part 1 - Do log odds have any intuitive meaning?

The goal of this series of posts is to show that practically all the features of logistic regression can be explained in terms of information theory.

Let’s start with definitions. Binary logistic regression model with dichotomic outcome Y is defined by equation

ln[P(Y = 1)/P(Y = 0) = ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k                 (1)

where p is the predicted probability of EVENT (Y = 1) and 1 – p is the probability of NONEVENT (Y = 0)) , X = ( X₁, X₂, … X_k) is a vector of predictors or explanatory variables, b = ( b₀ , b₁, b₂, … b_k) are the corresponding coefficients. For comparison we define linear probability model as

p = a₀ + a₁X₁ + a₂X₂ + … + a_kX_k                                                                   (2)

Note we use different notations for the coefficients in (1) and (2). The coefficients (a_i) are estimated by using Ordinary Least Squares (OLS) method, the coefficients (b_i) - by using Maximum Likelihood (ML) approach.

As said in Paul Allison (2012), p. 26:

“For the linear probability model, a coefficient of 0.25 tells you that the predicted probability p of the event increases by 0.25 for every 1-unit increase in the explanatory variable. By contrast, a logit coefficient of 0.25 tells you that the log odds increases by 0.25 for every 1-unit increase in the explanatory variable. But who knows what a 0.25 increase in log odds means?”

Paul von Hippel in his post of July 5, 2015 supports this statement in even stronger terms:

“In the linear model, if a₁ is (say) .05, that means that a one-unit increase in X₁ is associated with a 5 percentage point increase in the probability that Y is one. Just about everyone has some understanding of what it would mean to increase by 5 percentage points their probability of, say, voting, or dying, or becoming obese. The logistic model is less interpretable. In the logistic model, if b₁ is .05, that means that a one-unit increase in X₁ is associated with a .05 increase in the log odds that Y is 1. And what does that mean? I’ve never met anyone with any intuition for log odds” (in both citations bold print is mine).

Some researches question interpretability and intuition not only of log odds but even odds themselves (see discussions on the subject in Allison (2015) and (2017), and von Hippel (2015). Hosmer and Lemeshow (1989), p. 42 moderate discussions by saying “The interpretation given for the odds ratio is based on the fact that in many instances it approximates a quantity called the relative risk.”

And nevertheless, log odds or logits do have quite a simple and intuitive interpretation on their own. It is known (see, for example, Rissanen (1989) p. 38) that for each random event E there are necessarily two numbers, the event’s probability p = P(E) and its information I(E), i.e. the information contained in the message that E occurred. P(E) and I(E) are related to each other by with equation

I(E) = - lnP(E)                                                                                                  (3)

where ln is natural logarithm, so information is measured in nats, not in bits (1 nat ≈ 1.44 bits). It may be argued that using I(E) instead of P(E) provides a more natural scale, especially in case of rare events. For example, three values of P(E) = 0.02, 0.01, and 0.001 are almost equidistant, but the corresponding values of I(E) are 3.91, 4.61; 6.91 (all in nats) which are more realistic. We see that rarer events provide more information when observed. Note that (3) can be also associated with the minimal word length to convey the information content. So information is as fundamental a concept as probability. Now, let y₁, y₂,..., y_i,... y_n be independent observations of variable Y; n - the total sample size; p = P(E) = P(Y = 1) and 1 – p = P(NE) = P(Y = 0). In these notations, we have the following chain of equalities:

ln[p/(1-p)]=ln[p(E)/(1-p(E)]=ln[p(E)] - ln[(1-p(E)]= I(NE) - I(E)                 (4)

Thus, the logit is nothing but the discriminative information difference between event E and nonevent NE - (DID(E, NE)). From equations (3) and (4) we see that

(a) the less probable event E is, the more information it contains, i.e. more amount of nats is needed to store it, and vice versa;
(b) as the value of p goes from 0 to 1, the corresponding value of DID (log odds) goes symmetrically from minus infinity to plus infinity;
(c) if p = 0.5, then DID(E, NE) = 0, i.e. E and NE cannot be discriminated in terms of information content;
(d) in general the DID (log odds) value for probability p is minus the DID (log odds) value for probability 1-p;

If anybody is uncomfortable with possible negative values of DID (if p < 0.5), then the absolute value of DID can be used. Though in this case the information about sign of DID will be lost.

DID(E,NE) can serve as an information measure of the ability to discriminate between E and NE. In this regard, it is interesting to compare logistic regression with linear discriminant analysis. These techniques are defined by the same mathematical equations, but they work under different assumptions. According to Hosmer and Lemeshow (1989), pp 34-36, and Menard (2010), pp 319-320 logistic regression is more preferable when the basic assumption of LDA – multivariate normality of predictors is not satisfied. And yet in practice, logistic regression and LDA often give similar results. See, for example, Hastie, Tibshirani, and Friedman (2013), pp. 121-122:

“It is our experience that the models give very similar results, even when LDA is used inappropriately, such as with qualitative predictors…We have seen that linear discriminant analysis and logistic regression both estimate linear decision boundaries in similar but slightly different ways”

After estimating coefficients (b₀ , b₁, … b_k) in (1) by maximum likelihood we have to take decision whether our current case is EVENT (Y = 1) or NONEVENT (Y = 0). For this purpose, we can choose some decision cut-off either in terms of predicted probability or in terms of log odds (DID). Actually, in a logistic model, units of log odds or DID, (i.e. nats), are converted back to units of probability by formula

p = exp(ln[p/(1-p)] /(1+exp(ln[p/(1-p)]) =

1 / (1 + exp(- ln[p/(1-p)])) = 1 / (1 + exp(-( b₀ + b₁X₁ + b₂X₂ + … b_kX_k)                (5)

where exp is the exponential function. Though the predicted probabilities may be used in decision, it seems more convenient and natural to do it in terms of log odds, i.e. information. According to Allison (2017), in many applications researches are not really interested in predicted probabilities. And then Allison in this article added emphatically “… there’s nothing sacred about probabilities. An odds is just as legitimate a measure of the chance that an event will occur as a probability. And with a little training and experience, I believe that most people can get comfortable with odds”. We can extend the statement above by saying: there is nothing sacred about both probability and odds. Log odds i.e., discriminative information difference (DID) is as legitimate as a probability and odds. Moreover, in terms of logit (and not probability) finding a decision threshold becomes a linear problem – to choose a cut-off, C, such that

Declare EVENT if ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k > C                         (6)

Declare NONEVENT if ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k <= C

Of course, a cut-off point depends on the relative consequences of a false positive and a false negative.

Thus, in this post we have showed that logit is easily interpreted in terms of information as discriminative information difference (DID). Since there exists a linear relationship between logit and coefficients (b_i) it is natural to expect a similar interpretation for coefficients (b_i). According to Hosmer and Lemeshow (1989), p.39: “Proper interpretation of the coefficient in a logistic regression model depends on being able to place meaning on the difference between two logits”. It can be easily showed when we have a sole independent variable of a continues or a binary type. First, consider the case of a sole binary predictor, say, X, with values ‘1’ for ‘female’ and ‘0’ for ‘male. Then, b can be interpreted as the difference in ability to discriminate (DID) between ‘female’ and ‘male’. In the same way, with a sole continues predictor, say, age in years, the coefficient b can be interpreted as a change in DID per year of age. In general case of multiple logistic regression this interpretation is not as straightforward, and the estimated coefficients must be interpreted with care.

In our next blog posts, it will be showed that the main estimation method for logistic regression - Maximum Likelihood (ML) - has a simple and natural interpretation in terms of information theory. As a result, R-Squared measures based on the log likelihood (see our post Shtatland (2018)) acquire an information interpretation. And so do some goodness-of-fit tests (for example, the deviance test or the likelihood ratio test).

References

Allison, P.D. (2012) “Logistic Regression Using SAS: Theory and Application”. Cary, NC: SAS Institute.

Allison, P.D. (2014) “Measures of fit for logistic regression”. https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf

Allison, P.D. (2017) “In Defense of Logit – Part 1”. https://statisticalhorizons.com/in-defense-of-logit-part-1

Hastie, T. ,Tibshirani, R. and Friedman, J. (2013) “The Elements of Statistical Learning: Data Mining, Inference, and Prediction”. Springer

Hosmer, D.W. and Lemeshow, S. (1989) “Applied Logistic Regression”. New York: Wiley.

Shtatland, E.S. (2018) “Do we really need more than one R-Squared in logistic regression?” http://statisticalmiscellany.blogspot.com/

Von Hippel, P. (2015) Linear vs. Logistic Probability Models: Which is Better, and When? http://statisticalhorizons.com/linear-vs-logistic

Do we really need more than one R-Squared in logistic regression?

In the previous post we advocated McFadden R-Square as our preferred choice for binary logistic regression, in other words we joined the so-called “McFadden Camp” (see Paul Allison’s post “What’s the best R-Squared for logistic regression” (2013)). But not all researchers think this way. And as it is said in DeMaris (1992), p. 56: “… It may not be prudent to rely on only one measure for assessing predictive efficacy - particularly in view of the lack of consensus on which measure is most appropriate. Perhaps the best strategy is to report more than one measure for any given analysis. If the model has predictive power, this should be reflective in some degree by all of the measures discussed above”.

Many different R-Squared statistics have been proposed in the past four and a half decades (see, e.g., review publications: Windmeijer (1995), Cameron and Windmeijer (1997), Mittlbock and Schemper (1996), Menard (2000), (2010), Smith and McKenna (2013), Walker and Smith (2016)). These statistics can be divided into three main categories:

(1) R-Squared measures based on likelihoods;

(2) R-Squared measures based on the sum of squares;

(3) Measures based on squared correlations of observed output and estimated probabilities. 

Standing alone is the so-called Tjur R-Squared.

R-Squared measures based on model likelihoods

 

This category is most populated. It includes familiar to us from the previous post R^2(MF) (McFadden R^2) and R^2(CS) (Cox - Snell R^2). Here and further we use notations partially from our previous post, Menard (2000) and Smith – McKenna (2013)):

(1)  R^2(MF) = [lnL(0) – lnL(M)] / [lnL(S)] – lnL(0)] = 

      1 – lnL(M) / lnL(0)

(2)  R^2(CS) = 1 – [L(0) / L(M)]^(2/n)

(3)  R^2(N) = R^2(CS) /max R^2(CS) = R^2(CS) / (1 – L(0)^(2/n))

Also included are:

(4)  Aldrich – Nelson R-Squared, defined by          

R^2(AN) = G(M) / (G(M) + n) = 

2(lnL(M) - lnL(0)) /[2(lnL(M) - lnL(0)) + n]          

(see Smith – McKenna (2013))

(5)  Veall – Zimmermann R-Squared, given by formula 
R^2(VZ) = 

2(lnL(M) - lnL(0)) /[ 2(lnL(M) - lnL(0)) + n]*[(-2lnL(0) + n) / -2lnL(0)] =  R^2(AN) / max R^2(AN) 

(see Walker and Smith (2016)). R^2(VZ) was introduced as a correction to R^2(AN) which cannot reach the value of one. It’s easy to see that max R^2(AN) is equal to 2ln2/(2ln2 + 1) ≈0.581 (compare to max R^2(CS) = 0.75)

(6) Estrella R-Squared, defined by formula 

R^2(Est) = 1 – [lnL(M) / lnL(0)]^ (-2lnL(0)/n) 
(see Walker and Smith (2016)) 

We don’t consider here corrections for the number of predictors of the R-Squared measures mentioned above. From this abundance of R-Squared indexes we choose only two: R^2(MF) and R^2(CS). As it is shown in Shtatland et al. (2000) and (2002), there exists an important and a very simple functional relationship between R^2(MF) and R^2(CS) (see also our previous post, Menard (2010), p. 50), and our comment to Allison (2013):

R^2(CS) = 1 – exp(-R^2(MF)*T)                          (1)                          

where the quantity T is equal to -2lnL(0) / n. Since the maximized log-likelihood for the null model can be written as

lnL(0) = n[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]

the formula for T can be rewritten as follows

T = -2[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]                                       (2)

Thus, T is nothing but doubled entropy of Bernoulli distribution with probability ȳ (the base rate).

Desirable properties of an R-squared include interpretation in terms of the information content of the data (see Cameron and Windmeijer (1997)). It is shown in Shtatland et al. (2000) and (2002) (see also our previous post) that R^2(MF) has an intuitively reasonable, immediate interpretation in terms of information theory: it can be considered as ratio of information gain (IG) when moving from the null model with intercept only to the current one by adding available predictors (2(lnL(M) - lnL(0)) to the quantity of all potentially recoverable information (-2lnL(0)). Note that R^2(CS) also allows some information interpretation, though not as direct as for R^2(MF). Indeed, let us transform the equation

R^2(CS) = 1 – [L(0) / L(M)]^(2/n)

which defines R^2(CS) to

-ln (1 - R^2(CS)) = 2(lnL(M) – lnL(0))/n = IG /n = IGR

Here information gain rate, IGR, is nothing but information gain per observation. The logarithmic function -ln(1 – x) is a well-known measure from information theory (Theil and Chung (1988)). Using -ln(1 - R^2(CS)) rather than R^2(CS) provides a more natural scale for interpretation. Taking the first derivative of both sides of this equation with respect to R^2(CS), we have

dIGR/dR^2(CS) = 1 / (1 - R^2(CS))

From this formula we arrive at the following important conclusions:

If R^2(CS) = 0, then dIGR/dR^2(CS) = 1, i. e. the increase in R-Square is equivalent to the increase in IGR;

If R^2(CS) = 0.5, then dIGR/dR^2(CS) = 2, i. e. the increase in IGR is twice as large as the increase in R-Square;

If R^2(CS) = 0.75 (maximal value possible), then dIGR/dR^2(CS) = 4, i. e. the increase in IGR is four times larger than the increase in R-Square.

Note that equations (1) and (2)) can be also found in Sharma and McGee (2008), though without informational - entropic interpretation, which is one of the most desirable properties of an R-squared (see, for example, Cameron and Windmeijer (1997)).

Concluding our comparison R^2(CS) and R^2(MF), note that these measures are related to the base rate ȳ quite differently. According to Menard (2000), R^2(MF) stands out for its relative independence from the base rate, comparative to R^2(CS) and other R-Squared measures. At the same time, R^2(CS) is highly correlated with ŷ (see Menard, 2000, p. 23). From formulas (1), (2), it can be seen that R^2(CS) depends on the base rate ȳ basically through the quantity T. As a result, R^2(CS) demonstrates a rather anti-intuitive and odd trait: it increases as the base rate (more exactly, ȳ or 1 – ȳ, whichever is smaller,) increases from 0 to 0.5, absurdly implying that ȳ itself can be used as some R^2 measure.

It is easy to see that each candidate for R^2 in logistic regression expressed as a function of statistic [lnL(M) – lnL(0)] can be also rewritten through R^2(MF) and T. For example, R^2(AN), R^2(VZ) and R^2(EST) can be shown as:

R^2(AN) = R^2(MF) / (R^2(MF)+1/T)                            (3)

R^2(VZ) = R^2(MF) * (1 + T) / (1 + T* R^2(MF))         (4)

R^2(Est) = 1 – exp[T*ln(1 - R^2(MF))]                            (5)

Thus, it can be concluded from the formulas above that R^2(MF) plays a central role in the category of R-Squared statistics based on the model likelihoods - if we know R^2(MF) and T, then all other members of this category can be calculated by hand.

Furthermore, as it is shown above, R^2(MF) (and partly R^2(CS)) has a clear and direct interpretation in terms of the information content of the data, potentially recoverable information, information gain due to predictors adding, etc. (see Shtatland (2002), Menard (2010) p .48, and our previous post). This interpretation is no less natural and important than the one used in the OLS approach. At the same time R^2(AN), R^2(VZ) and R^2(EST) have lost this interpretation and look as ad hoc measures.  Summarizing we can conclude that R^2(MF) is obviously more preferable measure than any other R-Squared in this category.

R-Squared measures based on the sum of squares – OLS analogs

We consider here only the most popular measures. First one, based on sums of squared residuals, is used under various names: R^2(O), R^(SS), R^(Efron), R^2(res), etc., and is given by familiar to us from our previous post formula:

R^2(O) = 1 - ∑(yi –  ŷi) ^2 / ∑(yi – ȳ ) ^2 󠄝                         (6)

Second one, the so-called model R-Squared, R^2(mod) defined by formula:

R^2(mod) = ∑( ŷi – ȳ) ^2 /∑(yi – ȳ ) ^2                             (7)

Let us remind that yi is the observed outcome (1 or 0), ŷi is the predicted value of yi (actually, the predicted probability), and ȳ is the arithmetic mean of yi. R^2(O) looks exactly like the ordinary least square R^2(OLS) for linear regression except that ŷi are calculated by maximizing likelihood function and not minimizing sum of squares. It is frequently used as a standard for comparison between other R-Squared measures. For some researchers (see, for example, Mittlbock and Schemper (1996)), R^2(O) is a measure of choice. Also, it is concluded in Menard (2010), p. 56 that there are certain benefits in using R^2(O) not instead R^2(MF), but as a supplemental measure. Though, R^2(O) has two serious disadvantages: it does not automatically increase when the model is extended by an additional predictor, and can be negative in some rare, rather degenerate cases.
 

Measures based on squared correlations

There are plenty of them. In Mittlebock and Schemer (1996) six measures are discussed. We will mention only the most popular measure in this category – based on squared Pearson correlation:

R^2(cor) = 

{[∑(yi – ȳ )( ŷi – ȳ) / [∑(yi – ȳ )^2]^0.5] [∑(ŷi – ȳ )^2]^0.5]}^2

It is known that R^2(cor) is very close to R^2(O), moreover they are almost identical numerically (Liao and McGee (2003). Therefore, it is reasonable to use only one of them, and we prefer R^2(O).

Tjur R-Squared

Finally, we consider Tjur R-Squared (R^2(Tjur). Since it is not so easy to define R^2(Tjur) by formula, we refer for a verbal definition to Tjur (2009) and Allison (2013), (2014). Actually, this measure was introduced earlier in Cramer (1999), so it would be fair to use the double name: Cramer - Tjur. According to Allison (2013), the measure has a lot of intuitive appeal, it is easy to calculate, and also is closely related to R-Squared measures based on sum of squares:

R^2(Tjur) = (R^2(O) + R^2(mod)) /2                                (8)

R^2(Tjur) = (R^2(mod) * R^2(cor))^0.5                           (9)

Following Allison (2013), we propose to use R^2(Tjur) as an attractive alternative and as a good supplement to R^2(MF) (the best R-Squared based on likelihoods) and R^2(O) (our preferred choice in the category of OLS analogs).

Conclusion

Let us return to DeMaris quotation: “… It may not be prudent to rely on only one measure for assessing predictive efficacy - particularly in view of the lack of consensus on which measure is most appropriate. Perhaps the best strategy is to report more than one measure for any given analysis”. Following this advice, we propose to report three measures: R^2(MF), R^2(0) and T^2(Tjur) altogether, hoping that they make up a good ”three pillars of R-Squared measures” and a good pathway from “the jungle of R-Squareds” for binary logistic regression.
        

References:

Allison, P. D. “What’s the best R-squared for logistic regression?” (2013) (http://www.statisticalhorizons.com/r2logistic).

Allison, P. D. “Measures of fit for logistic regression” 2014) https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf.

Cameron, C. A. and Windmeijer, F. A. G. (1997) "An R-squared measure of goodness of fit for some common nonlinear regression models", Journal of Econometrics, Vol. 77, No.2, pp. 329-342.

Cox, D. R. and Snell E. J. (1989) “Analysis of binary data” Second Edition, Chapman & Hall, London.

Cramer, J. S. (1999) “Predictive performance of the binary logit model in unbalanced samples”, The Statistician, 48, 85 – 94.

DeMaris, A. (1992) “Logit modeling” Sage University Paper Series.

Efron, B. (1978), “Regression and ANOVA with zero-one data: Measures of residual variation”, Journal of the American Statistical Association, 73, 113 – 121.

Hosmer, D. W. and Lemeshow, S. (1989), “Applied logistic regression”, New York: Wiley.

Kent, J. T. (1983). “Information gain and a general measure of correlation”,  Biometrika, 70, 163 – 173.

Menard, S. (2000) “Coefficients of determination for multiple logistic regression analysis”, The American Statistician, 54, 17 – 24.

Menard, S. (2002), “Applied logistic regression analysis”, Sage University Paper Series (Second edition).

Menard, S. (2010), “Logistic regression: From introductory to advanced concepts and applications”, Sage University Paper Series, Chapter 3, pp. 48 – 62.

McFadden, D. (1974), “Conditional logit analysis of qualitative choice behavior”, pp. 105 -142 in Zarembka (ed.), Frontiers in Econometrics. Academic Press.

Mittlebock, M. and Schemper, M. (1996), “Explained variation in logistic regression”, Statistics in Medicine, 15, 1987 – 1997.

Sharma D. and McGee, D. (2008), “Estimating proportion of explained variation for an underlying linear model using logistic regression analysis”, Journal of Statistical Research,42, 59 – 69.

Shtatland, E. S., Moore, S. and Barton, M. B. (2000) “Why we need R^2 measure of fit (and not only one) in PROC LOGISTIC and PROC GENMOD”, SUGI 2000 Proceedings, Paper 256 - 25, Cary, SAS Institute Inc.

Shtatland, E. S., Kleinman, K. and Cain, E. M. (2002) “One more time about R^2 measures of fit in logistic regression”, NESUG 2002 Proceedings.

Smith, T. J. and McKenna, C. M. (2013) “A comparison of logistic regression pseudo R^2 indices”, Multiple Linear Regression Viewpoints, 39, 17 - 26. 

Theil, H. and Chung, C.F. (1988) “Information-theoretic measures of fit for univariate and multivariate linear regressions”, The American Statistician, 42, 249 - 252.

Tjur, T. (2009) “Coefficients of determination in logistic regression models – a new proposal: the coefficient of discrimination”, The American Statistician, 63, 366 – 372.

Walker, D. A. and Smith, T. J. (2016) “Nine pseudo R^2 indices for binary logistic regression models”, Journal of Modern Applied Statistical Methods, 15, 848 – 854.

Windmeijer, F. A. G. (1995) “Goodness of fit measures in binary choice models”, Econometric Reviews, 14, 101 – 116.

And yet, once again: What’s the best R-squared for logistic regression?

This title is a paraphrase of the titles of the two publications: “What’s the best R-squared for logistic regression?” by Allison (2013) in http://www.statisticalhorizons.com/r2logistic and “One more time about R-Squared measures of fit in logistic regression” by Shtatland et al. (2002) in http://www.lexjansen.com/nesug/nesug02/st/st004.pdf.  

A comprehensive and almost exhaustive review of this topic  can be found in Menard (2010), Chapter 3, pp. 48 – 62, and we strongly recommend this publication.

For simplicity, we will discuss only the case of binary logistic regression with single-trial syntax, though a number of our results can be generalized to the case of multinomial logistic regression. Following Menard (2000), we introduce now minimal necessary notations:    

Y is the dependent variable taking values coded “1” in the case of EVENT or “0” in the case of NOT-EVENT;

n is the total sample size, i. e. the number of observations of dependent variable y₁, y₂,..., y_i,... y_n;

L(0) is the maximized likelihood of the model containing the intercept only, so called “Null model”;                                                    

L(M) is the maximized likelihood of the current model M containing all predictors available;

L(S) is the maximized likelihood of the saturated model that contains as many predictors as observations;

ŷ_i is the predicted value of y_i obtained from the model; actually ŷ_i is a continuous probability with a value between 0 and 1, unlike y_i which takes values either “1” or “0”;

ȳ is the arithmetic mean of the dependent variables, also a continuous probability, ȳ is known also as base rate or prevalence or marginal proportion.

There are many candidates for R-Squared measure in logistic regression (see Menard (2010) for a review). We are interested, first and foremost, only in two of them because they are more popular among researchers and most often frequently reported in statistical software. They are R²_CS and  R²_MF, which we define and discuss later. Nevertheless, we will start with another R-Squared, determined in our notations above as follows:

R²_O = 1 - ∑(y – ŷ)² / ∑(y – ȳ)²                                (1)

since R²_O is frequently used as a standard for comparison between other R-Squared measures in logistic regression. Note that R²_O looks like ordinary least squares R²_OLS in linear regression except that ŷ_i are calculated by maximizing likelihood function in logistic regression and not minimizing sum of squares.

Now, let us introduce R²_CS and R²_MF, using notations above:

R²_CS = 1 – exp{2[lnL(M) – lnL(0)] / n}                  (2)

R²_MF = 1 - lnL(M) / lnL(0 )                                     (3)

R²_CS is usually attributed to Cox and Snell (1989), which explains notation “CS”. R²_MF is usually referred to McFadden (1974) – that is why notation “MF”.

Note that R²_MF is a particular case of more general, entropy-based R-Squared (R²_Ent) given by formula

R²_Ent = [lnL(M) – lnL(0)] / [lnL(S) – lnL(0)]          (4)

It is known that for binary logistic regression with single-trial syntax lnL(S) = 0, and R²_Ent is reduced to R²_MF. Note that R-Squared (4) can also be interpreted as deviance-based measure R²_DEV (see Hosmer and Lemeshow (1989, pp. 147 – 148) and Cameron and Windmeijer (1997)). Thus, in our case we have R²_MF = R²_Ent = R²_DEV.
Desirable properties of R-Squared measures mentioned above include interpretation in terms of the information gain (IG) when using the model with available predictors in comparison with the null model (see Kent (1983), Cameron and Windmeijer (1997), and Shtatland et al. (2000)).          

As far as we know, all publications on R-Squared measures in logistic regression that refer to R²_CS and R²_MF consider them as independent measures. But this is not the case. There exists a very interesting and very important functional relationship between them. It was shown in our SUGI (2000) and NESUG (2002) presentations that

R²_CS = 1 – exp(-R²_MF * T)                                      (5)

where

T = - 2lnL(0) / n                                                        (6)

Since the maximized log-likelihood for the null model lnL(0) can be written as

lnL(0) = n[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]

formula (6) can be rewritten as follows

T = -2[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]                              (7)

It means that quantity T is nothing but doubled entropy of Bernoulli distribution with probability ȳ. Note that in our previous publications we have used another notation instead of R²_CS, namely R²_SAS. It’s because in SAS at that time, there existed only this R-Squared measure (not to mention the unfortunate so-called Nagelkerke adjustment). Entropy properties predetermine properties of our key quantity T. They are discussed below.

T is defined in the interval [0, 1], and is a symmetrical function of ȳ with respect to ȳ = 1 /2. It is equal to 0 at the ends ȳ = 0 and ȳ = 1, increases from 0 to 2ln2 on [0, 1 /2], attains its maximum 2ln2, and then decreases to 0. In Shtatland et al. (2002), it was shown that if 1 < T <= 2ln2, which corresponds to 0.2 < ȳ < 0.8, then R²_CS is slightly greater than R²_MF. Otherwise, R²_CS is smaller than R² _MF and very substantially when ŷ is close to either 0 or 1. It is interesting that exactly in this interval, linear probability model presents a good approximation to logistic regression (see Allison, 2017). The properties of T, R²_MF and R²_CS mentioned above provide us with theoretical justification of empirical results on the comparison between R²_CS and R²_MF when ȳ takes values around 0 (or 1) vs. when ȳ is close to 0.5 (see Allison (2013) and numerous examples in Menard (2000), Mittlbock and Schemper (1996) and Shtatland et al. (2002). In addition, formula (5) explains directly why the maximal value of R²_CS is equal to 0.75 and not to 1 like R²_MF.

Entropy and information are related so much that they are sometimes called “two sides of the same coin”. That is why the “entropy loss” (see Efron, 1978) is nothing more than the equivalent “information gain” (see Kent 1983, Shtatland et al, 2002).

Note also that if ȳ = 0.5, then we are in the state of maximal uncertainty or maximal possible entropy. Adding some relevant predictors can only increase information, or decrease entropy. Thus, the notions of entropy and information are fundamental for understanding the model building process. Any step-by-step construction of better and better models is always accompanied by an entropy decrease / information gain process.

There exists a physical (thermo-dynamical) analogy of this process. The system in thermo-dynamical equilibrium is characterized by maximum entropy. To move the system from equilibrium, it is necessary to spend some energy, i.e. to do some work. In our context, it means that some additional predictors, i.e. explanatory variables, have to be introduced into the model.

Summarizing, we can conclude that R²_MF is certainly much more preferable measure than R²_CS at least in two respects. First of all, R²_MF has a desirable intuitively reasonable, immediate interpretation in terms of information-gain or entropy loss. This interpretation for R²_CS is not as direct as for R²_MF. Second, according to Menard (2000) p. 24, R²_MF stands out for its relative independence from the base rate, comparative to other R-Squared measures. At the same time, R²_CS is highly correlated with ŷ (see Menard, 2000, p. 23). From formulas (5), (7), it can be seen that R²_CS depends on the base rate ȳ basically through the quantity T. As a result, R²_CS demonstrates a rather anti-intuitive and odd trait: it increases as the base rate (more exactly, ȳ or 1 – ȳ , whichever is smaller,) increases from 0 to 0.5. That is why we recommend to use R²_MF and not R²_CS. All of this perfectly well explains the fact that more and more researchers have joined the “McFadden camp” since 2000. Of course, it does not prevent them from using R²_O as a supplemental measure. As to a new, so-called Tjur R², we will consider this measure elsewhere in the future.

As some researchers are still in doubts about using R-Squared measures or not, we would like to cite Menard (2010 p. 52): “If you want R², why not use R²?”

References:

Allison, P. D. “What’s the best R-squared for logistic regression?” (2013) (http://www.statisticalhorizons.com/r2logistic).

Allison, P. D. “Measures of fit for logistic regression” (2014) https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf.

Allison, P. D. “In Defense of Logit – Part 1” (2017) https://statisticalhorizons.com/in-defense-of-logit-part-1.

Cameron, C. A. and Windmeijer, F. A. G. (1997) "An R-squared measure of goodness of fit for some common nonlinear regression models", Journal of Econometrics, Vol. 77, No.2, pp. 329-342.

Cox, D. R. and Snell E. J. (1989) “Analysis of binary data” Second Edition, Chapman & Hall, London.

Efron, B. (1978), “Regression and ANOVA with zero-one data: Measures of residual variation”, Journal of the American Statistical Association, 73, 113 – 121.

Hosmer, D. W. and Lemeshow, S. (1989), “Applied logistic regression”, New York: Wiley.

Kent, J. T. (1983). “Information gain and a general measure of correlation”,  Biometrika, 70, 163 – 173.

Menard, S. (2000) “Coefficients of determination for multiple logistic regression analysis”, The American Statistician, 54, 17 – 24.

Menard, S. (2002), “Applied logistic regression analysis”, Sage University Paper Series (Second edition).

Menard, S. (2010), “Logistic regression: From introductory to advanced concepts and applications”, Sage University Paper Series, Chapter 3, pp. 48 – 62.

McFadden, D. (1974), “Conditional logit analysis of qualitative choice behavior”, pp. 105 -142 in Zarembka (ed.), Frontiers in Econometrics. Academic Press.

Mittlebock, M. and Schemper, M. (1996), “Explained variation in logistic regression”, Statistics in Medicine, 15, 1987 – 1997.

Shtatland, E. S., Moore, S. and Barton, M. B. (2000) “Why we need R^2 measure of fit (and not only one) in PROC LOGISTIC and PROC GENMOD”, SUGI 2000 Proceedings, Paper 256 - 25, Cary, SAS Institute Inc.

Shtatland, E. S., Kleinman, K. and Cain, E. M. (2002) “One more time about R^2 measures of fit in logistic regression”, NESUG 2002 Proceedings.