Logistic Regression and Information Theory: Part 4 -

How Good Is Our Model?

After estimating logistic regression coefficients – usually by using maximum likelihood approach – we inevitably come to the question: How good is our model? According to Allison (2014):

“There are two very different approaches to answering this question. One is to get a statistic that measures how well you can predict the dependent variable based on the independent variables. I’ll refer to these kinds of statistics as measures of predictive power . . . Predictive power statistics available in PROC LOGISTIC include R-square, the area under the ROC curve, and several rank-order correlations . . . The other approach to evaluating model fit is to compute a goodness-of-fit statistic. With PROC LOGISTIC, you can get the deviance, the Pearson chi-square, or the Hosmer-Lemeshow tests. These are formal tests of the null hypothesis that the fitted model is correct . . . What many researchers fail to realize is that measures of predictive power and goodness-of-fit statistics are testing very different things. It’s not at all uncommon for models with very high R-squares to produce unacceptable goodness-of-fit statistics. And conversely, models with very low R-squares, can fit the data very well according to goodness-of-fit tests.”

Thus, to answer the question "How good is our model?", we have to understand clearly - for what purpose. It is known that any regression model (including logistic regression) has two primary goals:

(1) to predict dependent variable (aka response or output variable) based on corresponding independent variables (aka predictors), this is a practical goal, which can be achieved by statistician or statistical programmer;
(2) to explain the relationship between response variable and independent or explanatory variables, this is rather a theoretical goal, requiring subject-matter knowledge.

In this blogpost, we limit ourselves to the measures of predictive power mentioned above, more specifically to R² based on likelihood.

Basic notations

Given a sample of observations or subjects (y_i, X_i), i=1,…, n, let:

(1) y_i denote the dependent class variable with values ‘1’ (EVENT) or ‘0’ (NONEVENT) where EVENT can be meant as Disease, Buy, Vote, Out-of-Order, etc. and correspondingly NONEVENT – No-Disease, No-Buy. No-Vote, In-Order, etc.;

(2) X_i denotes a vector of independent variables (x₁, x₂,…, x_k), or covariates, or predictors, which can be understood as the attributes of i-th subject, such as age, education, test results and so on, depending on context;

(3) Class variable y_i and vector of attributes X_i are related to each other by equation

Ln[(P(y_i = 1) / (1 – P (y_i = 1))] = b₀ + b₁x₁ + … + b_kx_k       (1)

or equivalently

P (y_i = 1) = exp(b₀ + b₁x₁ + … b_kx_k) / (1 + exp(b₀ + b₁x₁ + … b_kx_k))       (1’)

where coefficients b₀, b₁, b₂, … b_k have to be estimated (usually by using maximum likelihood method);

(4) Estimates b^*₀, b^*₁, …, b^*_k put into (1’), provide us with an estimate for P(y_i = 1) and based on this estimate we have to predict the real value of y_i, which is still unknown at this point;

(5) The likelihood of the logistic regression model (1) is given by equation

L(y₁, y₂,..., y_i ... y_n; X₁, X₂, … X_n) = Π n_i=1 P(y_i | X_i)       (2)

Here, we have dropped the vectors of parameters b = (b₀ , b₁, b₂, … b_k) for notational brevity.

(6) The logarithm of maximized likelihood is as follows

lnL(y₁, y₂,..., y_i ... y_n; X₁, X₂, … X_n) = Σⁿ_i=1 ln[P( y_i | X_i) =       (3)

Σⁿ_i=1[y_i [ln(P(y_i | X_i)] + (1 - y_i) (1 - [ln(1 - P(y_i | X_i)

R² measures based on likelihood

There are several R-Squared measures based on likelihood (see Shtatland (2018)). The most popular are R²_MF (McFadden R²) and R²_CS (Cox - Snell R²) defined as follows

R²_MF = 1 – lnL(M) / lnL(0) = (lnL(0) – lnL(M)) / lnL(0) = (lnL(M) – lnL(0))/(-lnL(0))       (4)

R²_CS = 1 – [L(0) / L(M)](2/n)       (5)

Here, L(M) and L(0) are maximized likelihoods of the model M with all available

predictors (x₁, x₂,…, x_k), and the null model ”0” without predictors, respectively.

It is interesting that R²_CS can be expressed as a simple function of R²_MF:

R²_CS = 1 – exp(-2H* R²_MF)       (6)

where

H(ŷ)= -[ŷln ŷ + (1 – ŷ)ln(1 – ŷ)] and ŷ = (Σⁿ_i=1 y_i)/n       (7)

Note that quantity H is nothing but the entropy of Bernoulli distribution with parameter ŷ. Equations (6) and (7) were originally derived in Shtatland et al (2000) and (2002). It is easy to get similar formulas in terms of R²_MF and H for other R² measures based on likelihood (see, for example, Sharma (2006), Sharna and McGee (2008), Sharma et all (2011), and Shtatland (2018)), but they are rather bulky and difficult to interpret. Besides, those R² measures are not as popular as R²_MF and R²_CS.

R²_MF and R²_CS - information interpretation

Note also that information theoretic interpretation of R² measures in logistic regression is one of the most desirable properties on a level with such standard properties as: (1) 0 ≤ R² ≤ 1; and (2) R² is nondecreasing as predictors are added (see, for example, Cameron and Windmeijer (1997) and references herein). We are in this, “ R² information interpretation camp” since 2000: Shtatland et al (2000), Shtatland et al (2002), Shtatland (2018). It has been shown in these publications that both R²_MF and R²_CS have an explicit and direct interpretation in terms of the information content of the data, potentially recoverable information and information gain due to added predictors. This interpretation is based on the fact that negative loglikelihood can be understood as the information contained in the observed data. As a result we can interpret quantity (- lnL(0)) for the model without predictors as potentially recoverable information in data (y_i, i=1,…, n) and quantity (-ln(M)) for the model with predictors as the information content of enlarged data (y_i,X_i, i=1,…, n). Thus, it is natural to interpret quantity ln(M) – lnL(0) as information gain (IG) due to switching from the null model with intercept only to the current model with available predictors. Note that R²_MF as the ratio of information gain, IG, to all potentially recoverable information shows not only how good model M is compared to the null model, but also how bad it is compared to the saturated model.

R²_CS and R²_MF scales vs information scale

After getting information interpretation for R²_CS and R²_MF, it is interesting to compare their scales with information scale. As to R²_MF, the situation is simple. According to formula (4), which can be re-written as

R²_MF = (lnL(M) – lnL(0))/(-lnL(0)) =
= [(1/n) (lnL(M) – lnL(0))]/[- (1/n)lnL(0)] = IGR/H       (8)

where (-lnL(0)) does not depend on (X_i, i = 1, 2, … , n) and can be treated as constant; information gain rate, IGR, is information gain per observation, and at the same time is the estimate of the entropy in the available data. R²_MF scale is the same as information scale (up to this constant). That is information capacity per R²_MF unit is the same along the interval 0 ≤ R²_MF ≤ 1. As to R²_CS, the situation is more sophisticated and interesting. Indeed, let us transform the equation (5)

R²_CS = 1 – [L(0) / L(M)]^(2/n)

which defines R²_CS to

–ln (1 - R²_CS) = 2(lnL(M) – lnL(0))/n =2 IG /n = 2IGR       (9)

Note that there is an alternative interpretation of (IG/n) as n → ∞. Indeed, this limit can be interpreted as entropy loss associated with using model M with predictors instead of the null model 0 without predictors. The logarithmic function -ln(1 – x) is a well-known measure from information theory (Theil and Chung (1988)). Using -ln(1 - R²_CS) rather than R²_CS provides a more natural scale for interpretation. Taking the first derivative of both sides of equation (9) with respect to R²_CS, we have

dIGR/ dR²_CS = 1 / (1 - R²_CS)       (10)

From this formula we arrive at the following important conclusions:

If R²_CS = 0, then dIGR/dR²_CS = 1, i. e. the increase in R-Square is equivalent to the increase in IGR;

If R²_CS = 0.5, then dIGR/dR²_CS = 2, i. e. the increase in IGR is twice as large as the increase in R-Square;

If R²_CS = 0.75 (maximal value possible), then dIGR/dR²_CS = 4, i. e. the increase in IGR is four times larger than the increase in R-Square.

Asymptotic behavior of R²_MF and R²_CS

Note that all characteristics of logistic regression discussed above: likelihoods, maximized likelihoods, significance tests, and R² measures, including R²_MF and R²_CS are statistics thus random quantities. Some of these statistics depend on (y_i i=1,…, n) only, others depend on (y_i, X_i, i=1,…, n). It is very interesting and important to study their asymptotic behavior when n → ∞. The driving force in our analysis is the Law of Large Numbers (LLN). Indeed,

(1) ŷ = (Σⁿ_i=1 y_i)/n → π (unknown real population proportion)

as n → ∞ (here and below symbol → means convergence in probability);

(2) - lnL(0)/n = H(ŷ) = -[ŷln ŷ + (1 – ŷ)ln(1 – ŷ)] → -[π ln π + (1 – π)ln(1 – π)] = H(π) as n → ∞; where H(π) is entropy of Bernoulli distribution with parameter π and H(ŷ) is the estimate of H(π);

(3) It can be showed that as n → ∞, the quantity (1/n) (lnL(M) – lnL(0)) converges to some nonrandom finite limit (see Hu, Shao and Palta (2006)).

It is reasonable to interpret this limiting value of (1/n) (lnL(M) – lnL(0)) as an averaged information gain or entropy loss due to switching from the null model with intercept only to the current model with available predictors. A natural notation for this limit is

lim(1/n) (lnL(M) – lnL(0)) → H(M) – H(0)

where H(M) = - lim(1/n) lnL(M) is an averaged entropy associated with model M

and H(0) - an averaged entropy associated with the null model. As we have seen

above H(0) = H(π) = H(Y). Thus, we have

R²_MF → 1 - H(M) / H(0)       (11)

R²_CS → 1 - exp2(H(M) - H(0))       (12)

Formula (12) was derived in Hu, Shao and Palta (2006) using somewhat different approach and notations.

The quantities R²_MF and R²_CS should be treated as estimators of their limiting values in assessing the predictive model strength for large data sets.

References

Allison, P.D. (2014) “Measures of fit for logistic regression”. https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf

Cameron, C. A. and Windmeijer, F. A. G. (1997) "An R-squared measure of goodness of fit for some common nonlinear regression models", Journal of Econometrics, Vol. 77, No.2, pp. 329-342.

Cox, D. R. and Snell E. J. (1989) “Analysis of binary data” 2nd Edition, Chapman & Hall, London.

Hosmer, D.W. and Lemeshow, S. (1989) “Applied Logistic Regression”. New York: Wiley.

Hu B., Shao J. and Palta M. (2006) “Pseudo-R² in Logistic Regression Model” Statistica Sinica, 16, 847- 860.

McFadden, D. (1974), “Conditional logit analysis of qualitative choice behavior”, pp. 105 -142 in Zarembka (ed.), Frontiers in Econometrics. Academic Press.

Sharma, D.R. (2006) “Logistic Regression, Measures of Explained Variation and the Base Rate Problem”, Ph.D. Thesis, Florida State University, USA.

Sharma, D. and McGee, D. (2008) “Estimating proportion of explained variation for an underlying linear model using logistic regression analysis” J. Stat. Res., 42, No. 1, pp. 59-69.

Sharma, D., McGee, D., and Golam Kibria, B.M. (2011) “Measures of Explained Variation and the Base-Rate Problem for Logistic Regression”, American Journal of Biostatistics, 2 (1): 11-19

Shtatland, E. S., Moore, S. and Barton, M. B. (2000) “Why we need R^2 measure of fit (and not only one) in PROC LOGISTIC and PROC GENMOD”, SUGI 2000 Proceedings, Paper 256 - 25, Cary, SAS Institute Inc.

Shtatland, E. S., Kleinman, K. and Cain, E. M. (2002) “One more time about R^2 measures of fit in logistic regression”, NESUG 2002 Proceedings.

Shtatland, E.S. (2018) “Do we really need more than one R-Squared in logistic regression?” http://statisticalmiscellany.blogspot.com/.

Theil, H. and Chung, C. F. (1988) “Information-theoretic measures of fit for univariate and multivariate linear regressions”, The American Statistician, 42, No 4, 249 – 252.

Windmeijer, F.A.G. (1995), “Goodness-of-fit measures in binary choice models”, Econometric Reviews, 14, 101-116

Logistic Regression and Information Theory: Part 3 - Maximum Likelihood Is Equivalent To Minimum Cross Entropy and Kullback-Leibler Divergence

Binary logistic regression model with dichotomic outcome Y is defined by equation

ln[P(Y = 1)/P(Y = 0) = ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k       (1)

where p is the predicted probability of EVENT (Y = 1) by model (1) and 1 – p is the predicted probability of NONEVENT (Y = 0); X = ( X₁, X₂, … X_k) is a vector of predictors or explanatory variables; b = ( b₀ , b₁, b₂, … b_k) are the related coefficients. The actual or real probability π is unknown and estimated by p. It is well known (see, for example, Hosmer and Lemeshow (1989) or Menard (2010)) that maximum likelihood is a basic method for estimating the parameters b = ( b₀ , b₁, b₂, … b_k) in (1). It means that with independent observations (class labels) y₁, y₂, ..., y_i, ... y_n of output variable Y and the corresponding vectors of predictors X₁, X₂, … X_n, we have to find values of coefficients b^* = (b^*₀, b^*₁, b^*₂, … b^*_k) which maximize likelihood

L( y₁, y₂,..., y_i ... y_n; X₁, X₂, … X_n ; b₀ , b₁, b₂, … b_k) = Πⁿ_i=1 P(Y = y_i |Xi,; b₀ , b₁, b₂, … b_k) (2)

But maximizing this product by using derivatives is much more inconvenient and cumbersome than maximizing the sum. Also, if sample size n is large enough, we will operate with very small numbers and will get inevitably into an arithmetic underflow effect. That is why the logarithm of likelihood, lnL, is used:

lnL( y₁, y₂,..., y_i ... y_n; X₁, X₂, … X_n) = Σⁿ_i=1 ln[P(Y = y_i |Xi)]       (3)

Here, we have dropped the vectors of parameters b = ( b₀ , b₁, b₂, … b_k) for notational brevity, but keeping in mind that probabilities and likelihoods do depend on these variables.

Due to the relationship between the probability of event E and the information associated with this event (see formula (3) in Shtatland (2019)), the right side of (3) can be rewritten in terms of information as follows:

Σⁿ_i=1 ln[P(Y = y_i | Xi)] = - Σⁿ_i=1 I(Y = y_i | Xi)       (4)

Thus, maximizing likelihood L (or log likelihood lnL ) is equivalent to minimizing the negative log likelihood or information content of the event that we observe the sequence y₁, y₂,..., y_i ... y_n. So, in addition to technical advantages mentioned above, taking logarithm of likelihood allows us to acquire an information interpretation of negative log likelihood. Further, we will come to the same conclusion from a different angle.

In the case of binary logistic regression with y_i equals either to 1 or 0 (we consider only this case in our post) the equation for negative log likelihood can be rewritten as:

-Σⁿ_i=1 ln[P(Y = y_i |Xi,) = -Σⁿ_i=1[ y_i [ln(P(Y = y_i |Xi,)] + (1 - y_i) (1 - [ln(1 - P(Y = y_i|Xi)       (5)

(see equation (1.4) in Hosmer and Lemeshow (1989)). It is interesting that the right side of (5) can get an alternative meaning of the so-called cross-entropy, a well-known measure in machine learning (Hastie, Tibshirani, Friedman (2009)).

Generally, in information theory, the cross-entropy between two probability distributions p(ω) and q(ω) on the same underlying finite set of outputs Ω = {ω} is defined as:

H(p;q) = - Σ_ω p(ω)lnq(ω)       (6)

It is a central concept in information theory. There are two other fundamental concepts closely related to cross-entropy: entropy itself, H(p), and Kullback-Leibler divergence, D_KL (see Cover and Thomas (2006)). Entropy of a probability distribution p(ω) is defined as follows:

H(p) = - Σ_ω p(ω)lnp(ω),       (6')

i.e., entropy is just the averaged information associated with the distribution p(ω).

Kullback-Leibler divergence is defined by formula:

D_KL(pǁq) = - Σ_ω p(ω)ln[p(ω)/(q(ω)].]

It is easy to see that:

(a) H(p) ≥ 0; and H(p) = 0 if and only if p is a degenerate distribution;

(b) H(p;q) ≥ H(p) ≥ 0; and H(p;q) = H(p) if and only if distributions p and q are identical (p≡q);

(c) D_KL(pǁq) = Σ_ω p(ω)ln[p(ω) - Σ_ω p(ω)ln(q(ω) = H(p;q) - H(p) ≥ 0       (7)

and D_KL(pǁq) = 0 if and only if distributions are identical (p≡q).

Kullback-Leibler divergence is known under a variety of names, including relative entropy, Kullback–Leibler distance, information divergence, and information for discrimination. In spite of the name “Kullback–Leibler distance”, D_KL(pǁq) is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality. But it does satisfy the following inequality:

½[Σ_ω|p(ω) - (q(ω)|]2 ≤ D_KL(pǁq)       (8)

where Σ_ω|p(ω) - (q(ω)| is the so-called total variation distance. This inequality is known as Pinsker’s inequality. It is useful, particularly in proving convergence results, and giving bounds of one measure of dissimilarity in terms of another. One of the implications of inequality (8) is that convergence in relative entropy implies convergence in total variation. In applications, {p(ω)} typically represents the true distribution of data, which is unknown. Observations y₁, y₂,..., y_i ... y_n are drawn from this distribution, while {q(ω)} represents a model distribution that approximates {p(ω)}. In order to find the distribution that is closest to {p(ω)}, we have to build a model with predictive distribution q that minimizes D_KL(pǁq) or cross-entropy H(p;q), which is the same due to (7) since H(p) does not depend on the parameters of the model.

In the case of binary logistic regression, we have Ω = {1, 0}, i.e., the set of label values associated with EVENT and NONEVENT correspondingly. If the class label y_i is interpreted as the probability of being class 1, then logistic regression provides an estimate of the probability that the data point is in class 1. Probability distribution p in this case is a degenerate observed label distribution { y_i , 1 - y_i } with values (1,0) or (0,1) depending on y_i = 1 or y_i = 0 respectively. Probability distribution q is estimated from logistic regression model with q(ω =1) = P( Y_i = 1| X_i) and q(ω = 0) = P( Y_i = 0| X_i). Thus, the term − [ y_i ln(P(Y = y_i| X_i) + (1 - y_i ) (1 - ln(1 - P(Y = y_i)| X_i)] in (5) is nothing but the binary cross-entropy corresponding to the i-th observation ( y_i, X_i )) between label distribution { y_i, 1- y_i} and {P(Y = y_i | X_i,), (1 - P(Y = y_i | X_i,))}. At the same time, this term is the contribution to the negative log likelihood for the pair ( y_i , X_i ). As a result, (5) is just the total cross-entropy measure and simultaneously the total negative log likelihood. Hence, negative log likelihood and binary cross-entropy are identical measures, but negative log likelihood is the basic measure in statistics (including logistic regression) and cross-entropy and D_KL are the favorite measures in machine learning. So, maximizing the log likelihood is equivalent to minimizing the binary cross-entropy or Kullback - Leibler divergence, which are frequently used in problems of model selection, testing for goodness of fit, etc.

References

Cover, T. M. and Thomas, J. A. (2006) “Elements of Information Theory”, 2nd Edition. John Wiley, New Jersey.
Hastie, T., Tibshirani, R., and Friedman, J. (2009) “The Elements of Statistical Learning: Data Mining, Inference, and Prediction”, Springer Series in Statistics, 2nd Edition, John Wiley, New Jersey.
Hosmer, D.W. and Lemeshow, S. (1989) “Applied Logistic Regression”. New York: Wiley.
Menard, S.W. (2010) “Logistic Regression: From Introductory to Advanced Concepts and Applications”, Thousand Oaks, CA: Sage.
Shtatland, E.S. (2019) “Logistic Regression and Information Theory: Part 1 - Do log odds have any intuitive meaning?” https://statisticalmiscellany.blogspot.com/2019/09/logistic-regression-and-information.html

Logistic Regression and Information Theory: Part 2

Once Again in Defense of Logit (Log Odds)

The Birth (1944) - in the Shadow of Probit - The Triumph

Following Allison (2017) “In Defense of Logit”, we are choosing the title “Once again in Defense of Logit“. Now a little bit of history.
The term “logit” was coined by Joseph Berkson in 1944, by analogy to the “probit”. Though, a related term “logistic function” appeared much earlier. It was invented in 1838 by the Belgian mathematician Verhulst to describe population growth. Later it was used in modeling autocatalytic chemical reactions and in bioassays (see, for example, Cramer (2003)). According to Cramer (2003):
“As far as I can see the introduction of the logistic as an alternative to the normal probability function is the work of a single person, namely Berkson... The issue of logit versus probit was tangled by Berkson’s simultaneous attacks on the method of maximum likelihood and his advocacy of minimum chi-squared estimation instead... it was Berkson who persisted and fought a long and spirited campaign which lasted for several decades ...Berkson’s case for the logit was not helped by his simultaneous attacks on the established wisdom of maximum likelihood estimation and his advocacy of minimum chi-squared. The unpleasant atmosphere in which this discussion was conducted can be gauged from the acrimonious exchanges between R.A. Fisher and Berkson in Fisher (1954).”
Said above explains why logit was waiting for its recognition in statistical community for such a long time.
From Jaynes (2003) we can learn about some unsuccessful attempts to introduce other names for logit: “deciban” (Allan Turing in 1941 in classified cryptographic work in England during World War II to decipher Nazi’s “Enigma”); “lods” (I.J. Good in 1950); and “evidence” (E.T. Jaynes in 1956).
Also, it should be noticed that there exists an additional, very interesting and rather surprising interpretation of logit in terms of the general system theory. In accordance with Voit and Knapp (1997), logistic model in epidemiological context is a natural and essential consequence of the formulation of a disease process as the dynamic model in a S-system form. The corresponding S-system of ordinary nonlinear differential equations describes epidemic development in time while logistic regression characterizes epidemic steady state. We mentioned this fact in our early work, Shtatland and Barton (1998). Summarizing, we can see that logit does have meaningful interpretations in various, seemingly different fields.

References

Allison, P.D. (2017) “In Defense of Logit – Part 1”, https://statisticalhorizons.com/in-defense-of-logit-part-1
Cramer, J.S. (2003) “The origins and development of the logit model”, Cambridge University Press, pp.10 – 11. https://pdfs.semanticscholar.org/7218/daab6499b46759f0a16d173d01d348bed906.pdf
Jaynes E. T. (2003) “Probability Theory: The Logic of Science”, Cambridge University Press: New York.
Shtatland, E.S. (2019) “Logistic Regression and Information Theory: Part 1 - Do log odds have any intuitive meaning?” https://statisticalmiscellany.blogspot.com/2019/09/logistic-regression-and-information.html
Shtatland, E.S. and Barton M.B. (1998) “Information Theory Makes Logistic Regression Special”. NESUG Proceedings
Voit, E. O. and Knapp, R. G. (1997). Derivation of the linear-logistic model and Cox’s proportional hazard model from a canonical system description. Statistics in Medicine, 16, 1705-1729

Logistic Regression and Information Theory: Part 1 - Do log odds have any intuitive meaning?

The goal of this series of posts is to show that practically all the features of logistic regression can be explained in terms of information theory.

Let’s start with definitions. Binary logistic regression model with dichotomic outcome Y is defined by equation

ln[P(Y = 1)/P(Y = 0) = ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k                 (1)

where p is the predicted probability of EVENT (Y = 1) and 1 – p is the probability of NONEVENT (Y = 0)) , X = ( X₁, X₂, … X_k) is a vector of predictors or explanatory variables, b = ( b₀ , b₁, b₂, … b_k) are the corresponding coefficients. For comparison we define linear probability model as

p = a₀ + a₁X₁ + a₂X₂ + … + a_kX_k                                                                   (2)

Note we use different notations for the coefficients in (1) and (2). The coefficients (a_i) are estimated by using Ordinary Least Squares (OLS) method, the coefficients (b_i) - by using Maximum Likelihood (ML) approach.

As said in Paul Allison (2012), p. 26:

“For the linear probability model, a coefficient of 0.25 tells you that the predicted probability p of the event increases by 0.25 for every 1-unit increase in the explanatory variable. By contrast, a logit coefficient of 0.25 tells you that the log odds increases by 0.25 for every 1-unit increase in the explanatory variable. But who knows what a 0.25 increase in log odds means?”

Paul von Hippel in his post of July 5, 2015 supports this statement in even stronger terms:

“In the linear model, if a₁ is (say) .05, that means that a one-unit increase in X₁ is associated with a 5 percentage point increase in the probability that Y is one. Just about everyone has some understanding of what it would mean to increase by 5 percentage points their probability of, say, voting, or dying, or becoming obese. The logistic model is less interpretable. In the logistic model, if b₁ is .05, that means that a one-unit increase in X₁ is associated with a .05 increase in the log odds that Y is 1. And what does that mean? I’ve never met anyone with any intuition for log odds” (in both citations bold print is mine).

Some researches question interpretability and intuition not only of log odds but even odds themselves (see discussions on the subject in Allison (2015) and (2017), and von Hippel (2015). Hosmer and Lemeshow (1989), p. 42 moderate discussions by saying “The interpretation given for the odds ratio is based on the fact that in many instances it approximates a quantity called the relative risk.”

And nevertheless, log odds or logits do have quite a simple and intuitive interpretation on their own. It is known (see, for example, Rissanen (1989) p. 38) that for each random event E there are necessarily two numbers, the event’s probability p = P(E) and its information I(E), i.e. the information contained in the message that E occurred. P(E) and I(E) are related to each other by with equation

I(E) = - lnP(E)                                                                                                  (3)

where ln is natural logarithm, so information is measured in nats, not in bits (1 nat ≈ 1.44 bits). It may be argued that using I(E) instead of P(E) provides a more natural scale, especially in case of rare events. For example, three values of P(E) = 0.02, 0.01, and 0.001 are almost equidistant, but the corresponding values of I(E) are 3.91, 4.61; 6.91 (all in nats) which are more realistic. We see that rarer events provide more information when observed. Note that (3) can be also associated with the minimal word length to convey the information content. So information is as fundamental a concept as probability. Now, let y₁, y₂,..., y_i,... y_n be independent observations of variable Y; n - the total sample size; p = P(E) = P(Y = 1) and 1 – p = P(NE) = P(Y = 0). In these notations, we have the following chain of equalities:

ln[p/(1-p)]=ln[p(E)/(1-p(E)]=ln[p(E)] - ln[(1-p(E)]= I(NE) - I(E)                 (4)

Thus, the logit is nothing but the discriminative information difference between event E and nonevent NE - (DID(E, NE)). From equations (3) and (4) we see that

(a) the less probable event E is, the more information it contains, i.e. more amount of nats is needed to store it, and vice versa;
(b) as the value of p goes from 0 to 1, the corresponding value of DID (log odds) goes symmetrically from minus infinity to plus infinity;
(c) if p = 0.5, then DID(E, NE) = 0, i.e. E and NE cannot be discriminated in terms of information content;
(d) in general the DID (log odds) value for probability p is minus the DID (log odds) value for probability 1-p;

If anybody is uncomfortable with possible negative values of DID (if p < 0.5), then the absolute value of DID can be used. Though in this case the information about sign of DID will be lost.

DID(E,NE) can serve as an information measure of the ability to discriminate between E and NE. In this regard, it is interesting to compare logistic regression with linear discriminant analysis. These techniques are defined by the same mathematical equations, but they work under different assumptions. According to Hosmer and Lemeshow (1989), pp 34-36, and Menard (2010), pp 319-320 logistic regression is more preferable when the basic assumption of LDA – multivariate normality of predictors is not satisfied. And yet in practice, logistic regression and LDA often give similar results. See, for example, Hastie, Tibshirani, and Friedman (2013), pp. 121-122:

“It is our experience that the models give very similar results, even when LDA is used inappropriately, such as with qualitative predictors…We have seen that linear discriminant analysis and logistic regression both estimate linear decision boundaries in similar but slightly different ways”

After estimating coefficients (b₀ , b₁, … b_k) in (1) by maximum likelihood we have to take decision whether our current case is EVENT (Y = 1) or NONEVENT (Y = 0). For this purpose, we can choose some decision cut-off either in terms of predicted probability or in terms of log odds (DID). Actually, in a logistic model, units of log odds or DID, (i.e. nats), are converted back to units of probability by formula

p = exp(ln[p/(1-p)] /(1+exp(ln[p/(1-p)]) =

1 / (1 + exp(- ln[p/(1-p)])) = 1 / (1 + exp(-( b₀ + b₁X₁ + b₂X₂ + … b_kX_k)                (5)

where exp is the exponential function. Though the predicted probabilities may be used in decision, it seems more convenient and natural to do it in terms of log odds, i.e. information. According to Allison (2017), in many applications researches are not really interested in predicted probabilities. And then Allison in this article added emphatically “… there’s nothing sacred about probabilities. An odds is just as legitimate a measure of the chance that an event will occur as a probability. And with a little training and experience, I believe that most people can get comfortable with odds”. We can extend the statement above by saying: there is nothing sacred about both probability and odds. Log odds i.e., discriminative information difference (DID) is as legitimate as a probability and odds. Moreover, in terms of logit (and not probability) finding a decision threshold becomes a linear problem – to choose a cut-off, C, such that

Declare EVENT if ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k > C                         (6)

Declare NONEVENT if ln[p/(1-p)] = b₀ + b₁X₁ + b₂X₂ + … b_kX_k <= C

Of course, a cut-off point depends on the relative consequences of a false positive and a false negative.

Thus, in this post we have showed that logit is easily interpreted in terms of information as discriminative information difference (DID). Since there exists a linear relationship between logit and coefficients (b_i) it is natural to expect a similar interpretation for coefficients (b_i). According to Hosmer and Lemeshow (1989), p.39: “Proper interpretation of the coefficient in a logistic regression model depends on being able to place meaning on the difference between two logits”. It can be easily showed when we have a sole independent variable of a continues or a binary type. First, consider the case of a sole binary predictor, say, X, with values ‘1’ for ‘female’ and ‘0’ for ‘male. Then, b can be interpreted as the difference in ability to discriminate (DID) between ‘female’ and ‘male’. In the same way, with a sole continues predictor, say, age in years, the coefficient b can be interpreted as a change in DID per year of age. In general case of multiple logistic regression this interpretation is not as straightforward, and the estimated coefficients must be interpreted with care.

In our next blog posts, it will be showed that the main estimation method for logistic regression - Maximum Likelihood (ML) - has a simple and natural interpretation in terms of information theory. As a result, R-Squared measures based on the log likelihood (see our post Shtatland (2018)) acquire an information interpretation. And so do some goodness-of-fit tests (for example, the deviance test or the likelihood ratio test).

References

Allison, P.D. (2012) “Logistic Regression Using SAS: Theory and Application”. Cary, NC: SAS Institute.

Allison, P.D. (2014) “Measures of fit for logistic regression”. https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf

Allison, P.D. (2017) “In Defense of Logit – Part 1”. https://statisticalhorizons.com/in-defense-of-logit-part-1

Hastie, T. ,Tibshirani, R. and Friedman, J. (2013) “The Elements of Statistical Learning: Data Mining, Inference, and Prediction”. Springer

Hosmer, D.W. and Lemeshow, S. (1989) “Applied Logistic Regression”. New York: Wiley.

Shtatland, E.S. (2018) “Do we really need more than one R-Squared in logistic regression?” http://statisticalmiscellany.blogspot.com/

Von Hippel, P. (2015) Linear vs. Logistic Probability Models: Which is Better, and When? http://statisticalhorizons.com/linear-vs-logistic

Do we really need more than one R-Squared in logistic regression?

In the previous post we advocated McFadden R-Square as our preferred choice for binary logistic regression, in other words we joined the so-called “McFadden Camp” (see Paul Allison’s post “What’s the best R-Squared for logistic regression” (2013)). But not all researchers think this way. And as it is said in DeMaris (1992), p. 56: “… It may not be prudent to rely on only one measure for assessing predictive efficacy - particularly in view of the lack of consensus on which measure is most appropriate. Perhaps the best strategy is to report more than one measure for any given analysis. If the model has predictive power, this should be reflective in some degree by all of the measures discussed above”.

Many different R-Squared statistics have been proposed in the past four and a half decades (see, e.g., review publications: Windmeijer (1995), Cameron and Windmeijer (1997), Mittlbock and Schemper (1996), Menard (2000), (2010), Smith and McKenna (2013), Walker and Smith (2016)). These statistics can be divided into three main categories:

(1) R-Squared measures based on likelihoods;

(2) R-Squared measures based on the sum of squares;

(3) Measures based on squared correlations of observed output and estimated probabilities. 

Standing alone is the so-called Tjur R-Squared.

R-Squared measures based on model likelihoods

 

This category is most populated. It includes familiar to us from the previous post R^2(MF) (McFadden R^2) and R^2(CS) (Cox - Snell R^2). Here and further we use notations partially from our previous post, Menard (2000) and Smith – McKenna (2013)):

(1)  R^2(MF) = [lnL(0) – lnL(M)] / [lnL(S)] – lnL(0)] = 

      1 – lnL(M) / lnL(0)

(2)  R^2(CS) = 1 – [L(0) / L(M)]^(2/n)

(3)  R^2(N) = R^2(CS) /max R^2(CS) = R^2(CS) / (1 – L(0)^(2/n))

Also included are:

(4)  Aldrich – Nelson R-Squared, defined by          

R^2(AN) = G(M) / (G(M) + n) = 

2(lnL(M) - lnL(0)) /[2(lnL(M) - lnL(0)) + n]          

(see Smith – McKenna (2013))

(5)  Veall – Zimmermann R-Squared, given by formula 
R^2(VZ) = 

2(lnL(M) - lnL(0)) /[ 2(lnL(M) - lnL(0)) + n]*[(-2lnL(0) + n) / -2lnL(0)] =  R^2(AN) / max R^2(AN) 

(see Walker and Smith (2016)). R^2(VZ) was introduced as a correction to R^2(AN) which cannot reach the value of one. It’s easy to see that max R^2(AN) is equal to 2ln2/(2ln2 + 1) ≈0.581 (compare to max R^2(CS) = 0.75)

(6) Estrella R-Squared, defined by formula 

R^2(Est) = 1 – [lnL(M) / lnL(0)]^ (-2lnL(0)/n) 
(see Walker and Smith (2016)) 

We don’t consider here corrections for the number of predictors of the R-Squared measures mentioned above. From this abundance of R-Squared indexes we choose only two: R^2(MF) and R^2(CS). As it is shown in Shtatland et al. (2000) and (2002), there exists an important and a very simple functional relationship between R^2(MF) and R^2(CS) (see also our previous post, Menard (2010), p. 50), and our comment to Allison (2013):

R^2(CS) = 1 – exp(-R^2(MF)*T)                          (1)                          

where the quantity T is equal to -2lnL(0) / n. Since the maximized log-likelihood for the null model can be written as

lnL(0) = n[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]

the formula for T can be rewritten as follows

T = -2[ȳ*lnȳ + (1 – ȳ)*ln(1 – ȳ)]                                       (2)

Thus, T is nothing but doubled entropy of Bernoulli distribution with probability ȳ (the base rate).

Desirable properties of an R-squared include interpretation in terms of the information content of the data (see Cameron and Windmeijer (1997)). It is shown in Shtatland et al. (2000) and (2002) (see also our previous post) that R^2(MF) has an intuitively reasonable, immediate interpretation in terms of information theory: it can be considered as ratio of information gain (IG) when moving from the null model with intercept only to the current one by adding available predictors (2(lnL(M) - lnL(0)) to the quantity of all potentially recoverable information (-2lnL(0)). Note that R^2(CS) also allows some information interpretation, though not as direct as for R^2(MF). Indeed, let us transform the equation

R^2(CS) = 1 – [L(0) / L(M)]^(2/n)

which defines R^2(CS) to

-ln (1 - R^2(CS)) = 2(lnL(M) – lnL(0))/n = IG /n = IGR

Here information gain rate, IGR, is nothing but information gain per observation. The logarithmic function -ln(1 – x) is a well-known measure from information theory (Theil and Chung (1988)). Using -ln(1 - R^2(CS)) rather than R^2(CS) provides a more natural scale for interpretation. Taking the first derivative of both sides of this equation with respect to R^2(CS), we have

dIGR/dR^2(CS) = 1 / (1 - R^2(CS))

From this formula we arrive at the following important conclusions:

If R^2(CS) = 0, then dIGR/dR^2(CS) = 1, i. e. the increase in R-Square is equivalent to the increase in IGR;

If R^2(CS) = 0.5, then dIGR/dR^2(CS) = 2, i. e. the increase in IGR is twice as large as the increase in R-Square;

If R^2(CS) = 0.75 (maximal value possible), then dIGR/dR^2(CS) = 4, i. e. the increase in IGR is four times larger than the increase in R-Square.

Note that equations (1) and (2)) can be also found in Sharma and McGee (2008), though without informational - entropic interpretation, which is one of the most desirable properties of an R-squared (see, for example, Cameron and Windmeijer (1997)).

Concluding our comparison R^2(CS) and R^2(MF), note that these measures are related to the base rate ȳ quite differently. According to Menard (2000), R^2(MF) stands out for its relative independence from the base rate, comparative to R^2(CS) and other R-Squared measures. At the same time, R^2(CS) is highly correlated with ŷ (see Menard, 2000, p. 23). From formulas (1), (2), it can be seen that R^2(CS) depends on the base rate ȳ basically through the quantity T. As a result, R^2(CS) demonstrates a rather anti-intuitive and odd trait: it increases as the base rate (more exactly, ȳ or 1 – ȳ, whichever is smaller,) increases from 0 to 0.5, absurdly implying that ȳ itself can be used as some R^2 measure.

It is easy to see that each candidate for R^2 in logistic regression expressed as a function of statistic [lnL(M) – lnL(0)] can be also rewritten through R^2(MF) and T. For example, R^2(AN), R^2(VZ) and R^2(EST) can be shown as:

R^2(AN) = R^2(MF) / (R^2(MF)+1/T)                            (3)

R^2(VZ) = R^2(MF) * (1 + T) / (1 + T* R^2(MF))         (4)

R^2(Est) = 1 – exp[T*ln(1 - R^2(MF))]                            (5)

Thus, it can be concluded from the formulas above that R^2(MF) plays a central role in the category of R-Squared statistics based on the model likelihoods - if we know R^2(MF) and T, then all other members of this category can be calculated by hand.

Furthermore, as it is shown above, R^2(MF) (and partly R^2(CS)) has a clear and direct interpretation in terms of the information content of the data, potentially recoverable information, information gain due to predictors adding, etc. (see Shtatland (2002), Menard (2010) p .48, and our previous post). This interpretation is no less natural and important than the one used in the OLS approach. At the same time R^2(AN), R^2(VZ) and R^2(EST) have lost this interpretation and look as ad hoc measures.  Summarizing we can conclude that R^2(MF) is obviously more preferable measure than any other R-Squared in this category.

R-Squared measures based on the sum of squares – OLS analogs

We consider here only the most popular measures. First one, based on sums of squared residuals, is used under various names: R^2(O), R^(SS), R^(Efron), R^2(res), etc., and is given by familiar to us from our previous post formula:

R^2(O) = 1 - ∑(yi –  ŷi) ^2 / ∑(yi – ȳ ) ^2 󠄝                         (6)

Second one, the so-called model R-Squared, R^2(mod) defined by formula:

R^2(mod) = ∑( ŷi – ȳ) ^2 /∑(yi – ȳ ) ^2                             (7)

Let us remind that yi is the observed outcome (1 or 0), ŷi is the predicted value of yi (actually, the predicted probability), and ȳ is the arithmetic mean of yi. R^2(O) looks exactly like the ordinary least square R^2(OLS) for linear regression except that ŷi are calculated by maximizing likelihood function and not minimizing sum of squares. It is frequently used as a standard for comparison between other R-Squared measures. For some researchers (see, for example, Mittlbock and Schemper (1996)), R^2(O) is a measure of choice. Also, it is concluded in Menard (2010), p. 56 that there are certain benefits in using R^2(O) not instead R^2(MF), but as a supplemental measure. Though, R^2(O) has two serious disadvantages: it does not automatically increase when the model is extended by an additional predictor, and can be negative in some rare, rather degenerate cases.
 

Measures based on squared correlations

There are plenty of them. In Mittlebock and Schemer (1996) six measures are discussed. We will mention only the most popular measure in this category – based on squared Pearson correlation:

R^2(cor) = 

{[∑(yi – ȳ )( ŷi – ȳ) / [∑(yi – ȳ )^2]^0.5] [∑(ŷi – ȳ )^2]^0.5]}^2

It is known that R^2(cor) is very close to R^2(O), moreover they are almost identical numerically (Liao and McGee (2003). Therefore, it is reasonable to use only one of them, and we prefer R^2(O).

Tjur R-Squared

Finally, we consider Tjur R-Squared (R^2(Tjur). Since it is not so easy to define R^2(Tjur) by formula, we refer for a verbal definition to Tjur (2009) and Allison (2013), (2014). Actually, this measure was introduced earlier in Cramer (1999), so it would be fair to use the double name: Cramer - Tjur. According to Allison (2013), the measure has a lot of intuitive appeal, it is easy to calculate, and also is closely related to R-Squared measures based on sum of squares:

R^2(Tjur) = (R^2(O) + R^2(mod)) /2                                (8)

R^2(Tjur) = (R^2(mod) * R^2(cor))^0.5                           (9)

Following Allison (2013), we propose to use R^2(Tjur) as an attractive alternative and as a good supplement to R^2(MF) (the best R-Squared based on likelihoods) and R^2(O) (our preferred choice in the category of OLS analogs).

Conclusion

Let us return to DeMaris quotation: “… It may not be prudent to rely on only one measure for assessing predictive efficacy - particularly in view of the lack of consensus on which measure is most appropriate. Perhaps the best strategy is to report more than one measure for any given analysis”. Following this advice, we propose to report three measures: R^2(MF), R^2(0) and T^2(Tjur) altogether, hoping that they make up a good ”three pillars of R-Squared measures” and a good pathway from “the jungle of R-Squareds” for binary logistic regression.
        

References:

Allison, P. D. “What’s the best R-squared for logistic regression?” (2013) (http://www.statisticalhorizons.com/r2logistic).

Allison, P. D. “Measures of fit for logistic regression” 2014) https://support.sas.com/resources/papers/proceedings14/1485-2014.pdf.

Cameron, C. A. and Windmeijer, F. A. G. (1997) "An R-squared measure of goodness of fit for some common nonlinear regression models", Journal of Econometrics, Vol. 77, No.2, pp. 329-342.

Cox, D. R. and Snell E. J. (1989) “Analysis of binary data” Second Edition, Chapman & Hall, London.

Cramer, J. S. (1999) “Predictive performance of the binary logit model in unbalanced samples”, The Statistician, 48, 85 – 94.

DeMaris, A. (1992) “Logit modeling” Sage University Paper Series.

Efron, B. (1978), “Regression and ANOVA with zero-one data: Measures of residual variation”, Journal of the American Statistical Association, 73, 113 – 121.

Hosmer, D. W. and Lemeshow, S. (1989), “Applied logistic regression”, New York: Wiley.

Kent, J. T. (1983). “Information gain and a general measure of correlation”,  Biometrika, 70, 163 – 173.

Menard, S. (2000) “Coefficients of determination for multiple logistic regression analysis”, The American Statistician, 54, 17 – 24.

Menard, S. (2002), “Applied logistic regression analysis”, Sage University Paper Series (Second edition).

Menard, S. (2010), “Logistic regression: From introductory to advanced concepts and applications”, Sage University Paper Series, Chapter 3, pp. 48 – 62.

McFadden, D. (1974), “Conditional logit analysis of qualitative choice behavior”, pp. 105 -142 in Zarembka (ed.), Frontiers in Econometrics. Academic Press.

Mittlebock, M. and Schemper, M. (1996), “Explained variation in logistic regression”, Statistics in Medicine, 15, 1987 – 1997.

Sharma D. and McGee, D. (2008), “Estimating proportion of explained variation for an underlying linear model using logistic regression analysis”, Journal of Statistical Research,42, 59 – 69.

Shtatland, E. S., Moore, S. and Barton, M. B. (2000) “Why we need R^2 measure of fit (and not only one) in PROC LOGISTIC and PROC GENMOD”, SUGI 2000 Proceedings, Paper 256 - 25, Cary, SAS Institute Inc.

Shtatland, E. S., Kleinman, K. and Cain, E. M. (2002) “One more time about R^2 measures of fit in logistic regression”, NESUG 2002 Proceedings.

Smith, T. J. and McKenna, C. M. (2013) “A comparison of logistic regression pseudo R^2 indices”, Multiple Linear Regression Viewpoints, 39, 17 - 26. 

Theil, H. and Chung, C.F. (1988) “Information-theoretic measures of fit for univariate and multivariate linear regressions”, The American Statistician, 42, 249 - 252.

Tjur, T. (2009) “Coefficients of determination in logistic regression models – a new proposal: the coefficient of discrimination”, The American Statistician, 63, 366 – 372.

Walker, D. A. and Smith, T. J. (2016) “Nine pseudo R^2 indices for binary logistic regression models”, Journal of Modern Applied Statistical Methods, 15, 848 – 854.

Windmeijer, F. A. G. (1995) “Goodness of fit measures in binary choice models”, Econometric Reviews, 14, 101 – 116.