Peter E. Kennedy

Simon Fraser University

Journal of Statistics Education Volume 10, Number 1 (2002)

Copyright © 2002 by Peter E. Kennedy, all rights reserved.

This text may be freely shared among individuals, but it
may not be republished in any medium without express
written consent from the authors and advance notification
of the editor.

**Key Words:** Detrending; Multicollinearity; Omitted regressor; Regression graphics; Teaching statistics.

A Venn diagram capable of expositing results relating to bias and variance of coefficient estimates in multiple regression analysis is presented, along with suggestions for how it can be used in teaching. In contrast to similar Venn diagrams used for portraying results associated with the coefficient of determination, its pedagogical value is not compromised in the presence of suppressor variables.

In a recent article in this journal, Ip (2001) discussed the use of Venn diagrams for enhancing instruction of multiple regression concepts. The purpose of this note is to inform readers of several applications of Venn diagrams to regression analysis that were not included in Ip's presentation, and to demonstrate some ways of using these applications to improve student understanding of ordinary least squares (OLS) regression. Because these alternative applications are not new to the literature, the main contribution of this paper consists of suggestions for how this approach can be used effectively in teaching.

The first use of Venn diagrams in regression analysis appears to be in the textbook by Cohen and Cohen (1975). A major difficulty with its use occurs in the presence of suppressor variables, a problem discussed at length by Ip (2001). No one denies that Venn diagrams can mislead, just as no one denies that ignoring friction in expositions of physical phenomena misleads, or using Euclidian geometry misleads because the surface of the earth is curved. Such drawbacks have to be weighed against the pedagogical benefits of the "misleading" expository device. As recognized by Ip, in the case of applying the Venn diagram to regression analysis, reasonable instructors could disagree on the pedagogical value of the Venn diagram because of the suppressor variable problem.

Ip's article is confined to the use of Venn diagrams for analyzing the coefficient of determination *R ^{2}*, partial correlation, and sums of squares. In these cases, exposition is compromised in the presence of suppressor variables. But there are other concepts in regression analysis, thought by many to be of considerably more importance than

Kennedy (1981) extended the Venn diagram to the exposition of bias and variance in the context of the classical linear regression (CLR) model, written as
*y = Xb +
e
*.
Here the dependent variable

Figure 1.

Figure 1. Venn diagram for regression.

Although the purple area represents the variation in *y* explained by *X*, just as in Ip’s application,
Kennedy extends its interpretation in three substantive ways:

The purple area represents information used by the OLS formula when estimating

*b*; if this information corresponds to variation in_{x}*y***uniquely**explained by variation in*X*, the resulting estimate of*b*is unbiased._{x}A larger purple area means that more information is used in estimation, implying a smaller variance of the

*b*estimate._{x}The black area, the variation in

*y*that cannot be explained by*X*, is attributed by OLS to the error term, and so the magnitude of this area represents the magnitude of the OLS estimate of , the variance of the error term. Notice a subtlety here. The purple area is interpreted as information; the black area interpretation is quite different - its magnitude reflects the magnitude of a parameter estimate.

To analyze multiple regression, Kennedy adopted the three intersecting circles diagram of
Cohen and Cohen (1975), which they named the
"Ballantine" because of its resemblance to the logo of a brand of beer;
Kennedy reinterpreted the areas as described above, and to emphasize this new interpretation changed its spelling to “Ballentine.”
Such a diagram is shown in
Figure 2, in which a new circle marked *W* is
added (with an associated slope
*b _{w}*),
representing variation in another explanatory variable. In the presentations of
Cohen and Cohen (1975) and Ip (2001), the overlap between the

Figure 2.

Figure 2. Ballentine Venn diagram.

Kennedy does not attempt to exposit *R ^{2}*, but rather looks at a completely
different set of OLS properties - bias and variance. The following section describes what
in my experience are some particularly effective ways of using this diagram to teach the
properties of OLS estimates in the CLR model.

I begin by carefully explaining the interpretations of the areas in
Figure 1
as described above. Following this I put up
Figure 2
on the overhead and ask the class what will happen using OLS when there is more than one explanatory variable, drawing to their attention that it is not obvious what role is played by the red area. I note that if we were to regress *y* on *X* alone, OLS would use the information in the blue plus red areas to
create its estimate of
*b _{x}*,
and if we were to regress

- Continue to use blue plus red to estimate
*b*and green plus red to estimate_{x}*b*._{w} - Throw away the red area and just use blue to estimate
*b*and green to estimate_{x}*b*_{w} - Divide the red information into two parts in some way, and use blue plus part of red to estimate
*b*and green plus the other part of red to estimate_{x}*b*._{w}

I point out that several special cases of option c are possible, such as using blue and all of red to estimate
*b _{x}*
and only green to estimate

After setting this up I inform students that they are to guess what OLS does, and ask them to vote for one of these options. (Voting has to be done one by one, because if the class at large is asked to vote, invariably nobody votes for anything; Kennedy (1978) is an exposition of this pedagogical device.) I have never had a majority vote for the correct answer. Next I ask the class why it would make sense for an estimating procedure to throw away the information in the red area. (It is this throwing away of the red area that allows this application of the Venn diagram to avoid being compromised by the presence of a suppressor variable.) The ensuing discussion is quite useful, with good students eventually figuring out the following.

The information in the red area is bad information - it has

*y*"moving together" with*X*and also with*W*so that we don’t know if the movements in*y*are due to*X*or due to*W*, so to be on the safe side we should throw away this information.If only the blue area information is used to estimate

*b*and the green area information is used to estimate_{x}*b*, unbiased estimates are produced, because the blue area corresponds to variation in_{w}*y*uniquely attributable to*X*and the green area corresponds to variation in*y*uniquely attributable to*W*.

An instructor may wish to elaborate on point b by noting that the variation in *y* in the red area is actually due to joint movements in *X* and *W* because the red area corresponds to *X* and *W* “moving together” as well as to *y* and *X* moving together and *y* and *W* moving together. Suppose that in the red area when *X* changes by one *W* changes by two, so that a joint movement of one by *X* and two by *W*
gives rise to a movement in *y* of
*b _{x}*
+ 2

Instructors presenting an algebraic version of this material can demonstrate this result by
working through the usual derivation of the OLS estimate of
*b _{x}*
as

The instructor can finish by noting that the yellow area in Figure 2 represents the magnitude of , the variance of the error term. The OLS estimating procedure uses the magnitude of the area that cannot be explained by the regressors as its estimate of . In this case this is the yellow area, which is the correct area, so the OLS estimator of is unbiased. This sets up the Venn diagram for investigation of several features of OLS estimation, of which three are discussed below.

Ask the students how a greater degree of multicollinearity would manifest itself on the Venn diagram. They should be able to guess that it is captured by increasing the overlap between the *X* and *W* circles, as shown by moving from
Figure 3a to Figure 3b.
Next ask them if the higher collinearity causes bias. I go around the class and ask everyone to commit to yes, no, or don’t know. (Those answering don’t know are not asked later to explain their rationale.) Once this has been done, have someone who voted with the majority offer an explanation, and then have someone who voted with the minority offer a counter-explanation. Work with this until everyone sees that because the OLS formula continues to use the information in the blue and green areas to estimate the slopes of *X* and *W*, these estimates remain unbiased - the blue and green areas continue to correspond to variation in *y* uniquely attributable to *X* and *W*,
respectively.

Figures 3a and 3b.

Figures 3a and 3b. Ballentine Venn diagrams displaying modest and considerable collinearity.

Next ask the students what the higher collinearity does to the variance of the estimate of
*b _{x}*
(or

In summary, it should now be clear that higher collinearity increases variance, but does not cause bias. Finish by asking what happens in the diagram if *X* and *W* become perfectly collinear - the blue and green areas disappear and estimation is impossible.

Return to Figure 2, reproduced here as
Figure 4, specifying that both
*X* and *W* affect *y*, and ask
how the properties of the OLS estimate of
*b _{x}*
would be affected if

Figure 4.

Figure 4. Ballentine Venn diagram.

First, is bias created whenever a relevant explanatory variable is omitted? I ask everyone to commit to yes, no, or don’t know. After this voting, discussion should continue until everyone sees that if *W* is omitted the OLS formula uses the blue plus
red area to estimate
*b _{x}*, and so is clearly biased because the red area is contaminated information. The direction of the bias cannot be determined from the diagram. Before leaving this
the instructor can ask under what special circumstance would no bias be created? The students should be able to guess that when the

Second, what can we say about the variance of the OLS estimator compared to when *W* is included? I ask everyone to commit to larger, smaller, no change, or don’t know. After this voting, the discussion should continue until everyone sees that because the blue plus red area information is used (instead of just the blue area information),
**more** information is being used, so that variance should be smaller. What if the omitted explanatory variable *W* is orthogonal to *X*? In this case the OLS estimator continues to use just the blue information, so variance is unaffected.

At this stage the instructor might want to note that by omitting a relevant explanatory variable it should be clear that bias is created, a bad thing, but variance is reduced, a good thing, and comment that the mean square error criterion becomes of interest here because it is a way of trading off bias against variance. A good example to use here is the common procedure of dropping an explanatory variable if it is highly collinear with other explanatory variables. Ask the students how the results developed above could be used to defend this action. They should be able to deduce that omitting a highly-collinear variable can markedly reduce variance, and so may (but may not!) reduce the mean square error.

Third, what can we say about our estimate of
(the variance of the error term)? I ask everyone to commit to unbiased, biased upward, biased downward, or don’t know. After this voting, the discussion should continue until everyone sees that the OLS procedure uses the magnitude of the yellow plus green area to estimate the magnitude of the yellow area, so the estimate will be biased upward.
The instructor can follow up by asking if this bias disappears if the omitted explanatory variable *W* is orthogonal to *X*.

In summary, omission of a relevant explanatory variable in general biases coefficient estimates, reduces their variances, and causes an overestimate of the variance of the error term. If the omitted variable is orthogonal to the included variable, estimation remains unbiased, variances are unaffected, but is nonetheless overestimated.

Suppose *W* is a time trend. How will the
*b _{x}*
estimate be affected if the time trend is removed from

They are then told to perform the following three estimations, following which they are asked to use the Ballentine Venn diagram to explain their results:

- Regress
*y*on*X*and*W*to obtain*b*, and its estimated variance_{x}*vb*.^{*} - Regress
*X*on*W*, save the residuals*r*, and regress*y*on*r*to get*c*, the estimate of the^{*}*r*coefficient, and its estimated variance*vc*.^{*} - Regress
*y*on*W*, save the residuals*s*, and regress*s*on*r*to get*d*, the estimate of the^{*}*r*coefficient, and its estimated variance*vd*.^{*}

With their data my students obtain results reported in Table 1.

**Table 1.** Results from estimating with residualized data.

Coefficient | Estimate | Estimated Variance |

b^{*} | 1.129427 | 0.00210754 |

c^{*} | 1.129427 | 0.00987857 |

d^{*} | 1.129427 | 0.00208904 |

Students are surprised that these three estimates *b ^{*}*,

Figure 5.

Figure 5. Ballentine Venn diagram.

Trouble begins when they try to explain the estimated variance results. Because exactly the same information is being used to produce *b ^{*}*,

They ignore this problem, pretending that all they have to do is explain why the slope estimates are identical. Or they don’t realize that the three variances are equal, and so believe that these differing numbers do not need comment.

They claim that all three numbers are identical except for rounding error.

They note that

*vb*and^{*}*vd*are close enough that we can legitimately claim they are identical except for rounding error, note that^{*}*vc*is markedly higher, but are unable to offer an explanation.^{*}They realize that although the true variances are equal, the estimated variances may not be equal, and use the Venn diagram to explain how this happens. When estimating

*b*the variation in^{*}*y*not explained is the yellow area, so , the variance of the error term, is estimated by the magnitude of the yellow area. Similarly, when estimating*d*(by regressing blue plus yellow on orange plus blue) the variation in the dependent variable not explained is the yellow area, so in this case is also estimated by the magnitude of the yellow area. But in the case of estimating^{*}*c*(by regressing the entire^{*}*y*circle on orange plus blue) the variation in*y*not explained is the yellow plus red plus green areas. As a result, in this case is overestimated. This overestimation causes*vc*to be larger than^{*}*vb*and^{*}*vd*.^{*}

Instructors may wish to supplement this explanation by noting that the formula for the
variance of an OLS estimator involves both and variation in the explanatory variable data which is “independent” of variation in other explanatory variables.
(In this example variance would be given by the formula
^{t}M_{w}X)^{-1}.*X* is reflected by the blue plus orange areas, so the relative magnitudes of the estimated variances depend entirely on the estimates of
.

How does all this relate to regressing on detrended data? If *W* is a time trend, then *s* is detrended *y* and *r* is detrended *X*, so that regressing *s* on *r* produces estimates identical to those of regressing on raw data. One concludes that it doesn’t matter if one regresses on raw data including a time trend, or if one removes the linear trend from data and regresses on detrended data. Similarly, if *W* is a set of quarterly dummies, it doesn’t matter if one regresses on raw data plus these dummies, or if one regresses on data that have been linearly deseasonalized. More generally, this reflects the well-known result that slope estimates are identical using raw data or appropriately residualized data.

The Ballentine Venn diagram is not new to the literature. Kennedy (1998) exposits the applications presented earlier, as well as discussing the implications of adding an irrelevant explanatory variable, and the rationale behind instrumental variable estimation. In Kennedy (1989), the Ballentine is used to exposit tests for non-nested hypotheses. The main contribution of this paper, beyond drawing readers’ attention to this use of Venn diagrams, is to describe some particularly effective ways of using this diagram when teaching regression analysis.

There do exist special situations in which the Ballentine can mislead. For example, the Ballentine erroneously suggests that
the OLS slope estimates from regressing *y* on *X* and *W* can be reproducing by the following
procedure: obtain *r* by taking the residuals from regressing *X* on *W*, obtain *v* by taking
the residuals from regressing *W* on *X*, and then regress *y* on *r* and *v*.
My experience with its use in the classroom has been overwhelmingly positive, however; confined to standard analyses, the advantages of this Venn diagram interpretation as a pedagogical device are too powerful to ignore.

Cohen, J., and Cohen, P. (1975), *Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences*, Hillside, NJ: Lawrence Erlbaum Associates.

Ip, E. H. S. (2001), "Visualizing Multiple Regression," Journal of Statistics Education [Online], 9(1). (ww2.amstat.org/publications/jse/v9n1/ip.html)

Kennedy, P. E. (1978), "Democlass: A Variation on the Question/Answer Technique," *Journal of Economic Education*, 9, 128-130.

----- (1981), "The 'Ballentine': A Graphical Aid for Econometrics," *Australian Economic Papers*, 20, 414-416.

----- (1989), "A Graphical Exposition of Tests for Non-nested Hypotheses," Australian Economic Papers, 28, 160-165.

----- (1998), *A Guide to Econometrics* (4th ed.), Cambridge, MA: MIT Press.

Peter Kennedy

Department of Economics

Simon Fraser University

Burnaby, B.C.

Canada V5A 1S6 kennedy@sfu.ca

Volume 10 (2002) | Archive | Index | Data Archive | Information Service | Editorial Board | Guidelines for Authors | Guidelines for Data Contributors | Home Page | Contact JSE | ASA Publications