Reasoning and Communicating in the Language of Statistics

Carol S. Parke
Duquesne University

Journal of Statistics Education Volume 16, Number 1 (2008), jse.amstat.org/v16n1/parke.html

Copyright © 2008 by Carol S. Parke all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the author and advance notification of the editor.

Key Words: Conceptual understanding; Confidence; Interpretation of results; Verbal communication; Written communication.

Abstract

Although graduate students in education are frequently required to write papers throughout their coursework, they typically have limited experience in communicating in the language of statistics, both verbally and in written form. To succeed in their future careers, students must be provided with opportunities to develop deep understandings of concepts, develop reasoning skills, and become familiar with verbalizing and writing about statistics. The instructional approach described here spans the entire semester of a statistics course and consists of several aspects including cognitively rich individual assignments, small group activities, and a student-led scoring activity. To demonstrate the impact of this approach on student learning, qualitative and quantitative data were collected from students in two statistics courses. Several assessments indicate improvement in students’ reasoning and understanding, written and verbal communication, and confidence.

1. Introduction

A reform movement has emerged over the past two decades within the teaching and learning of statistics. Instructional emphasis is shifting from rote memorization of formulae, computational skills, and procedural rules to conceptual understanding, making connections among statistical concepts, use of real-world data, interpreting results, and making appropriate conclusions. Educators and researchers use terminology such as statistical reasoning, thinking, comprehension, citizenship, and literacy to describe desirable goals of instruction and assessment. As early as the 1970s, researchers wrote about statistical reasoning. Chervaney, Collier, Fienberg, Johnson, & Neter (1977) presented a three-step process of reasoning that focused on comprehending the problem, planning and executing appropriate methods to solve the problem, and evaluating and interpreting the outcomes. However, the early research did not describe how to operationalize these ideas in the statistics classroom.

Recent literature defines student learning outcomes and describes instructional techniques that promote a solid understanding of statistics. A series of articles published in 2002 discussed the varied definitions of three domains of statistical development (literacy, reasoning, and thinking) and provided illustrations of ways to incorporate the domains into the teaching, learning, and assessment of statistics. One perspective considers the three domains to be mainly independent of each other while recognizing some degree of overlap. Another perspective views statistical literacy as the overarching goal serving as the umbrella under which reasoning and thinking are subsumed (delMas, 2002).

Garfield (2002) describes statistical reasoning as making sense of statistical information, making inferences, interpreting results, and having a conceptual understanding of important ideas. She presents a five-stage model of statistical reasoning for sampling distributions that progresses from faulty reasoning and misconceptions to integrated process reasoning in which "the student has a complete understanding of the process of sampling and sampling distributions, …can explain the process in his or her own words, and makes correct predictions with confidence" (p. 7).

Statistical thinking is described by Chance (2002) as the ability to view the statistical process as a whole, to explore data in ways beyond those given in textbooks, to pose new questions, to understand the meaning of variation, and to understand "why". She provides illustrations of how to instill these mental "habits" in students during instruction. They include examining data collection issues to ensure that the right data is collected to answer the right questions, providing opportunities to be skeptical of statistical results in the media, examining limitations of inferential techniques, relating the data to the context, and understanding the relevance of statistics.

Finally, statistical literacy is defined by Rumsey (2002) as encompassing two distinct learning outcomes: competence and citizenship. Statistical competence is the basic knowledge underlying statistical reasoning and thinking. Statistical citizenship is the goal of becoming educated consumers of quantitative data who are able to think critically and make good decisions from the information. Rumsey also states that correctly calculating numerical results does not necessarily demonstrate a good understanding of concepts. Students should be given opportunities to collaborate with each other and to explain ideas and interpret results in their own language.

1.1 Written and Verbal Communication in Statistics

An essential aspect of statistical literacy is the ability to communicate concepts and results in written and verbal form. In her article on literacy, Rumsey (2002) distinguishes between interpretation skills that demonstrate whether a student understands a concept and communication skills that involve sharing the information clearly with others. In the late 1980s and early 1990s, much of the research on writing in statistics reflected the "writing across the curriculum" movement in colleges and universities. Instructors in all programs were required to incorporate writing into their courses. In statistics classes, the requirement was often met by adding a writing component to the course. The component typically asked students to write a paper about a topic learned in class, complete a data analysis project, summarize a method, or respond to essay items on an exam (e.g., Iversen, 1991; Peterson, 1988). Writing was not integrated throughout the course, but rather it was an "add-on". Asking students to write about statistics in a paper, on an exam, or in an assignment, does not automatically increase their understanding or improve their communication skills.

Altering the structure of a statistics course is one way that some instructors have successfully incorporated writing. Spurrier (1997, 2000) describes a one-credit capstone course for statistics majors. Students are placed in the role of a statistician and are asked to solve problems for companies or organizations. The capstone experiences are designed to make connections among research and statistical topics. Students make decisions based on statistical and mathematical results, prepare a written report that summarizes the analysis and makes recommendations to the company or organization, and prepare a 5-minute oral presentation that addresses items in the written report.

For post-residency physicians, Samsa and Oddone (1994) organized a statistics course into modules. The first class in each module uses a lecture format and describes the elements that should be included in students’ statistical presentations of the topic (e.g., confidence intervals, test statistic, and p-value). The second class uses a discussion format in which students present data analyses and journal article critiques. A series of guidelines are also available for reporting statistical results in medical journal articles (Bailar & Mosteller, 1992). One guideline says to provide enough detail for the statistical methods so that if someone had access to the data they could replicate the study’s results. 

At an even broader level, writing was incorporated into an entire teacher training program for all PhD students in an arts and science graduate school (Hertzberg, Clark, & Brogan, 2000). The intent of the program is to prepare future statistics teachers.  Students receive formal instruction in pedagogy through two courses. The first course uses writing as a pedagogical tool, and a second course focuses on pedagogy in the students’ specific discipline area (e.g., biostatistics). Later in the program, students become teaching associates and practice their communication skills in statistical consultation settings.

The above illustrations are from statistics courses or programs in which students have strong mathematics or statistics backgrounds. Instructors of statistics courses for undergraduate or graduate students in social sciences and education face a different situation when they try to incorporate writing into their classes. These students do not usually have strong math backgrounds. They often lack confidence in their quantitative abilities, view statistics as irrelevant to their career, and have a high degree of anxiety towards statistics and research (Onwuegbuzie, 1997).  Stromberg and Ramanathan (1996) identify five reasons why students in introductory statistics classes have difficulty when writing about statistics: 1) lack of understanding of material, 2) unfamiliar with technical writing, 3) unable to develop cogent arguments from facts, 4) instructions not followed, and 5) multiple drafts not written. In order to improve students’ writing ability, they incorporate journal writing and use note cards with one-sentence answers to various questions. They also ask students to find a survey in a newspaper or magazine and write a letter to the editor discussing the positive and/or negative aspects of the survey. Students evaluate each other’s first drafts in terms of the organization, sentence structure, word choice, mechanics and grammar. Each component is graded as 0 or 1. Data showed that the final grades for this project improved when peer evaluations were used.

Another example from an introductory statistics course describes several projects organized around one set of data (Holcomb and Ruffer, 2000). For each project, students work in groups to analyze data and respond to a series of questions. A 100-point analytic rubric is used to evaluate the groups’ projects. Mathematical and statistical accuracy accounts for 50 points, and the quality of writing (clarity, organizational layout, mechanics, thoroughness, spelling/punctuation, and professionalism/style) accounts for the other 50 points. The rubric is shared with students so that they know what is expected of them.

Finally, two excellent books were recently published on writing about statistics (Miller, 2004, 2005). As Miller explains, they are not intended to be statistics textbooks nor writing manuals, but instead they describe how to communicate numerical results so that readers can understand how they answer a question. "A naked number sitting alone and uninterpreted is unlikely to accomplish its purpose." (p. 5) The first section of the 2004 book presents a series of principles for writing about numbers, such as setting the context, stating statistical significance, using examples, and defining terminology. The second section contains suggestions on the numbers to report, use of conventions and standards, creation of tables and graphs, and incorporation of good examples. The third section of the book focuses on writing reports for scientific and non-scientific audiences. The second book, organized in the same manner, is about writing in advanced statistical topics (Miller, 2005).

1.2 The Value of Incorporating Communication into Graduate Statistics Classes in Education

The purpose of this article is to describe an approach to teaching statistics that emphasizes communication and reasoning. The course is for graduate students in education who are teachers, school administrators, counselors, or school psychologists. These students are frequently asked to write papers throughout their coursework, but they have limited experience in writing about quantitative results. A typical student’s first impression of a statistics course is that it primarily involves computational and procedural knowledge. A commonly held belief by students is that once a correct numerical answer is obtained or after a correct statistical decision is made (such as rejecting or not rejecting a null hypothesis), then their work is finished. Moreover, when students are drafting their dissertation chapters, they may be quite eloquent in their description of the study and in synthesizing related literature, but they struggle with how to explain the statistical procedures they used to answer the research questions and how to communicate their interpretations of results to others.

For nearly two decades, the mathematics education community has recognized that communication is essential to the learning, understanding, and doing of mathematics. In fact, communication is one of the standards described in publications of the National Council of Teachers of Mathematics (NCTM, 1989; 2000). In the Principles and Standards for School Mathematics (2000), the Communication standard states that instructional programs should enable students to: "organize and consolidate their mathematical thinking through communication; communicate their mathematical thinking coherently and clearly to peers, teachers, and others; analyze and evaluate the mathematical thinking and strategies of others; and use the language of mathematics to express mathematical ideas precisely" (NCTM, 2000, p. 60).

Although the NCTM documents were developed for K through 12 educators, the benefits of incorporating verbal, written, and graphical forms of communication into instruction and assessment are the same for educators at all levels. Most importantly, it forces students to go beyond the numerical answers and focus on the meaning of the results. Once students begin to verbalize their thoughts and reasonings, the misconceptions and confusions are brought to light and can be addressed by the instructor. With frequent opportunities to write and talk about statistics, students’ conceptual understanding is ultimately improved.

When students share their interpretations of data analysis with each other, they realize there are multiple ways to describe and present results. Each student’s interpretation is different. There are inaccurate ways to describe results, but there is no one correct way. Radke-Sharpe (1991) describes this benefit as creativity. Reading published research also allows students to see the many different ways of displaying and writing about results. 

Learning how to communicate will help educational practitioners in their future careers as they are called upon to interpret results and share them with a variety of audiences in their school districts, including parents, teachers, counselors, administrators, and community leaders. Numerous quantitative reports come across their desks, and the ability to understand what is in those reports, evaluate the accuracy of results and conclusions, and communicate them to others is an essential skill. 

Finally, by fostering a classroom environment that encourages students to express their statistical thinking, reasoning, and interpretations in a variety of forms and contexts, students’ confidence levels are increased and anxiety is reduced. A large literature base has formed over the years surrounding statistical anxiety (e.g., Bell, 2001; Onwuegbuzie, DaRos, & Ryan, 1997; Zeidner, 1991). As an example, Onwuegbuzie (1997) delineated four dimensions of statistical anxiety that graduate students may experience: 1) perceived lack of relevance of statistics to their future careers, 2) fear of statistical language, 3) fear of application of statistics knowledge, and 4) interpersonal anxiety, which is the fear students experience when they have to ask for help from peers or the instructor. In a class in which the norm is to communicate statistically by speaking, writing, reading, and listening, students’ fears may be eased, and thus their confidence in their ability is improved.

A mathematics educator made the following reflection when she was incorporating mathematical discourse into her lessons.

"When I started asking students to explain themselves more often, they were uncomfortable. They hesitated, sometimes so long that I did not know whether to keep waiting patiently or to let them off the hook. I remember many times telling them, ‘I’m not saying you are wrong, we’re just interested in your thinking,’ but this statement did not seem to reassure them very much. Quite a bit of time and perseverance were needed to get the students to accept this habit (Van Zoest & Enyart, 1998, p. 153)."

Fortunately, the outcomes are usually worth the initial time and effort. As the teacher further reflects,

"When they realized that explaining was not frightening after all, so many students wanted to explain…that we could barely accommodate all the sharing they wanted to do. ...When we spent more time discussing thinking, an added bonus was that students were more willing to let me know when they could not understand something. The stigma of ‘not getting it’ was lessened (Van Zoest & Enyart, 1998, p. 153)." 

A middle school math teacher made this reflection about the students in her classroom. College and university educators who are encouraging communication in their statistics classrooms may have similar sentiments.

1.3 Overview of Paper

The first purpose of this paper is to describe a systematic, integrated approach to incorporating statistical communication throughout an intermediate statistics course in graduate education. Students in the course are in the educational administration, instructional leadership, counseling education, or school psychology doctoral programs. The majority has limited mathematics and statistics backgrounds. Most have full- or part-time employment as teachers, principals, counselors, or school psychologists. In their future careers as practitioners who must interpret and communicate results or as researchers who conduct their own studies, it is essential that these students have strong statistical reasoning skills, solid conceptual understandings, and the ability to make accurate and complete interpretations of results.

The instructional approach was designed with the overarching goal to improve students’ verbal and written communication in the language of statistics. Two additional goals were to enhance students’ statistical reasoning and conceptual understanding and to increase their level of confidence in doing statistics and talking about it with others.  The framework for developing the approach was guided in part by NCTM’s Communication standard and the description of what students should be able to do (2000).

This paper adds to the body of literature on statistical writing in several ways. The approach is broad. Students are communicating verbally and in writing throughout the entire course, not just on one or two papers or projects. Detailed illustrations are provided to describe what the instructor is doing, what the students are doing, and the intent is to paint a picture of the classroom environment. For each statistical topic, students see a variety of written examples that interpret results. Examples are from previous students or from published research articles. They vary in their quality and use of text, tables, graphs, and other forms of representation. Students also receive a great deal of feedback from the instructor and from their peers. The instructor gives individual feedback on written communication. Students are able to practice their verbal communication skills weekly through whole-class discussions and small-group activities with their peers. After students’ become more familiar with communicating statistically, they participate in a scoring activity in which they evaluate sample paragraphs of results. They score each paragraph as low, medium, or high quality based on rubric criteria they develop. For each paragraph, they provide a rationale for why they assigned the particular score. Finally, to show evidence of the successfulness of the approach, quantitative and qualitative data is provided.

2. Instructional Approach

Four areas describe how the approach integrates statistical communication throughout the course: 1) weekly individual assignments, 2) in-class, small-group activities, 3) use of published journal articles, and 4) a "scoring" activity where students develop rubrics and evaluate paragraphs of results. Detailed illustrations are given within each area, along with the benefits they provide to students and teachers.

2.1 Individual Assignments

Instruction focuses on the importance of including statistical evidence to support statements made about results, writing conclusions that refer back to the original research questions, and communicating results in a way that they can be easily understood. Weekly assignments give students the opportunity to practice these skills and receive feedback on their communication.

Because most students come to the course not used to writing in a "math" class, the first few assignments are more structured than the later assignments. Early assignments ask them to state the independent and dependent variables, indicate the null hypothesis, summarize the descriptive statistics, state the F-values and degrees of freedom (df), state the statistical decision regarding the null hypothesis, and write a one-sentence conclusion about the results of the study. After they become familiar with writing statistically in their own words, the prompts on the assignment become more open-ended. For example, "write a paragraph or two that describes and interprets your statistical analyses as though you are writing in the results section of a research journal article" or "write a paragraph as though you are explaining the results to someone who has a limited amount of statistical knowledge."

2.1.1 Stating Conclusions

This illustration is from an assignment early in the course that asked students to conduct an analysis of variance (ANOVA) on data from a study examining the effects of violence in television on the aggression of viewers. There were three groups of participants. All groups watched a 30-minute segment of a television program, but each group’s segment had a different number of violent scenes. After viewing the segment, participants took an aggression inventory.

Discussion of the completed assignment the following week focuses on the one-sentence conclusions. Students received written, individual feedback on their own conclusions, and now they have an opportunity to see other examples of written work. Using previous students’ samples, I organize a wide variety of sentences into categories. We discuss each category, and students can see where their own sentence fits.

One category is labeled as "too general". These statements could apply to any study that shows a statistically significant difference among means (e.g., "At least one pair of groups is significantly different" or "The null hypothesis of equality of means was rejected."). Another category is "incomplete/unclear" (e.g., "The groups showed different reactions in aggression to the independent variable"). This statement does not describe the independent variable. A third category is "incorrect" (e.g., "There is no significant difference between violence and aggression".) This is a common misconception of students who are beginning to learn about inferential statistics. Although they understand that an ANOVA tests for significant differences, they incorrectly believe that it examines the difference between an independent and dependent variable rather than a difference among means for levels of an independent variable.

Examples of "good" statements are shown below. It is helpful to provide students with several in order to demonstrate that there is more than one way to write a complete and correct conclusion.  All the statements below specifically mention the independent and dependent variables and correctly state that a significant difference was found. Although all of the statements fail to include the word "mean" when referring to a significant difference, the last statement is the most complete.

There is at least one significant difference in aggression scores among groups who were exposed to different amounts of violence in a movie.

The amount of violence viewed does have a significant effect on the amount of aggression displayed.

There is at least one significant difference in aggression scores across groups that watched a movie with differing amounts of violent scenes.

The number of violent scenes during a 30-minute television segment has a significant effect on the amount of fantasy aggression displayed as shown on the Thematic Apperception Test.

2.1.2 Editing Conclusions to Improve Their Quality

A few weeks into the course, students begin to examine lengthier conclusions. In a homework assignment on post hoc testing, students analyzed tax return data. The dependent variable was the efficiency of processing tax returns, defined as the number of days between receipt of the tax return and final processing. The independent variable was the region where the tax-processing center was located: east, mid-west, south, and west. One question on the assignment asked students to "write a few sentences to describe the results of the Tukey test. After receiving individual feedback on their own paragraphs, students were given several complete and correct paragraphs written by previous students. Two of them are shown below:

Tukey post hoc analysis, using a significance level of .05, revealed one significant different between the centers. The regional tax-processing center in the East took more days (mean=54.9, sd=9.8) to complete final processing of tax returns than the tax-processing center in the West (mean=44.7, sd=7.6). There were no significant differences between the mean length of time required for processing between east and mid-west, east and south, mid-west and south, mid-west and west, and south and west.

Follow-up Tukey analyses indicated that the Western tax-processing centers were significantly more efficient than the Eastern processing centers in terms of days needed to process tax returns (p<.05). The mean numbers of days for the East and West centers were 54.9 (sd=9.8) and 44.7 (7.6), respectively. Results revealed no significance for the remaining five pairwise comparisons of regions. 

Both paragraphs explicitly describe the dependent variable, indicate the direction of the significant difference (East had a higher mean number of days than the West), include the sample means and standard deviations, state the significance level used, and account for all unique pair-wise comparisons tested in the Tukey analyses by stating that the remaining pairs were not significant.

Next, students were given examples of incomplete or incorrect paragraphs and asked to describe how they can be improved. For example, the following paragraph correctly indicates the significant difference between the East and West centers and correctly states the dependent variable. However, it does not indicate which region was more efficient nor mention the other regional comparisons. Students said that a person reading this paragraph might assume that the other pairs were not significantly different, but it should be stated explicitly.

A post hoc test was performed on the data. Results indicate that there is a significant difference between 2 of the groups in terms of efficiency of tax return processing. The East and West groups were significantly different at the .05 level.

Another paragraph, below, does not mention the dependent variable. Students said a reader would not know what was being compared. Also, they indicated that one of the main problems with the summary is that it only describes the statistically based decision regarding the null hypothesis. One student said, "...it does not state what the rejection and non-rejection of the hypotheses means in terms of the variables in the study." In essence, there was no real conclusion. A correct aspect noted by students, however, was that all pair-wise comparisons were addressed.

Using the Tukey, there is one significant pair-wise comparison between the East and the West. Therefore, the null hypothesis should be rejected. The results of the study revealed that we failed to reject the null hypothesis for the pair-wise comparisons between East and Midwest, East and South, Midwest and South, Midwest and West, and West and South.

Through the individual assignments and the discussions that follow in class, students become familiar with the elements that constitute a complete and correct interpretation. As they edit the paragraphs to improve their quality, students learn how to be precise when communicating. This process encourages self-assessment and reflection of their own work. A benefit for instructors is that it gives them a more accurate perspective of what students know, and do not know, about a statistical concept. They are made aware of misconceptions that often remain hidden when only a numerical answer or statistical decision is required.

2.1.3 Making and Testing Predictions

The individual assignments are not only used for conducting analyses and writing summaries of results. Students are also presented with cognitively rich questions and problems that require them to think further about the problem situation. For example, in the violence and aggression assignment described in Section 2.1.1, students were asked to predict what would happen if two aggression scores were removed and the analysis was conducted again. To test their prediction, they obtained new output, identified the values that changed, and described why.

Students noticed that the new group means were closer together, and they also recognized that the Mean Square between (MSb) and error (MSe) changed. Most students correctly described that MSb was smaller because there was less variation among group means. However, some students were not sure why MSe was also smaller. The homework sparked an interesting discussion during the following class session when students shared their reasons about why MSe changed. One student explained that, "there was less variance in aggression scores within each of the two groups that had scores removed." Someone also noted that the degrees of freedom for error and total were smaller. The class talked about how these changes would have a bigger impact on the results of a small data set versus a larger one. A similar conversation took place with regard to the magnitude of the values deleted from the data set.

This question on the assignment encourages students to think more deeply about the relationships between the elements in an ANOVA summary table. By doing so, they develop a conceptual understanding of the F-statistic and the crucial role of variance in inferential statistics. At the beginning of the course, not all students are able to write or verbalize their thoughts. However, given repeated opportunities through weekly assignments and whole-class dialogue about them, students gradually develop the ability to communicate their reasoning and thinking. 

2.2 In-class Activities

During each class, students participate in small group activities that give them immediate practice in embedding the newly learned topics into their minds and stimulate statistical discussion. They begin to develop ways of verbally communicating with each other that feel comfortable to them. Students who are less comfortable contributing to a statistical discussion in a whole-class setting especially benefit from the chance to ask questions and get help in a small-group environment of their peers. Written communication skills are also enhanced because the group must decide how to clearly and accurately record, in writing, a summary of their group’s discussion. A benefit to the instructor is that it creates time in the class session to interact with students on an individual and small-group level, to give immediate feedback, and to determine students’ strengths and weaknesses.

A variety of scenarios and problem situations are used. Some activities require them to compare various data situations and the statistical procedures that can be performed. Other activities allow students to make connections among ideas learned in previous class sessions or make sense of numbers provided in output from statistical software. We talk about how the software rarely gives "errors" and that it is the responsibility of the user to select the appropriate analysis to answer the research question and determine whether the variables have the appropriate characteristics.  Below are two illustrations of in-class activities.

2.2.1 Making Decisions about Appropriate Data Analysis

In an activity following instruction on the homogeneity of variance assumption, students compared three data scenarios. Each scenario had three levels of an independent variable, but the sample sizes and standard deviations differed. Students first used the Fmax test to evaluate the assumption for each scenario. Then, they were asked to determine whether the validity of the ANOVA results was in jeopardy for those scenarios in which the assumption was violated. In one scenario, students stated that because the group sample sizes were approximately equal (the ratio for the largest versus smallest group size was 1.12), the F result was robust to violation of the assumption. In the other scenario, students considered the group sizes to be unequal, and because the group with the largest sample variance had the smallest sample size, they said the F-test result would be liberal. This meant that if the null hypothesis for an ANOVA were rejected, there would be a concern about the validity of results because of the possibility of a Type I error.

The final question asked students to describe the possible alternative plans of action if violation of the assumption was of concern. Alternatives suggested by students included using a nonparametric test such as Kruskal-Wallis, using a parametric ANOVA with a more stringent significance level such as .001, or applying a variance stabilizing transformation such as square root or logarithmic.

2.2.2 Comparing Results Across Several Data Sets

This activity takes place in a computer lab using a data set with age and cholesterol levels for 200 subjects. It is referred to as the "original" data set and serves as a point of reference for subsequent analyses conducted on modified versions of the data. Using technology during instruction is beneficial for several reasons. It allows for exploration and discovery. Because statistical software alleviates the need to perform calculations repeatedly, data values can easily be added, deleted, or modified. Instruction then focuses on having students speculate about the changes that occur in the results and the reasons why they occur. The intent is not to encourage students to modify data in order to get better results, but rather to see how changes to data affect the overall results. It helps students understand the formulas and the relationships among the elements, thus developing a deeper understanding of the concepts. These activities also encourage meaningful student interaction with peers and with their instructor. As instruction progresses, students tend to pose their own "what if" questions.

First, students used SPSS to conduct a simple linear regression on the "original" data set. Means and standard deviations for each variable, regression results, case-wise diagnostics to identify potential outliers, and a scatter plot with the regression line were produced. Students were also asked to save the standardized and un-standardized residuals to the data file. In pairs, they talked about the information that each number in the output provided. For example, there was a moderate correlation between cholesterol and age (r = .411 (p < .001). Regression results showed that age was a significant predictor of cholesterol (F_(1,198)= 40.31, p<.001), with 16.9% of the variance in cholesterol accounted for by age. One subject with a standardized residual of 4.051 was identified as an outlier. The actual cholesterol value was 520 compared to the predicted value of 279.

Then students were asked to predict how the values in the output would change if the outlier was removed. They tested their prediction by modifying the "original" data set to remove the outlier. Most students accurately predicted that the correlation and R² would increase and that the standard error would be smaller. Results showed that r=.430 and R²=.185. Some students were surprised that the results only improved slightly. They talked about why this was the case. One student said that although the standardized residual was above 3.00, it was not "that much above it". She speculated that "a bigger difference in r and R² would be found if the outlier had a standardized residual of say 9.00 or 10.00". Another student said that there were 200 cases and only 1 was removed. "It would have a bigger impact if the total sample size was smaller, for example, 50, or if more outliers were found and removed."

Next, students were asked to modify the "original" data set by removing a case with a cholesterol value close to the mean of 283.93. The changes in results were negligible, but during the discussion, one group noticed that their output differed slightly from other groups. They realized that two subjects had the same cholesterol level of 283, but different ages. A student explained that the regression equation produces a different predicted cholesterol value for each age. Based on the difference in the groups’ results, students determined that the cholesterol level for the 45-year old subject was more accurately predicted than the 52-year old subject. They double-checked this by examining the residuals and found that the younger subject had a lower residual than the older subject.

At this point, students began posing their own questions and modifying the data set to answer them. For example, a few groups wanted to remove subjects with the highest and lowest cholesterol levels. One group removed subjects with the five highest levels, another group removed subjects with the lowest five levels, and a third group removed all ten of these subjects. When comparing results, they discovered that all three analyses resulted in smaller r and R²values compared to the original results of .411 and .169. A student then said that, "it doesn’t necessarily matter if a subject has a high or low cholesterol level compared to the rest of the data set. I think what matters more is how accurate the predicted value is. Let’s try removing the ten largest absolute values of the residuals." When they did so, the results showed an improvement in the prediction and amount of variance explained (r=.477 and R²=.228).

Students became increasingly engaged as the activity progressed. In part, this was due to how the activity unfolded. There was a certain degree of systematic progression. It is helpful to have in mind the direction you want to take the activity and the modifications you want students to make. At the beginning, students were told how to change the data set. However, as they became more involved in the activity, they were given the freedom to decide the avenues they wanted to explore.

This activity was used again in a later class session on multiple regression. The statistical software produces an abundance of measures to examine the regression model. Residuals plots are used to check model assumptions and a variety of measures are used to evaluate outliers and influential data points (e.g., hat elements, Mahalanobis’ distance, Cook’s distance, DFFITS, DFBETAS). The purpose of the activity was not to encourage students to modify data to get a preconceived fit. Instead, the environment was one of posing and answering questions to gain a more meaningful understanding of evaluating a regression model rather than simply memorizing the available measures.

2.3 Examining Journal Articles

2.3.1 Narrative and Visual Representations of Results

Typically, near the end of instruction on each concept, students are given several excerpts from journal articles that show how results can be represented in narrative paragraphs, tables, graphs, and other visual displays. Students examine each representation to determine what details are intended to be conveyed to readers of the article. Students gain practice interpreting results, and it serves as another method of reinforcing complete and correct ways to communicate. It also provides them with a multitude of tools to use when presenting their own research.

As an example, there are many ways to display the outcomes of post hoc analysis. One common method is called the "triangular table format". It is similar in structure to a correlation matrix, but the rows and columns represent groups of the independent variable. Mean differences and p-values are usually provided in the cells of the table. This format allows for results of all possible pair-wise comparisons to be displayed to the reader in an efficient manner. Another graphical way to display post hoc results is by using vertical bars where each bar is a group and the height of the bar represents the mean (Chollet, Delaplace, Pelayo, & Tourny, 1997).

Post hoc results are also displayed within tables of means and standard deviations by using letters. If the same letter is attached to any two means, then the means are not significantly different (e.g., Weger & Polcar, 2002). Other tables use an underlining method instead of letters, to show nonsignificance (e.g., Gerstein & Duffey, 1992). A line is drawn under means that are not significantly different. Finally, tables might have a column that uses "greater than" and "less than" signs to show which groups are significantly different.  This table is especially useful when multiple dependent variables are analyzed (e.g., Miller, Greif, & Smith, 2003).

2.3.2 Journal Articles

Approximately twice during the semester, students read and reflect on a published quantitative article in a research or practitioner journal. They are familiar with critiquing articles in other education courses, but the focus is not usually on the methodology and results. To organize students’ reflections, they are given several prompts. Some relate to research design issues, while others relate to the statistical procedures and analysis. Examples of the latter are: Identify the variables in the study and describe what the author’s data file might look like (i.e., how many columns are needed, what is the scale of measurement for each variable). How many statistical tests were conducted (e.g., how many ANOVAs were run)?  Evaluate the strengths and weaknesses of the results section. What do you like about the way the author described the results? What didn’t you like? Were the authors’ conclusions supported from the statistical evidence?  Are rival hypothesis plausible and did the author provide them? What would you most like to ask the author about the results if you had the opportunity?

One article that I often use encourages rich discussions around several statistical concepts. Cullerton-Sen and Crick (2005) conducted research to examine teacher, peer, and self-reports of physical and relational peer victimization for fourth grade boys and girls. The first set of analyses examines relationships among the multiple perspectives of victimization. Correlation coefficients between teacher, peer, and child reports of victimization are obtained and Fisher’s z statistic is used to determine the level of cross-informant agreement for each type of victimization. The second aspect of the article utilizes one-between and one-within repeated measures ANOVA to examine gender differences in children’s experiences of relational and physical forms of victimization. Third, the article utilizes regression to assess the unique information provided by teacher reports of relational victimization in the prediction of adjustment.

This study incorporates many variables and analyses, but the authors do an excellent job of clearly and succinctly presenting the results in narrative form and tables. When students first read the article, they may be overwhelmed, but as we break it down into sections and talk about it over the semester focusing on one aspect at a time, they are better able to understand the analyses. Students have reported that it is helpful to see the concepts applied in a real research study. It also helps them to appreciate the amount of work that goes into producing an article of high quality. By discussing full articles, students not only gain exposure to the ways in which researchers and practitioners summarize and interpret their results for various outlets, but they also gain a better appreciation for how the statistical analysis fits into the broader context of a study.

2.3.3 Student-Guided Discussion of Articles

Once per semester, students select an article that utilizes a statistical concept covered in class. Prior to the class session, each student is required to: 1) make the article available to the other group members electronically through the university library, 2) create a list of discussion questions, and 3) read the articles from other group members. On the day of class, each student is responsible for facilitating his/her own article’s group discussion using the questions he/she created.

There are numerous benefits to this activity. Students can choose an article that is of interest to them. The small group format encourages student participation. Students feel a sense of ownership when they are responsible for guiding group discussion. Finally, a class session of two to three hours in length allows sufficient time to have a thorough discussion of each article in the group and compare how the same statistical procedure was applied in different research studies. 

2.4 Scoring Activity

In this activity, students create rubrics to score other students’ interpretations of results. Typically, it is used about two-thirds into the semester after students have had experience writing their own summaries of data (as described in Sections 2.1.1 and 2.1.2). As students evaluate the work of others they are crystallizing their ideas about the desirable qualities of good written communication, and it deepens self-reflections on their own writing. The steps in creating and implementing the activity are given below.

Step 1: First, an assignment is developed in which students conduct an analysis on a data set, obtain output, and write paragraphs of results. The activity works well with any statistical analysis, and the data set can be simple or complex. The assignment illustrated here involves an analysis of covariance (ANCOVA). The data is relatively straightforward, consisting of an independent variable (type of instruction: computer-assisted or not computer-assisted), a dependent variable (assessment score after two months), and a covariate (aptitude).

There are three parts to the assignment: 1) conduct a one-factor ANOVA to determine if instruction influences performance and describe results in a paragraph, 2) conduct a one-factor ANCOVA using aptitude as the covariate and describe results in a paragraph, and 3) compare the results from the two analyses and explain why they did, or did not, lead to the same conclusion. The instructions regarding the paragraphs can be general ("write a summary as though you were writing for a journal article") or specific ("your summary should include the sample means that are being compared, the F value, df, p-value, the decision regarding the statistical hypothesis being tested, and the conclusion based on the variables in the study.") depending on students’ prior experiences in the course.

Step 2: Prepare a set of 10 to 15 paragraphs written by students from previous classes. The illustration shown here includes 11 paragraphs from part 2 of the assignment (see Appendix A). It is important that the paragraphs vary in terms of numerical accuracy, level of reasoning, quality of interpretive statements, use of terminology, accuracy of conclusions, clarity, completeness, and length.

Step 3: When students bring the completed assignment to class, discuss the results with them. This is to ensure that all students have the correct output and an accurate understanding of the ANOVA and ANCOVA results before they move on to evaluating other students’ work.

Step 4: Provide each student with the set of paragraphs. They should individually read each paragraph first.  This allows them to write their own thoughts about the quality of the paragraphs before discussing them with their group.

Step 5: Provide each group with a blank scoring sheet and rubric form as shown in Figure 1. First, ask them to sort the paragraphs into three levels based on the overall quality: low, medium, or high. More levels could be used, but if this is students’ initial experience in developing criteria, three levels seem to work best. Also, it helps to have each paragraph on a separate piece of paper so that students can physically put them into groups and shuffle them during their conversations. After reaching group consensus on a score for each paragraph, the group records the score on the scoring sheet and writes a rationale for why they assigned that score. They also record the scoring criteria they created for each level.

Step 6: After scoring is complete, tally the group scores for each paragraph. The whole-class discussion can begin by focusing on paragraphs that were assigned the same score by all groups. Students can give their rationales for the score and talk about why the response so clearly fit into one of the levels. Then, the focus can turn to the paragraphs with different scores. This is the point where the discussion becomes quite animated as groups support their score and defend their rubric criteria. (see Section 3.2 for a summary of how this activity played out in the classroom).

Figure 1.  Example of scoring sheet (top) and rubric form (bottom) used in the scoring activity.

3. Impact on Student Learning and Communication

This portion of the paper examines qualitative and quantitative data collected from doctoral students in school psychology and educational leadership/administration programs in order to describe the impacts of the instructional approach on student learning and determine if the initial goals were realized. The data are organized into four areas of evidence: 1) individual student growth over the semester, 2) impact of the "scoring" activity, 3) comparison of interpretive summaries, and 4) student perceptions and confidence.

3.1 Individual Student Growth

All individual assignments and in-class group activities were collected to examine students’ solution strategies, explanations, and interpretations in terms of how they progressed over the semester. In addition, two class sessions were videotaped and transcribed in order to examine dialogue among students during the scoring activity, in-class activities, and whole-class discussions.

Students across the ability continuum appeared to benefit from the instructional approach. The first example is Linda. Her work on the first few individual assignments displayed a lack of conceptual understanding and poor communication skills. In whole-class discussions, she rarely participated except when called upon.  In small group work on the activities, she preferred to be the "recorder", rarely making a contribution other than to say, "what should I write?" Over time, there was a noticeable change in Linda’s performance and confidence level. Her work showed a better grasp of the underlying meaning of concepts, and although her written interpretations were sometimes unclear and awkward, there were fewer errors. She remained hesitant during whole-class discussions, but participated more in her small group. She began to feel comfortable describing what she did not understand and sought feedback from her peers. As I walked into the room before class one evening, I heard her asking a classmate a high-level statistical question about the way in which the group means are adjusted along the regression line when a covariate is in the model. The fact that she was able to communicate the question in a meaningful way, and felt comfortable asking a peer, demonstrated her growth over the course of the semester.

The second example is Jackie. From the beginning of the course, her assignments showed relatively few errors or misconceptions. Explaining answers in written form appeared natural for her. She participated often in whole-class discussions and was actively engaged in her small group. Over time, her verbal communication during whole-class discussions continued to increase. She often responded to questions from other students, which led to meaningful and interactive dialogue. Jackie demonstrated a sense of confidence in her abilities and a willingness to help others understand. Other students often looked to her for help in reasoning and understanding the concepts. She developed a level of statistical maturity that is gratifying to see as an instructor.

3.2 Scoring Activity

The scoring activity took place during a 2 hour and 40 minute class session and was videotaped. Student-created rubrics, scores for each paragraph, and written rationales for each score were collected from the groups. Individual student reflections about the activity were also obtained.

Student engagement during the scoring activity was extremely high and group discussions were cognitively intense. Each group developed its own unique approach to evaluate the paragraphs for correctness and completeness. Some groups evaluated all numerical values first, and then examined the wording. Other groups used a sentence-by-sentence and word-by-word approach. One group had a lengthy discussion of Rhonda’s paragraph (see Appendix A). They eventually came to a consensus that the paragraph was a "medium", and their written rationale on the scoring sheet was that the "word choices were often poor, grammar was not great, thoughts could be broken down and more succinct, and the phrasing was not always correct. But all the numerical test results were correct, except for the degrees of freedom." They methodically talked about each sentence. They agreed that the information in the first sentence was mostly accurate, but the wording was awkward and unclear. A student said, "This sentence is crazy. She’s trying to bring too much information into one sentence, she needs to break it down." They made corrections to individual words, such as changing the word "covariance" to "covariate" in the first sentence. The group also did not like the phrasing of the next to last sentence in Rhonda’s paragraph that said, "there were statistically significant differences among groups to begin with." They discussed various edits of that sentence to make it accurate and clear. The following are excerpts from the general criteria this group developed and applied to score the paragraphs.

• Low: incorrect data, important values are omitted, interpretation of data or concepts is confusing, failure to explain or describe variables, incorrect conclusions;
• medium:  vague but correct interpretation, either too wordy or needs further elaboration, minor numerical reporting errors, failure to completely articulate variables;
• high: reported all data correctly, accurately interpreted results, clearly articulated results, correct conclusions.

Discussions were most lively when students disagreed with each other about a score and had to defend their position. A few brief comments from the transcribed discussions are:

• How did you give Hank a ‘medium’, when you gave Lisa a ‘medium’? Hank’s paragraph was much worse.
• Why did he use the words ‘treatment effect’? I think a treatment effect is like treating with a drug or an instruction method. That’s not what the study is about.
• You can’t say there’s a significant difference between aptitude and assessment. That’s not what you’re testing. One is a covariate, the other is a dependent variable. They should report the "r" value to show the relationship.
• They didn’t label and describe the groups consistently.
• He reports original means instead of the adjusted means based on the covariate.
• These sentences are jumbled and misleading.
• Gosh, it’s so wordy that you get lost. That’s probably me (laughs). I have a lot of trouble being succinct.
• This is really hard. Now I know what it’s like for Dr. Parke when she grades our work and gives feedback.

After groups finished, a class tally was taken to compare scores. Conversations across groups were meaningful and mathematically rich. Students shared their rubric criteria with each other and enthusiastically debated scores for each paragraph. They talked about what constituted a high level paragraph and eventually created a sample "complete and correct" paragraph (see Appendix A).

At the end of the course, students were asked to provide a written reflection on the activity and describe the types of paragraphs that caused the most animated discussion in their group, 73% of students indicated that the ‘lows’ led to good discussions because of the errors in numerical values and the inappropriate interpretations. Other students said both ‘lows’ and ‘highs’ were discussed at length: "The examples that were poor generally led to a discussion of everything that was missing, and the examples that were considered good could be used as models to compare our own work to." A few students said they could have used more than three score levels because it was often difficult to choose between a ‘medium’ or ‘high’. They said, "it wasn’t like, ‘these paragraphs are all high and they are all the same’, There are high ‘highs’ and low ‘highs’."

Finally, students were asked to reflect on the ways in which the activity contributed to their learning experience in the course. All students responded positively. Here are a few examples:

• The scoring activity allowed me to see flaws in my own communication in relation to interpreting statistical results. By observing how others interpreted results, it deepened my awareness of how important it is to understand what is being asked in the research question.
• It helped me become more aware of my own writing style and allowed me to self-evaluate.
• The scoring activity really helped me see what sort of explanation was needed to convey all the necessary information for people to understand what you were saying. When reading the examples, I thought about the types of written output I had been providing and learned how to better communicate my understanding.
• It’s beneficial to see the right and wrong ways to put statistical data and results into words.  This is definitely a very important aspect of this class.
• We were able to take what we learned and actually apply it. In rating the student responses we needed to determine if the student understood the statistical procedure and understood the results. We also rated how well the student was able to communicate his understanding.
• I enjoyed the activity because it allowed me to examine more concrete examples of student interpretation. It also allowed me to utilize feedback that I had gotten and apply it.
• I benefit greatly from the simple act of having others around to ‘talk statistics’. The language becomes more fluent, contributing to an overall understanding.

3.3 Comparison of Interpretive Summaries

An opportunity arose to compare interpretive summaries from students who experienced the instructional approach to those who did not. Data was collected in a Measurement Theory and Practice course in which both groups of students were enrolled after their statistics course. An assignment in the measurement course required students to conduct a reliability analysis on a survey that measured attitudes toward computers. Students were asked to conduct a t-test to determine if two groups of subjects (one group from University A, the other from University B) differed significantly in terms of computer attitude. They were required to interpret the t-test results and write a summary as though they were including it in a results section of an article.

It should be noted that this was not a true experimental comparison due to lack of random selection and assignment. There is also a potential confounding variable of instructor. In other words, the instructional approach cannot be separated from the instructor. However, there were many similarities across the two groups. All students were enrolled in the educational leadership doctoral program, had similar occupations, and had similar mathematics and statistics backgrounds. Both groups received the statistics course in the semester prior to the measurement course. They used the same statistics textbook and followed the same sequence of topics. The structure and length of the course was also identical.

A 5-point holistic rubric was developed to score each summary. Criteria in the rubric took into account the accuracy of numerical results, appropriate use of terminology, and the types of statistical evidence used to support the decision that computer attitudes differed significantly. Several precautions were taken to minimize bias. The instructor and an assistant with a statistics background scored the summaries. Office staff blinded the summaries to remove names and other potentially identifying information from the pages so that the raters would not know the students or the class from which they came. The rubric was developed and agreed upon by the raters. Scoring took place after the measurement course was completed and grades were assigned. Inter-rater reliability was 91%. When disagreements occurred, raters reached a consensus on the scores.

Results showed a significant difference in the mean quality of summaries across the two groups. Students who experienced the instructional approach had a higher mean score M = 4.00, sd = 1.07, n = 28) than students who did not experience the instructional approach (M = 2.67, sd = 1.22, n = 21). The value of the two-sample t-statistic is 4.05 with a P-value of 0.0002. To further examine the difference between the summaries, they were also evaluated using an analytic coding scheme to identify the presence or absence of eight elements.  The numbers in Table 1 indicate the percentages of students that included the element in their summary.