Deborah A. Curtis
San Francisco State University
University of Pittsburgh
Journal of Statistics Education v.6, n.1 (1998)
Copyright (c) 1998 by Deborah A. Curtis and Michael Harwell, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.
Key Words: Doctoral student preparation; Statistics education; Survey research.
Although numerous research studies have focused on issues related to the teaching of statistics, few studies have focused on the training of people who may become statistics teachers. The purpose of this study was to examine doctoral students' preparation in statistics in the field of education. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and asked to describe and evaluate the training of QM and non-QM doctoral students at his or her institution. The vast majority of professors indicated that most or all of the students in their QM programs received training in the "old standard" procedures -- ANOVA, multiple regression, and traditional multivariate procedures, whereas fewer than half of the professors indicated that most or all of their QM students received training in more recent procedures such as bootstrapping and multilevel models. Professors were also asked to rate the skills of their QM students in areas such as mathematical statistics and computing on a scale from "Weak" to "Strong." Most professors gave high ratings to their QM students' skills with statistical packages, but gave much more mixed ratings of their QM students' training in mathematical statistics. Nearly half of the professors thought that most of their QM students could have benefited from one or two additional statistics courses. Results are discussed in terms of training future doctoral students.
1 Much of the literature on statistics education focuses on basic issues of designing and delivering statistics courses. Thus, a good deal of attention has been paid to issues such as what topics to teach in statistics courses, how to teach specific topics in statistics, how to organize classrooms for effective instruction, which textbooks to use, which computer programs to use, and what kinds of examples to use. Substantially less attention has been paid to examining the graduate school preparation of the statistics educators themselves. This seems unfortunate, because the knowledge and skills of these statistics educators will likely have a direct impact on their effectiveness as classroom instructors, statistical consultants, and mentors. The purpose of this paper is to address this topic by taking a closer look at doctoral training in statistics in one particular academic discipline, namely the field of education.
2 Consider doctoral students who are enrolled in quantitative methods (QM) programs in the U.S. Most doctoral programs in education have QM programs, loosely defined here to include doctoral students interested in research design, educational measurement, and educational statistics. Many of the graduates of these QM doctoral programs will go on to teach introductory and advanced statistics, measurement, and research design courses in colleges and universities across the nation. Moreover, these graduates may serve as statistical educators in ways that are not related to direct classroom instruction, including serving as consultants with colleagues; serving on thesis and doctoral committees; serving on editorial boards of research journals; and training and supervising future doctoral students. Although there may be disagreement among faculty members over what should be the specific content of doctoral training programs in quantitative methods, there is substantial agreement on the importance of adequately preparing students for their future roles as teachers, researchers, and consultants. As faculty members who are responsible for training QM doctoral students in research methods and statistics and who recognize the variety of skills our graduates need, we have a special responsibility to train students in techniques that will allow them to expertly discharge their professional responsibilities.
3 Next consider doctoral students in education in the U.S. who are not enrolled in QM programs. These non-QM doctoral students in education vary widely in their areas of specialization. Typical specializations offered in U.S. universities include educational leadership, educational psychology, curriculum and instruction, special education, mathematics education, science education, bilingual education, language and literacy, and counseling. When studying the statistics preparation of doctoral students in education, we considered these non-QM students to be an important group to study because non-QM graduates who take post-secondary faculty positions in education may also play a part in the statistical education of students at their institutions, either directly, as classroom instructors, or indirectly, as advisors, mentors, and consultants.
4 This paper presents the results of a survey addressing one particular aspect of doctoral student preparation, namely, their skills and knowledge in the area of statistics. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and given a questionnaire to complete. These professors were asked to describe and evaluate the training of both QM and non-QM doctoral students at their institutions.
5 The literature on training doctoral students in educational statistics is quite sparse. A survey of statistics course content for doctoral students has not been conducted in the field of education, although studies of statistical preparation have been conducted in psychology (see Aiken, West, Sechrest, and Reno 1990) and medicine (see Dawson-Saunders, Azen, Greenberg, and Reed 1987). The lack of systematic research on what doctoral students are learning in their statistics classes is unfortunate because this information could help facilitate discussions of whether this training should be restructured and, if so, how it should be restructured. This is not to suggest that discussions of what should be taught do not take place. For example, Tukey (1980), among others, has argued that students need much more experience with exploratory data analysis in their statistics training. Similarly, Noether (1980) argued for teaching nonparametric methods in introductory statistics classes because nonparametrics can provide a strong conceptual framework for students. Brogan (1986) also suggested that nonparametrics should be taught, especially in social science and nursing disciplines. Brogan further recommended that in a required two-course statistics sequence, graduate students should learn about general linear models, including multiple linear regression, ANOVA, and possibly categorical data analysis, and should be trained to use statistical computer packages.
6 Content analyses of the statistical procedures and research designs currently being used in published research can serve as a foundation for discussions of what we should be teaching. A number of such content analyses have been performed over the years. Elmore and Woehlke (1988, 1996), for example, conducted a content analysis of statistical methods used in the American Educational Research Journal (AERJ), Educational Researcher, and Review of Educational Research for all articles published. In their 1988 publication, these researchers focused on articles published between 1978 and 1987, and in their 1996 paper, they focused on articles published between 1988 and 1995. Consider their findings for AERJ articles published between 1988 and 1995. Here, Elmore and Woehlke (1996) found that the predominant statistical method used was ANOVA/ANCOVA, which was used in 64 out of 245 articles that they coded. The next most frequent procedures were multivariate procedures, which were used in 12% of the articles, followed by bivariate correlation (12%), multiple regression/correlation (11%), t-tests (9%), structural equation modeling (9%), and nonparametrics (8%).
7 Aiken, West, Sechrest, and Reno (1990) examined the adequacy of doctoral student preparation in statistics in psychology programs. Because our research is similar to theirs, we will describe their study in detail. Aiken et al. surveyed all 222 psychology departments or schools identified by the American Psychological Association as granting doctoral degrees. From these 222 units, 186 responded, for a response rate of 84%. The goals of their research were (a) to describe the current content of the statistics, measurement, and research design courses; (b) to look at differences in requirements between sub-disciplines in psychology; and (c) to gather data on professors' perceptions of students' abilities to apply statistical and measurement techniques in their own research. In addition, these researchers distinguished between "elite" and "other" institutions, and presented data comparing the two types of institutions.
8 In terms of demographics, Aiken et al. (1990) found that 89% of the departments offered an introductory graduate statistics sequence, and of the departments offering this sequence, 77% were one year in length. A minority of institutions -- 17% -- offered doctorates in a quantitative area of specialization, although one-third of these did not have any first-year students. In terms of curriculum offerings (including optional and required courses), they found that the majority of programs offered at least a partial course (defined as at least a half a semester or full quarter) in ANOVA (88% of the programs), multiple regression (58%), and MANOVA (53%). More recent statistical topics such as structural equation models and time series models were less frequently covered (18% and 6%, respectively).
9 In terms of the content of the required introductory statistics course sequence, the authors distinguished between "old standards of statistics" and "more advanced statistical considerations," and collected information on the percent of institutions offering in-depth coverage of specific procedures. ("In-depth" was defined as coverage to the point that students could perform the analysis in question themselves). In-depth coverage of the "old standards" was provided in the majority of institutions: multifactor ANOVA, multiple comparison procedures, repeated measures via traditional factorial ANOVA, and multiple regression were covered in-depth in the required course sequence in 73%, 69%, 73% and 63% of the institutions, respectively. Topics considered more advanced were covered in-depth in required courses less often. For example, 39% of the institutions provided in-depth coverage of ANCOVA in their required course sequence, 38% provided in-depth coverage of ANOVA as a special case of regression, 20% provided in-depth coverage of exploratory data analysis, 21% provided in-depth coverage of multivariate procedures, and 18% provided in-depth coverage of statistical power analysis.
10 Aiken et al. (1990) interpreted these findings to mean that the statistical and methodological curriculum in psychology has changed little in 20 years. They also concluded that typical first-year courses only serve those students who undertake traditional laboratory research. They summarized their work by stating that new Ph.D. recipients are competent to handle traditional statistical techniques but not newer and often more useful procedures (p. 721).
11 Guo and Nitko (1996) surveyed 80 programs that offer graduate training in educational measurement. Among the 48 programs that responded and award doctorates, 19 offer training in measurement alone, 16 offer degrees in the broader area of research methodology, and 13 offer degrees in other areas. Guo and Nitko's focus was on surveying the content of courses offered to QM students and assessing agreement on what is taught. Although their focus was on educational measurement, they also presented aggregated information that reflected the frequency of coverage of various statistical topics. At the doctoral level, students in the 19 programs in measurement take 21 credits of statistics on average, and those in research methodology programs take 24 credits of statistics. The statistics courses include regression, factor analysis, multivariate analysis of variance (MANOVA), and other unspecified advanced topics.
12 Like Aiken et al. (1990), our general purpose was to survey the course-related training of doctoral students in statistics. One goal was to assess the statistical methods and procedures used in training QM and non-QM students in education. Another goal was to learn about the perceptions of QM professors regarding the competence of QM and non-QM doctoral students with various statistical techniques. This information should add to the statistical education literature on training doctoral students in QM.
13 The universities we initially targeted were educational research institutions identified in a University of Illinois study as constituting the top thirty-one Schools and Colleges of Education, ranked in terms of academic productivity and prestige (West and Rhee 1995). A list of these institutions is provided in Table 1. Our target population consisted of those universities that had QM programs, and our plan was to identify one QM professor from each institution as a contact person who would complete a questionnaire. The use of the 31 schools listed in West and Rhee (1995) as our initial sample necessarily narrows our inferences to these or similar institutions. In order to identify these contacts, several strategies were used. First, we simply identified people that we knew personally at specific universities, with the belief that these people would be most likely to complete questionnaires. Second, we used the membership roster of the American Educational Research Association (AERA) Educational Statisticians Special Interest Group. Because the initial results of this study would be presented at an AERA conference, we believed that using this membership roster would increase our chance of having the questionnaires completed and returned. Third, we used a book entitled Graduate Study in Educational and Psychological Measurement, Quantitative Psychology, and Related Fields, published by the Pennsylvania State University, which lists quantitatively-oriented programs in the United States and provides names of appropriate contact persons. Finally, in a few instances, we used personal contacts to identify a non-quantitative faculty member at a particular university who then gave us the name of a QM professor at that university. There were only two instances where we could not identify a contact person at a given university, leaving an effective sample of 29 universities.
|1. University of Wisconsin -- Madison|
|2. University of Illinois|
|3. Ohio State University|
|4. Stanford University|
|5. University of Minnesota|
|6. Indiana University -- Bloomington|
|7. Michigan State University|
|8. Columbia -- Teachers College|
|9. University of Georgia|
|10. Pennsylvania State University|
|11. University of Maryland|
|12. University of Texas -- Austin|
|13. University of Michigan|
|14. Arizona State University|
|15. University of California -- Los Angeles|
|16. University of Washington|
|17. University of California -- Berkeley|
|18. University of Chicago|
|19. Harvard University|
|20. University of Virginia|
|21. Vanderbilt University -- Peabody|
|22. University of North Carolina|
|23. University of Florida|
|24. Florida State University|
|25. Syracuse University|
|26. University of Arizona|
|27. University of Nebraska|
|28. Virginia Polytechnic|
|30. University of Missouri|
|31. University of Pittsburgh|
14 Because our goal was to survey universities with QM programs, the next step was to determine whether each university had a QM program. To do this, QM programs were broadly defined -- thus, some programs might place a heavy emphasis on measurement, whereas others might place a heavy emphasis on educational statistics. To figure out which universities had a QM program, we again used our personal knowledge of various universities. In cases where we were not sure, we contacted someone at that university. Of the 29 universities, all but two had QM programs. Thus, our final sample consisted of 27 universities. (The two Schools/Colleges without QM programs still offered doctoral training in statistics.)
15 In November 1995, all participants received a copy of the questionnaire (described below) to complete. A second mailing was conducted approximately one month later, in December 1995. In January 1996, this second mailing was followed by e-mail and/or phone calls asking that participants return their questionnaires. All questionnaires received within three months of the initial mailing were then analyzed. Twenty-three of the 27 universities with QM programs responded for a response rate of 85%. This rate is consistent with that reported by Aiken et al. (1990) and Guo and Nitko (1996). We believe that this response rate is large enough to generate credible results, although it is possible that there is some unknown but systematic difference between QM faculty members who responded to the survey and those who did not respond.
16 A questionnaire (see Appendix) was used to collect information about the statistics courses offered in QM programs and the perceptions of professors of QM. The questionnaire was divided into two sections, one directed toward students majoring in QM and one for non-QM students.
17 On the portion of the questionnaire pertaining to QM programs and training of QM doctoral students, we collected the following data:
18 The development of the questionnaire addressing the training of non-QM students was more challenging because it needed to be sufficiently flexible to accommodate the various ways in which Schools and Colleges of Education are organized. Specifically, a School/College of Education might consist of a series of schools, divisions, departments, or other organizational units, each with separate programs. Each organizational unit could conceivably have its own statistics requirement and its own statistics course sequence. A non-QM student with one area of specialization at a particular institution could, therefore, take one required statistics course taught in one particular organizational unit, whereas another student with a different area of specialization could take a different required statistics course taught in a different organizational unit.
19 Considering these complexities, it became apparent that completion of the survey of what non-QM students learn in their statistics courses would require input from all of the statistics instructors in each organizational unit at each institution. We thought that such a survey would be too unwieldy, and decided to shift the focus for the non-QM students away from a survey of course content and course requirements and towards a survey of quantitative professors' perceptions of non-QM doctoral students' preparation in educational statistics. We asked one QM professor at each institution to evaluate non-QM doctoral students' training in educational statistics at his or her institution, and to frame this evaluation in terms of doctoral students with whom he or she had direct contact over the past few years -- through committees, research assistantships, classes, and so forth. In this way, we hoped to circumvent potential problems of having professors speculate about what was taught in statistics classes they might not have thorough knowledge of and about the statistical competence of doctoral students whom they might not know well.
20 For the non-QM students, we collected the following data:
21 In designing the questionnaire, we were initially concerned about QM faculty members' ability to provide accurate information regarding the preparation of non-QM students. First, we were concerned that some QM professors might not be familiar with the statistics requirements of non-QM students. These concerns led us to conduct informal conversations with some of our colleagues at other institutions, who did, in fact, appear to be familiar with the statistics requirements at their universities. Our concerns were further alleviated by the results of the survey itself: Of the 23 QM professors we surveyed, only one person was unfamiliar with the statistics requirements of the non-QM students at the university. Second, we were concerned that the QM professors might not have adequate familiarity with the non-QM students to accurately evaluate their competence with statistics. To address this problem, we decided to ask the professors to evaluate only the non-QM doctoral students with whom they had direct contact over the previous five years -- through dissertation committees, classes, consulting, and so forth. While this strategy is still problematic from a design perspective, we believe that the results provide a solid first step in looking at the adequacy of non-QM doctoral students' training in educational statistics.
22 It is important to point out three limitations of the current research. First, sampling the schools listed in the West and Rhee (1995) paper limits the generalizability of the results, as does the use of only one QM faculty member per institution in the survey. Second, because of time and resource constraints we did not collect survey data on doctoral students' preparation in the areas of educational research methods or measurement (see Guo and Nitko 1996), thus limiting the picture we can paint about doctoral student training. Third, no thorough reliability or validity information was obtained for the questionnaire items. On the positive side, we believe that the survey has several advantages, among them, that it is not a time-consuming research method and it is not particularly intrusive.
23 General information for QM programs and students is presented in Table 2.1 and Table 2.2. The median number of full-time tenured and tenure-track faculty in these QM programs is four, with three full-time faculty regularly teaching statistics during the academic year. The distribution of new doctoral students enrolling in QM programs was positively skewed, with a median of four students admitted annually during the previous five years. All of the institutions required students to take at least one statistics course in the School or College of Education; nearly half required students to take one or more courses in the Mathematics or Statistics Departments; and two (9%) required students to take one or more courses in the Psychology Department.
|Number of full-time tenured or tenure-track faculty in QM program||5.1||4||2.2||23|
|Number of full-time faculty who regularly teach statistics during the academic year||3.6||3||1.7||23|
|Number of lecturers and adjunct faculty members who regularly teach statistics in QM program||1.8||1||2.8||23|
|Approximate number of students admitted to QM program each year over past five years||7.2||4||8.8||22|
Table 2.2. Location of Required Statistics Courses in QM Program (N = 22)
|School/College of Education||100%|
24 To assess the extent to which doctoral students in QM programs received training in specific procedures, we developed a list that we believe reflects a breadth of statistical procedures, including those considered to be "recent," such as meta-analysis, bootstrapping/jackknifing, multilevel models, and structural equation models. We asked faculty to indicate those procedures and methods in which most or all of their QM students received training. Results are presented in Table 3.
|Repeated measures designs||96%|
|Power/sample size calculations||96%|
|Thorough coverage of multiple comparison procedures||74%|
|Cell means models||57%|
|Complex designs (e.g., fractional factorial)||39%|
|Ordinary least squares estimation||100%|
|Weighted least squares estimation||70%|
|Traditional Multivariate Procedures|
|Principal components analysis||78%|
|Repeated measures tests (e.g., Friedman, Cochran)||30%|
|Asymptotic relative efficiency||26%|
|Other Topics and Procedures|
|Structural equation models||74%|
|Multilevel models/Hierarchical linear models||35%|
|Times series models||4%|
25 The data suggest that the vast majority of QM doctoral students receive training in the "old standards" -- ANOVA, multiple regression, and traditional multivariate procedures. The exceptions to this appear to be logistic regression and log-linear models, where more than half of the respondents indicated that their QM students receive training. Nonparametric procedures, on the other hand, appear to be less prevalent. Although a more detailed list of nonparametric procedures was not provided, it seems reasonable to use the Kruskal-Wallis test as a benchmark and to note that only half of the respondents indicated that their QM students received training in this procedure. Finally, in terms of more recent procedures, we see that the majority of programs train their students in meta-analysis and structural equation models (74%), and over one-third of the programs train their students in jackknifing/bootstrapping (39%) and multilevel models (35%).
26 We also asked faculty to rank the mathematical statistics training and computer skills of most of their QM doctoral graduates using a 4-point scale with "1" indicating "weak" skills and "4" indicating "strong" skills. Results are presented in Table 4. Faculty ratings were reasonably symmetrically distributed, with most of the faculty rating their students midway on the scale, either as a "2" (38% of faculty) or as a "3" (48% of faculty). In terms of computer skills, the vast majority (91%) thought their students had "strong" skills in standard data analysis programs such as SAS and SPSS. Faculty perceptions of student competence in other computer skills -- database management and programming -- were more varied.
|1. Mathematical statistics training of most QM graduates||1||8||10||2||2.6||.74||21|
|2. Computer skills: Standard data analysis programs (e.g., SAS, SPSS, SYSTAT)||0||0||2||21||3.9||.28||23|
|3. Computer skills: Database management||1||6||14||2||2.7||.69||23|
|4: Computer skills: Programming (e.g., FORTRAN, PASCAL, C)||7||5||9||2||2.3||1.01||23|
NOTE: For this question, the anchors were 4 = Strong, perhaps took multiple courses in mathematical statistics and probability theory, would probably be well acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc.; and 1 = Weak, little or no coursework in mathematical statistics or probability theory.
27 Colleagues have occasionally remarked that they believe earlier graduates of QM programs were better trained than more recent graduates. We investigated these kinds of comments by asking faculty in the survey if they thought that doctoral students who had graduated in the last year or two were better prepared or less well prepared to do publishable statistical work, as compared to students who graduated five to six years earlier. Data from Table 5 do not support the claim that earlier graduates are perceived to be better trained. In fact, none of the faculty indicated that they thought their recent QM students were less well prepared than graduates from five to six years prior. Over half of the faculty (56%) thought that there was no difference between past and present graduates, and the remaining 44% believed that recent graduates are better prepared.
|Faculty who believe that recent graduates are better prepared than graduates from 5-6 years ago||44%|
|Faculty who believe that recent graduates are less well prepared than graduates from 5-6 years ago||0%|
|Faculty who believe that there is no difference between recent graduates and graduates from 5-6 years ago||56%|
28 Finally, we asked faculty members to indicate whether they thought their QM students could benefit from additional (probably specialized) statistics coursework. Specifically, they were asked to assess (a) whether more than half or fewer than half would have benefited from one or two additional courses, and (b) whether more than half or fewer than half would have benefited from three or more additional statistics courses. Data are presented in Table 6. Nearly half of the respondents (48%) indicated that they thought more than half of their students could have benefited from one or two additional courses in statistics. Moreover, 20% indicated that they thought more than half of their students could benefit from three or more additional courses.
|Statistics Coursework||<50%||>50%||Can't Judge|
|1-2 additional (probably specialized) statistics courses (N = 21)||43%||48%||10%|
|3 or more additional (probably specialized) statistics courses (N = 20)||65%||20%||15%|
NOTE: Data are in response to the following question: In hindsight, of the quantitative methods program doctoral students who graduated from your program in the last 5 years, what percent do you think could have profited from (the following additional coursework)?
29 For the non-QM doctoral students, we began by determining whether each institution had a minimal statistics requirement for all doctoral students in education. Descriptive statistics are presented in Table 7. In nearly half of the institutions (44%), students in some programs can graduate without taking a statistics course, whereas in 39% of the institutions, students in all programs are required to take at least one statistics course. Only 13% of the institutions had a uniform two-course statistics requirement. (Note that this statistic does not estimate how many students actually graduate without a statistics course in a given institution.)
|Students in some programs can graduate without taking a statistics course||44%||Students in all programs are required to take at least one statistics course||39%||Students in all programs are required to take at least two statistics courses||13%||Don't know||4%|
30 We speculated that students might be taking fewer statistics courses currently than in previous years, due to the increased prevalence of qualitative research methods. The data in Table 8, however, do not indicate a decreasing trend in the number of students taking statistics over the past five years. In approximately one-third of the institutions surveyed, students are taking fewer statistics classes, but in a roughly equal number of institutions, they are taking more statistics classes.
|Doctoral students have tended to take fewer statistics courses||27%||Doctoral students have tended to take more statistics courses||36%||Doctoral students have tended to take about the same number of statistics courses||27%||Don't know/No Response||9%|
31 We asked faculty to rate non-QM doctoral students in terms of their ability to critically read and interpret research articles utilizing specific statistical procedures (see Table 9). In particular, faculty were asked to think about the non-QM doctoral students with whom they had contact over the past five years, and to indicate whether they thought more than half or fewer than half of these students would be competent with a particular statistical method. Notice that this question requires a lower threshold of "competence" than a question that asks about doctoral students' abilities to carry out and analyze research utilizing these procedures. The procedures listed -- including ANOVA, ordinary least squares (OLS) regression, MANOVA, and nonparametric rank tests -- were not meant to be an exhaustive list of procedures, but rather were selected because they appeared to represent a cross-section of "common" and "advanced" statistical methods found in published educational research.
32 Results indicate that the faculty perceived the students to be most competent with ANOVA: 63% of the faculty thought that more than half of their graduates could critically read an article using ANOVA. Only 41% of the faculty thought that more than half of their graduates could critically read an article using OLS regression. Only a few of the 22 QM professors thought that more than half of their students could critically read and interpret substantive articles that utilized more advanced procedures, namely MANOVA, log-linear models, and structural equation models.
|<50%||>50%||N||(Pct)||N||(Pct)||ANOVA||8||(36%)||14||(63%)||OLS regression||13||(59%)||9||(41%)||Nonparametric rank tests||19||(86%)||3||(14%)||MANOVA||20||(91%)||1||( 5%)||Log-linear models||21||(95%)||1||( 5%)||Causal models||20||(91%)||1||( 5%)|
NOTE: Data are based on responses to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting, and so forth. Of these non-quantitative students, what percent do you think are competent to critically read and interpret research articles that utilize the following procedures? (1) Less than half, (2) More than half, (3) Can't judge.
33 Faculty were asked to think about non-QM students with whom they had contact over the past five years, and who had graduated, and to assess whether more than half or fewer than half would have benefited from additional statistics coursework. Results are shown in Table 10. Virtually all of the faculty thought that more than half of the graduates could have benefited from additional (probably specialized) statistics coursework: 44% of the faculty thought that over half of the graduates could have benefited from one or two additional courses, and 43% of the faculty thought that at least half of the graduates could have benefited from three or more courses.
|Statistics Coursework||<50%||>50%||Can't Judge|
|1-2 additional (probably specialized) statistics courses (N = 23)||52%||44%||4%|
|3 or more additional (probably specialized) statistics courses (N = 21)||48%||43%||10%|
NOTE: Data are in response to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting and so forth. Of these non-quantitative students, what percent do you think could have profited from (the following additional coursework)?
34 Has this survey accomplished anything? Despite some methodological shortcomings, we think so. For the portion of the survey addressing the preparation of QM students, we offer three general findings. First, we now have empirical evidence of the nature of the curriculum offered in a sample of QM programs. We found much of this information encouraging because it indicated widespread professional agreement in the way that doctoral students majoring in quantitative methods are trained in statistics. For example, almost all of the surveyed programs emphasized traditional data-analytic procedures, a result that agrees with the work of Elmore and Woehlke (1988, 1996). There was also evidence that most respondents thought recent graduates were as well or better trained than earlier graduates. The fact that a number of QM programs also offered instruction in more recent or advanced procedures such as meta-analysis, structural equation models, and jackknifing/bootstrapping implies that a number of faculty believe this kind of training is important enough to incorporate into the curriculum. A second implication of our findings is their relationship to those reported by Aiken et al. (1990). In general, the two sets of findings are similar in terms of what is being taught in QM programs in education and psychology, and in terms of the perceptions of QM professors of students. One area of modest disagreement is that our findings suggest that newer statistical techniques are being taught more frequently in Schools and Colleges of Education than in Departments of Psychology. Still, the results suggest that much of the information provided by the two studies is similar and can be combined to draw stronger conclusions and guide future research.
35 Perhaps the most important implication of our research on QM programs is that the findings offer a platform from which we can ask (and in some cases provide tentative answers to) some important questions about the training of doctoral students in quantitative methods. For example, about half of the quantitative methods programs surveyed have incorporated newer techniques into their curricula. In our view, this places our findings in a somewhat less positive light. What accounts for the failure to offer training in these newer techniques? Is it simply the result of constraints on credit hours, too few faculty, or that training students who are not majoring in quantitative methods is now the primary business of many quantitative methods programs, and that this has a severe effect on the curriculum?
36 We believe the questions raised above (and others) are important and thus provide directions for additional work in the area of preparing doctoral students in quantitative methods. One obvious need in future work is to survey at least two faculty per program to allow within- and between-institution variability to be examined; another is to collect information on additional variables that should controlled, such as rank of the school and required statistics training for QM and non-QM students. It is also important to widen the participation of QM programs and to obtain information about the kinds of positions taken by graduates. As the statistical education literature has argued, curricula must be responsive to the needs of the institutions and industries hiring graduates. We believe that the emergence of information about the training of students in quantitative methods programs should encourage a long-overdue dialogue among educational statisticians.
37 For the portion of the survey that addressed the preparation of non-QM students, the research findings suggest that a number of QM faculty think many non-QM doctoral students are under-prepared to be critical consumers of quantitatively-oriented research articles. Specifically, we found that 36% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing ANOVA, and 59% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing OLS regression. Moreover, the vast majority of faculty thought that fewer than half of the non-QM doctoral students could critically read articles utilizing more advanced procedures, namely MANOVA, log-linear models, nonparametric rank tests, and structural equation models. These research findings provide preliminary data suggesting that we may have cause for concern about non-QM doctoral students' preparation in statistics.
38 Although these findings on the preparation of non-QM doctoral students are only tentative, they do suggest a direction for future research and discussion. One line of research suggested by this investigation involves studying faculty members' opinions about why some non-QM doctoral students at their universities are under-prepared in educational statistics, and what they think can be done to improve these students' preparation. Another line of research suggested by this investigation involves studying editors of educational research journals. Because these editors review so many manuscripts (both publishable and unpublishable), they are in a good position to evaluate the statistical and methodological training of today's educational researchers. In addition to journal editors, it might prove fruitful to study the opinions of the doctoral students themselves regarding the adequacy of their own training. Lastly, the perceptions of non-QM faculty would help inform a discussion on the statistical preparation of non-QM doctoral students.
This questionnaire is part of a national survey of statistics training in doctoral programs in education. The first part of the questionnaire addresses training of doctoral students in quantitative methods (QM) programs in education. The second part of the questionnaire addresses statistical training of doctoral students in education who are not in a quantitative methods program. Your responses to this questionnaire are anonymous and confidential.
_____ Nonorthogonal designs
_____ Covariance analysis
_____ Repeated measures designs
_____ Cell means models
_____ Random-effects models
_____ Mixed-effects models
_____ Complex designs (e.g., fractional factorial)
_____ Thorough coverage of multiple comparison procedures
_____ Power/sample size calculations
_____ Other: Please describe
_____ Ordinary least squares estimation
_____ Weighted least squares estimation
_____ Nonlinear-in-the-predictors models
_____ Nonlinear-in-the-parameters models
_____ Logistic regression
_____ Other: Please describe
_____ Canonical correlation
_____ Discriminant analysis
_____ Principle Components Analysis
_____ Factor analysis
_____ Log-linear models
_____ Other: Please describe
_____ Kruskal-Wallis test
_____ Repeated measures tests (e.g., Friedman, Cochran, Hodges-Lehmann block test)
_____ Asymptotic relative efficiency
_____ Exact tests
_____ Rank-transform tests
_____ Other: Please describe
_____ Causal models
_____ Time-series models
_____ Multilevel models/hierarchical linear models
_____ Matrix algebra
4 = Strong, perhaps took multiple courses in mathematical statistics, probability theory, would probably be well-acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc).
1 = Weak, little or no coursework in mathematical statistics or probability theory, probably not familiar with Gauss-Markoff Theorem, etc.).
Weak Strong 1 2 3 4
Weak Strong Standard data analysis programs (e.g., SAS; SPSS; SYSTAT; BMDP) 1 2 3 4 Database management 1 2 3 4 Computer programming (e.g., FORTRAN; PASCAL; C) 1 2 3 4
Less More Can't than Half than Half Judge 1-2 additional (probably <50%>50% CJ specialized) statistics courses 3 or more additional <50%>50% CJ (probably specialized) statistics courses 1-2 additional qualitative <50%>50% CJ methods courses
If yes, please explain.
Directions: We are now interested in finding out about methodological training for doctoral students in your School or College of Education who are NOT in your Quantitative Methods program.
*Of these non-quantitative students, what percent do you think are competent to critically read and interpret research articles that utilize the following procedures? (circle your answers)
Less More Can't than Half than Half Judge ANOVA <50%>50% CJ OLS regression <50%>50% CJ MANOVA <50%>50% CJ Log-linear models <50%>50% CJ Nonparametric rank tests <50%>50% CJ Causal models <50%>50% CJ
*Of these non-quantitative students, what percent do you think could have profited from:
Less More Can't than Half than Half Judge 1-2 additional (probably <50%>50% CJ specialized) statistics courses 3 or more additional (probably <50%>50% CJ specialized) statistics course 1-2 additional qualitative <50%>50% CJ methods courses
Aiken, L. S., West, S. G., Sechrest, L., and Reno, R. (1990), "Graduate Training in Statistics, Methodology, and Measurement In Psychology: A Survey Of Ph.D. Programs In North America," American Psychologist, 45, 721-734.
Anonymous (1995), Graduate Study in Educational and Psychological Measurement, Quantitative Psychology, and Related Fields, College of Health and Human Development, Pennsylvania State University.
Brogan, D., and Kutner, M. H. (1986), "Graduate Statistics Service Courses," The American Statistician, 40, 252-254.
Dawson-Sanders, B., Azen, S., Greenberg, R. S., and Reed, A. H. (1987), "The Instruction of Biostatistics in Medical School," The American Statistician, 41, 263-266.
Elmore, P. B., and Woehlke, P. L. (1988), "Statistical Methods Employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978 To 1987," Educational Researcher, 19-20.
----- (1996), "Research Methods Employed in American
Educational Research Journal, Educational Researcher,
and Review of Educational Research From 1978 To 1995,"
paper presented at the annual meeting of the American
Educational Research Association, New York.
Guo, F., and Nitko, A. J. (1996), "Graduate Programs that
Prepare Educational Measurement Specialists,"
Educational Measurement: Issues and Practice, 15,
Noether, G. E. (1980), "The Role Of Nonparametrics in
Introductory Statistics Courses," The American
Statistician, 34, 22-23.
Tukey, J. W. (1980), "We Need Both Exploratory and
Confirmatory," The American Statistician, 34, 23-25.
West, C. K. and Rhee, Y. (1995), "Ranking Departments or
Sites within Colleges of Education Using Multiple
Standards: Departmental and Individual Productivity,"
Contemporary Educational Psychology, 20, 151-171.
Deborah A. Curtis
DAIS/College of Education
San Francisco State University
1600 Holloway Avenue
San Francisco, CA 94132
Department of Psychology in Education
5B27 Forbes Quad
University of Pittsburgh
Pittsburgh, PA 15260
Guo, F., and Nitko, A. J. (1996), "Graduate Programs that Prepare Educational Measurement Specialists," Educational Measurement: Issues and Practice, 15, 28-31.
Noether, G. E. (1980), "The Role Of Nonparametrics in Introductory Statistics Courses," The American Statistician, 34, 22-23.
Tukey, J. W. (1980), "We Need Both Exploratory and Confirmatory," The American Statistician, 34, 23-25.
West, C. K. and Rhee, Y. (1995), "Ranking Departments or Sites within Colleges of Education Using Multiple Standards: Departmental and Individual Productivity," Contemporary Educational Psychology, 20, 151-171.
Deborah A. Curtis