Developing a Biostatistical Collaboration Course in a Health Science Research Methodology Program

Lehana Thabane, Stephen D. Walter, Steven Hanna, Charles H. Goldsmith, and Eleanor Pullenayegum
McMaster University

Journal of Statistics Education Volume 16, Number 2 (2008), jse.amstat.org/v16n2/thabane.html

Copyright © 2008 by Lehana Thabane, Stephen D. Walter, Steven Hanna, Charles H. Goldsmith, and Eleanor Pullenayegum all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.

Key Words: Biostatistical collaboration course; Mentorship; Mentoring; Internship; Health research; Biostatistics training; Collaborative research.

Abstract

Effective statistical collaboration in a multidisciplinary health research environment requires skills not taught in the usual statistics courses. Graduates often learn such collaborative skills through trial and error. In this paper, we discuss the development of a biostatistical collaboration course aimed at graduate students in a Health Research Methodology PhD program with Specialization in Biostatistics. The objectives of the course are to promote enthusiasm and commitment to excellence in statistical collaboration in clinical research; to enhance communication of statistical issues to non-statistician collaborators; to build statistical self-sufficiency and develop skill in applied statistics; and to enhance a culture of collaboration among statisticians and non-statistician researchers. The course uses a combination of lectures and tutorials led by faculty members, videotaped consulting practice sessions, and internship with mentoring of each student by an experienced biostatistician.

1. Introduction

Training statisticians is one of the main functions of graduate statistics programs. The curricula used to achieve this goal vary between programs, although there is some consensus on the key skills (Committee on Training of Statisticians 1980; Tobi 2001; Iman 1995; Hammond 1980; Hogg 1991; Federer 1978; Watts 1970).

Statisticians play a major role in biological, health, and medical research endeavors often through collaborative research in multidisciplinary teams. Effective statistical collaboration in a multidisciplinary health research environment requires skills not usually taught in statistics courses, and so graduates often learn such skills through trial and error. It is, therefore, crucial to design statistics education programs appropriately to help students acquire the skills needed for their future responsibilities. Several authors have advanced ideas on how to modernize statistics training programs to equip trainees with the skills necessary for effective multidisciplinary work (DeMets 1994; Lethoczy 1995; McCulloch 1985; Killion 1995; Snee 1996)

Formerly known as the Design, Measurement and Evaluation (DME) program, the current Health Research Methodology (HRM) program is a program in Graduate Studies at McMaster University based in the Department of Clinical Epidemiology and Biostatistics (http://fhs.mcmaster.ca/ceb) in the Faculty of Health Sciences (http://fhs.mcmaster.ca). The HRM program offers advanced studies in a wide spectrum of clinical and health-related academic disciplines including medicine, nursing, rehabilitation sciences, social work, mathematics and statistics, economics, and other social sciences. With its emphasis on interdisciplinary education, the program focuses on evaluative frameworks and research methods derived from clinical epidemiology, biostatistics, epidemiology, health economics, health policy analysis, and many of the social sciences. Figure 1 provides a summary of the educational model used in the HRM program.

The overall goal is to train students to integrate theory and methods from various perspectives so as to create innovative research and evaluation methods that will enhance and strengthen healthcare. Mentoring and apprenticeship also play a key role in the program.

Since September 2007, HRM students have been able to enroll in specialized fields of Clinical Epidemiology, Biostatistics, Health Services Research, Population and Public Health, and Health Technology Assessment as well as pursue the current HRM areas of health economics, health policy analysis, health ethics, health education evaluation, and knowledge translation (http://fhs.mcmaster.ca/grad/hrm/msc/Fields.htm).

The purpose of this paper is to describe the process of developing a biostatistical collaboration course which was designed as part of the core requirements for the new Biostatistics Field.

In the next section, we describe the general objectives of the HRM program with Specialization in Biostatistics (henceforth to be referred to as Biostatistics Field). In Section 3, we present the objectives and outline of the Biostatistical Collaboration course. We also describe the evaluation process used in the course. Section 4 provides some experiential results from the first offering of the course, insight from student evaluations of the course, and areas for improvement. In Section 5, we discuss some changes made in the second year of offering the course. Concluding remarks are found in Section 6.

2. The Biostatistics Field

Although the HRM program as a whole accepts applicants for both Masters and PhD studies, the Biostatistics Field was designed for PhD candidates who want to specialize in Biostatistics (http://fhs.mcmaster.ca/grad/hrm/msc/Fields.htm). The requirements for admission into the Biostatistics Field include a Masters in Statistics, Biostatistics or equivalent.

The goal of the Biostatistics Field is to educate or train individuals who, upon graduation, will have acquired sufficient skills to be actively involved in independent and collaborative applied health research or in the teaching of biostatistics. Therefore graduates will be expected to have acquired the following skills:

• a strong foundation in biostatistical concepts/techniques and their application in study design and data analysis;
• the ability to critically appraise methodological aspects of research proposals and manuscripts;
• the ability to develop or contribute to the development of grant proposals as methodologists;
• the ability to teach or communicate biostatistical concepts effectively to non-biostatisticians; and
• the ability to adapt existing statistical techniques or to develop new techniques to solve problems.

The motivation for developing the program was the perceived strong need for the training of biostatisticians in the skills needed for collaborative research – particularly students trained in statistics who have limited experience in applied health research and want to work in this area. The Biostatistical Collaboration course was designed as a pillar of the program. Some of the key reasons for introducing the course included:

• to teach effective statistical collaboration skills:
• skills of this type are often learned elsewhere by trial and error;
• resources (based on peer-reviewed literature) are available to advance knowledge on effective interdisciplinary research;
• to promote enthusiasm for research collaboration;
• to promote commitment to excellence in statistical collaboration in health research;
• to enhance communication of statistical issues to non-statistician collaborators;
• to build statistical self-sufficiency and develop skill in applied statistics; and
• to enhance a culture of collaboration among statisticians and non-statistician researchers through provision of sound statistical advice to the scientific community.

3. HRM-739: The Biostatistical Collaboration Course

Designated as one of the courses for graduate students registered in the HRM Biostatistics Field, the objectives of the Biostatistics Collaboration course (henceforth to be referred to as HRM 739) are:

• to understand the consulting environment and the roles of biostatisticians in multidisciplinary research;
• to learn strategies required to elicit information from clinical collaborators, so as to assist with study design;
• to learn how to translate clinical or applied research questions into statistical questions;
• to learn how to facilitate provision of statistical support on design, sampling and analytic plans;
• to learn strategies of communicating the sampling plan, experimental design, and statistical analysis to collaborators;
• to facilitate provision of support on statistical programming; and
• to facilitate provision of help with write-up of methods and reporting of the results of studies.

The course is structured to include two key elements: 1) in-class discussion on specific topics with more emphasis on the process issues than the technical aspects, and 2) mentoring of each student by an experienced statistician to enable their experiential learning.

The prerequisite for the course includes advanced statistics methodology courses at the Masters level. For example, we expect students to have taken advanced courses in probability theory, statistical inference, linear models, design of experiments, and survey sampling. The annual enrolment in the course is not expected to exceed five students – this limit is driven by the expected number of new biostatistics PhD per year, which in turn is influenced resource limitations, including funding for graduate students.

3.1 Course outline

Below is a summary of the topics covered in the course, with corresponding objectives. (See Table 1 for detailed descriptions with key references.) For each session, the discussions focus primarily on the process of handling or dealing with the underlying issues. The session instructors draw heavily from their own experience in underlining the approaches to handling the consultee’s problem, using statistical principles.

SESSION 1: Introduction to Statistical Consulting
This session discusses issues that include the ethical norms of the statistical profession, the role of statistics in science, and the role of the statistician in considering research ethics.

SESSION 2: Statistical Consulting Environment
In this session, we discuss the types of environments in which statisticians work, the role of the statistician within a research environment, and how to communicate with researchers from other disciplines.

SESSION 3: Communications Skills
This session covers topics on verbal, non-verbal and written communication. We focus on how to write statistical reports, how to make effective presentations, and effective verbal and non-verbal communication.

SESSION 4: Statistics and Ethics: Case Studies
This session covers the role of the statistician and the techniques used in clarifying research questions and hypotheses, choosing study designs, the use of placebo controls in randomized controlled trials, and determination of primary outcomes, covering the pros and cons of surrogate outcomes versus clinical outcomes, and composite outcomes versus individual outcomes, as examples.

SESSION 5: Methodologic Issues I: Reporting of Scientific Publications
This session covers issues on how to report results of scientific studies. In particular, we focus on the role of a statistical collaborator in ensuring appropriate reporting standards on design, methods, analysis and results, and discussion of authorship issues.

SESSION 6: Methodological Issues II: Sample Size Issues
This session covers techniques for eliciting information from clinician collaborators to calculate sample size. We also discuss issues related to reporting of the results of the calculation in

• a protocol for granting agency or Research Ethics submission,
• an abstract for a scientific meeting, and
• a manuscript for publication.

SESSSION 7: Video Practice Session I
This session involves videotaping of a practice session with a clinician researcher and a subsequent discussion on the experience. The consultee usually also participates in the discussion. The objective is to allow the student, who acts as consultant, to put some skills into practice while other students learn by observing the consulting session. The videotapes will form the basis for discussion of the experiences afterwards.

SESSION 8: Computational Aspects
This session involves a lab and discussions of different features of common and special statistical software such as SAS, STATA, SPSS, MINITAB and WinBUGS, including special software for sample size calculation and meta-analysis. We also discuss different features of database design/management software such as ACCESS and Teleform.

SESSION 9: Video Practice Session II
This session is a continuation of Session 7. We are able to schedule 2-3 consulting sessions for each of the two Sessions.

SESSIONS 10, 11, 12: Consulting Experience/Project
These are sessions are held one per week. The weeks corresponding to these sessions are dedicated to formal internship for experiential learning. Students are attached to a research group at the beginning of the course and may start working with the group as soon as feasible. Each intern is assigned a mentor within the group to monitor the student’s consulting experience and performance. The mentor meets regularly with the student (preferably after each project meeting) for debriefing and performance assessment. Each mentor is provided with an evaluation form to assess the performance of the student.

SESSION 13: Career Development
This session covers discussions of how to create and maintain a curriculum vitae (CV); aspects considered in evaluation of CVs for promotion, career awards, research awards and so on; how to create and maintain a teaching dossier; aspects considered in evaluation of one’s contributions to education through a teaching dossier; and strategies for career development and improving the publication record. We also discuss study CVs which record study-related activities such as publications, presentations, abstracts, etc. The overall goal is to teach students how to effectively document their research collaboration activities.

SESSION 14: Project Presentations
This session is used for the presentation of projects that students were involved in during the internship. The projects can be based on design of new studies, analysis of existing data, or can report on experiences during the internship period. For example, students could describe their experience in consulting with clinicians, being part of a team to develop a protocol or grant.

3.2 Mentoring

Incorporation of mentorship is an important and innovative part of the course. Several scientists have long advocated the use of mentoring as a way to facilitate the acquisition of important career skills (Thabane 2006; Alberts B. Science 1999; Hoover 2005; E. Garfield 1992). The HRM program provides experiential learning through internships. The Biostatistical Collaboration course was designed to extend this model by assigning every student to a mentor whose primary role includes ensuring that the student is exposed to tasks that a statistician would be expected to help with in any collaboration with non-statisticians. These include providing opportunities to the protégé such as helping with or providing guidance on

• clarifying the research question;
• clarifying hypotheses;
• choice of design;
• formatting/coding data;
• database design;
• generating a randomization schedule;
• reviewing protocols/manuscripts;
• data analysis or interpretation of results; and
• explaining statistical issues to non-statisticians.

The mentors are asked to meet with the students for at least one hour per week to monitor progress. The choice of mentors was made by matching student and mentor interests. Gender was not taken into consideration in assigning mentors. However, the student’s primary thesis supervisor was declared ineligible as that student’s mentor for the course so as to broaden the student’s overall learning experiences. While the protégé -mentor relationship is established in general terms at the beginning of the course, we have dedicated the three sessions of consulting experience/project as time for hands-on learning. At the end of the course, the mentors provide an evaluation of the student’s progress (see Appendix A).

3.3 Course Evaluation

The overall evaluation of the course involves four parts: Assignments (15%); In-class participation (15%); Internship (30%); and Final Project (40%).

• Assignments
  There are regular assignments on a variety of topics. These may include brief write-ups on small-scale data analyses; others may involve summaries of readings as well as videotape viewings.


• In-class Participation
  In-class instruction covers a broad spectrum of issues as stated in the outline. Students are expected to do prior reading for every session. In addition to formal lectures and discussions, some use is made of videotapes. Critical appraisal of videotapes of role playing is used. The course is delivered through guest lectures on diverse topics, with occasional invitation of clients from various health sectors: government, industry and hospitals. The evaluation form (Appendix B) is used to evaluate student participation in class discussions.


• Internship
  
  • The course coordinator meets with each mentor to discuss the specific arrangements for mentoring and evaluation of the student’s progress. The mentor is responsible for ensuring that the student gets a wide spectrum of experiences as noted in Appendix A. This may be achieved through various measures such as introducing the student to other members of the group and explaining the importance of the internship. However, the student is not to be used as a primary data-analyst or methodologist for the group.
  • The student is responsible for follow-up on the decisions/suggestions made during these weekly meetings with their mentor.
  • The course coordinator meets with the mentors twice (half-way through the course and at the end) to discuss the process and address any challenges. This can also be done by email or phone.
  • At the end of the internship, the mentor communicates the progress report to the course coordinator with an overall grade for the student’s performance. A form (Appendix A) is used to evaluate the internship component.


• Final project
  The final project involves actual consulting on a study, with oral presentation and a written report. The project covers, among other things, 1) the objectives of the study; 2) how the student was able to identify key issues in their meeting with investigators; 3) what the student’s recommendations to address the issues were; and 4) in some cases, the results of the project. The evaluation of the final projects will be based on the form shown in Appendix C.

4. Findings from the First Offering of the Course in 2006/07

At the beginning of the process, it took some time and effort to arrange mentors and explain their role in the course. Organizing instructors or guest speakers for different sessions was relatively easy. This is possibly because instructors enjoy the opportunity to share their consulting experiences with students.

Three male students, all of whom were registered in the Biostatistics Field, took the course in its first year of offering. Overall, the course seems to have made a successful debut as attested by positive student evaluations described below. There were two male mentors, and one female mentor. The mentors used a variety of approaches to engage their mentees:

• Student I worked on verifying the results obtained by another data analyst, using different approaches. He then made recommendations as to what approach would be best and why.


• Student II summarized his experiences consulting with various people that were introduced to him by the mentor. He also evaluated his own performance – something that was not expected of the students.


• Student III described his experiences as a member of team that wrote an application for funding to the Canadian Institutes of Health Research. He was part of this process from the beginning of the idea to the submission of the grant.

The video sessions went smoothly with help from an administrative assistant from the HRM program who organized the audio and visual equipment from the University Audio and Visual Instruction Materials Office. The equipment was supplied at no cost. It is possible that this could be costly in universities that do not have similar facilities. Each video session ran for about 30 minutes. One client brought a question about how to design a study and calculate the sample size for developing a tool used by clinicians to determine whether an adverse event (AE) was caused by a study drug. The objective was to assess agreement between clinicians with each of the 25 clinicians, evaluating AEs for five patients. The design involved generating an incomplete block design. The second consulting project was about prediction of journal citation counts based on information available at the point of publication. The objective was to quickly identify journal articles that would be important to the research community. Statistical issues involved usage of large data bases, non-normal data, and collinearity between predictors of high citation counts. The third project involved a secondary analysis of a randomized clinical trial of use of antibiotics. The consultation involved the possibility of using some relatively complex analyses such as marginal structural models, propensity scores, and random effects regressions involving a number of confounding variables. All three consultees remained after their formal video taping session, to provide feedback to the group on how things had gone from their perspective. Possible next steps were also discussed a little more in some cases.

At the end of the course, a representative of the HRM program came to class to elicit students’ quantitative and qualitative evaluations of the course. Overall, the quantitative assessments were positive. Based on a group of three students who took the course, the ratings on a 7-point scale [1=Poor, 2=Mediocre, 3=Fair, 4=Average, 5=Good, 6=Very good, 7=Excellent], were as follows: 1) overall rating of the course [mean (SD) = 6.0(1.0)]; 2) clarity of course objectives [5.7(0.6)]; 3) organization of the course [5.7 (0.6)]; 4) relevance to career, educational goals and interests [6.3(0.6)]; 5) usefulness of class discussions [6.3(0.6)]; 6) usefulness of class materials [6.3 (1.0)]; 7) how student needs and expectations were met [6.3(1.0)]. Some of the qualitative responses on what were the best features of the course included

• The mentorship was a good learning experience…";
• "The mentorship aspect was great. The video session was also great…";
• "Class discussion. The topics of discussion".

Although we did not systematically elicit feedback from instructors or guest speakers, most expressed enthusiasm and interest in continuing to participate in the course in the future. Some of the activities in which mentors involved students included attending research meetings, reviewing journal manuscripts, reviewing proposals, attending research ethics board meetings, consulting with clinician researchers, and training in statistical issues such as interpretation of results, statistical software issues, research designs and methods.

5. Some Changes in the Second Year of Offering the Course

The academic year 2007/08 marked the second year of running the course. Based on feedback from the first year, we took out the three-week internship. The students felt that the time could be better used discussing other issues, since their internship basically started at the beginning of the course when their mentor included them in their research program. We added two new sessions:

• New Session 1:Grant Writing
  • To discuss the basic principles of scientific writing.
  • To discuss how to write statistical sections for grants and manuscripts.
  • To discuss how to write statistical section for REB submissions.
  • To discuss the guidelines for writing a book review.
    Key references
    
    1. Boice R. Professors as Writers: A elf-Help Guide to Productive Writing. New Forums Press: Oklahoma, 1990
    2. Struck W (Jr), White EB. The Elements of Style, 4th Edition. Longman: New York, 2000
    3. Peat J, Elliott E, Baur L, Keena V. Scientific Writing. BMJ Books: London, 2002
    4. Stausser J. Painless Writing. Barron’s: New York, 2001.
• New Session 2: Stress and Time Management
  • To discuss strategies for stress management.
  • To discuss strategies for time management in collaborative research.
    Key references
    
    1. Tracy B. Time Power: A Proven System for Getting More Done in Less Time Than You Ever Thought Possible. AMACOM (American Management Association): New York, NY 2007.
    2. Tracy B. Goals! How to Get Everything You Want - Faster Than You Ever Thought Possible. Berrett-Koehler Publishers: San Francisco, CA 2004.
    3. Time Management Lecture Video by Randy Pausch (from Carnegie Mellon University): http://www.youtube.com/watch?v=oTugjssqOT0

In 2007/08, we again had three students enrolled in the course (two males and one female). Overall, the quantitative evaluations for the second offering were consistent with those of the first offering. The qualitative sentiments were also similar: one student described it as " the best statistics course ever taken". Comments on the best feature of the course included:

• "Having several tutors with different areas of statistical expertise…most importantly, each student has a mentor".
• "The course covers aspects of biostatistics that are not commonly found in the literature. Used practical approach and experiences from instructors".

6. Concluding Remarks

We have developed a course in biostatistical collaboration which aims to close the gap between classroom instruction on biostatistical techniques and the application of the techniques in real-life collaborations. The course combines classroom discussion, mentoring and internship to advance both the technical and non-technical statistical skills necessary for biostatisticians to be effective in health research collaborations. In addition to helping them become effective collaborators, we hope that the graduates will receive valuable pre-employment training from the mentors, and enrich their statistical training with some work experience as they actively take part in health research under their mentor’s guidance. While it is too early to predict the potential impact of the initiative, the results from the first and second years of offering are promising.

Table 1 : Course Outline: Session Objectives and Key References

Appendix A: INTERNSHIP EVALUATION

Student Name: ____________________________________________________

Mentor (Evaluator): ________________________________________________

Date (yyyy:mm:dd): ___________________________

A.  Internship goals: Rate the student’s ability on the following items on a scale of 1 – 7, with 7 being Outstanding and 1 being Poor. Please circle your response.

B.  This section includes items about a variety of skills that are considered essential for effective biostatistical collaboration. Please give an overall rating of each aspect and also check any of the specific strengths or areas that need improvement. Please circle response.

C.  This section includes specific items that the student could have done during the internship period. Please check any of the items that you observed the student do and rate your impressions of the student’s effectiveness on each on a scale of 1 – 7, with 7 being Outstanding and 1 being Poor. Please circle your response.

D.  This section is for general feedback about your role as a mentor:

Appendix B: IN-CLASS EVALUATION

Student Name: ____________________________________________________

Evaluator: ________________________________________________________

Week: ____________                                 Date (yyyy:mm:dd):_______________

Appendix C: FINAL PROJECT EVALUATION

Student Name: ____________________________________________________

Evaluator: ________________________________________________________

Date (yyyy:mm:dd): ___________________________

A.  This section includes items that form the objectives of the course. Please rate the student’s demonstration of their ability on the items as judged from the presentation on a scale of 1-7 where 7 is "Outstanding" AND 1 is "poor":

Acknowledgements

We thank Kristina Vukelic and Christina Rizzato for their help in organizing the needed infrastructure (booking class room, booking and operating video equipment for video sessions) and Dr Kathy Bennett for her guidance in putting together the proposal for the Fields Initiative. We also thank Susan Tomlinson for help in managing the scheduling for the course and editing this manuscript. We thank the editor and the reviewer for valuable comments and suggestions that led to improvements in the presentation.

References

Alberts B. Science and World’s Future. 136^th Annual Meeting of the National Science Academy Speech, Washington DC, 1999 (available at http://www.nasonline.org/site/DocServer/1999address.pdf?docID=941) [last date of access: September 7, 2007]

Committee on Training of Statisticians for Industry Preparing Statisticians for Careers in Industry: Report of the ASA Section on Statistical Education. The American Statistician 1980;34(2):65-75.

DeMets DL, Anbar D, Fairweather W, Louis TA, O'Neill RT. Training the next generation of biostatisticians. The American Statistician 1994;48(4):280-4.

Federer WT. Some remarks on statistical education. The American Statistician 1978;32(4):117-21.

Garfield E., Mentoring Young Scientists is an Ethical Imperative and a Pragmatic Necessity. The Scientist 1992;6(24):411-2.

Hammond D. The training of clinical trials statisticians: a clinician's view. Biometrics 1980;36:679-85.

Hogg RV. Statistical education: Improvements are badly needed. The American Statistician 1991;45(4):342-3.

Hoover EL. Mentoring surgeons in private and academic practice. Arch Surg 2005;140(6):598-608.

Iman RL. New paradigms for the statistics profession. JASA 1995;90(429):1-6.

Killion RA. Developing the Affective Side of Technicians: Consulting and Managing Skills. Proceedings of the Section on Government Statistics, American Statistical Association, 1995.

Lethoczy J. Modernizing Statistics Ph.D. Programs. The American Statistician 1995;49:12-7

McCulloch C, Boroto D, Meeter D, Polland, R, Zahn, D. An Expanded Approach to Educating Statistical Consultants. The American Statistician 1985;39:159-67.

Snee RD. Nonstatistical Skills That Can Help Statisticians Be More Effective. Center for Quality and Productivity Improvement, University of Wisconsin-Madison, Report No. 140, 1996.

Thabane L, Thabane M, Goldsmith CH. Mentoring Young Statisticians: Facilitating the Acquisition of Important Survival Skills. African J Statistics 2006;2:31-42

Tobi H, Kuik DJ, Bezemer PR, Ket R. Towards a curriculum for the consultant biostatistician: identification of central disciplines. Stat Med. 2001;20:3921-9.

Watts DG. A Program for Training Statistical Consultants. Technometrics 1970:12(4):737-40.

Lehana Thabane
Department of Clinical Epidemiology and Biostatistics
McMaster University
Ontario, Canada
thabanl@mcmaster.ca

Stephen D. Walter
Department of Clinical Epidemiology and Biostatistics
McMaster University
Ontario, Canada
walter@mcmaster.ca

Steven Hanna
Department of Clinical Epidemiology and Biostatistics
McMaster University
Ontario, Canada
hannas@mcmaster.ca

Charles H. Goldsmith
Department of Clinical Epidemiology and Biostatistics
McMaster University
Ontario, Canada
goldsmit@mcmaster.ca

Eleanor Pullenayegum
Department of Clinical Epidemiology and Biostatistics
McMaster University
Ontario, Canada
pullena@mcmaster.ca