Keywords: Reliability, Validity, Survey Instruments, Multilevel, Guidelines
Surveys to evaluate the work environment of multilevel organizations (i.e. hospitals) aggregate data from individual level responses to work-group level summary scores. To ensure that summary scores are reliable and valid, researchers set criteria for a minimum number of respondents or overall response rates from work groups. The National Database of Nursing Quality Indicators® Annual RN survey requires a minimum response rate of 50% and 5 nurses per unit for inclusion in research or benchmarking; however, the exact origins of these guidelines are unclear and arbitrarily assigned. If the criteria are too strict, valuable data may be excluded; too broad and reliability and validity may be compromised. Our goal is to evaluate the appropriateness of these criteria for establishing reliability and validity in multilevel survey data.. We are interested in assessing the effects of within unit variability (wide vs narrow bands), number of eligible nurses per unit (10, 25, 50, 100), and response rate (25%, 50%, 75%) on the bias of summary score estimates. Understanding what the appropriate criteria are will improve estimate reliability without unnecessarily eliminating data.