|
Abstract:
|
Educational surveys such as the National Assessment of Educational Progress (NAEP) assess broad frameworks, requiring more items than can be completed by any single student. Items form separately timed blocks, and pairs of blocks form books according to a Balanced Incomplete Block (BIB) design. Parametric Item Response Theory is used to combine the data for estimates of subgroups of students' academic proficiency. To support valid inferences, model-data fit must be adequate. Douglas and Cohen (2001) use a nonparametric item response function to create an item fit measure, using resampling to construct a null sampling distribution. Work on this method has only examined linear test forms. It is unknown how they function in the context of BIB. This Monte Carlo study examined these methods with BIB data. Data were simulated according to the design of a 9-block NAEP assessment. Numbers of students, items per block, and (lack of) item fit were varied. A given item occurs in one block, but that block occurs in several books. Thus, three kinds of analysis (block, book, and pooling data of books) were examined in terms of Type I error rate and power.
|
