TITLE:
A Procedure for Diagnostically Modeling Extant Large-Scale Assessment Data: The Case of the Programme for International Student Assessment in Reading
AUTHORS:
Jinsong Chen, Jimmy de la Torre
KEYWORDS:
CDM, Q-Matrix, Large-Scale Assessment, Fit Measures, PISA
JOURNAL NAME:
Psychology,
Vol.5 No.18,
November
24,
2014
ABSTRACT: Cognitive
diagnosis models (CDMs) are psychometric models developed mainly to assess
examinees’ specific strengths and weaknesses of a set of skills or attributes
within a domain. Recently, several methodological developments have been added
to the CDM literature, which include the development of general and reduced
CDMs, various absolute and relative fit measures at both the test and item
levels, and a general Q-matrix validation procedure. Building on these
developments, this research proposes a systematic procedure to diagnostically
model extant large-scale assessment data. The procedure can be divided into
four phases: construction of initial attributes and Q-matrices, construction of
final attributes and Q-matrix, evaluation of reduced CDMs, and crossvalidation
of the selected model. Working with language experts, we use data from the PISA
2000 reading assessment to illustrate the procedure.