Author(s): JOACHIM HÄUSLER | MARKUS SOMMER
Journal: Psychology Science Quarterly
ISSN 1866-6140
Volume: 50;
Issue: 1;
Start page: 75;
Date: 2008;
VIEW PDF
DOWNLOAD PDF 
Original page
Keywords: adaptive testing | test economy | success probability | test taking motivation
ABSTRACT
Recent research on the psychological effects of different design decisions in computerized adaptive tests indicates that the maximum-information item selection rule fails to optimize respondents’ test-taking motivation. While several recent studies have investigated psychological reactions to computerized adaptive tests using a consistently higher base success rate, little research has so far been conducted on the psychometric (primarily test reliability and bias) and psychological effects (e.g. test-taking motivation, self-confidence) of using mixtures of highly informative (p = .50) and easier items (p = .80) in the item selection process. The present paper thus compares these modifications to item selection with a classical maximum-information algorithm. In a simulation study the effect of the different item selection algorithms on measurement precision and bias in the person parameter estimates is evaluated. To do so, the item pool of the Lexical Knowledge Test, measuring crystallized intelligence and self-confidence, is used. The study indicated that modifications using base success probabilities over p = .70 lead to reduced measurement accuracy and - more seriously - a bias in the person parameter estimates for higher ability respondents. However, this was not the case for the motivator item algorithm, occasionally administering easier items as well. The second study (n = 191) thus compared the unmodified maximum-information algorithm with two motivator item algorithms, which differed with regard to the percentage of motivator items presented. The results indicate that respondents yield higher self-confidence estimates under the motivator item conditions. Furthermore, the three conditions did not differ from each other with regard to the total test duration. It can be concluded that a small number of easier motivator items is sufficient to preserve test-taking motivation throughout the test without a loss of test economy.
Journal: Psychology Science Quarterly
ISSN 1866-6140
Volume: 50;
Issue: 1;
Start page: 75;
Date: 2008;
VIEW PDF
DOWNLOAD PDF Original pageKeywords: adaptive testing | test economy | success probability | test taking motivation
ABSTRACT
Recent research on the psychological effects of different design decisions in computerized adaptive tests indicates that the maximum-information item selection rule fails to optimize respondents’ test-taking motivation. While several recent studies have investigated psychological reactions to computerized adaptive tests using a consistently higher base success rate, little research has so far been conducted on the psychometric (primarily test reliability and bias) and psychological effects (e.g. test-taking motivation, self-confidence) of using mixtures of highly informative (p = .50) and easier items (p = .80) in the item selection process. The present paper thus compares these modifications to item selection with a classical maximum-information algorithm. In a simulation study the effect of the different item selection algorithms on measurement precision and bias in the person parameter estimates is evaluated. To do so, the item pool of the Lexical Knowledge Test, measuring crystallized intelligence and self-confidence, is used. The study indicated that modifications using base success probabilities over p = .70 lead to reduced measurement accuracy and - more seriously - a bias in the person parameter estimates for higher ability respondents. However, this was not the case for the motivator item algorithm, occasionally administering easier items as well. The second study (n = 191) thus compared the unmodified maximum-information algorithm with two motivator item algorithms, which differed with regard to the percentage of motivator items presented. The results indicate that respondents yield higher self-confidence estimates under the motivator item conditions. Furthermore, the three conditions did not differ from each other with regard to the total test duration. It can be concluded that a small number of easier motivator items is sufficient to preserve test-taking motivation throughout the test without a loss of test economy.