Evaluating the Definition of a Test Domain
LEARNING OBJECTIVES
- Explain two fundamental aspects of a task on a test: content and process.
- Evaluate the representation of those two aspects in a test manual’s domain specification.
- Evaluate codifications of the two aspects as dimensions in a test blueprint.
- Evaluate how well the blueprint has been used in test development and in domain communication.
Abstract
Psychological processing is often addressed in theories that give rise to assessments, but careful domain specification and sampling plans for processing is often lacking in commercially available tests. In achievement testing more specifically and educational testing more generally the definition of a test domain, as well as its blueprint (the sampling plan of the domain) often delineates only content, if even that.
The cognitive processing dimension, crucial to a full understanding of what an examinee can do, is often not specified, except perhaps after the fact in evaluating the scope of a given assessment. A similar dimension is also relevant in psychological assessments designed to discover in what way and/or how people process material. The material itself is independent of the person, but is nevertheless important because the processing is only meaningful in relation to it. Yet only with an understanding of processing can it be ensured that test stimuli elicit the range of internal activities intended.
This presentation will discuss this prevalent deficiency in many, if not most, published tests. Possible reasons why this deficiency exists, the effects of the deficiency on interpretability, and how domain specifications can improve test documentation for users and development of new forms by test publishers and authors will be discussed.
Bio
William (Bill) Schafer (EdD, University of Rochester, 1969) is an Affiliated Professor and Associate Professor Emeritus in the Educational Measurement, Statistics, and Evaluation program at the University of Maryland College Park. He was with the university from 1969 until his retirement in 2000. From 1997 to 1999 he was on leave from the university, serving as director of student assessment with the Maryland State Department of Education. He has served as a Mental Measurements Yearbook (MMY) reviewer since 1990, completing 17 reviews in 9 yearbooks, and was recognized as a Distinguished MMY Reviewer in 2008. His current professional interests are focused on applied assessment and accountability in educational contexts. He has actively served in various leadership positions with the Association for Assessment in Counseling and Education (AACE) and American Educational Research Association (AERA) and has chaired several committees and work groups that prepared statements about responsible test use. Dr. Schafer is a fellow of the American Counseling Association and received awards for his research from the AERA Classroom Assessment SIG in 2001 and AERA Measurement and Research methodology Division in 2006. He is a former editor of Measurement and Evaluation in Counseling and Development, is currently on the editorial board of Applied Measurement in Education, and is co-editor of Practical Assessment, Research, and Evaluation, a free journal available at PAREonline.net.