Dr. Thomas P. Hogan

Title

How to Evaluate and Review a Test: Reflections of a Long-Time Mental Measurements Yearbook Reviewer

Abstract

This webinar outlines thought processes that might be involved in completing a Mental Measurements Yearbook review of a test. Some parts of the review process present little difficulty. Other parts entail considerable challenge. Identifying the test’s purpose and describing the test’s materials and processes usually present little difficulty. Perhaps surprisingly, evaluating the standardization (the norms) is often straightforward. Four aspects of the review process frequently present special challenges. First, many test manuals present an overwhelming amount of data on reliability and validity. The reviewer must sort through all this information and formulate overall conclusions that can be conveyed in relatively succinct language. Second, many tests provide a host of scores. An important, and sometimes tricky, part of the review process is spotting all the scores (and suggested interpretations) and determining if adequate technical information is available for all of them. Third, fairness has emerged as a major concern, now ranking with the traditional “big three” of norms, reliability, and validity. Many tests have not yet adequately treated this subject. Some have covered it, but only in the most cursory manner. Fourth, in formulating an overall conclusion about the test, the reviewer faces the difficult task of deciding whether to comment exclusively about the test under review or to place it in the wider context of other tests in the same domain.

Bio

Thomas P. Hogan, Ph.D. is Distinguished Professor of Psychology at the University of Scranton, where he previously served as Dean of the Graduate School, Director of Research, and Interim Provost/Academic Vice President. He was formerly Associate Vice Chancellor for Graduate and Professional Programs, University of Wisconsin-Green Bay and Co-Director of the statewide Wisconsin Assessment Center. He has an extensive history as a reviewer for the Mental Measurements Yearbook, completing his first review in 1978 and contributing 15 reviews in eight different yearbooks. He was recognized as a Distinguished Mental Measurements Yearbook Reviewer in 2014. He is the author or co-author of four books on measurement and research methods, several nationally used standardized tests, and over 200 published articles, chapters, and presentations related to psychological and educational measurement. He has won all three of his university’s teaching awards. He frequently serves organizations on matters of test design, evaluation, and interpretation. He is a long-time member of the American Psychological Association, American Statistical Association, American Educational Research Association, and National Council on Measurement in Education. He holds a bachelor degree from John Carroll University and both master and doctoral degrees in Psychology from Fordham University, with a specialization in Psychometrics.