Sample Mental Measurements Yearbook review

This review appeared in The Fourteenth Mental Measurements Yearbook (2001). Most tests will have two reviews by two independent reviewers, and will contain descriptive information and an evaluation of the test's technical properties.

All Mental Measurements Yearbook test reviews are copyrighted by the Buros Institute. Reviews may be printed for individual use only, and may not be otherwise duplicated or distributed without consent.

SAQ-Adult Probation III.

Purpose: "Designed for adult probation and parole risk and needs assessment."

Population: Adult probationers and parolees.

Publication Dates: 1985-1997.

Acronym: SAQ.

Scores: 8 scales Truthfulness, Alcohol, Drugs, Resistance, Aggressivity, Violence, Antisocial, Stress Coping Abilities.

Administration: Group.

Price Data: Available from publisher.

Time: (30) minutes.

Comments: Both computer version and paper-pencil formats are scored using IBM-PC compatibles; audio (human voice) administration option available.

Author: Risk & Needs Assessment, Inc.

Publisher: Risk & Needs Assessment, Inc.

Cross Reference: For a review by Tony Toneatto, see 12:338.

Review of the SAQ--Adult Probation III

DESCRIPTION. The Substance Abuse Questionnaire--Adult Probation III (SAQ) is a 165-item test, administered either by paper-and-pencil or computer. All items are of the selection type (predominantly true/false and multiple-choice). Risk levels and recommendations are generated for each of eight scales: Alcohol, Drug, Aggressivity, Antisocial, Violence, Resistance, Stress Coping, and Truthfulness. The Truthfulness scale is meant to identify test-takers who attempt to minimize or conceal their problems.

Nonclinical staff can administer, score, and interpret the SAQ. Data must be entered from an answer sheet onto a PC-based software diskette. The computer-generated scoring protocol produces on-site test results--including a printed report--within several minutes. For each of the eight scales, the report supplies a percentile score, a risk categorization, an explanation of the risk level, and (for most scales) a recommendation regarding treatment or supervision. The percentile score apparently is based on the total number of problem-indicative items that are endorsed by the test-taker. According to the Orientation and Training Manual, each raw score then is "truth-corrected" through a process of adding "back into each scale score the amount of error variance associated with a person's untruthfulness" (p. 8). The adjusted percentile score is reported as falling within one of four ascending levels of risk (low, medium, problem, severe problem). The responsible staff person is expected to use information from the report, along with professional judgment, to identify the severity of risk and needs and to develop recommendations for intervention.

DEVELOPMENT. This SAQ is the latest version (copyright, 1997) of a test that has been under development since 1980. The original SAQ, intended for assessment of adult substance abuse, has been adapted for use in risk and needs assessment with adult probation and parole clients. Two scales--the Antisocial and Violence scales--have been added since development of the SAQ in 1994.

Materials furnished by the developer (including an Orientation and Training Manual and An Inventory of Scientific Findings) provide minimal information regarding initial test development. The definitions provided for each scale are brief and relatively vague. The constructs underlying several scales appear to overlap (e.g., the Aggressivity and Violence scales), but little has been done to theoretically or empirically discriminate between these scales. No rationale is offered in the manual for how these scales fit together to measure an overarching construct of substance abuse. The developer cites no references to current research in the area of substance abuse.

TECHNICAL. Information describing the norming process is vague. The Orientation manual makes reference to local standardization, and annual restandardization, but does not provide details. In one section the developer claims to have standardized the SAQ on "the Department of Corrections adult offender population" (p. 7). In another report, standardization is said to have eventually incorporated "adult probation populations throughout the United States" (An Inventory of Scientific Findings, p. 5). One might assume, based on the citing of SAQ research studies involving literally thousands of probationers that the recency and relevance of norms is beyond question. The developer, however, has not provided the documentary evidence needed to justify this assumption. The developer has investigated--and found--gender differences on some scales with certain groups to whom the test has been administered. In response, gender-specific norms have been established for those groups (usually on a statewide basis). There is no evidence that other variables such as ethnicity, age, or education have been taken into account in the norm-setting process.

The items selected for use in the test have several commonalities. Most items focus on personal behaviors, perceptions, thoughts, and attitudes and are linked in a direct and very obvious way to the content of associated scales (e.g., "I am concerned about my drinking," from the Alcohol scale). Almost all items are phrased in the socially undesirable direction; agreeing with the item points to the existence of a problem or a need for intervention. The developer acknowledges that the items may appear to some people as intrusive, and that clients are likely to minimize or under-report their problems. In the SAQ, the response to this concern has been the inclusion of the Truthfulness scale and calculation of "truth-corrected" scale scores. Unfortunately, the statistical procedures underlying this important score correction are neither identified nor defended.

Internal consistency for the individual subscales of the SAQ has been well-established by a large number of developer-conducted studies that report Cronbach alpha estimates generally in the .80s to .90s. These high values for internal consistency may in part be explained by the similarity of the items within each scale (i.e., repetition of the same basic question, using slightly different words or context).

Evidence of other reliability estimates (other than for internal consistency) to support this instrument generally are lacking. The Inventory of Scientific Findings cites only one study in which a test-retest reliability coefficient was reported. Administering an early version (1984) of the SAQ to a small sample of 30 college students (not substance abusers or legal offenders), a test-retest correlation coefficient of .71 was found across an interval of one week.

Evidence to support the validity of the SAQ is limited. Some concurrent validity evidence is presented, in the form of multiple studies showing modest correlations between some SAQ scales and subscales of the Minnesota Multiphasic Personality Inventory (MMPI). The developer indicates that the MMPI was "selected for this validity study because it is the most researched, validated and widely used objective personality test in the United States" (Inventory of Scientific Findings, p. 14). This explanation, however, does not suffice as a rationale for use of the MMPI to support concurrent validity; and no theoretical framework is provided about how the SAQ subscales relate to the personality constructs underlying the MMPI.

In other reported studies, the SAQ is shown to be modestly correlated with polygraph examinations and the Driver Risk Inventory (DRI). Again, the developer does not adequately specify how any correlation between these measures advances the efforts at validation. The studies cited, and the validation process in general, do not meet accepted psychometric standards for substantiating validity evidence established in the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999). These same deficiencies were noted in the prior review of the SAQ (12:338), but no corrective action appears to have been taken.

COMMENTARY. The value of the SAQ as a measure of substance abuse severity with criminal justice populations seems to be compromised on a number of levels. First, the test lacks a clear focus. Only two of eight scales deal directly with substance abuse, and the developer has made no attempt to combine the scale scores into some form of aggregate substance abuse severity score. Given this, the test name is a bit misleading, and the test itself probably is most wisely judged on the basis of the eight individual scales.

Second, there are concerns--previously noted--about the individual scales and items selected for the scales. Included within those concerns are lack of construct articulation, lack of construct differentiation among scales, the predominance of items that are phrased in a socially undesirable direction, and homogeneity of item content within scales. Item phrasing and the bluntness of the items (e.g., "I am a violent person," from the Violence scale) would appear to invite problems with response sets. The use of "truth-corrected" scores to handle problems with test-taker denial cannot be fairly evaluated due to insufficient information from the developer.

Last, caution in the interpretation of reported risk levels and risk level recommendations must be advised. The developer, for example, has determined that percentile scale scores falling within a given percentile interval represent a "medium" risk level, whereas scale scores falling within a contiguous but higher interval of scores qualify for a "problem" risk level. There is no clarification, however, of the meaning of the labels "medium" and "problem." Further, there are no statements regarding how the two risk levels are to be discriminated from one another, and no identification of outcomes (or probabilities of outcomes) that are tied to the levels. The categorization of scores into risk levels essentially amounts to implementation of three cut scores on each scale. Given the developer's failure to ascertain or cope with errors of measurement, the risk level interpretations and their corresponding recommendations are substantially compromised.

SUMMARY. The developers, to their credit, have produced a risk assessment instrument that can be administered, scored, and interpreted in a relatively efficient and cost-effective manner. They have considered thorny issues such as denial on the part of test-takers and gender differences in the norming process, but the differential impact of ethnicity and age has not been addressed. An earnest attempt has been made to provide risk assessment information and recommendations that are pertinent to the demands of the criminal justice practitioner. On balance, however, the SAQ falls far short of the mark. Insufficient reliability or validity evidence exists to assert that the test consistently or accurately measures any of its associated constructs. There is continued doubt, in the words of the prior reviewer of the SAQ, that the test "conveys any useful information additional to simply asking the client if they have an alcohol-drug problem, if they are violent, and how they cope with stress" (Toneatto, 1995, p. 891). Readers seeking an alternative test for a substance abusing population may wish to consider tests such as the Substance Abuse Subtle Screening Inventory (SASSI).

REFERENCES

Toneatto, T. (1995). [Review of the SAQ--Adult Probation [Substance Abuse Questionnaire].] In J. C. Conoley & J. C. Impara (Eds.), The twelfth mental measurements yearbook (pp. 889-891). Lincoln, NE: Buros Institute of Mental Measurements.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.