Editors Note: When the manuscript for the following article came to the attention of the editor of this journal (Psychology of Espionage Reports), it was first being considered for publication in the other journal (Electronic Journal of Forensic Psychonomics) supervised by this same editor. Although the subject of this manuscript is not at all focused specifically upon espionage or similar matters, it reports the seeing of a very major security concern, involving psychological personnel characteristics, regarding the quality of security police protection that was seen operating in one of our Country’s major intelligence organizations. For this reason, this manuscript was assigned publication status in the Psychology of Espionage Reports as it is believed that the research reported will (or should be!) of increased interest to readers who possess some type of connection to or association with U.S. intelligence agencies, in general. 

The article ("Oh No, Not a Negative Direction Validity Association!") quite briefly describes a segment of aptitude test battery revalidation research that was carried out about a half decade ago in a large and ‘sensitive’ US. Government agency. Readers of this article will not be surprised to learn that the results of this specific reported research have not been widely circulated during this past half-decade time period; in fact, it can be believed that the results actually have been hidden! It is not known whether the conclusions that can be drawn from the reported research can be generalized any further with other ‘sensitive’ governmental agencies. Hopefully, they cannot! 
 

     No author or agency/institutional identification is being given here for reasons somewhat similar to why the author of the novel, "Primary Colors: A Novel of Politics," originally chose to (or at least try to) remain unidentified (i.e., Anonymous, 1996). The to-be-reported findings of conducted research though are believed to be so unusual that they deserve a chance for some public documentation that will then serve as a record that such results can be seen (hopefully, almost never) in the real world. 

     We were involved in a major psychometric project of rebuilding and validating a multiple aptitude testing battery that historically had been used in a large governmental agency for personnel selection purposes. The rebuilt or new version of the aptitude testing batteries (ATB) ended up being a more simplified version of the previously employed batteries and consisted only of nine subtests. These nine subtests, based upon rather strong psychometric data and logic were believed to measures three underlying cognitive ability factors (i.e., verbal ability, spatial/perceptual ability, and clerical ability). 

     The job performance measures that served as the validity criterion variable were based upon supervisors’ ratings of close to two thousand employees and were based upon behavioral-examples, anchored rating scales for 18 abilities and 18 knowledge areas. 
 

2 

     Quite acceptable judgmental reliabilities were found with most of these rated performance measures across the number of studied jobs. With the studied security guard (this was not the ‘formal’ job name) job, an alpha coefficient of 0.93 was found associated with its six rated performance dimensions. A rather large range of aptitude test validity coefficients (based upon multiple regression determinations) were obtained, across the many different jobs, that ranged from a high of 0.45 to a low of 0.25. 

     Of course, it is well known that multiple correlations cannot be negative as they represent the correlation between the best predicted scores with the obtained scores. However, it is quite possible to see negative correlations between the individual predictor variables with the criterion variable. It is this latter situations that will mainly be described in the present article. 

     As indicated in a prior paragraph, the to-be-focused-upon job classification involves a protective police type of job that some may regard mainly (and not incorrectly) as a security guard (SG). ATB scores, as well as complete job performance ratings, were obtained for 156 SGs. (Note - at the time this research was being conducted the number of SGs in the involved agency’s work force was several hundred.) The first-order correlations between the job performance ratings and the scores on the nine ATB subtests were found to be mostly negative in direction. These correlations for the Pattern Matching, Matrices, English Usage, Word Meaning, Spelling, Directions, Arithmetic, Name/Number Checking, and Artificial Language subtests were: -.17, -.04, 02, .07, -.05, .07, -.01, -.18, and -.04, respectively. The first and eighth correlation coefficients (both negative in direction) were the only ones that reached statistical significance (p < .05). 

     When a multiple correlation (based upon a nonstepwise solution) was computed involving this particular job classification, a coefficient of 0.34 was found and could be regarded as being statistically significant (F = 2.15, df = 9 and 146, p < .03). If we stopped here, most experienced psychometricians would be most likely to easily assume that measured cognitive ability was simply not too systematically related, in any reliable fashion, to rated job performance for this particular job classification, even though some indications of statistical significance were seen. To conceptualize that cognitive ability to be negatively associated with job performance, in a real-life government work situation, is a bit mind-boggling and is quite inconsistent with the positions put forth by Ree, Earles and Teachout (1994) and others such as Schmidt and Hunter (e.g., 1977). 

     To better examine the matter of predicting job performance from a measure of g (or general intelligence), for each studied SG, his/her factor score based upon the unrotated first principal component of an analysis of the nine ATB subtests was computed. The relationship of this factor score (which is believed to be fairly widely accepted as a measure of g) with the overall job performance ratings for the SGs was -0.03 (N = 166; r is quite obviously nonsignficant). 
 

3 

     However, some additional information renders this situation much more interesting. All of the SGs were graduates of a well-regarded U.S. governmental police academy that involves a training period of eight weeks. With the studied sample of SGs, it was found that the final overall academic grade score for this eight-week training period was available (from a previously conducted job performance prediction investigation) for 146 SGs. Since no one is permitted to remain employed by the specific involved agency as a SG, after completing this police academy training, if one obtains a final score below 70, it explains why the observed final grade score range did not drop below 70 for any of the studied SGs. It can be therefore believed that the studied group of SGs could be considered to represent a truncated sample of all those who attended the academy as it is likely that some did obtain nonpassing final grades. 

     When we included the final grade scores (from the police academy) into the already studied ATB scores and job performance ratings database, ten new correlation coefficients were additionally computed. Inspection of these coefficients revealed that the nine aptitude subtests all positively correlated (range was from 0.04 to 0.42), one of which was significant at the .05 level and four were significant at the .01 level). When this correlational information was input into a multiple regression analysis, a highly significant multiple R of 0.48 was found (F =4.83, df = 9 and 146, p < .00001). In other words, the final academic grade for an eight-week police academy exposure is highly correlated with multiple aptitude scores on a cognitive ability test battery that was administered anywhere from about two to seven years following the police academy training experience. Such is really not at all surprising as it is well understood that cognitive ability is generally found to be quite well associated with academic performance (e.g., Ree & Earles, 1991). 

     The correlation coefficient between the police academy final grade scores with the job performance ratings (which were made anywhere from two to seven years following the completion of the police academy experience was found to be 0.01 (N = 99; quite obviously no existing systematic linear relationship is indicated). Howver, the correlation between the police academy final grades with the previously described measure of g (the factor score from the unrotated first principal component) was quite different in its magnitude - it was 0.46 (p < .01, based upon N = 99). This particular correlation coefficient is quite consistent with the argument put forth several years ago by Ree and Earles (1991). 

     What does this all mean? Quite simply put, it most certainly suggests that the involved supervisors of the studied approximately 150 SGs seem to be more likely to give more favorable job performance ratings to those SGs who apparently possess lesser degrees of cognitive ability and that they assign lesser favorable job performance ratings to those SG subordinates who possess higher levels of cognitive ability. It should be noted that the SG sample quite uniformly demonstrated, when compared to studied employees in the agency’s many other studied jobs, a low position on the cognitive ability dimension. 
 

4 

     Although there were a couple very unusual ‘outriders," for almost the entire studied SG sample, their scorings on the ATB tended to be rather low. On the average, they (i.e., the SGs) had the lowest ATB scores of any of the other many different job employee samples associated with the involved agency. 

    These results are not what one would expect, especially when one considers historically the literature pertaining to the usually found relationships between measured cognitive ability and subsequently obtained measures of job performance. A most likely explanation for the obtained results is believed to involve the cooperativeness or obedience characteristics of the studied SGs. Those who are more likely to question orders, operational procedures and who are generally more ‘threatening’ to their supervisors are probably those who tend to be more cognitively blessed than those who never do anything to suggest that they might be regarded as being "boat-rockers." This would seem to be somewhat similar to the situation when the U.S. Government was still drafting young males into the U.S. Army. It was not at all uncommon at that time to hear draftees complain about how ‘dumb’ and uneducated were many of their enlisted NCOs or even those who had enlisted instead of waiting to be drafted. At the same time, it was common for many of the NCOs to regard the draftees as being nonrespectiveful or even troublesome, when compared to those who had enlisted. 

     Among those who were involved in a recruitment effort for a good number of the studied SGs, it is widely believed that many of the SG applicants had been misled to believe that the SG job entailed much more sophisticated and responsible police functions than actually took place. Seemingly, the most usual and frequent kind of work assignment for a SG is to function almost solely as a nicely uniformed door guard or monitor. Boredom amongst the more intellectual SGs was said to be widespread; if bored then job dissatisfaction would seem to be the next serial consequence. If this occurs, then it would not be unlikely that lower than average job performance ratings might be assigned to those exhibiting some communicated signs of job dissatisfaction. Although it would be most desirable to know the causes or underlying reasons for the reported relationships between measured cognitive ability, training scores, and job performance, such is not essential as a raison d’être for this research results documentation. 

     In any event, the situation described in this paper was, at the time of the research completion, communicated to organizational management. It has been hoped that some major changes, as a result of this knowledge, did take place to correct the basis for the obtained results. No organization wants to be in the position whereby if one were tasked to predict which individual employees, in a specified job classification that is alleged to be very complex and highly responsible, would be the best performers, one would have to, based upon statistical prediction logic, choose those with the lesser levels of cognitive ability. Needless to say, the involved governmental agency was not at all expecting (nor wanting) these described results. Quite obviously, within the involved governmental agency, aptitude test scores should now be only very carefully used in any SG 
 

5 

employment selections. It was wondered, by those who conducted this reseach, whether these kinds of findings with governmentally employed SGs, might be more widespread than suspected and not entirely unique to the agency involved. 
 

Anonymous (1996). Primary colors: A novel of politics. New York: Random House. 

Ree, M. J., Earles, J. A., & Teachout, M. S. (1994). Predicting job performance: Not much more than g. Journal of Applied Psychology, 79, 518-524. 

Ree, M. J., & Earles, J. A. (1991). Predicting training success: Not much more than g. Personnel Psychology, 44, 321-332.

Schmidt, F. L. & Hunter, J. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. 
 

                                                                               Return to Dr. Stone’s Home Page