Personnel Security Standards Psychological Questionnaire (PSSPQ)
Note – In recent months, Dr. LeRoy A. Stone, the developer and current commercial offer of this test, has been repeatedly asked, mostly by trained psychologists, to provide the psychometric statistical information that is descriptive of past reliability and validity estimations for the PSSPQ. Although the several other Web pages, which have been on the Web for generally over a year, that are descriptive of the PSSPQ, provide a good deal of description of both the historical as well as the technical background of this test, no in-depth description of the reliability/validity determinations have yet been presented. Communication of this information is the raison d’etre of this current presentation of Web sub-pages that are devoted to an adequate description of PSSPQ reliability/validity estimation determinations. Due to the extreme length of the following presentation, it is suggested that readers print out a ‘hard copy’ to facilitate study and their understanding.
As indicated in the Note-paragraph above,
additional information regarding the PSSPQ’s estimations of reliability
and validity has been mainly requested by psychologists, especially those
with strong interests and backgrounds in psychometrics. To satisfy these
requests, the following set of Web sub-pages is presented. These pages
will most likely present a kind of technical information that either will
not be understood by a goodly number of people, as it has been written
especially for those with sufficient psychometric expertise backgrounds.
However, for those with true or bona fide interest in the PSSPQ
and possible usage of it, a reading of the following pages should be considered
to be essential. The contents of the following pages can be considered
to provide a sound basis for truly believing that the PSSPQ can accomplish
that what it as been described as being capable of doing. The
PSSPQ is, without question, a highly reliable and valid psychometric testing
instrument!
Estimations of Reliability
Test-Retest
Estimation of the PSSPQ’s reliability was
initially computed while the PSSPQ was in the early stages of development.
After the final selection of items was accomplished, which narrowed a starting
pool of prospective items numbering about 400 down to 72, which is now
what comprises the PSSPQ. This collection of 72 items was then administered
to a group of 74 adults (none of whom was involved in the original standardization/validation
group). None in this groups was being processed for any security clearances
at the time of the testing. Although never recorded the time, is was recalled
that approximately one-half of this group (N =74) was one gender or the
other. This group was then again administered the PSSPQ 10 days following
the first administration. These data were then used to calculate a test-retest
reliability estimation; which was found to be very high (r = 0.94,
p
< .0001). This particular test-retest measure was of the PSSPQ Total
Scores (the sum of the first 11 scales scores which correspond to the 11
DCID 6/4 adjudication concerns that are addressed when one is being evaluated
for TS-SCI clearance status). [Actually, when research on the PSSPQ first
started, the Governmentally-made adjudication decisions regarding the granting
or non-granting of TS-SCI clearance status was guided by the Director of
Central Intelligence Directive 1/14 (or DCID 1/14) was in effect. In 1998
some fairly minor changes were made and the new governmental document was
now labeled as the DCID 6/4.] In some ways employment of these 74 individuals
for a test-retest determination of reliability was somewhat ideal in that
a rather conservative under-estimation could be expected due to characteristics
of this group. Namely, since they had no interest in security clearance
matters as they were college students. Compared to those who would be highly
personally interested in security clearance adjudication concerns, this
student group could be expected to be more careless, less motivated, and
certainly less involved in the task than would be the actual persons who
would be taking the PSSPQ because they were highly interested in learning
of their chances for being granted a security clearance.
Internal Consistency
After the PSSPQ had been initially standardized/validated
(based upon an N of 102.. This group consisted of employees of contractor
organization who were being processed for TS-SCI clearances and who at
the time of their PSSPQ testing had been already identified a persons who
had been identified for further additional processing due to the fact that
some difficulties had been seen so far in the processing that had already
taken place. This sample of 102 was obtained in over slightly more than
one year’s time. Seventeen percent were females; ages ranged from 22 to
60 years, the mean age being approximately 36 years (SD = 10). The
average education was 15.5 years. (SD = 2.7). Based on the results
of an administered verbal abilities intelligence test, the mean derived
deviation IQ score for the group was 107. Approximately, 25-30 different
contractor firms were represented which were sponsoring these individuals
for potential TS-SCI access. These subjects came from all over the country,
with the West Coast and East Coast being disproportionally over-represented;
however, there were some who were living and working in the central portions
of the Nation.
Based upon their MMPI scores (each subject was also administered this test in addition to the then beta version of the PSSPQ, this sample appeared to be rather representative of the general population as their mean MMPI scales scores seemed to be close to recognized norm values. In the studied sample of 102 individuals, 61 were eventually and finally successful in obtaining TS-SCI access; 41 therefore were not successful in obtain this access. This particular sample was especially well suited for the validity testing of the PSSPQ.
Based upon PSSPQ data from this standardization group, some internal consistency forms of reliability were determined. The internal consistency forms (i.e., based upon Kuder-Richardson models) were employed with the PSSPQ and its scales and such estimation attempts resulted in very encouraging and supportive results. When the five-level item responses are ‘collapsed’ to only two response levels, Kuder-Richardson Formula 20 reliability estimates for the Close Relatives and Associates, Sexual Considerations, Undesirable Character Traits, Financial Irresponsibility, Alcohol Abuse, Illegal Drugs and Drug Abuse, Emotional and Mental Disorders, Record of Law Violations, Security Violations scales, PSSPQ Total Score and LIE Scale are as follows (respectively): 0.93, 0.90, 093, 0.80, 0.94, 0.96, 0.91, 0.80, 0.93, 0.97, and 0.86. Again it should be noted that these Kuder-Richardson reliability estimates are based upon the sample of studied contractor employees who were, at the time, being processed and considered for possible security clearance status.
Split-Half
Even though the number of items in the
PSSPQ sub-scales are low (ranging from 1 to 10 items per scale), split-half
(corrected for length by employing the Brown-Spearman correction formula)
reliability estimations were computed. With several of these quite short
PSSPQ sub-scales, surprisingly high split-half reliability estimates were
obtained. For example, with the Illegal Drugs and Drug Abuse Scale, the
Emotional and Mental Disorders Scale, and the Record of Law Violations
Scale, the reliability estimates were: 0.83, 0.90, and 0.81, respectively.
The reliability estimations for the involved
remaining sub-scales were lower were (with one exception) statistically
significant at the .001 level (the exception was significant at the .01
level). The exception sub-scale was the three-item Security Violations
Scale) and it is believed that this scale was much too short a scale for
appropriate utilization of the split-half reliability estimation paradigm.
Of course it was impossible to utilize this reliability estimation paradigm
with the two PSSPQ sub-scales which involved only one item each. In general,
utilization of split-half methodology has resulted in quite favorable reliability
estimations of the PSSPQ scales. Again, it should be noted that these estimations
were computed from the contractor employee (N = 102) sample.
The Reliability Estimations Hold Up in Cross-Validation Research
When the PSSPQ was cross-validated with
an larger and independent sample (i.e., N = 179), approximately
one year following the earlier initial validation (i.e., with an N
of 102), reliability estimations, of the kinds (with the exception of the
test-retest model) described in the several preceding paragraphs, were
again computed. Basically what was found, just about the same size reliability
estimations were again calculated. Just about all of the differences between
comparable reliability coefficients was about what one would expect on
the basis of chance alone. Based upon all the test administration data,
coming from both the original and the cross-validation samples, when submitted
to reliability estimation paradigms of the internal consistency (i.e.,
Kuder-Richardson Formula 20) and split-half (when corrected for scale length)
types, a reassuring appearance of highly acceptable test reliability can
be inferred.
Estimations of Validity
Face/Content Validity
Validity for the PSSPQ was studied in several
different fashions. It, of course, possesses almost perfect ‘face’ or ‘content’
validity as it was directly constructed from a very careful study of the
11 adjudication concern areas, as described in the DCID 6/4. With great
care for accuracy and completeness, each item in the large original collection
of hundreds of items was developed so that the items would all have face/content
validity. Even some of the reduction of number of items down to the final
72 item collection made use of idea of wanting a high degree of face/content
validity for the items. All of this was true for all PSSPQ scales except
for one, the LIE Scale. It, of course, would be hoped to possess very low
‘face’ validity. Much of the item selection was based upon determination
of the degree of correlation between response to the item and the final
validity criterion of success/failure to be granted high-level security
clearance status.
Empirical [Prediction] Validity
However, there were a couple other validity
criteria employed in addition to the success/failure to obtain hoped-for
clearance criterion. These were the degree of relationship (i.e., correlation)
with the interviewing psychologist’s favorable or unfavorable recommendations
and with the chief psychologist’s case review recommendations of favorableness
or unfavorableness. Empirical validity was studied (using the original
102 contractor employees or applicants for employment) by correlating PSSPQ
scorings with the three just described validity criterion variables. Multiple
correlation coefficients (in this specific situation such could also be
referred to as simple discriminant functions) were computed between the
12 PSSPQ scales scores and each of the validity variables. The multiple
correlation for the 12 PSSPQ scales with the interviewing psychologists
dichotomous recommendations was. 051 (p < .05); the multiple
correlation for the 12 PSSPQ scales with the chief psychologists more final
recommendation was 0.53 (p < .05); and the multiple correlation
for the 12 PSSPQ scales with the actual final adjudicated decision as to
whether the individual would or would not be granted a clearance was 0.51
(p < .05). Inspection of the computed multiple correlation coefficients
and their associated statistical information does clearly show that not
all of the PSSPQ scales are strong or important predictors of whether persons
will be eventually successful in being granted TS-SCI access. However,
some of these scales were not actually expected, with this particular sample
of subjects, to show any relationship. For example, the first PSSPQ sub-scale,
Loyalty, was not expected to be a valid statistical predictor with this
sample simply because no one in the sample endorsed (nor were they expected
to endorse) a statement of non-loyalty to the United States. Another example
here would be the 11th PSSPQ sub-scale, that dealing with a
prior history of having committed security violations; only a couple of
the studied 102 contractor employees did in fact admit having been involve
in some prior security violation (generally a minor kind of violation when
they had been in military service). With only an N of 102, one would
not normally expect any significant number to admit to, based upon actual
experience, having been caught in any real security violation in the past.
Anyone who would admit to such, along with admitting to being non-loyal
to the United States, would not be expected, by their own volition, to
be in a group of individuals actually being processed for TS-SCI access
unless they perhaps had already defined plan for espionage.
In attempting to ascertain how PSSPQ results might be used to maximize accuracy in the prediction of whether individuals would or would not be granted TS-SCI access, it was discovered that utilization of specific sets of PSSPQ items, rather than the originally constructed PSSPQ scale entities, would achieve significantly higher and improved levels of prediction success. For example, it was found that an optimally weighted sum of information, based on a specific set of 25 PSSPQ items, would correlate extremely high with the validity criterion of whether, in fact, the involved individuals (N = 102) would or would not be granted TS-SCI access. The actual multiple correlation coefficient, in this specific 25 predictor items situation was 0.79 (p < .001). It has been found that the number of PSSPQ items can be as low as 15 and still the validity coefficient remains quite high (e.g., above 0.73).
When the best predictive 25 PSSPQ items are considered, they allow for extremely accurate, early-on predictions of whether an individual will eventually be granted or not granted TS-SCI access. For example, with the studied 102 personnel, when the 25 items response information was entered into a developed multiple regression equation (or a simple discriminant function model), such allowed for quite accurate, early predictions of what actually later occurred. Out of the 102 persons only eight errors of prediction occurred. Seven of the errors might be regarded as being of the ‘false positive’ type in that these seven were predicted to obtain TS-SCI clearances and in fact they were not successful in this regard. Only one error of prediction, of the other type, occurred in that the involved individual was predicted (based on only information from the 25 PSSPQ items) to not obtain TS-SCI access and in fact was successful in this regard. The involved empirically determined accuracy rate therefore was quite high; 92.2% accuracy was achieved in correctly predicting whether an individual would or would not be eventually granted high-level clearance status.
If one made use of additional statistical prediction information, such as the standard error of the multiple regression, then even more highly improved predictions can be made. For example, no errors of prediction were noted for those individuals who achieved prediction scores at or above the mean prediction score for ‘successful’ individuals; also, no errors of prediction were noted for those who achieved prediction scores at or below the mean prediction score for ‘unsuccessful’ individuals. Only two errors of prediction occurred for individuals who scored on the multiple regression prediction scale, which were within one standard error of the multiple regression prediction scale of their respective means on this scale. Therefore, almost all (six out of eight) errors of prediction did occur for individuals who were almost midrange between the ‘successful’ and ‘unsuccessful’ means on the multiple regression prediction scale.
The prediction accuracy described in the above couple of paragraphs can be more completely and dramatically understood by inspecting a tabular presentation of this set of statistical prediction information. Such is shown below:
Not
Selected
Selected
Prediction Scores
2.25 – 2.49
0
3
2.00 - 2.24
0
11
1.75 - 1.99
1
27
1.50 - 1.74
6
19
1.25 - 1.49
12
1
1.00 - 1.24
13
0
0.75 - 0.99
9
0
Inspection of the above tabular presentation clearly shows that the prediction scale (based upon the multiple regression prediction model involving 25 PSSPQ items) ranged from a possible low of 0.75 up to a possible high of 2.49. The mean score on this scale for the "Not Selected" group (which consisted of those 41 individuals who were not successful in later obtaining high-level security clearance status.) was 1.00; the mean score on this same scale for the "Selected" group (consisting of those 61 individuals who were later successful in obtaining their hoped-for security clearances) was 2.00. It can be easily seen that the higher and the lower scores on this prediction scale, with just about perfect accuracy, predicted whether, in fact, that the associated individuals would or would not be eventually granted TS-SCI access. A small number of prediction error (eight totally) occurred only for those individuals who had rather scores in a quite narrow mid-range segment
Readers (most likely, psychometric oriented psychologists) of this presentation should be aware that the obtained 0.79 predictive validity correlation coefficient is unusually high for any psychological test used in a ‘real world’ situation. As has been discussed by Cronbach (1960), applied psychologists have almost "abandoned their insistence on validity coefficients of .70 or .80 for all tests" (p. 349) as such are very seldom encountered. He noted that, based on 30 years of practical testing experience, "we cannot obtain such standards." He used this as an introduction to the idea that validity coefficients "as low as .30 are of definite value" (p. 349), In this regard, one is reminded that Strong (1943) [one of the "greats" in the field of psychometrics] commented that he had observed that test critics who are hostile towards the idea of lower value validity coefficients (i.e., correlations) are quite willing to accept information of no greater dependability "when (they) play gold or employ a physician." According to Strong, the correlation of golf scores between the first and second 18 holes in championship play is about .30, and the reliability of medical diagnosis is near.40 (i.e., p. 55).
To further show that PSSPQ scorings are valid in a predictive sense, it should be noted that multiple correlations between the set of 25 PSSPQ items with interviewing psychologists dichotomous recommendations was seen to be 0,718 (p < .001) and with the chief psychologists later, further-on, dichotomous recommendations was just barely higher, 0.725 (p < .001).
The particular 25 PSSPQ items, along with their multiple-regression B weights will not be described in this presentation as this particular prediction information of considered to be proprietary information and will not be publicly shared. However, it can be stated that the selected 25 items were rather representative of the total 72 items, although not all of the PSSPQ scales had items in this item collection. A principal components factor analysis was done with the 25 item collection and did contribute some further understanding of what was being measured by this particular collection of items. It was interesting to note that two of the 25 items did not come from PSSPQ scales based upon DCID 6/4 adjudication concern areas, but rather came from the 10 item length LIE Scale.
Cross Validation
Several attempts to cross validate the
PSSPQ have been accomplished. The first attempt was based upon a psychometric
maneuver sometimes known as the "folding over technique" and also involved
the "best" 25 items approach. This cross-validation attempt employed data
from the first 50 individuals from the original validation study (i.e.
where the final total N was 102). A multiple regression prediction
model was constructed (using the adjudication decision as the criterion
variable). This was used to predict the final adjudication status of the
51st subject; then another multiple regression prediction model
was constructed but this time was based on the now 51 subjects in the sample.
This model was used to predict adjudication status for the 52nd
subject and then this subject was combined with the preceding 51 other
subjects in the sample. Then a new multiple regression model was computed
to predict adjudicated status for the next obtained subject, the 53rd
one and so forth until the last subject was obtained, the 102nd
subject.
Then a multiple regression model was computed using all of the PSSPQ items information from all 102 subjects. This model was then used to predict adjudication status for each of the studied 102 subject and such was compared to what was their actual adjudication status with respect to TS-SCI access. For only two of the last obtained 52 subjects did their prediction status change when it was based upon the multiple regression model which was computed using all 102 subjects information. For one of these two subjects his predicted status, based upon a prediction model not employing his scores in the development of the model, was that he would not be successful in obtain TS-SCI access, whereas when employing the prediction model, based upon all 102 subjects, he was predicted to obtain TS-SCI access. Exactly the opposite prediction results occurred with the other subject. As noted earlier, this form of cross-validation has been referred to as a "folding-over technique" and is especially useful in multiple regression (or discriminant function) situations where N size is not overly large. It should be noted that both of the above described individuals who changed predicted status when the prediction model included their own scores were seen as having positions n the prediction scale which was most central in the midrange region on the scale continuum. Such results do strongly suggest a contention that, with the limitation of only having employed a sample size of 102 studied subjects, very favorable cross-validation has been demonstrated
Approximately a year or so later, the PSSPQ had been administered to another independent sample of contractor personnel who were also involved in the process of being evaluated for potential TS-SCI clearance status granting. Basically, somewhat similar demographic description of this group was seen when compared to that noted for the first or initial validation group. One difference was that the level of non-success in later being granted was of a lesser degree than was noted with the first or initial validation group. For the earlier group, it was roughly about 41% whereas with this second group (N = 179) it was quite a bit less, about 24%. When a multiple correlation coefficient (or simple discriminant function) coefficient was computed for the "best" 25 PSSPQ items and the success/failure determination, regarding the granting of high-level security clearance status, it was found to be only slightly less than the numerical value of the original validity coefficient; the cross-validation coefficient was seen to be 0.74, as compared to the original validation coefficient of 0.79. In terms of predictive validity, the PSSPQ most certainly can be regarded as showing excellent cross-validation predictive validity.
Construct Validity
Construct validity can also be shown with
some of the PSSPQ scales, namely those which would involve or which would
be expected to involve mental and emotional pathology. Correlations were
computed with MMPI scales (which had also been administered in addition
to the PSSPQ. There were no surprises here to psychologists who are familiar
with psychological testing of candidates for high-level security clearances.
There were seen a number of rather low-level, some statistically significant
correlations, between the PSSPQ scales and some of the MMPI scales. Although
no elaborate or full description will be attempted here to describe the
specific correlations between PSSPQ and MMPI scales (a more lengthy description
of these relationships has been described elsewhere), some overall ideas
of relationships will be suggested. The PSSPQ scales pertaining to sexual
considerations, undesirable character traits, financial irresponsibility,
alcohol abuse, illegal drugs and drug abuse, emotional and mental disorders,
and record of law violations all seem to show (in the expected directions)
statistically significant correlations with certain MMPI scales (especially
the K and Scales 4, 6, 7, 9, and 0). The PSSPQ LIE Scale correlated .53
with the MMPI L Scale. The multiple correlation of the 10 LIE Scale items
with the MMPI L Scale was quite high (R = 0.62, p < .001).
In general, if one were to inspect carefully a correlation matrix which
included the PSSPQ scales along with the MMPI scale, an overall conclusion
of the presence of a high-level of construct validity could be easily entertained.
Factorial Validity
An attempt to secure factorial validity
was accomplished by a factor analysis of the 12 PSSPQ scales (11 adjudication
concerns and the one LIE Scale). The method of principal factors was employed;
four factors were extracted and these were submitted to a Kaiser-type varimax
rotation. The four factors accounted for over 56% of the common variance
of the involved scales. Communalities for the scale variables were not
overly large and were quite variable (range: 0.62 to 0.15). The obtained
and rotated factor structure seemed easily and readily interpretable. For
example, Factor I (which accounted for about 20% of the common variance)
involved only two high positive loadings, those of the Close Relations
and Associates Scale and of the Cohabitation Scale. Factor II (which accounted
for almost 15% of the common variance) was especially interesting in that
it was markedly bipolar –high positive loadings for the Emotional and Mental
Disorders Scale, the Loyalty Scale, and the Undesirable Character Traits
Scale with a high negative loading for the LIE Scale. Factor III (which
accounted for almost 12% of the common variance) involved moderate-sized
positive loadings for the following listed PSSPQ scales: Alcohol Abuse,
Security Violations, Undesirable Character Traits, Law Violations, Financial
Irresponsibility, and Sexual Considerations. This third factor seemed to
be the closest thing to consider as a type of ‘general factor’ for suitability
concerns. Factor IV (which only accounted for about 10% of the common variance)
really only had one high loading and that was for the Illegal Drugs and
Drug Abuse scale. Rather interestingly, on all four factors, the LIE Scale
had negative loadings; with a very high negative loading on Factor II (see
description of Factor II above).
This factor analysis of the PSSPQ scales (which reflect the DCID 6/4 adjudication area concerns) clearly show that "suitability-nonsuitabilitiy for TS-SCI access is not a simple construct but is something which must be understood multidimensionally. A separate research investigation was completed a few years back that employed this PSSPQ data, and was focused upon study of the DCID 1/14 (i.e., which is an earlier version of the now-current DCID 6/4) adjudication area concerns. This ‘other’ research employed a principal components factor analysis paradigm. In this other research, almost the same factorial type results were obtained. Ina the principal factors analysis, the principal factor, Factor I was almost identical to the principal components Factor III; the principal factors Factor III was almost identical to the principal components Factor I; the principal factor Factor III was almost identical to the principal components Factor II; and the principal factors Factor IV bore some major similarity to the principal components Factor IV. Communalities in the principal components analysis were uniformly higher than in the principal factors analysis. In the generalized paradigm of factorial validity, the obtained factor loadings results (actually, from two different factor analyses of basically the same data) can be considered to represent validity coefficients with respect to the obtained factors.
Although no real description will be attempted in the current presentation, factor analyses (i.e., using the principal components model, followed by varimax rotations) of the 10 (after omitting the LIE Scale and the single-item Loyalty Scale) PSSPQ scales has been accomplished. In other words, for each of the 10 scales, the items comprising the scale have been factor analyzed. As a result the factor structure for each of the involved PSSPQ scales has been obtained. In general, the results have proven to be readily interpretable and therefore have enlarged our understanding of the very specific nature of behaviors and misbehaviors associated with DCID 6/4 adjudication concerns.
Another area of PSSPQ research that has
been extensive, but which is not going to be described here, is that involving
the correlation of many PSSPQ scorings (e.g., items scores, scale scores,
factor scores, etc.) with other test scores (e.g., MMPI, verbal intelligence
measures, etc.) as well as with many biographic/demographic variables (e.g.,
age, gender, education levels, etc.) These many hundreds of correlations
have been entered into extensive matrices and have been subjected to a
number of multidimensional analyses (e.g., factor and principal components
analyses, canonical analyses, discriminant analyses, etc.). These research
endeavors have led to a very large number of understandings of what is
measured by the PSSPQ. It can be comfortably stated that much of these
understandings can be considered to represent a good deal of support to
add to and further support construct validity arguments for the PSSPQ.
Actually, some of this additional research that has extensively explored
correlations of a great many variables (biographic, demographic, psychometric
testing variables, etc.) adds even more credence to even further factorial
validity arguments.
What About the Reliability of the Adjudication Decisions Pertaining to the Granting or Non-Granting of High-Level Security Clearance Status
Readers of this PSSPQ development description presentation, mainly focused upon reliability/validity concerns, who truly have a psychometric background and who are not youngsters in the field, likely have some familiarity in the theory of measurement error (as found in Gulliksen, 1950 or Nunnally, 1967) and will quite easily understand what is argued in the following paragraph. If one is willing to assume in that with any two sets of measures, errors from each set are uncorrelated between sets and that error on either set is uncorrelated with ‘true’ scores, then the measurement model for the correction for attenuation can be developed. Students of theories of measurement error are quite familiar with the fact that from the classic formula for computing the correction for attenuation another formula can be derived provided that one make the assumption that the correlation between ‘true’ scores (i.e., without measurement error being involved) from both the variables involved be equal to unity. This particular formula is:
r12 = ( Ö r11) ( Ö r22)
In other words, the upper limits size of the correlation between variables 1 and 2 (e.g., the PSSPQ and the decison to grant or not grant TS-SCI security clearance status) is equal to the product of the square root of the reliability of variable 1 and the square root of the reliability of variable 2. In this fashion, it can be seen that a correlation coefficient between any two variables is a function of the reliabilities of the involved variables. One of the major uses of the above shown formula is that its elements can be rearranged so as to allow for the estimation of a reliability coefficient for one set of measurements if the correlation coefficient between the two measurement sets is known or established in some fashion and if the reliability coefficient for the other set of measurements has been established. Such is the situation with our PSSPQ data at the present time. A correlation coefficient of 0.79 (i.e., the multiple correlation coefficient) has been clearly described as having been found to describe the relationship between the a weighted sum of information associated with the so-called ‘best’ PSSPQ items with the final adjudication decision (made by the U. S. Government) to grant or not grant high-level security clearances. Also developed have been two different estimates of the reliability of the PSSPQ total scores measure; i.e., 0.94 based on the test-retest paradigm and 0.97 based on the Kuder-Richardson internal consistency model. If one accepts that the value of 0.95 (the average of the two reliability estimates) can represent a reliability coefficient for the PSSPQ, then the heretofore unknown and never-before estimated reliability for Government made adjudicational decisions, regarding the granting or non-granting of TS-SCI access, can now be known (or at least, argued)
When 0.95 (the PSSPQ’s averaged reliability estimation) and the 0.79 (the Multiple R involving 25 of the PSSPQ’s items with the final adjudication decision) are entered into the correction for attenuation formula given above in one of the previous paragraphs, then the up-to-now, unknown reliability for the adjudication decision variable (favorable or unfavorable) can be determined and specified. This particular determined reliability value, in the present situation can be easily shown to be equal to 0.656, which is not an overly impressive reliability coefficient. However, for a measurement that is only dichotomous in kind, such a sized reliability estimation value seems not to be surprising to those who have studied judgmental reliability for diagnostic and prognostic evaluations regarding human beings. It can be noted that that the estimated reliability coefficient, having a numerical value of 0.656, can be thought of only as an ‘upper-limit’ value, as this estimation value was obtained by having to assume that the correlation of ‘true’ PSSPQ scores with the ‘true’ adjudication decisions was perfect and equal to unity. Psychometrically speaking, it is almost certain that the actual reliability for the Governmentally made adjudication decision is much lower than the 0.656 value presented and discussed here.
The major conclusion which can be made, based upon information and logic described in the couple preceding paragraphs, is that if one wishes to increase the correlational prediction accuracy of the PSSPQ with respect to it’s ability to predict subsequent adjudication decisions, then it would be far more productive to attempt to increase the reliability for the making or formulating the adjudication decision-making rather than attempting to change, modify, or add-to the PSSPQ itself. All of the psychometric evidence points to a rather clear conclusion that the PSSPQ possesses excellent reliability. If there is a reliability problem in what is being measured, such is much more a problem with the Governmentally-made adjudication decision variable.
This is what has been observed frequently
in the past in the applied areas of psychology. When using well-constructed
psychometric tests in applied situations, it is also the rule and not the
exception, that the tests possess far superior reliability than do the
involved validity criterion measures themselves. A good example, are the
correlational relationships that are found between extremely well developed
and constructed mental ability tests with school grades. Academic grading
has been notorious for being known to possess very poor measurement reliability.
Another example that has been well discussed, in the industrial psychology
literature, relates to poor reliability for the job interview type situation
(e.g., Guilford, 1959; Ulrich & Trumbo, 1965) The advice in this type
of situation is not that we need to improved psychological tests in order
to more accurately predict final adjudication decisions but rather what
is needed are better adjudication decision-making procedures which can
allow for improved reliability of such decision making. Readers of this
paper are urged to NOT tell or suggest to Dr. Stone (the developer and
current purveyor of the PSSPQ) that more research should be done so that
this testing instrument should be improved, added-to, or in some way changed.
If one wants to improve the accuracy of using PSSPQ scorings to predict
success/failure of individuals to be granted or not granted TS-SCI clearance
status, the way to increase the prediction accuracy is very simple – IMPROVE
THE RELABILITY OF THE GOVERNMENTAL DECISION MAKING PROCEDURE FOR ADJUDICATING
DECISIONS TO GRANT OR NOT GRANT HIGH-LEVEL SECURITY CLEARANCE STATUS!!!
References
Cronbach, L. J. (1960). Essentials of psychological testing (2nd
ed.), New York: Harper &
Brothers.
Guilford, J. P. (1959). Personality. New York: McGraw-Hill.
Gulliksen, H. (1950). Theory of mental tests. New York, Wiley.
Nunnally, J. C. (1967). Psychometric theory. New York, McGraw-Hill.
Strong, E. K., Jr. (1943). Vocational interests of men and women.
Stanford:
Stanford University
Press.
Ulrich, L., & Trumbo, D. (1965). The relationship of validity
coefficients to the practical
effectiveness of tests in selection: Discussion and
tables. Journal of Applied Psychology, 23,
565-578.
Addendum
Based upon the content of the above web-pages,
along with the format design of the Web Site in which they are a component
part, it is expected that most readers of these particular pages got to
these pages using a link from one of many subsections in Dr. Stone’s overall
Web Site that pertain to the PSSPQ. However, for those readers who got
to the above web-pages (counted on MicroSoft Word to consist of about 16
or 17 pages) in some fashion other than directly through a link on one
of Dr. Stone’s pages, some link to these indicated "other pages’ within
Dr. Stone’s Web Site that pertain the the PSSPQ and its use. Links to a
good number of these pages that involve presentation and discussion of
content that pertain to the PSSPQ test are as follows:
|
http://www.home.earthlink.net/~lastone2/psspq.html (The above should be regarded as the index or "first page" to read when wanting knowledge about the PSSPQ) http://www.home.earthlink.net/~lastone2/individualsales.html (The above was designed
for those who might be interested as individuals in ‘taking’
http://www.home.earthlink.net/~lastone2/hrandsecdirectors.htm (The above was designed
especially to present information to an organization's HR and
http://www.home.earthlink.net/~lastone2/psspqfaq.html (The above was created to include many "frequently asked questions" [along with answers, regarding the PSSPQ, that have been posed to Dr. Stone during the past couple of years) http://www.home.earthlink.net/~lastone2/increasesuccesschances.html (The above was written so as
to explain how the obtaining, possession, and use of PSSPQ
http://www.home.earthlink.net/~lastone2/trustworthinesstests.html (The above was presented to document
a new test for trustworthiness, the Probity/
http://www.home.earthlink.net/~lastone2/psspqcertificate.html (This set of pages details Dr.
Stone's recent decision to 'award' an attractive certificate that
http://www.home.earthlink.net/~lastone2/psspqcomparison.html (This section describes in detail how
the PSSPQ is exclusively the only actually available
http://www.home.earthlink.net/~lastone2/psspqhelpgetclearance.html (This set of web site pages is devoted
to description of advantages of becoming knowledgeable
Click
Here to Go to
|
