Limitations of Self-Report Data free essay sample

Limitations of Self Report Data Abstract Self-report data may be obtained from a test or an interview format of a self-report study. The format of self-report study that will be used to discuss limitations of self-report data will be a test and a personality disorder test will be used as an example. For specific example answers for the test I completed the results all rated â€Å"low† for all personality disorders. Limitations arise from decreased reliability and validity and issues with credibility of responses due to response bias. Content validity, construct validity and criterion-related validity as well as test-retest reliability will be presented. The forms of response biases that will be discussed are social desirability, acquiescence, halo effect, extreme responding, midpoint responding, random responding, negative/positive bias, memory recall bias and cultural bias. The discussion will conclude with the role self-report data plays in psychology today. Limitations of Self-Report Data Self-report data has the advantage that they come directly from the test taker themselves and not from a third party; who would know someone better than themselves? Self-report studies can examine a large number of variables such as feelings, behaviours, attitudes and beliefs. They are easy to administer, inexpensive and most take a modest amount of time to complete (McDonald, 2008). Self-report tests are able to assess matters that are not observable such as feelings, beliefs and opinions. The scoring of self-report tests is standardized whether it be a scoring template for pen and pencil formats or computer-based interpretations for tests whose answers are entered into a data file (Hood Johnson, 2007). However, it does have some limitations. The validity of causal conclusions drawn from self-report data may be of concern because it can be influenced by factors such as the psychometric properties (validity and reliability) of the test, the context (environment/situation) of the test, and response biases (Dodd-McCue Tartaglia, 2010). . Validity Content Validity Content validity is the extent to which a test is measuring the domain it is supposed to be measuring (Hood Johnson). Meaning for example; does the 4degreez. com Personality Disorder Test measure for personality disorders? Computers allow for much more complex self-report test scoring, and are accurate and thorough. A test’s construction can also play a key role. The content validity of some tests relies on the appropriate qualifications and level of expertise of the designers of the test and of the database used for computer-based test interpretations (Hood Johnson). For the 4degreez. com Personality Disorder Test the designers decide what questions to include in order to identify the different features of personality disorders. The creators of the database decide which answers correlate with a personality disorder. For self-report testing the fundamental way a question is worded can influence a person’s interpretation and thus their response. In the 4degreez. com Personality Disorder Test this may lead to the erroneous detection of the presence or no presence of diagnostic criteria. For example in the: Q1 â€Å"Do you believe you have more difficulty with relationships than the average person your age? † (4degreez. com, n. d. , p. 1). I didn’t know how to interpret or answer this question; do they mean social relationships or intimate relationships? Had I understood it as both and answered â€Å"yes† my answer would have been measured as a diagnostic criteria for Schizoid Personality Disorder. When I completed the test I answered â€Å"no† because neither situation applied for myself. There was no information given in regards to the designers of the 4degreez. com Personality Disorder Test nor for the database used for the computer-based test interpretations used to score the test; therefore it is unknown on how it faired for content validity. In addition it is questionable whether the number of questions (#71, excluding â€Å"What sex are you? †) were adequate to accurately assess the presence of a personality disorder. Construct Validity Construct validity is the extent to which a test measures a theoretical construct (Dyce, n. d. ); that is, can the 4degreez. com Personality Disorder Test measure the presence of the different behaviours described by the diagnostic criteria for the different personality disorders? There are two subcategories of construct validity: convergent validity and discriminant validity. In the case of a personality disorder test convergent validity is the degree to which the test that should be theoretically related to a behaviour associated with a given personality disorder is in fact related. This form of validity is an example in which results should be taken in a person’s context or in conjunction with results of other forms of testing. For example, Q11 of the 4degreez. com Personality Disorder Test (n. d. ) â€Å"Do you have a difficult time relating to others? † (p. 1). If a person’s contacts are of at a lower education level their language or ideas may or may not be understood. For discriminant validity it is the degree to which the test that should not be theoretically related to a behaviour associated with a given personality disorder is in fact not related. No information was available to know how the 4degreez. com Personality Disorder Test faired on testing for construct validity. Howard (1994) claims that the construct validity coefficients of self-report testing are superior to those of other forms of testing. Criterion-Related Validity As Dyce (n. d. ) explained criterion-related validity is the relationship between the test score and a criterion measure such as a behaviour. In the 4degreez. com Personality Disorder Test responders may think they possess a great deal of self-knowledge when in fact they may not therefore their answers may not accurately reflect their behaviours. Concurrent criterion-related validity is when the test score and the criterion are measured at approximately the same time. Predictive criterion-related validity is when the criterion measure is collected after the test score is collected. When self-deceptive socially desirable responding (SDR) in self-report testing, as defined in the social desirability subheading below, is controlled the predictive criterion-related validity is reduced therefore this form of SDR should not be controlled (Paulhus, 1991). Reliability Test-Rest Reliability Test-retest reliability compares the results of two of the same tests taken at different times (Dyce, n. d. ). A person’s state of mind and conditions of he environment may influence responses. The length of time between which the tests are taken should be recorded so it can be taken into account when interpreting the reliability coefficient. Changes in environmental conditions (e. g. , temperature, noise level, or lighting) or state of mind can all affect test-retest reliability. I performed the 4degreez. com Personality Disorder Test on two separate days. The f irst time it was completed all ten personality disorders were rated â€Å"low†; whereas the second time one was rated â€Å"high†, two were rated â€Å"moderate†, and the other seven were rated â€Å"low†. Thus on this occasion the 4degreez. com Personality Disorder Test was not very test-retest reliable. This may be attributed to poor sleep the night before and not haven eaten yet, therefore I was a bit irritable. In addition students like myself, who have high cognitive demands, may take mental shortcuts in completing the test depending which will thus decrease reliability. Response Bias Social Desirability Socially desirable responding (SDR) is when a person answers questions in such a way that they â€Å"look good† to others. Such responses will obviously skew results. The issue of SDR in measuring personality, psychopathology, attitudes and sensitive behaviours has been of concern for over sixty years (Paulhus). Subsequently, various methods of measuring (but not controlling) SDR were developed. The â€Å"Lie scale† used in the MMPI to detect obvious dishonesty is an example of one such method (McDonald). Self-report tests such as personality disorder tests rely heavily on the honesty of the responder; however SDR may be done consciously or unconsciously. Another form of SDR is: 1) self-deceptive positivity which is an honest but overly positive self-presentation which is linked to a MMPI factor and 2) impression management which is a conscious and purposeful positive self-presentation (Paulhus). SDR is more likely to occur with questions that address personal issues such as beliefs and integrity. Many of the questions in the 4degreez. com Personality Disorder Test were laden with vulnerability for SDR; for example, Q21 â€Å"Do you sometimes profit at the expense of others, without being bothered by pain or damage you may cause them? (p. 1). It would take a lot of trust and honesty for a person to answer â€Å"yes† if it was in fact true. Computer based self-report testing helps reduce SDR because there is more sense of anonymity. Using ipsative (forced choice; e. g. , yes/no, agree/disagree or true/false) questions in self-report testing also helps reduce SDR. Had I completed the 4degreez. com Personality Disorder Test a cou ple years back I would most probably have been influenced by this type of response bias; however â€Å"looking good† is no longer of importance to me. When I completed the 4degreez. om Personality Disorder Test I did so honestly and truthfully so as the results would be accurate. Acquiescence Acquiescence is a response style dubbed â€Å"yea-saying† which is the tendency to answer â€Å"yes† or agree irrespective of the content of the question. One explanation for acquiescence lies in the personality trait of the responder associated with impulsiveness (Dodd-McCue Tartaglia). â€Å"Yes† answers are more likely when the question is ambiguous, neutral in desirability or vague (Dodd-McCue Tartaglia) and the responder is uncertain on how to answer. Most tests are now designed with evenly distributed negatively and positively worded questions, in order to maintain reliability. This form of balancing only controls â€Å"agreement acquiescence† (agreeing with similar types of questions) and not sufficient for â€Å"acceptance acquiescence† (agreeing with all questions irrespective of type) (Paulhus). On examination of the 71 questions in the 4degreez. com Personality Disorder Test it was concluded that the majority were negatively worded, thus it appears that appropriately balanced wording of questions was not incorporated.