- Research article
- Open Access
Test - retest reliability of two instruments for measuring public attitudes towards persons with mental illness
BMC Psychiatry volume 11, Article number: 11 (2011)
Research has identified stigmatization as a major threat to successful treatment of individuals with mental illness. As a consequence several anti-stigma campaigns have been carried out. The results have been discouraging and the field suffers from lack of evidence about interventions that work. There are few reports on psychometric data for instruments used to assess stigma, which thus complicates research efforts. The aim of the present study was to investigate test-retest reliability of the Swedish versions of the questionnaires: FABI and "Changing Minds" and to examine the internal consistency of the two instruments.
Two instruments, fear and behavioural intentions (FABI) and "Changing Minds", used in earlier studies on public attitudes towards persons with mental illness were translated into Swedish and completed by 51 nursing students on two occasions, with an interval of three weeks. Test-retest reliability was calculated by using weighted kappa coefficient and internal consistency using the Cronbach's alpha coefficient.
Both instruments attain at best moderate test-retest reliability. For the Changing Minds questionnaire almost one fifth (17.9%) of the items present poor test-retest reliability and the alpha coefficient for the subscales ranges between 0.19 - 0.46. All of the items in the FABI reach a fair or a moderate agreement between the test and retest, and the questionnaire displays a high internal consistency, alpha 0.80.
There is a need for development of psychometrically tested instruments within this field of research.
Negative public attitudes towards persons suffering from mental illness have been identified as an obstacle for recovery from illness and becoming full participants in society . The concept of stigmatization has been described and operational definitions have been made by various researchers . Although there is no clear consensus, stigmatization of people with mental illness is usually described as a complex of problems emanating from a lack of knowledge about mental health problems, negative attitudes and excluding or avoiding behaviours towards individuals suffering from mental health problems. For those who are subjected to stigma and their relatives this leads to a perception of the public as ignorant, prejudiced and discriminating in their view of mental illness .
It has been shown that stigmatization reduces life chances for persons suffering from mental illness. The negative effects include less access to mental health services and to advances in psychiatric treatment [3, 4], to psychosocial stress, to low socio-economic status and to delay in appropriate help seeking . Relatives of mentally ill persons identify stigma as a key problem that causes their ill relatives a lowered self-esteem, difficulties in making and keeping friends, and a tendency to deny that they have a mental illness . The relatives also experience burden due to the stigma associated with mental illness. In a study investigating family stigma one group of relatives reported thoughts of life and death in terms of their relatives and themselves.. Eighteen per cent of the relatives had sometimes felt that the patient would have been better off dead. Ten per cent of the family members in the same study reported that they now and then had experienced suicidal thoughts .
Persons with a diagnosis of mental disorder also present difficulties in seeking and keeping work. In a study of people suffering from schizophrenia in five European countries the employment rate ranged from 5% (England) to a maximum of 23% (Italy) . One explanation for the low employment rate is that employers discriminate against persons who have a history of psychiatric treatment. In a study of 200 Human Resource Officers in UK companies it was shown that a history of mental illness significantly reduced the chances of employment in comparison with a history of diabetes . Two studies in California found that employers perceived people with mental illness as being as undesirable as ex-convicts and that the only group they were more reluctant to hire were people with active tuberculosis .
To suffer from mental health problems also leads in many cases to a process of identity transformation where persons lose their previously held or desired identities and adopt a stigmatised view of themselves. This process is referred to as self-stigma , internalised stigma  or anticipated stigma . Self-stigma has been associated with reductions in protective psychological variables such as hope , self-esteem [15, 16], self-efficacy , empowerment [17, 18] and recovery beliefs . It is also associated with lower quality of life [15, 17] and an increase in avoidant coping strategies such as withdrawal and secrecy.
Several programs to fight stigma have been initiated as a consequence of the research findings that have indicated stigmatization as a threat to successful treatment and rehabilitation of individuals with mental illness. Some of these programs have been international, such as the World Psychiatric Association's Global Programme against Stigma and Discrimination because of Schizophrenia , others have been national such as the Changing Minds Campaign by the Royal College of Psychiatrists in the United Kingdom [20, 21], the PSYKE-Campaign in Sweden  and the National Mental Health charity SANE Australia . The overall ambitions have been to increase public knowledge about mental illness and its treatment possibilities, to create more positive attitudes towards people with mental illness and to reduce prejudice and discrimination. To some extent the aims of these campaigns have been achieved [19, 24]. However, despite all efforts to reduce stigma through public campaigns the results are not convincing and the field suffers from a lack of evidence about what works . The process appears to be a slow one. In a recent study 73% of a cohort of persons suffering from mental illness believed that most employers would pass over an application from a former psychiatric patient in favour of another applicant and that they are not considered as trustworthy as the average citizen (67%) or as intelligent as other people (56%) . In another recent study a list of 250 derogatory labels used among fourteen year old schoolchildren to stigmatize people with mental illness were described. The authors suggest that this wide-ranging and negative number of terms about mental illness appears to emanate from the media and from family and peers . All these findings indicate a quite persistent negative attitude in the general population.
There is a need for further work in the struggle against stigma and valid and reliable instruments need to be tested in order to measure effects of interventions. Link et al  have reviewed different methods for measuring mental health stigma but there is little information about psychometric properties of the instruments used in the various studies. There is often information about the internal consistency of the different scales but nothing about their test-retest reliability. The reliability in this respect is crucial when using the scales to evaluate the rate of success associated with different interventions. The present study was carried out to determine psychometric properties of two instruments used in earlier studies. The instruments chosen were: "The self-report inventory of fear of and behavioural intentions toward the mentally ill (FABI) [28, 29] and the set of questions used in the campaign "Changing Minds: Every Family in the Land" initiated by the Royal College of Psychiatrists . The purpose was to create Swedish versions of instruments that could be used in research on attitudes towards persons with mental illness. Being as no data concerning psychometric properties concerning these scales have been presented it is important to conduct a trial to establish this information.
The aims of the study were:
to investigate test-retest reliability of the Swedish versions of the self-report inventory: FABI and of the "Changing Minds"- questionnaire
to examine the internal consistency of the two instruments
Permission to translate the instruments was obtained from the original authors. The instruments were first translated into Swedish and then retranslated into English. The language in the different versions were compared and adjusted to the final Swedish versions.
The self-report inventory of fear of and behavioural intentions toward the mentally ill (FABI) is a ten item questionnaire with a five point response scale for each question. Responses are coded from 1 to 5 and the items include:
Fear: (response alternatives: strongly agree, agree, neutral, disagree and strongly disagree)
"I am afraid of people with mental illness"
Behavioural intentions: (response alternatives: very likely, likely, uncertain, unlikely, very unlikely)
"Would you object to having mentally ill people living in your neighbourhood?"
"Would you avoid conversations with neighbours who had suffered from mental illness?"
"Would you be willing to work with somebody with a mental illness?"
"Would you invite somebody into your home if you knew they suffered from mental illness?"
"Would you be worried about visiting somebody with a mental illness?"
"If somebody had been a former psychiatric patient, would you have them as a friend?"
"If somebody who had been a former psychiatric patient came to live next door to you, would you greet them occasionally?"
"Would you have casual conversations with neighbours who had suffered from mental illness?"
Changing Minds: Every family in the Land
The instrument was, in the original study, administered as an interview and two sets of data were obtained . The first set collected data about household composition and individual demographic and employment-related circumstances. This set was not used in the present study. The second set was designed to obtain perceptions about seven mental disorders, identified in pre-interview focus groups and then targeted by the Royal College of Psychiatrists campaign. The focus groups had indicated that these disorders were familiar to the general public. The disorders were severe depression, panic attacks, schizophrenia, dementia, eating disorders, alcoholism and drug addiction. The questions about the disorders were based on findings presented in a literature review on stigmatization of people with mental disorders by Hayward and Bright . The key perceptions of persons with mental illness are that they are dangerous, unpredictable, are difficult to talk with, have only themselves to blame, are able to pull themselves together, have a poor outcome and don't respond very much to treatment according to this review. For each disorder the respondents could rank their perception on a five point scale from one extreme e.g. "dangerous to others", to the opposite, "not dangerous to others", "unpredictable" to "not unpredictable" and so forth. In the present study the questions were administered as a self-rating questionnaire. Information about the activities and the evaluation of the "Changing minds" campaign can be found on the website: http://www.rcpsych.ac.uk/campaigns/previouscampaigns/changingminds.aspx
Sample and Data collection
The sample consisted of 51 students in a three year educational programme in nursing. They were in their second semester and during the data collection period they were studying physiology and anatomy. Nothing in their present education had, to our knowledge, a content that could affect their view of mental disorders. The first data collection was made in September, 2007 and the retest data collection was made three weeks later. No major media discussions concerning mental illness occurred during the time period between the test and retest data collection. Being as this investigation is a part of a larger study concerning the impact of formal education in psychiatry on attitudes towards mental illness baseline data from a larger database was used to examine internal consistency. This larger database consisted of 1001 students from educational programmes including psychiatry in their curriculum. The following student categories were represented: nursing, medicine, social work, physiotherapy, occupational therapy, psychology, public health work, and police trainee.
Weighted kappa coefficients were calculated for each item in the two questionnaires in order to investigate the test - retest reliability for the different instruments. The reference values for the strength of agreement are from Altman  who considers < 0.20 as poor agreement, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as good and 0.81-1.00 as a very good agreement. Weighted kappa is used because it takes into account the magnitude of the discrepancy in ordinal data. A weakness of ordinary kappa statistics is that it treats all disagreements equally whether they are very large or very small . Internal consistency was examined by calculating Cronbach's alpha coefficient for the FABI. In the questionnaire from the Changing Minds campaign the Cronbach's alpha coefficient was calculated for the responses concerning each disorder. The statistical software used was SPSS 15.0 and Vasserstats .
In accordance with Swedish legislation research that not includes interventions does not require formal approval from a research ethic committee. The methods of evaluation conformed to the Helsinki Declaration.
The results from the "Changing minds" questionnaire display at best a moderate agreement between the test and retest. The items concerning unpredictability and whether it is difficult or not to talk to a person with a certain disorder appear in particular to have low test-retest reliability. The calculation of Cronbach's alpha shows that none of the batteries of questions about the different disorders has a sufficient level of internal consistency in order to be considered as scales (table 1).
The complete "Changing minds" questionnaire consists of 56 items and 10 of these (17.9%) present poor test-retest reliability, 26 items (46.4%) fair and finally 20 items (35.7%) moderate test-retest reliability.
The weighted kappa values for the self-report inventory of fear and behavioural intentions towards the mentally ill (FABI) also attain at best moderate test-retest reliability. However, none of the items fall below a fair agreement level between the test and retest. The questionnaire shows high internal consistency (table 2).
The investigation of the test-retest reliability in attitudes towards the mentally ill during a three week period indicates a certain level of instability in attitudes as measured with the "Changing minds" questionnaire and FABI. The sample consisted of nursing students studying in their first year of the educational programme. No demographic data for the group investigated was collected but a fair estimation is that it consists of a majority of females with a stable middle class background and with a mean age between 20 and 25. They can thus not be considered to be representative for any group other than nursing students. There are, however, no plausible hypotheses that state that a group consisting of a majority of young women should be more inconsistent in their attitudes than any other group. The fact that the students were trainees in a helping profession might be of interest. If they, after the first completion of the questionnaire, became interested in the issues presented and found out more about these then this may have contributed to a change in their attitudes. Though, health care professionals have in several studies shown attitudes even less favourable than members of the general public [33–35].
Comparisons are difficult to make being as no previous psychometric data for the instruments have been published. The internal consistency reliability of scales measuring social distance tends to be good or excellent , which is also repeated for the self-report inventory of fear of and behavioural intentions towards the mentally ill (FABI) in the present study. The questions in FABI deal with intentions that to a high degree reflect social distance, so in that respect the results are quite consistent with earlier findings.
The "Changing minds" questionnaire displays poor internal consistency reliability and rather problematic overall test-retest reliability. This is quite unexpected; the scale has high face validity and is based on ambitious research . A similar technique to construct questionnaires was used by Olmsted and Durham . They studied attitudes among college students towards the mentally ill using a scale containing pairs of adjectives representing opposites, such as "valuable - worthless, clean - dirty, predictable - unpredictable, etcetera". The results from analyses of three different cohorts, sociology students from 1962, from 1971 and a stratified sample of "general public" show a very high correlation in attitudes between both groups and time points. This indicates a high stability in attitudes at a group level. Whether this also mirrors high test-retest reliability cannot be deduced due to the lack of data. The fact that the "Changing minds" questionnaire was used as a self-rating instrument and the information in the original studies was obtained through interviews might make a difference. It is, however, difficult to imagine the potential significance of this difference being as no comparable data exist. The research of stigma towards the mentally ill has increased and a number of different instruments are in use . Further research is thus needed to create psychometrically sound instruments for securing valid and reliable results.
Psychometrically tested instruments for measuring public attitudes towards persons with mental illness are generally lacking. The present study indicates low test-retest reliability for the two scales investigated. To ensure the usefulness of further research within this particular research field a development of reliable and valid measurements is needed.
Thornicroft G: Shunned: Discrimination against People with Mental Illness. 2006, Oxford University Press, Oxford, UK
Link BG, Phelan JC: Conceptualizing Stigma. Annu Rev Sociol. 2001, 27: 363-85. 10.1146/annurev.soc.27.1.363.
Sartorius N: Diminishing the stigma of schizophrenia. Adv Schizophr Clin Psychiatry. 2004, 1: 50-54.
Schultze B, Angermeyer MC: Subjective experiences of stigma. A focus group study of schizophrenic patients, their relatives and mental health professionals. Soc Sci & Med. 2003, 56: 299-312.
Link BG, Phelan JC: Stigma and its public health implications. The Lancet. 2006, 367: 528-529. 10.1016/S0140-6736(06)68184-1.
Wahl OF, Harman CR: Family views of stigma. Schizophr Bull. 1989, 15 (1): 131-9.
Ostman M, Kjellin L: Stigma by association: psychological factors in relatives of people with mental illness. Br J Psychiatr. 2002, 181: 494-498. 10.1192/bjp.181.6.494.
Thornicroft G, Tansella M, Becker T, Knapp M, Leese M, Schene A, et al: The personal impact of schizophrenia in Europe. Schizophr Res. 2004, 69 (2-3): 125-132. 10.1016/S0920-9964(03)00191-9.
Glozier N: Workplace effects of the stigmatization of depression. J Occup Environ Med. 1998, 40 (9): 793-800. 10.1097/00043764-199809000-00008.
Brand RC, Clairborn WL: Two studies of comparative stigma: employer attitudes and practices toward rehabilitated convicts, mental and tuberculosis patients. Community Ment Health J. 1976, 12 (2): 168-175. 10.1007/BF01411387.
Corrigan PW, Watson AC: The paradox of self-stigma and mental illness. Clin Psychol. 2002, 9: 35-53.
Van Brakel WH, Anderson AM, Mutatkar RK, Bakirtzief Z, Nicholls PG, Raju MS, Das-Pattanayak RK: The Participation Scale: measuring a key concept in public health. Disabil Rehabil. 2006, 28: 193-203. 10.1080/09638280500192785.
Thornicroft G, Brohan E, Rose D, Sartorius N, Leese M: Global pattern of experienced and anticipated discrimination against people with schizophrenia: a cross sectional survey. Lancet. 2009, 373: 408-415. 10.1016/S0140-6736(08)61817-6.
Yanos PT, Roe D, Marcus K, Lysaker PH: Pathways between internalised stigma and outcomes related to recovery in schizophrenia spectrum disorders. Psychiatr Serv. 2008, 59: 1437-1442. 10.1176/appi.ps.59.12.1437.
Lysaker PH, Roe D, Yanos PT: Toward understanding the insight paradox: internalized stigma moderates the association between insight and social functioning, hope, and self esteem among people with schizophrenia spectrum disorder. Schizophr Bull. 2007, 33: 192-199. 10.1093/schbul/sbl016.
Lysaker PH, Tsai J, Yanos P, Roe D: Associations of multiple domains of self-esteem with four dimensions of stigma in schizophrenia. Schizophr Res. 2008, 98: 194-200. 10.1016/j.schres.2007.09.035.
Vauth R, Kleim B, Wirtz M, Corrigan P: Self-efficacy and empowerment as outcomes of self stigmatizing and coping schizophrenia. Psychiatry Res. 2007, 150: 71-80. 10.1016/j.psychres.2006.07.005.
Ritsher JB, Otillingam PG, Grajales M: Internalized stigma of mental illness: psychometric proporties of a new measure. Psychiatry Res. 2003, 121: 31-49. 10.1016/j.psychres.2003.08.008.
Sartorius N, Schultze H: Reducing the stigma of mental illness. A Report from a Global Programme of the World Psychiatric Association. 2005, Cambridge University Press, Cambridge, UK
Crisp AH, Gelder MG, Rix S, Meltzer HI, Rowlands OJ: Stigmatization of people with mental illnesses. Br J Psychiat. 2000, 177: 4-7. 10.1192/bjp.177.1.4.
Crisp A, Gelder M, Goddard E, Meltzer H: Stigmatization of people with mental illnesses: a follow-up study within the Changing Minds campaign of the Royal College of Psychiatrists. World Psychiat. 2005, 4:2: 106-113.
Handikappsombudsmannen Psykekampanjen - en sammanfattande utvärdering (The Psyke campaign - a concluding evaluation by the Swedish Disability Ombudsman). 2001
Hocking B: Reducing mental illness stigma and discrimination - everybody's business. Med J Aust. 2003, 178: 47-48.
Crisp A, Cowan L, Hart D: The College's Anti-Stigma Campaign, 1998-2003. Psych Bull. 2004, 28: 133-136. 10.1192/pb.28.4.133.
Lundberg B, Hansson L, Wentz E, Björkman T: Sociodemographic and clinical factors related to devaluation/discrimination and rejection experiences among users of mental health services. Soc Psychiatry Psychiatr Epidemiol. 2006, 42: 295-300. 10.1007/s00127-007-0160-9.
Rose D, Thornicroft G, Pinfold V, Kassam A: 250 labels used to stigmatise people with mental illness. BMC Health Serv Res. 2007, 7: 97-10.1186/1472-6963-7-97. (2007)
Link BG, Yang LH, Phelan JC, Collins PY: Measuring mental health stigma. Schizophr Bull. 2004, 30 (3): 511-541.
Wolff G, Pathare S, Craig T, Leff J: Public education for community care. A new approach. Br J Psychiatry. 1996, 168 (4): 441-7. 10.1192/bjp.168.4.441.
Wolff G, Pathare S, Craig T, Leff J: Community knowledge of mental illness and reaction to mentally ill people. Br J Psychiatry. 1996, 168 (2): 191-8. 10.1192/bjp.168.2.191.
Hayward P, Bright J: Stigma and mental illness: A review and critique. J Ment Health. 1994, 6 (4): 345-354.
Altman DG: Practical statistics for medical research. 1991, Chapman & Hall, Padstow, Cornwall
VassarStats: Website for Statistical Computation. [http://faculty.vassar.edu/lowry/VassarStats.html]
Lauber C, Nordt C, Braunschweig C, Rossler W: Do mental health professionals stigmatize their patients?. Acta Psychiatr Scand Suppl. 2006, 113: 51-59. 10.1111/j.1600-0447.2005.00718.x.
Nordt C, Rossler W, Lauber C: Attitudes of mental health professionals toward people with schizophrenia and major depression. Schizophr Bull. 2006, 32: 709-714. 10.1093/schbul/sbj065.
Bjorkman T, Angelman T, Jonsson M: Attitudes towards people with mental illness: a cross-sectional study among nursing staff in psychiatric and somatic care. Scand J Caring Sci. 2008, 22: 170-177. 10.1111/j.1471-6712.2007.00509.x.
Olmsted DW, Durham K: Stability of mental health attitudes: A Semantic differential study. J Health and Soc Behav. 1976, 17:1: 35-44. 10.2307/2136465.
The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-244X/11/11/prepub
The authors declare that they have no competing interests.
All authors were involved in the design of the study and in the data collection for the larger study used for establishing the internal consistency of the scales. BS and TB made the data collection for the test-retest investigation. BS analyzed the data and wrote the manuscript together with MÖ and UM. All authors read and approved the final manuscript.
About this article
Cite this article
Svensson, B., Markström, U., Bejerholm, U. et al. Test - retest reliability of two instruments for measuring public attitudes towards persons with mental illness. BMC Psychiatry 11, 11 (2011). https://0-doi-org.brum.beds.ac.uk/10.1186/1471-244X-11-11
- Mental Illness
- Eating Disorder
- Behavioural Intention
- Nursing Student
- Public Attitude