Inter-rater reliability and concurrent validity of the MHNS

Research Highlights: The Mental Health Need Scale (MHNS) has excellent inter-rater reliability and good concurrent validity for all offender groups.

The Health Services Evaluation Recommendation 6 called for the Correctional Service of Canada (CSC) “to conduct analysis to verify the validity and reliability of the Mental Health Need Scale (MHNS).” In addition, it called for provisions to strengthen the process for recording and maintaining offender level of need data. A related recommendation was made in the recent Auditor General’s report Preparing Women Offenders for Release (Auditor General of Canada, 2017). Recommendation 5.84 called for CSC to “ensure that it appropriately identifies women offenders who need mental health services and assigns them to the appropriate level of care”.

This study examined the tool’s inter-rater reliability and aspects of its validity. The total sample consisted of 150 offenders from across CSC who were selected by staff as needing a MHNS assessment. Just over half of the sample was comprised of men and almost two-thirds were non-Indigenous. Almost 60% were rated as low or no need and 40% were rated as either moderate or high on the MHNS. The NHQ raters were trained by one of the scale developers prior to conducting interviews. Site raters received an information package with guidelines on the use of the assessment, but no formal training. NHQ staff and site staff independently and concurrently completed the MHNS and the Clinical Global Impressions (CGI) scale for each of the 150 offenders. The overall need ratings and the CGI ratings were compared to evaluate the consistency and reliability across different raters. Concurrent validity was assessed by comparing the ratings on the MHNS to that of the Clinical Global Impression Scale (CGI). Percentage agreement and Intra-class Correlation Coefficients were calculated for the overall sample as well as by men, women, non-Indigenous and Indigenous offenders.

The results produced three main findings: 1) excellent inter-rater reliability for the MHNS total score across the overall sample; excellent inter-rater reliability on the overall need score for women and Indigenous offenders; 2) strong concurrent validity between the CGI and the MHNS, for both institutional raters and NHQ raters (for the overall sample as well as across subsamples); and, 3) in general, good internal consistency of the tool in that most of the domains were significantly related to the overall rating. Very few offenders received overall ratings that did not match their domain ratings. Of note, however, the rating on concurrent disorders was only weakly associated with the overall rating indicating inconsistent rating on this domain.

The MHNS demonstrates excellent inter-rater reliability as well as substantial concurrent validity across men, women, and Indigenous offenders. This indicates that it is an appropriate tool to allocate offenders to a level of care. However, consideration should be given to highlighting to raters the importance of assessing concurrent substance abuse and mental health problems as part of the MHNS assessment process. Recent research within CSC has demonstrated that the presence of substance use problems in addition to other mental health disorders substantially complicates treatment and that offenders with the combined disorders are at greater risk for negative outcomes. Finally, given the psychometric strength of the tool, consideration could be given to its use to systematically reassess offenders following treatment to gauge the effectiveness of the intervention.

Stewart, L., Wanamaker, K., Wilton, G., & Toor, G. (2018). Inter-rater reliability and concurrent validity of the Mental Health Need Scale. (Research Report R-411). Ottawa, Ontario: Correctional Service of Canada.

To obtain a PDF version of the full report, or for other inquiries, please e-mail the Research Branch or contact us by phone at (613) 995-3975.

You can also visit the Research Publications section for a full list of reports and one-page summaries.