Skip to main content

Table 4 Expected versus observed agreement

From: The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies

a) Expected by stakeholders

b) Observed in the RELY studies

‘Maximum acceptable difference’a

Corresponding ‘Standard error of measurement’

  

‘Standard error of measurement’

Corresponding ‘Maximum acceptable difference’

25% WC

9.0% WC

Last job

RELY 1

26.0% WC

72.2% WC

20% WC

7.2% WC

RELY 2

23.9% WC

66.1% WC

15% WC

5.4% WC

Alternative job

RELY 1

24.6% WC

68.1% WC

10% WC

3.6% WC

RELY 2

19.4% WC

53.9% WC

  1. Legend: % WC = absolute percentage points in work capacity
  2. a derived from the stakeholder survey (Table 1) [6]
  3. This table compares the expectations of Swiss stakeholders of the agreement in WC ratings between two experts, expressed as ‘maximum acceptable differencea’, with the agreement observed in the RELY studies, i.e., the variation between experts, expressed as ‘standard error of measurement’. Converting ‘maximum acceptable difference’ into ‘standard error of measurement’ and vice versa allows comparison of the level of agreement
  4. a) Agreement expected by stakeholders: Treating and expert psychiatrists considered a difference of 25% WC between two experts as the ‘maximum acceptable difference’ (i.e. for example, expert A: 60% WC; expert B: 35% WC or 85% WC) which corresponds to a variation between experts of 9.0% WC ‘standard error of measurement’
  5. If the ‘maximum acceptable difference’ between two experts were only 15% WC (i.e. for example, expert A: 60% WC, expert B: 45% WC or 75% WC), the corresponding variation between experts would be as low as 5.4% WC ‘standard error of measurement’
  6. b) Agreement observed in the RELY studies: RELY 2last job found a level of agreement of 23.9% WC ‘standard error of measurement’ which corresponds to a (‘maximum acceptable’) difference in WC of 66.1% (i.e. for example, expert A: 30% WC; expert B: 96% WC)