Background
Verbal information and visual cues are major and primary ingredients of psychiatric assessment. The sounds and images transmitted through video-conferencing are equivalent to these two parameters respectively. Other factors such as empathy and rapport are also crucial and their influence on the outcome of assessment is well understood but not well quantified. The assumption that video-conferencing would provide results equivalent to those from face-to-face psychiatric interview is related to these corollaries and requires testing and quantification. Trust and confidence in using this technology can be greatly enhanced if this assumption is proved true. In view of rapid developments in hardware, wireless technology and data-transmission, psychiatric intervention through video-conferencing (telepsychiatry) can be an effective mode of service delivery, especially for remotely located population clusters.
Meeting mental health needs for remotely and sparsely populated communities has been a challenge to service providers due to various factors including resource constrains and difficulty in recruitment of mental health staff. Attempts have been made to address these concerns through the use of new and emerging technologies. When such new methods are used for clinical assessment, there is bound to be inherent uncertainties as to whether these new methods are as reliable, sensitive and accurate as existing methods of clinical assessment.
Although several studies
This study attempts to detect the level of intermethod agreement between telepsychiatric assessment and face-to-face interview for routine new outpatient referrals to the general adult psychiatric unit. It is anticipated that there is a high level of agreement between conclusions drawn from psychiatric interviews through video-conferencing (V) and the standard method of face-to-face psychiatric assessment (S) for diagnosis, risk assessment and clinical intervention. This study aims to test this assumption.
Methods
Setting and participants
The study was conducted at Hawkes Bay Health Care, which provides a National Health Service to the Hawkes Bay and Chatham Island areas of New Zealand (NZ). The study was approved by the Hawkes Bay Ethics Committee, New Zealand. The sample consisted of consecutive new adult psychiatric referrals to the Napier Community Mental Health Team (NCMHT). They belonged to the 19 to 65 age group, were not under care of the NCMHT and had not received care for any mental health issue from this unit for a period of at least 6 months at the time of referral. Cases requiring urgent assessment or home visit were excluded.
In clinical practice, the outcome of the standard method of face-to-face assessment (S) is always supposed to be accurate. Accordingly, using method S as the gold standard, the results from V can be classified as 'accurate' if the outcome is identical to that from S for a given attribute, or as 'inaccurate' if there is disagreement between methods S and V. For the purpose of this study, the accuracy ratio (AR) is defined as the risk ratio (RR) between the accurate outcomes of video-assessment (V) and the results from face-to-face assessment (S). Assuming an AR of 0.95 or above for face-to-face assessment and results of video-assessment at a significance level of 0.05 and a power of 0.8, a sample size of 34 for each method would suffice to detect a difference of 15% or more between these two methods of assessment
From the 40 consecutive new psychiatric referrals fulfilling above criteria, two cases declined to participate and one case could not be located. A written informed consent was obtained from all remaining 37 cases and they all went through their complete intended assessments. The referral period extended from 26 February 2001 to 15 May 2001 and the assessments were completed between 23 March 2001 and 17 May 2001.
Assessment procedure
The assessment order for each method and for each psychiatrist was predetermined using a method of random allocation. The whole list was randomly divided into the two sub-lists, then participants were randomly allocated to have their first assessment either by researcher R1 or R2. The randomly selected half of the cases of each researcher (R1 and R2) had their first assessment by method S and the remaining half had their first assessment by method V. The second assessment (S or V as appropriate) of each individual case was subsequently completed by the other researcher (R1 or R2). The details of the randomisations and assessment procedures have been displayed in the Figure
The sample randomisation
The sample randomisation. The numbers in the boxes are the serial numbers of the sample cases. Those crossed are drop-outs.
None of the researchers had prior experience of conducting formal telepsychiatric interviews for clinical care. Prior to initiating the research assessments, the researchers spent one session to familiarise with the equipment, and two sessions practising on known cases to evaluate and compare their findings in order to enhance their interrater agreement.
All face-to-face interviews were conducted at Hastings, while video assessment was carried out from Wairoa, 140 km away. All participants underwent both methods of assessment; each participant having one assessment on video by one psychiatrist and one face-to-face interview by the other psychiatrist. The interviewers utilised their own usual practice of clinical interview to resemble a standard outpatient setting.
The main confounding variables likely to influence the level of agreement are; bias between the researchers doing the assessments, duration of interview, use of interpreter, order-effect (effect on the second interview due to practice or residual memory from the first interview) and the time interval between the two methods of interview. The influence of such biases was minimised by adopting a crossover design and assigning an equal number of cases to each of the interviewing psychiatrists, to each interview methods, and to each order of assessments (S followed by V and V followed by S). The researchers were not aware of each others findings while assessing an individual participant. Both assessments for each participant were completed on the same date and each assessment lasted up to 60 minutes. If an interpreter was involved, he/she had to attend both sessions for that given case.
Video-Conferencing Units were available at Wairoa and Hastings. Both centres were equipped with a PictureTel Venue 2000 model 50 with 29 inch colour TV and were linked with a 384 KB (128 KB × 3) bandwidth ISDN line. Scanning and zooming of each of these video-conferencing units could be remotely controlled by the interviewer or by the interviewee. The Picture-in-picture (PIP) facility was not used on the interviewee side to prevent unnecessary distraction during the interview.
Diagnostic tools, scales and data
Diagnoses on the DSM-IV axes were based on the method described in the Decision Trees for Differential Diagnosis of the Handbook
The full diagnostic code for the DSM-IV-Axis-1, the presence or absence of diagnoses on Axis2 and Axis-3, the applicability or non-applicability of DSM-IV-Axis-4 questions and the score for DSM-IV-Axis-5 Global Assessment of Functioning (GAF) was recorded for each assessment. To confer uniformity, the numerical score of GAF was changed into ordinal type ranging from lower to high categories (A to E) based on a class interval method fulfilling the transformation criteria
Minor adjustments were made to present the data table in an n-by-n format as a pre-requisite for kappa calculation. One DSM-IV Axis-1 diagnosis of disorganised schizophrenia (V, 295.10) was changed to paranoid schizophrenia (295.30) giving a concordant entry; one case of cyclothymic disorder, (S, 301.13) was changed to bipolar disorder (296.56) giving a discordant entry; and one case of factitious disorder (V, 300.16) was changed to somatoform disorder (300.81) giving a concordant pair.
If the number of total diagnoses for a given case differed between methods of assessment, a category of 'NIL' was introduced to reflect the lack of identification of an equivalent diagnosis by the corresponding method. This led to the introduction of 2 concordant pairs and one discordant pair by the first method of adjustment and 3 discordant pairs arising from 'NIL' categories from the second method of adjustment. The resulting preponderance of discordant pairs over concordant pairs is likely to influence the interpretation against the research hypothesis, rather than in favour of it.
Statistical evaluation
The test statistics of AR, Risk Difference (RD) and CK were calculated and summarised in accordance with methods described in the standard texts
The re-sampling was performed in a manner that retained the structural consistency of each subgroup. The techniques of bootstrapping and re-sampling are well established statistical methods and yet are little known in medical literature. The required software codes were developed for these models by the principal author (SPS) using R (version 2.4.1)
Results
Of 37 participants, 20 were female and 17 male. Ethnically, there were 27 participants of European descent, 8 were of Maori origin and 2 were from other groups. Their ages ranged between 19.21 and 63.29 years; with an average age of 35.40 years and a standard deviation of 12.46 years.
The presence of statistical significance for the results of ARs in the Table
Comparison of results of telepsychiatric assessments and face-to-face interviews
Sample
Accuracy Ratio Statistics
Risk Difference Statistics
Primary attributes and accuracy
A
I
Size
AR
LCI
UCI
p
RD
LCI
UCI
p
01
DSM-IV Axis 1
49
5
54
0.91
0.83
0.99
0.025
-0.09
-0.17
-0.02
0.019
02
DSM-IV Axis 2
34
3
37
0.92
0.84
1.01
0.083
-0.08
-0.17
0.01
0.071
03
DSM-IV Axis 3
36
2
38
0.95
0.88
1.02
0.157
-0.05
-0.12
0.02
0.146
04
DSM-IV Axis 4 Q1
31
6
37
0.84
0.73
0.97
0.014
-0.16
-0.28
-0.04
0.007
05
DSM-IV Axis 4 Q2
31
6
37
0.84
0.73
0.97
0.014
-0.16
-0.28
-0.04
0.007
06
DSM-IV Axis 4 Q3
37
0
37
1.00
1.00
1.00
NA
0.00
0.00
0.00
NA
07
DSM-IV Axis 4 Q4
36
1
37
0.97
0.92
1.03
0.317
-0.03
-0.08
0.03
0.311
08
DSM-IV Axis 4 Q5
37
0
37
1.00
1.00
1.00
NA
0.00
0.00
0.00
NA
09
DSM-IV Axis 4 Q6
32
5
37
0.86
0.76
0.98
0.025
-0.14
-0.25
-0.02
0.016
10
DSM-IV Axis 4 Q7
37
0
37
1.00
1.00
1.00
NA
0.00
0.00
0.00
NA
11
DSM-IV Axis 4 Q8
34
3
37
0.92
0.84
1.01
0.083
-0.08
-0.17
0.01
0.071
12
DSM-IV Axis 4 Q9
1
36
37
0.03
0.00
0.19
0.000
-0.97
-1.03
-0.92
0.000
13
DSM-IV Axis 5
26
11
37
0.70
0.57
0.87
0.001
-0.30
-0.44
-0.15
0.000
14
Risk to Self Q1
27
10
37
0.73
0.60
0.89
0.002
-0.27
-0.41
-0.13
0.000
15
Risk to Self Q2
29
8
37
0.78
0.66
0.93
0.005
-0.22
-0.35
-0.08
0.001
16
Risk to Self Q3
28
9
37
0.76
0.63
0.91
0.003
-0.24
-0.38
-0.10
0.001
17
Risk to Self Q4
28
9
37
0.76
0.63
0.91
0.003
-0.24
-0.38
-0.10
0.001
18
Risk to Others Q1
28
9
37
0.76
0.63
0.91
0.003
-0.24
-0.38
-0.10
0.001
19
Risk to Others Q2
35
2
37
0.95
0.88
1.02
0.157
-0.05
-0.13
0.02
0.146
20
Risk to Others Q3
33
4
37
0.89
0.80
1.00
0.046
-0.11
-0.21
-0.01
0.034
21
Risk to Others Q4
32
5
37
0.86
0.76
0.98
0.025
-0.14
-0.25
-0.02
0.016
22
Risk to Others Q5
33
4
37
0.89
0.80
1.00
0.046
-0.11
-0.21
-0.01
0.034
23
Risk to Others Q6
29
8
37
0.78
0.66
0.93
0.005
-0.22
-0.35
-0.08
0.001
24
Admit-Discharge-Follow-up
34
3
37
0.92
0.84
1.01
0.083
-0.08
-0.17
0.01
0.071
25
Investigations
25
12
37
0.68
0.54
0.84
0.001
-0.32
-0.48
-0.17
0.000
26
Psychological Input
26
11
37
0.70
0.57
0.87
0.001
-0.30
-0.44
-0.15
0.000
27
Community Support
29
8
37
0.78
0.66
0.93
0.005
-0.22
-0.35
-0.08
0.001
28
Drug Type
64
11
75
0.85
0.78
0.94
0.001
-0.15
-0.23
-0.07
0.000
29
Drug Action
58
17
75
0.77
0.68
0.87
0.000
-0.23
-0.32
-0.13
0.000
30
Drug Indication
49
26
75
0.65
0.55
0.77
0.000
-0.35
-0.45
-0.24
0.000
Main Attributes
31
DSM-IV Axis 1 (1)
49
5
54
0.91
0.83
0.99
0.025
-0.09
-0.17
-0.02
0.019
32
DSM-IV Axis 2 (2)
34
3
37
0.92
0.84
1.01
0.083
-0.08
-0.17
0.01
0.071
33
DSM-IV Axis 3 (3)
36
2
38
0.95
0.88
1.02
0.157
-0.05
-0.12
0.02
0.146
34
DSM-IV Axis 4 (4:12)
276
57
333
0.83
0.79
0.87
0.000
-0.17
-0.21
-0.13
0.000
35
DSM-IV Axis 5 (13)
26
11
37
0.70
0.57
0.87
0.001
-0.30
-0.44
-0.15
0.000
36
Risk to Self (14:17)
112
36
148
0.76
0.69
0.83
0.000
-0.24
-0.31
-0.17
0.000
37
Risk to Others (18:23)
190
32
222
0.86
0.81
0.90
0.000
-0.14
-0.19
-0.10
0.000
38
Admit-Discharge-Follow-up (24)
34
3
37
0.92
0.84
1.01
0.083
-0.08
-0.17
0.01
0.071
39
Investigations (25)
25
12
37
0.68
0.54
0.84
0.001
-0.32
-0.48
-0.17
0.000
40
Psychological Input (26)
26
11
37
0.70
0.57
0.87
0.001
-0.30
-0.44
-0.15
0.000
41
Community Support (27)
29
8
37
0.78
0.66
0.93
0.005
-0.22
-0.35
-0.08
0.001
42
Drug Type (28)
64
11
75
0.85
0.78
0.94
0.001
-0.15
-0.23
-0.07
0.000
43
Drug Action (29)
58
17
75
0.77
0.68
0.87
0.000
-0.23
-0.32
-0.13
0.000
44
Drug Indication (30)
49
26
75
0.65
0.55
0.77
0.000
-0.35
-0.45
-0.24
0.000
Major Attributes
45
DSM Diagnosis (1:13)
421
78
499
0.84
0.81
0.88
0.000
-0.16
-0.19
-0.12
0.000
46
Risks (14:23)
302
68
370
0.82
0.78
0.86
0.000
-0.18
-0.22
-0.14
0.000
47
Non-Drug Intervention (24:27)
114
34
148
0.77
0.71
0.84
0.000
-0.23
-0.30
-0.16
0.000
48
Drug Intervention (28:30)
171
54
225
0.76
0.71
0.82
0.000
-0.24
-0.30
-0.18
0.000
Overall
49
Overall Result (1:30)
1008
234
1242
0.81
0.79
0.83
0.000
-0.19
-0.21
-0.17
0.000
Cohen's Kappa results of intermethod and interviewers assessments
Standard vs Video
Interviewers
SN
Primary Attributes
Size
Group
CK
LCI
UCI
CK
LCI
UCI
1
DSM-IV Axis 1
54
26
0.90
0.28
1.00
0.90
0.33
1.00
2
DSM-IV Axis 2
37
2
0.62
-0.04
0.87
0.63
0.21
0.87
3
DSM-IV Axis 3
38
2
0.84
0.49
0.93
0.84
0.49
0.93
4
DSM-IV Axis 4 Q1
37
2
0.57
0.25
0.82
0.57
0.22
0.83
5
DSM-IV Axis 4 Q2
37
2
0.17
-0.11
0.72
0.18
-0.09
0.68
6
DSM-IV Axis 4 Q3
37
2
0.85
0.66
0.91
0.85
0.66
0.91
7
DSM-IV Axis 4 Q4
37
2
0.87
0.65
0.94
0.87
0.65
0.94
8
DSM-IV Axis 4 Q5
37
2
0.66
NA
NA
0.66
NA
NA
9
DSM-IV Axis 4 Q6
37
2
0.55
0.18
0.84
0.54
0.13
0.84
10
DSM-IV Axis 4 Q8
37
2
0.36
-0.07
0.79
0.37
-0.08
0.54
11
DSM-IV Axis 5
37
5
0.89
0.81
0.94
0.89
0.81
0.94
12
Risk to self Q1
37
3
0.66
0.39
0.86
0.65
0.39
0.86
13
Risk to self Q2
37
3
0.68
0.34
0.86
0.68
0.37
0.86
14
Risk to self Q3
37
3
0.55
-0.11
0.73
0.55
0.22
0.80
15
Risk to self Q4
37
3
0.45
-0.11
0.68
0.45
0.12
0.79
16
Risk to others Q1
37
3
0.62
0.27
0.86
0.62
0.27
0.86
17
Risk to others Q2
37
3
0.82
0.30
1.00
0.82
0.37
1.00
18
Risk to others Q3
37
2
-0.04
-0.10
-0.03
-0.04
-0.10
-0.03
19
Risk to others Q4
37
3
0.55
-0.11
0.78
0.55
-0.06
0.78
20
Risk to others Q5
37
2
0.44
-0.07
0.84
0.44
-0.06
0.84
21
Risk to others Q6
37
3
0.41
-0.13
0.66
0.41
-0.12
0.66
22
Admit-Discharge-Follow up
37
3
0.76
0.37
0.93
0.76
0.37
0.93
23
Investigations
37
2
0.29
-0.03
0.59
0.30
-0.02
0.59
24
Psychological Input
37
2
0.16
-0.17
0.53
0.18
-0.12
0.54
25
Community Support
37
2
0.55
0.22
0.78
0.56
0.25
0.78
26
Drug Type
75
9
0.83
0.61
0.90
0.82
0.52
0.90
27
Drug Action
75
5
0.67
0.52
0.79
0.66
0.52
0.79
28
Drug Indication
75
11
0.59
0.35
0.74
0.59
0.35
0.76
Main Attributes
29
DSM-IV Axis 1 (1)
54
26
0.90
0.28
1.00
0.90
0.33
1.00
30
DSM-IV Axis 2 (2)
37
2
0.62
-0.04
0.87
0.63
0.21
0.87
31
DSM-IV Axis 3 (3)
38
2
0.84
0.49
0.93
0.84
0.49
0.93
32
DSM-IV Axis 4 (4:10)
259
14
0.65
0.52
0.78
0.65
0.52
0.78
33
DSM-IV Axis 5 (11)
37
5
0.89
0.81
0.94
0.89
0.81
0.94
34
Risk to self (12:15)
148
12
0.61
0.48
0.75
0.61
0.48
0.75
35
Risk to Others (16:21)
222
16
0.05
-0.01
0.11
0.05
-0.01
0.11
36
Admit-Discharge-Follow up (22)