In recent years out-of-hours primary care in the Netherlands has changed from practice-based to large-scale cooperatives. The purpose of this study is to determine patient satisfaction with current out-of-hours care organised in general practitioner (GP) cooperatives, and gain insight in factors associated with this satisfaction.
From March to June 2003, 2805 questionnaires were sent to patients within three weeks after they had contacted the GP cooperative in their region. The study was conducted in the province of Limburg in the South of the Netherlands. One-third of these questionnaires was sent to patients who had only received telephone advice, one-third to patients who attended the GP cooperative for consultation, and one-third to patients who received a home visit. Four weeks after the first reminder, a non-respondents telephone interview was performed among a random sample of 100 patients. Analyses were performed with respect to the type of consultation.
The total response was 42.4% (1160/2733). Sixty-seven percent of patients who received telephone advice only reported to be satisfied with out-of-hours care. About 80% of patients who went to the GP cooperative for consultation or those receiving a home visit, reported to be satisfied. Factors that were strongly associated with overall satisfaction included, the doctor's assistant's attitude on the phone, opinion on GP's treatment, and waiting time.
Patients seem generally satisfied with out-of-hours primary care as organised in GP cooperatives. However, patients who received telephone advice only are less satisfied compared to those who attended the GP cooperative or those who received a home visit.
In recent years, out-of-hours primary care in the Netherlands has been substantially reorganised. Formerly, general practitioners (GPs) used to perform these services in small locum groups (6 to 8 GPs) in which they joined a rota system. Nowadays, out-of-hours care is organised in large-scaled GP cooperatives (45 to 120 GPs) following examples in the UK and Denmark [1,2].
The initiative of reorganising out-of-hours care has come mainly from the profession itself, motivated by increased dissatisfaction with the organisation of former out-of-hours primary care services. This dissatisfaction was mainly due to the high perceived workload (after out-of-hours service a regular day of work followed), and poor separation between work and private life. The main advantage of the reorganisation was the substantial reduction of number of hours a GP has to be on call. Furthermore, the organisation of out-of-hours care became much more professional by installing management, employing doctor's assistants, and using chauffeured cars. Studies have indicated that GPs appear to be generally satisfied with out-of-hours care organised in cooperatives.
Not only did things change for doctors, but also patients experienced some important changes in out-of-hours primary care. Generally, the reorganisation caused a shift from more personal care to more anonymous care, with increased distance to the GP. Formerly, when patients needed primary care outside office hours, the probability of being seen by their own or a local GP with whom they were familiar, was higher. In addition, when patients contacted the GP during out-of-hours in the past, they were most likely to speak to the GP himself on the phone. Nowadays, the phone is staffed by a doctor's assistant who decides what action should follow the patient's call. Moreover, out-of-hours care used to be delivered by local GPs, indicating short distances to the GP's practice. In large-scale GP cooperatives, the distance to a GP outside office hours will have increased substantially for most patients.
We expected that patient satisfaction would have been reduced after the reorganisation, because factors that guaranteed personal out-of-hours care at a short distance, that may be important to patients, were changed substantially. Furthermore, in Denmark it has been shown that after the out-of-hours primary care reform patient satisfaction dropped significantly[4,5].
Patient satisfaction with out-of-hours primary care has quite often been investigated, especially in the UK [4-11]. Mostly, comparisons have been made between different types of out-of-hours services. Several of these studies focused on out-of-hours primary care as organised in GP cooperatives. These studies have shown that patients are generally satisfied with out-of-hours primary care organised in GP cooperatives[5,8,9,11]. Nevertheless, patients receiving telephone advice only, appear to be less satisfied compared to those attending the cooperative or those receiving a home visit. In addition, it has been shown that the patient's expectation about their contact with the GP cooperative strongly affects the patient's overall satisfaction with out-of-hours care. Other variables that appear to be related to overall satisfaction are, access to a car, age, and waiting time.
Insight in patient satisfaction with out-of-hours care supplies the health care provider with important information on the patient's perception of the quality of that care. During the last years, Dutch GP cooperatives have often received negative publicity in newspapers. The reorganisation has had some important implications for patients, and therefore research on their opinions about current out-of-hours care is warranted. The purpose of this study is to determine patient satisfaction with current out-of-hours care, and to determine how satisfaction is related to different aspects of the patient's contact with a GP cooperative.
The study was conducted in the province of Limburg in the South of the Netherlands. With respect to out-of-hours primary care, the province is organisationally divided in five regions. Two of these regions each have two GP cooperatives (NL and ML), one region (OZL) has one GP cooperative with two satellite centres, and in the other two regions (WM and MH) only one GP cooperative is operational. All cooperatives but one (MH) are organisationally separate from the emergency department of the local hospital, and are located nearby the hospital. This implies that patients may choose between attending the emergency department and the GP cooperative for medical problems during out-of-hours. The MH cooperative is located at the emergency department of the region's only hospital and sees all patients needing out-of-hours care, except for those having a referral for emergency care.
In total, these seven GP cooperatives cover a population of about 1.1 million people (the total Dutch population is over 16 million people), and are fully operational since the 1st of September 2001.
Development of the questionnaire
To determine relevant issues for the questionnaire we interviewed GPs and managers involved with out-of-hours primary care. In addition, we analysed the process for a patient contacting the GP cooperative for all three loci of care (telephone advice, consultation at the cooperative, and home visits) separately to make sure that all facets of the GP cooperative a patient faces would be incorporated in the questionnaire. Moreover, we also analysed unpublished Dutch questionnaires in this field, and the patient satisfaction questionnaire developed by McKinley et al. . Based on these three analyses, we identified a number of relevant elements (initial scales). Next a set of items was developed to enable us to produce multi-item scales. Subsequently, this list was sent to the patient organisation in our province, the two largest health insurance funds, and to the five GP cooperative organisations for commentary. These organisations were asked to critically review the list of items, and to add or remove items if they considered it necessary. After receiving all commentary the questionnaire was adjusted and was submitted to five people not involved in the development but with experience with out-of-hours primary care to check for clarity of the questions.
Finally three questionnaires were constructed for each of the three types of consultations (telephone advice, consultation at the cooperative, and home visit). The three questionnaires differed on items related to the specific type of contact, but general items were the same for all three questionnaires. In this way it was possible to avoid complex skip sections which lengthen the questionnaire and can reduce the response rate. We used a balanced Likert five point scale (strongly agree, agree, neutral, disagree, strongly disagree) to record responses.
The questionnaire related to telephone advice contained six initial scales measuring: accessibility of the cooperative by phone, doctor's assistant's attitude, questions asked by the assistant, advice given by the assistant, urgency of patient's complaint, and overall satisfaction.
The questionnaire related to consultations at the cooperative contained ten initial scales: accessibility of the cooperative by phone, doctor's assistant's attitude, questions asked by the assistant, urgency of patient's complaint, waiting time at the cooperative, waiting room, distance to the cooperative, GP's attitude, treatment by GP, and overall satisfaction.
The questionnaire related to home visits contained eight initial scales: accessibility of the cooperative by phone, doctor's assistant's attitude, questions asked by the assistant, urgency of patient's complaint, waiting time until GP arrives, GP's attitude, treatment by GP, and overall satisfaction.
In addition, patient characteristics such as, age, gender, level of education, and health insurance (as a measure of social economic status) were recorded. Patients were also asked which type of consultation they expected prior to their contact with the GP cooperative, and whether they thought that the right diagnosis had been made.
From March to June 2003 a sample of 2805 patients – who had contacted the GP cooperative in their region – received a questionnaire by mail. Patients received this questionnaire within three weeks after they had contacted the GP cooperative. Sampling was performed per GP cooperative within the four-month period. With respect to patients who received telephone advice only and those who attended the GP cooperative, a computer program randomly selected each fourth patient contact with the GP cooperative backwards from the moment of sampling. Since the number of home visits is limited, all 150 patients, who were visited by a GP from the cooperative, prior to the moment of sampling received a questionnaire. These procedures assured that the time between receiving the questionnaire and the contact with the GP cooperative was not more than three weeks.
Per region 450 questionnaires were sent out; 150 to patients who received only telephone advice, 150 to patients who visited the GP cooperative, and 150 to patients who received a home visit. Because of parallel research, more questionnaires were sent out in one of the regions (WM): 1005 questionnaires equally distributed among the three types of patient contact with the GP cooperative. The study size was chosen based on previous research by McKinley et al[7,13], who presented a study sample of about 1400 patients. We estimated that about half of all questionnaires would be returned, and therefore distributed 2805 questionnaires.
The study was approved by the Institutional Medical Ethics Board.
Reminder and non-respondents interview
Three to four weeks after the questionnaire had been distributed, a reminder was sent to patients who had not returned the questionnaire, with the exception of the WM area. Four weeks after the last reminder, a random sample of 100 patients who had not responded, was contacted by phone. They were asked about their reasons not to return the questionnaire, and about their opinion on the contact they had with the GP cooperative. This interview was performed during office hours, during a three-week period.
Principal components analysis with varimax rotation was used to test whether the items could be assumed to measure similar aspects or components of patients' opinions about their contact with the GP cooperative. Next, Cronbach's alpha coefficient was calculated to estimate the internal consistency as a measure for reliability for each component. Finally, scale scores were calculated per component by summing the scores per item and expressing the total result as a percentage of the maximum score for each scale[13,14]. Scale scores could range between 0 and 100.
The relationship between individual variables and overall satisfaction was analysed using multiple regression analysis, with subscale satisfaction scores as covariates. Variables that did not significantly contribute to the regression model were excluded from the final model. In case of missing data, listwise deletion of missing cases was applied. All data were analysed using SPSS-pc, version 10.0.5.
Seventy-two of the 2805 questionnaires were excluded, either because they could not be delivered (patient had moved or gave a wrong address), the patient had died, or the patient was sent a double questionnaire (multiple contacts). Eventually the response was 42.4% (1160/2733). Generally more women responded to the questionnaire, and about three-quarter of the respondents had public health insurance (table 1). The age of respondents of those who received telephone advice only was comparable with those who attended the GP cooperative for a consultation. The respondents who received a home visit were generally older; two-third was over sixty years of age.
Table 1. Patient characteristics.
Forty percent (366/908) of the patients who had received telephone advice only, returned the questionnaire. 67% of these patients responded to be satisfied (44.3%) or very satisfied (22.3%) with their contact with the GP cooperative, and 57% thought that the current out-of-hours care was an improvement compared to the former situation. We identified the same six scales that were initially set to represent patients' opinions on aspects of primary out-of-hours care (table 2). All six scales had Cronbach's alpha coefficients between 0.64 and 0.93. Detailed information on the scales and items can be found in table 7.
Table 2. Description of scales representing patients' opinion on different aspects of out-of-hours primary care.
Table 7. Patient satisfaction questionnaire. (Original items are in Dutch)
Overall satisfaction in this group was significantly related to five scales, with a variance explained of 62% (see table 3.). When patients judged that the right diagnosis had been made overall satisfaction was higher. We found that satisfaction also increased with age. When patients were satisfied with the accessibility of the cooperative by phone, the doctor's assistant's attitude on the phone, and the doctor's assistant's advice overall satisfaction was higher.
Table 3. Regression analysis with overall satisfaction with out-of-hours primary care as dependent variable of patients who received only telephone advice (adjusted R2 = 0.615).
Consultation at the GP cooperative
Forty-three percent (392/912) of the patients who attended the GP cooperative returned the questionnaire. Approximately 80% of these patients reported to be satisfied (54.6%) or very satisfied (26.3%) with their contact with the GP cooperative, and 61% thought that the current out-of-hours care was an improvement compared to the former situation. We identified nine scales that represent patients' opinions on aspects of primary out-of-hours care (table 2), with Cronbach's alpha coefficients between 0.62 and 0.93. Two initial scales have been merged into one scale; these were patient's opinion on the GP's attitude and the treatment by the GP. All other identified scales were the same as the initial scales. Detailed information on the scales and items can be found in table 7.
Seven variables proved to be predictors of overall satisfaction, with a variance explained of 51% (see table 4.). Patients, who expected prior to their contact with the cooperative that they were going to be asked to come to the GP cooperative, were generally more satisfied. Those who believed that their medical problem was urgent were less satisfied. Long waiting times and dissatisfaction with the distance to the cooperative also reduced overall satisfaction. When patients were satisfied with the accessibility of the cooperative by phone, the doctor's assistant's attitude on the phone, and the GP's treatment overall satisfaction was higher.
Table 4. Regression analysis with overall satisfaction with out-of-hours primary care as dependent variable of patients who went for consultation to the GP cooperative. (adjusted R2 = 0.501).
Almost forty-five percent (402/903) of the patients that received a home visit by a GP from the cooperative returned the questionnaire. About 81% of these patients reported to be satisfied (42.8%) or very satisfied (38.8%) with their contact with the GP cooperative, and 61% thought that the current out-of-hours care was an improvement compared to the former situation. We identified six multi-item scales that represented the patient's opinion on different aspects of out-of-hours primary care, with Cronbach's alpha coefficients between 0.73 and 0.96. Two initial scales have been merged into one scale; these were patient's opinion on the GP's attitude and the treatment by the GP. All other identified scales were the same as the initial scales. Detailed information on the scales and items can be found in table 7.
We found that five variables predicted overall satisfaction, with a variance explained of 51% (see table 5.). Similar to the group of patients who had received telephone advice only, patients who receive a home visit were generally more satisfied when they believed that the GP of the cooperative had made the right diagnosis. When patients were satisfied with the accessibility of the cooperative by phone, the doctor's assistant's attitude on the phone, and the GP's treatment overall satisfaction was higher. In addition, when patients were satisfied about the waiting time until the GP arrives, overall satisfaction increased.
Table 5. Regression analysis with overall satisfaction with out-of-hours primary care as dependent variable of patients who received a home visit from a GP from the cooperative. (adjusted R2 = 0.506).
The means of the three loci of care, adjusted for age, sex, insurance status, and education level, show that there is no difference between overall satisfaction in the group of patients who visited the GP cooperative (75.1 ± 1.31) and those who received a home visit (72.5 ± 1.37) (Table 6). However, patients who received telephone advice only (66.2 ± 1.30), were significantly less satisfied compared to the other two groups of patients.
Table 6. Adjusted means for overall satisfaction.
Out of 100 randomly selected patients, who had not returned the questionnaire, we were able to reach 63 by phone. Of these 63 non-respondents 35 (55.6%) were male and 28 (44.4%) were female. Many of them reported that they had forgotten to return the questionnaire (40%). A minority said not to be interested (6.7%) or did not find it needful (6.7%). Most non-respondents (46.7%) gave other reasons like, no time, too difficult, or had lost the questionnaire.
Of these patients, about 71% reported to be satisfied or very satisfied about their contact with the GP cooperative.
The results of this study indicate that patients were generally satisfied about their contact with the GP cooperative. Patients who received telephone advice only, however, were less satisfied compared to those who attended the GP cooperative and those who received a home visit. A small majority believes that current out-of-hours care is an improvement compared to the former situation.
The response rate in our study is not as high as presented previously by others who investigated patient satisfaction with out-of-hours primary care[5,7-9,11]. Reasons for patients not to return the questionnaire in our study were assessed through the non-respondents interview. We found that most patients gave reasons that were not directly related to their contact with the GP cooperative. Therefore, we assume that this reduced response rate may have had little effect on the outcome of our study. In addition, the overall satisfaction in the non-respondents group did not differ much from that of the respondents.
In the process of determining relevant aspects of out-of-hours care to patients, we consulted the province patient organisation and studied discussions on out-of-hours care in newspapers. We have not used patient interviews, although this might have identified other relevant domains of out-of-hours care. However, we think that the current questionnaire captures many relevant domains of out-of-hours care to patients as well as to health professionals.
Based on results of a Danish study [4,5], we expected overall patient satisfaction to be low because our study took place relatively shortly after out-of-hours care had been reorganised. However, we have not assessed patient satisfaction before the reorganisation, and therefore it remains unclear whether satisfaction has changed. Nevertheless, this study showed that more than half of the patients believe that the reorganisation has improved out-of-hours primary care.
We have no reason to believe that the results of this study cannot be generalised to other regions in the Netherlands. Most GP cooperatives in the Netherlands are comparable, with respect to organisation and population size, to those in this study. In addition, the region in our study includes both rural and urban areas. Despite the similarities with out-of-hours primary care in other countries such as Ireland, the UK and Denmark, there are also differences with respect to the way these cooperatives are organised, and therefore care should be taken when generalising these results to other countries.
We identified various factors that are closely related to overall satisfaction. These factors give important insight in aspects of the GP cooperative that really matter in the patient's opinion on out-of-hours care. The patient's opinion on the doctor's assistant's attitude on the phone proved to be the strongest predictor of overall satisfaction with respect to those having received telephone advice and those that received a home visit. Also for those attending the GP cooperative, this factor was a relatively strong predictor; in this group the patient's satisfaction with the GP's treatment was by far the strongest predictor of overall satisfaction. Thus, it appears that the patients' impression of the first contact they have with the cooperative, which is mostly through telephone, strongly influences overall satisfaction.
In accordance with other studies we found that patients who received telephone advice only, are generally less satisfied with the out-of-hours service, compared to those attending the GP cooperative and those receiving a home visit[4,5,8,9,11]. Patient's expectation of care is assumed to be an important factor that influences overall satisfaction. In our study, only 35% of the patients with telephone advice expected that they would receive this type of consultation. In contrast, 85% of the patients that were asked to attend the cooperative or received a home visit found this type of consultation in line with their expectations. This difference in expectation of care may very well explain the difference in overall satisfaction.
It is questionable whether extra information to the public on the process of the telephone triage process will adjust patients' expectations. Similar to what Salisbury et al suggested, we believe that a shift to an out-of-hours care organisation based predominantly on telephone advice may decrease patient overall satisfaction. Therefore, proper information about the telephone procedure at the GP cooperative is desirable. This information can be supplied by the doctor's assistant on the phone, and by written information through folders and posters in GP practices.
This study has shown that patients are generally satisfied with out-of-hours care, but that patients with telephone advice only are less satisfied than those attending the cooperative or receiving a home visit. The patient's opinion on several aspects of out-of-hours care can predict overall satisfaction, with different predictors regarding the three types of consultations. However, the accessibility by phone and the doctor's assistant's attitude on the phone are always significantly related to overall satisfaction, regardless of the type of consultation. This implies that when trying to improve overall satisfaction one should always focus on at least these two factors. The questionnaire used in this study has potential for use as a standardised instrument for assessing satisfaction with out-of-hours care in The Netherlands for either research or service monitoring.
The author(s) declare that they have no competing interests.
CU participated in the design of the study, performed the statistical analysis, and drafted this manuscript. AA, SH, PZ, and HC participated in the design of the study, supervised the project, and provided critical edits to this manuscript.
We would like to thank Marloes Elferink, Elles van Cromvoirt, and Dr. Mariette Hubben for their help during the study.
BMJ 1994, 309:1621-1623. PubMed Abstract
BMJ 1994, 309:1624-1626. PubMed Abstract
Br J Gen Pract 1998, 48:1497-1499. PubMed Abstract
Cragg DK, McKinley RK, Roland MO, Campbell SM, Van F, Hastings AM, French DP, Manku-Scott TK, Roberts C: Comparison of out of hours care provided by patients' own general practitioners and commercial deputising services: a randomised controlled trial. I: The process of care.
Murphy AW, Bury G, Plunkett PK, Gibney D, Smith M, Mullan E, Johnson Z: Randomised controlled trial of general practitioner versus usual medical care in an urban accident and emergency department: process, outcome, and comparative cost.
McKinley RK, Manku-Scott T, Hastings AM, French DP, Baker R: Reliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the United Kingdom: development of a patient questionnaire.
McKinley RK, Cragg DK, Hastings AM, French DP, Manku-Scott TK, Campbell SM, Van F, Roland MO, Roberts C: Comparison of out of hours care provided by patients' own general practitioners and commercial deputising services: a randomised controlled trial. II: The outcome of care.
The pre-publication history for this paper can be accessed here: