This study was conducted in the province of Quebec, Canada, where universal health coverage is provided through the provincial health insurance plan. Each Canadian province maintains a population-based registry of insured persons and claims for all physician visits remunerated on a fee-for-service basis. Physician claims include information on the diagnosis (recorded as an ICD-9 code), medical procedure, visit date, location, and cost of service. All claims also record unique physician and patient identifiers that can be used to create longitudinal histories of healthcare use. In the province of Quebec, 99% of residents have provincial health insurance and 85-95% of medical visits are remunerated on a fee-for-service basis 15 . In 2006, there were more than 7.6 million inhabitants in Quebec 16 , and 18,908 active registered physicians 17 . The availability of diagnostic information for nearly all medical visits to Quebec physicians represents an invaluable opportunity for assessing the validity of using diagnostic codes in physician claims for population-based surveillance, including syndromic surveillance.

The accuracy of diagnostic codes in physician claims for identifying syndromes was assessed by comparison to clinical information in the corresponding medical chart. To ensure representativeness, we used a population-based, 3-stage stratified random sample of 36,000 visits (Figure 1). In the first stage (Figure 1 Stage 1), the provincial health insurance agency identified all physicians who were eligible to be included in our study. To be eligible, physicians had to be a general practitioner, pediatrician, internist, geriatrician or general surgeon who practiced in the fee-for-service system in a private clinic, community health center, or hospital-based ambulatory care clinic during the 2-year study period (October 1, 2005 to September 30, 2007). Internists and general surgeons were included in our sample because, especially in rural-remote and underserved areas, these physicians may provide first-contact care and act as patients' family physician. From the 8,700 eligible physicians identified, the provincial health insurance agency selected a random sample of 3,600 (41.4%) physicians.

In the second stage (Figure 1 Stage 2), to facilitate chart retrieval for review, the health insurance agency randomly selected one eligible community practice location for each physician. The health insurance agency then sent the research team an anonymized file containing all physician claims billed by the 3,600 physicians from their respective selected community practice location during the 2-year study period (Figure 2 Step 1).

In the third stage (Figure 1 Stage 3), the research team randomly selected 5 syndrome-positive visits, i.e., 1 visit for each of fever, gastrointestinal, neurological, rash, and respiratory syndrome (including ILI), and 5 visits negative for all syndromes. Visits were classified as positive for a syndrome if a physician claim for the visit had an ICD-9 code that was part of the syndrome definition. Because syndromes have low population prevalence, to maximize data collection efficiency 18 , syndrome-positive visits were over-sampled relative to syndrome-negative ones, so as to yield 1 syndrome-positive visit per syndrome per physician and 5 syndrome-negative visits per physician. When sampling syndrome-positive claims, to maximize the number of syndrome-positive ICD-9 codes verified, we further stratified on ICD-9 code. Because two or more syndromes can occur concurrently in the same patient 19 , syndrome-negative visits were negative for all syndromes. Syndrome-negative visits were also matched to syndrome-positive visits on calendar month to avoid bias due to syndrome seasonality. To avoid bias due to visits being clustered within patients, restriction was used to ensure that each patient was only sampled once. The list of 10 sampled visits was enumerated for each of the 3,600 physicians, for a total of 36,000 visits. An anonymized unique identifier, the study number, was assigned to each sampled visit by the research team. The list of 36,000 sampled visits was then sent to the health insurance agency (Figure 2 Step 2).

We verified two sets of definitions for the 5 syndromes under study: the definitions developed and published by the US Centers for Disease Control and Prevention (CDC) in 2003 13 , and used by the US Department of Defense's (DoD) Electronic Surveillance System for Early Notification of Community-based Epidemics (ESSENCE), as well as the corresponding definitions in the University of Pittsburgh's Real-time Outbreak and Disease Surveillance (RODS) system 19 . For ILI, we used the large-group (sensitive) and small-group (specific) definitions developed for the DoD ESSENCE system 20 . These definitions are similar to the consensus syndrome definitions being developed by representatives from the 10 syndromic surveillance systems in place in the US 21 , which have not yet been mapped to ICD-9 codes.

To preserve physician and patient anonymity, the health insurance agency sent the list of 3,600 physicians and 36,000 visits sampled by the research team to the medical regulatory authority (Figure 2 Step 3). The medical regulatory authority has the legal right to access confidential physician and patient information, therefore the list it received included physician names and mailing addresses, as well as patient names, insurance numbers, and dates of birth. The medical regulatory authority acted as a trusted third party and recruited physicians to the study on behalf of the research team; it also provided physicians with information on the 10 sampled visits (Figure 2 Step 4). Interested physicians mailed their written consent and contact information to the research team (Figure 2 Step 5). Non-responding physicians were sent up to four reminders. Physician recruitment began in September 2008 and ended in August 2009. To maximize participation, physicians were offered $50 compensation for their participation and a summary of study findings.

The medical regulatory authority sent each physician the list of 10 sampled visits (Figure 2 Step 3). Lists sent to physicians included patients' first and last names, date of birth, health insurance number, and date of the visit to be verified, as well as the study number for each visit. Because the lists sent to physicians contained both patient information and study numbers, it enabled physicians to retrieve the relevant medical charts, and researchers to link the information collected through chart review to the anonymized physician claims file. During the chart review, interviewers and physicians referred to each visit using only the study number and visit date, thereby preserving patient anonymity.

Physician-facilitated medical chart reviews began in September 2008 and ended in December 2009. Using a previously published methodology 22 , trained interviewers contacted consenting physicians by telephone to perform the chart review (Figure 2 Step 6). For each of the 10 sampled visits, the interviewer asked the physician to list all diagnoses. For each diagnosis corresponding to a syndrome definition, the interviewer asked the physician about the signs, symptoms, and key findings recorded in the medical chart, as well as the most likely etiology for the diagnosis (based solely on information available at the time of the visit).

Physician responses were entered directly into a database by the interviewer. Diagnoses were selected from a searchable list of diagnoses (mapped to ICD-9 codes) or, if the physician had recorded the ICD-9 code in the medical chart, the ICD-9 code was entered directly. For each syndrome-positive diagnosis, a list of syndrome-specific signs and symptoms was elicited, and the interviewer recorded whether the sign or symptom had been present, absent, or not recorded in the medical chart. Symptoms or signs not in the list and other key findings, such as epidemiologic links to other diagnosed cases or known outbreaks, were recorded as free text in separate fields. The data collection tool was translated to French for use with French-speaking physicians, and back-translated to English to ensure comparability of data collection.

At the time of chart review, the physician and interviewer were both blinded to the ICD-9 code in the physician claim and the syndrome-positive or syndrome-negative status of the claim. To minimize measurement error due to inter-rater differences, interviewers were trained to use the data collection tool. Inter-rater reliability was assessed at baseline by having interviewers perform 2 simulated physician interviews of 10 visits each (for a total of 20 visits). To maintain data quality, interviewers underwent quality assurance monitoring every 3 months. Each assessment was comprised of 2 simulated physician interviews of 10 visits each (for a total of 20 visits). Agreement between raters was measured using the intraclass correlation coefficient (ICC).

The database containing the medical chart review data was linked to the physician claims file using the study number, physician identifier, and visit date. In our pilot study 14 , we found that the visit date in the chart sometimes differed slightly from the visit date on the claim. We considered the chart and the claim to refer to the same visit if the visit date in the chart was within 0 (identical date) to 3 days from the visit date in the claim.

Physician gender, preferred language (French or English), specialty, practice setting, and geographic location were obtained from the health insurance agency. Physician year of licensure was obtained from the medical regulatory authority. The number of days worked per year was calculated as the number of days when at least one claim was billed by the physician to the health insurance agency. The number of patients seen per day worked was calculated as the number of distinct patients for which one or more claim was billed by the physician per day worked. The number and prevalence of syndrome-positive visits were calculated for each physician using claims billed from the selected practice location during the 2-year study period.

For each visit, we assessed if the ICD-9 code in the physician claim and the diagnosis in the corresponding medical chart agreed as to the presence of each syndromes and ILI. For example, if the diagnosis in the claim was cough (786.2) and the diagnosis in the corresponding medical chart was acute bronchitis (466.0), then both the claim diagnosis and the chart diagnosis were positive for respiratory syndrome, therefore the claim was a true-positive for respiratory syndrome. If the diagnosis in the claim was cough (786.2) and the diagnoses in the chart were hypertension (401.9) and diabetes (250.0), then the claim diagnosis was positive for respiratory syndrome and the chart diagnoses were not, therefore the claim was a false-positive for respiratory syndrome.

The negative predictive value (NPV) of each syndrome definition was estimated directly from the data. Because we stratified syndrome-positive visits by ICD-9 code, we had to use an adjustment based on Bayes Theorem 23 to estimate the PPV of each syndrome. The PPV was estimated as a weighted average of each ICD-9 code's PPV, the weight being the number of visits with a given ICD-9 code divided by the total number of visits positive for that syndrome among participating physicians.

Because we verified more syndrome-positive visits than syndrome-negative ones, direct estimation of sensitivity and specificity using our data would lead to verification bias: sensitivity would be overestimated, and specificity underestimated 23 . Because verified claims were randomly sampled within syndrome-positive and syndrome-negative strata, unbiased estimation of these parameters was achieved by re-weighting for the verification fractions 23 . The sensitivity and specificity of physician claims for identifying each syndrome was estimated from the PPV and NPV 24 using the correction for verification bias 23 , re-weighting for the different sampling fractions. We estimated the 95% CI for the bias-corrected sensitivity and specificity using the methods described by Begg and Greenes 23 .

The research protocol for this study was reviewed and approved by the McGill University Institutional Review Board, the Quebec privacy commission (Commission d'accès à l'information du Québec), the legal department of the Quebec health insurance agency (Régie de l'assurance maladie du Québec), and the Quebec medical regulatory authority (Collège des médecins du Québec).

Of 3,600 physicians contacted, 172 (4.8%) had an incorrect address on file with the health insurance agency, and 170 (4.7%) were discovered to be ineligible (recently deceased, retired, on sick/maternity leave, no longer practicing at the selected practice location). Of the 3,258 remaining physicians, 1,129 (34.7%) physicians consented to participate in the study, 218 (6.7%) refused, and 1,911 (58.7%) did not respond. Of the 1,129 consenting physicians, 1,098 (97.3%) completed the physician-facilitated medical chart review, and 31 (2.7%) were unreachable or withdrew consent prior to interview. Participating and non-participating physicians were similar on all measured variables except two (Table 1): as compared to non-participants, participants had been in practice longer and had worked more days during the study period. Syndrome prevalence was similar among participating and non-participating physicians, and ranged from 5 per 1,000 visits for neurological syndrome and ILI small-group, to 126 per 1,000 visits for respiratory syndrome.

Agreement between raters was measured using simulated physician interviews shortly before the start of data collection and every 3 months thereafter. Agreement was perfect on all assessments (ICC = 1.00).

Of the 10,980 visits selected for verification (10 visits per participating physician), physicians were able to access the corresponding medical chart for 10,669 (97.2%). The most common reasons for being unable to access the chart were inability to locate the medical chart (151 charts) and medical chart in storage with retrieval fee (140 charts). For 10,465 (98.1%) of the sampled visits, the visit date in the medical chart was identical to the visit date on the claim. Allowing for potential date transcription errors during billing, an additional 64 (0.6%) visits with a date in the medical chart that was within 1-3 days of the visit date on the claim were identified, for a total of 10,529 visits for which both the medical chart and the claim was available and the visit dates were in agreement (within the 3 day time window).

Table 2 shows the accuracy of ICD-9 codes in physician claims for identifying syndromes, as compared to the medical chart. The sensivity of ICD-9 codes in physician claims for identifying syndromes was low, ranging from 0.11, 95% CI (0.10, 0.13) for fever syndrome to 0.44, 95% CI (0.41, 0.47) for respiratory syndrome. The PPV of ICD-9 codes in physician claims for identifying syndromes was moderate to high, ranging from 0.59, 95% CI (0.55, 0.64) for fever syndrome to 0.85, 95% CI (0.83, 0.88) for respiratory syndrome. Both the specificity and NPV of ICD-9 codes in physician claims were near-perfect for all syndromes studied.

Additional file 1 (excerpted in Table 3) shows the PPV of physician claims for identifying syndromes for each ICD-9 code individually. There was wide variation in PPV between different ICD-9 codes in a given syndrome. ICD-9 codes that were very rarely used by physicians, for example tularemia (ICD-9 code: 21.9), had a high probability of being false-positives, and therefore a very low PPV. ICD-9 codes for common symptoms, for example fever (ICD-9 code: 780.6), had a lower probability of being false-positives, and a higher PPV. ICD-9 codes that represent common diagnoses, for example acute bronchitis (ICD-9 code: 466.0), had the lowest probability of being false-positives, and the highest PPV.