The review sought to identify all studies (observational or experimental) that presented a comparison of self-report and direct measurement results to reveal differences in physical activity levels based on measurement in adult populations (18 years and over). Studies which examined only a self-report or direct measure, but not both were not included in the review. All study designs were eligible (e.g. retrospective, prospective, case control, randomized controlled trial, etc.) and both published (peer-reviewed) and unpublished literature were examined.

Only studies involving adult populations with a mean age of 18 years and older were considered. Abstracts and titles were examined for their mention of adult populations (using adult$.tw.), but the search relied mostly on the subject headings for adult age groups (exp adult/). This systematic review was conducted simultaneously with a systematic review of the same focus in child populations (mean age < 19 years). A separate pediatric review was carried out as a result of differences in measurement methodologies and hypothesized cognitive and recall abilities between adults and children 6.

The eligible self-report measures of physical activity included: diaries or logs; questionnaires; surveys; and recall interviews. Proxy-reports were excluded because they present issues of reliability due to the potential heterogeneity of reporters (e.g., spouse, trainer, coach, parent, caregiver). The eligible direct measures of physical activity included: doubly-labeled water (DLW), indirect or direct calorimetry, accelerometry, pedometry, heart rate monitoring (HRM), global positioning systems, and direct observation. Although no language restrictions were imposed in the search, only English language articles were included in the review. Abstracts were included if they provided sufficient details to meet inclusion criteria.

The following electronic bibliographic databases were searched using a comprehensive search strategy to identify relevant studies reporting the use of both self-report and direct measures for assessing individual physical activity levels: Ovid MEDLINE(R) (1950 to April Week 4 2007); Ovid EMBASE (1980 to 2007 Week 18); Ovid CINAHL (1982 to April Week 4 2007); Ovid PsycINFO (1806 to April Week 1 2007); SPORTDiscus (1830 to April 2007); Physical Education Index (1970 to April 2007); Dissertations and Theses (1861 to April 2007); and Ovid MEDLINE (R) Daily Update (May 4, 2007). The search strategy is illustrated using the MEDLINE search as an example (Table 1) and was modified according to the indexing systems of the other databases. The OVID interface was used to search MEDLINE, EMBASE, CINAHL, and PsycINFO; Ebscohost was used to search SPORTDiscus; Scholar's Portal was used to search Physical Education Index; and ProQuest for Dissertations and Theses. Grey literature (non-peer reviewed works) included published abstracts and conference proceedings, published lists of theses and dissertations, and government reports. Knowledgeable researchers in the field were solicited for key studies of interest. The bibliographies of key studies selected for the review were examined to identify further studies.

Two independent reviewers screened the titles and abstracts of all studies to identify potentially-relevant articles. Duplicates were manually removed. The full texts of all studies that met the inclusion criteria were then obtained and reviewed. When disagreements between reviewers occurred, consensus was achieved through discussion and/or with a third reviewer.

Standardized data abstraction forms were completed by one reviewer and verified by two others. Information was extracted on the type of study design, participant characteristics, sample size, and methods of physical activity measurement (self-report and direct measures employed, units of measurement, duration of direct measure, length of recall, and length of time between the self-report and directly measured estimates). Reviewers were not blinded to the authors or journals when extracting data.

The Downs and Black 7 checklist was used to assess the risk of bias. The Downs and Black instrument was recommended for assessing risk of bias in observational studies in a recent systematic review 8 and other assessments 9 and was employed in this review to assess study quality including reporting, external validity, and internal validity (bias). The Downs and Black checklist consists of 27 items with a maximum count of 32 points. A modified version of the checklist was employed with items that were not relevant to the objectives of this review removed. The adapted checklist consisted of 15 items, including items 1–4, 6, 7, 9–13, 16–18, and 20 from the original list, with a maximum possible count of 15 points (higher scores indicate superior quality). The risk of bias assessment was carried out by two independent assessors and when disagreements between assessors occurred, consensus was achieved through discussion.

Percent mean difference was used as the main outcome of this analysis; it was calculated using the formula: [(self-report mean – direct mean)/direct mean]. Only studies with units of measurement that were the same for both the self-report and direct measures were used to calculate percent mean differences. Units were converted where possible. These studies were included in the direct comparison analyses. Forest plots (graphical displays of the percent mean differences across the individual studies) were constructed to present overall trends in agreement of physical activity by direct measure and gender. As most studies did not employ the same units of measurement (e.g. kcal/week, MET/day, MET-min/day) and did not report a measure of variance (e.g. standard deviations or standard errors), pooled estimates and confidence intervals were not calculated.

The preliminary search of electronic bibliographic databases, reference lists and grey literature identified 4,463 citations (see Figure 1). Of these, 1,638 were identified in MEDLINE, 1,306 in EMBASE, 732 in CINAHL, 218 in PsycINFO, 133 in SportDISCUS, 34 in Physical Education Index, 3 in MEDLINE Daily Update, and 399 from Dissertations and Theses. After a preliminary title and abstract review, 296 full text articles were retrieved for a detailed assessment. Of these, 173 met the criteria for study inclusion. One hundred and forty-eight of these studies reported correlation statistics 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157. Seventy-four studies contained comparable data meaning the self-report and direct measurements were reported using the same units 1115171920233233444853565758596573747576778088909294100102105111114116119120121128131134135138139140143148151153154158159160161162163164165166167168169170171172173174175176177178179180181182183. These studies were included in the direct comparison analyses and their characteristics are described in Table 2. Common reasons for excluding studies included: populations with mean ages less than 18 years, the absence of directly measured and self-report data on the same population, non-English language, duplicate reporting of data, and the absence of comparable units between measures or the absence of a direct comparison.

Data abstraction identified three articles and two dissertations that analyzed and reported duplicate data in multiple papers 184185186187188. Authors of suspected duplications were contacted and in cases where several publications reported the same analyses from the same data source, only one study per data source/analysis was retained in order to avoid double counting. Studies were retained based on the most pertinent and most recent data, as well as the largest sample size. Studies included were published over a 24-year period from 1983 to 2007. All studies were written in English. Nineteen of the studies used randomized controlled trial designs 22242628305053618491124148149163165171181 and all others used observational designs (e.g. case control, cross-sectional, longitudinal). All included studies were published as journal articles except for 19 dissertations 162430343845496164697173747899107117163171.

Participants in the studies ranged from 10 to 101 years of age. Although the focus of the review was on those aged 18 and over, studies that had a range of ages less than 18 years were not excluded as long as the mean age of the sample was over 18 years. Sample sizes ranged from a low of six 21 to a high of 2,721 in Craig et al.'s work that assessed the validity of the International Physical Activity Questionnaire (IPAQ) 35. There were a greater number of studies reporting on female-only data than studies reporting on male-only data.

A total of five direct measures were used in the assessment of physical activity and included: accelerometers, DLW, indirect calorimetry, HRM, and pedometers. Of the studies included in the synthesis of directly comparable data (Table 2), accelerometers were the most frequently used direct measure and indirect calorimetry was the least used. A variety of self-report measures were employed, but the seven-day physical activity recall (7-day PAR) 189 was the most cited. Over half of the studies reported that the self-report and directly assessed physical activity levels were measured over the same length of time (e.g. seven days) and over the same period of time (i.e. no time lag between measurements). There were also a considerable number who reported measurements over the same period of time, but that did not measure the same length of time (e.g. self-report over seven days, directly measured over three days). Eleven of the studies in Table 2 lacked any mention of time 59131135138143159160164177178183.

Risk of bias was assessed for all included studies (n = 173) including those reporting only correlation data. The range of items met on the modified Downs and Black tool was 8 to 15 (maximum possible count was 15) with a mean of 11.7 ± 1.2. Results of the risk of bias assessment indicated that 38% (65/173) of the studies had lower quality (based on a median split count of < 12/15). All studies were given maximum points for describing study objectives. All but one study scored maximum points for describing the main outcomes to be measured and the interventions used (including comparison methods between measures). Although most studies carried out some sort of significance testing on results, most did not report the actual probability values associated with the estimates or their associated measures of random variability (e.g. standard error or confidence intervals). Most studies obtained a high number of items on the reporting section (maximum count of 8) with a mean of 6.9 ± 0.9.

The external validity section of the risk of bias assessment had a maximum count of three and consisted of reporting on the representativenessof the subjects and the testing conditions. Almost all of the studies (166/173) reported that the staff, places and facilities where the participants were tested were representative of the testing conditions that would be expected by most individuals (e.g. real-life and free-living situations). However, 87% (151/173) of the studies did not report on the representativeness of the subjects asked to participate in the study and 95% (165/173) of the studies failed to report on the representativeness of those who were prepared to participate (enrolled) compared to the entire population from which they were recruited (received a score of 0). As a result, the external validity ratings of most studies were poor with a mean of 1.1 ± 0.5.

In order to obtain the maximum number of items (four) in the internal validity section, studies must have reported whether any of the results of the study were based on "data dredging", whether the analyses adjusted for any time lag between the two measurements or different lengths of follow-up, whether the statistical tests used to assess the main outcomes were appropriate, and whether the main outcome measures were accurate (valid and reliable). Internal validity item counts were generally high with the majority of studies having obtained a four.

A qualitative analysis was conducted on the top seven (scores of 14 and 15 out of 15) and lowest seven studies (8 and 9 out of 15) based on scores from the risk of bias assessment. No conclusive patterns were identified from this analysis. The results from the accelerometer studies were further examined, as this was the only group of studies with a good distribution of low and high quality studies based on the accelerometer median split of bias scores. Findings from this analysis did not identify any clear patterns in the differences in agreement between physical activity measured by self-report compared to accelerometer when grouped by low and high quality.

One hundred and forty-eight studies 1011131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157190 reported correlation statistics between self-report and direct measurements of physical activity. Figure 2 is a plot of all extracted correlations and shows that overall, there is no clear trend in the degree of correlation between self-reported and directly measured physical activity, regardless of the direct method employed. Overall, correlations were low-to-moderate with a mean of 0.37 (SD = 0.25) and a range of -0.71 to 0.98. Mean correlations were higher in studies reporting results for males-only (r = 0.47) versus studies reporting results for females-only (r = 0.36), but with very similar ranges (males: -0.17 to 0.93 vs. females: -0.17 to 0.95).

Seventy-four studies contained comparable data on the measurement of physical activity based on self-report and directly measured values. Table 2 describes these studies and their subcomponents. Percent mean differences were calculated for all of these studies and are presented as forest plots in Figures 3 to 8. Negative values indicate that self-report estimates were lower than the amount of physical activity assessed by direct methods while positive values indicate values that are higher. Sixty percent of the percent mean differences indicated that self-reported physical activity estimates were higher than those measured by direct methods.

Studies with extreme percent mean differences (≥ 400%) were removed from the forest plots for clarity purposes 11139151181. All outlying data were from studies where physical activity was categorized by level of exertion (e.g. easy, moderate, vigorous) and outliers represent physical activity data categorized as vigorous or of high energy expenditure. While not all data categorized as vigorous had percent mean differences ≥ 400%, a pattern emerged whereby greater percent mean differences between the self-report and direct measures was larger for vigorous levels of physical activity than for light or moderate activities 114456134139151175181182.

Percent mean differences were examined separately for the five different direct measures. Accelerometers were the most used direct measure. Self-report measures of physical activity were generally higher than those directly measured by accelerometers (Figures 3 to 5). Studies reporting data for males and females combined (n = 58) had a mean percent difference of 44% (range: -78% to 500%), with similar findings for the male-only data (n = 32) (mean: 44%, range: -100% to 425%). However, female-only data (n = 60) identified that, on average, females self-reported higher levels of physical activity compared to accelerometers with a mean percent difference of 138% (range: -100% to 4024%).

The second-most common direct measure employed was DLW and comparable data with self-report measures are presented in Figures 6 to 8. Studies reporting on combined male and female data (n = 6) indicated that self-report measures of physical activity were lower when compared to DLW measures with a mean percent difference of -9% and a range of -1% to -26%. Results for male-only (n = 16) and female-only (n = 23) data were less distinct with percent mean differences and ranges of -4.5% (-78% to 37%) and 7% (-58% to 113%), respectively.

A greater number of HRM and self-report comparisons were observed for studies with both male and female participants (n = 11) or female-only populations (n = 13) versus male-only populations (n = 3). Female-only results showed a general trend toward higher levels of self-reported physical activity (mean 11%, range: -5% to 45%), while the male-only (mean -9%, range: -24% to 5%) and combined (mean -2%, range: -21% to 67%) data had a greater number of studies with lower self-reported physical activity levels when compared to results of HRM.

Pedometers and indirect calorimetry were the least commonly used direct measures for studies with comparable data. There were a total of eight comparisons from four studies for pedometers and 15 from two studies for indirect calorimetry (Figures 6 to 8) making it difficult to draw conclusions with regard to patterns of agreement between the self-report and direct measures. However, seven 197576167 of the eight pedometer comparisons reported higher levels of physical activity by self-report when compared to the pedometer results. The eighth comparison 19 which involved female-only data saw no difference between the two measures. The indirect calorimetry results were less straightforward and presented no obvious patterns in agreement.

Subgroups were qualitatively examined to assess whether any differences existed in the degree of agreement between self-reported and directly measured physical activity. No clear patterns emerged within studies reporting on elderly (range or mean ≥ 65 years) populations 23737792105116174 or within studies reporting on different time lags and periods of measurement. Few studies with comparable data reported exclusively on overweight/obese populations, but amongst those captured, the majority of studies reported higher levels of physical activity by self-report compared to the direct measures 139143148163164165172. However, it was not possible to compare the overweight/obese percent mean differences to those reported in general populations.

Meta-analyses were not possible due to the substantial heterogeneity in units of reporting for physical activity measured by the various self-report and direct methods across the studies, and the significant lack of data with comparable units across measures. As a result, we were unable to determine the sensitivity of the values and the associated measures of error for the studies. Overall effect sizes to summarize the magnitude of discrepancy across the various measures of physical activity could therefore not be calculated.