Enzyme-linked immunospot (ELISPOT) and cytokine flow cytometry (CFC) (or more specifically, intracellular cytokine staining (ICS)) are popular methods for single-cell analysis of antigen-specific T cell cytokine production. T cell production of IFNγ, and increasingly also IL-2, is taken as a measure of vaccine immunogenicity in experimental vaccine trials. Of the two types of assays, ICS has the advantage of a highly multiparametric read-out (flow cytometry) that allows for precise phenotyping of the responding T cell populations. It has also recently been adapted to a 96-well plate configuration 12, allowing for higher throughput analysis similar to that used for ELISPOT. However, while the precision of ELISPOT assays across sites has been recently documented 3, similar studies for ICS assays have been lacking.

Numerous phase I and phase II clinical trials have been initiated using candidate prophylactic HIV vaccines (reviewed in 4). Many of these trials use ICS as part of their immune monitoring. While most current HIV trials are not powered to determine efficacy, and cytokine production has not been validated as a surrogate marker of protection from HIV infection or progression, there is nevertheless a desire to measure immunogenicity of candidate vaccines as well as safety in early clinical trials 5. Because many different groups are performing immune monitoring for these clinical trials, there is currently a lack of standardization that would allow accurate comparisons of immunogenicity across candidate vaccines in different clinical trials.

There is some published literature on the intra-and inter-assay precision of ICS assays in whole blood 6. These values were determined to be about 8% and 20% C.V., respectively. Guidelines for performance of ICS assays have also been recently published 7. However, there are no existing data documenting the precision of ICS between laboratories, or comparing the precision of ICS using different sample types (e.g., whole blood versus cryopreserved PBMC). In order to allow more meaningful comparisons between laboratories and prioritization of emerging vaccine candidates, and thereby accelerate HIV vaccine development, this ICS standardization study was undertaken.

The objectives of the study were three-fold: (1) to assess the reproducibility of ICS assays using different sample types (shipped whole blood vs. cryopreserved PBMC); (2) to determine the inter-laboratory precision of ICS assays among major HIV vaccine clinical research laboratories; and (3) to improve the concordance of methodologies used in these laboratories. To achieve these objectives, joint experiments (Figure 1) were devised using (1) whole blood activated at a central site, then fixed, frozen and shipped to participating laboratories for processing and analysis; (2) fresh whole blood drawn at a central site and shipped to participating labs for activation, processing, and analysis; and (3) cryopreserved PBMC shipped from a central site to participating labs for activation, processing, and analysis. In the latter case, this experiment was also repeated with a larger number of participating laboratories, using pre-formatted microtiter plates containing lyophilised stimuli and lyophilised staining antibodies. In each experiment, raw data files were also sent by the participating labs to a central site for analysis, which was done using a dynamic gating template and batch analysis 1 (Figure 2).

This study examined the reproducibility of ICS assays across sites using different assay formats. It was not designed to compare ICS with other immune monitoring assays, comparisons of which have been published 15161718192021. The current study used 96-well plate-based protocols exclusively, as these were considered more convenient, and have recently been validated against tube-based protocols for both PBMC and whole blood 1.

Lyophilised reagents in plates were used for Experiment 4. These have been extensively compared to liquid reagents (22 and Figure 9) and shown to be largely equivalent. In addition to convenience of assay set-up, the lyophilised reagent plates offer long reagent stability, even at room temperature (>1 year, data not shown), and a potential reduction in errors caused by incorrect reagent addition. Intra-plate variability using lyophilised reagents was determined to be <10% in ICS assays (data not shown).

There are some potential drawbacks to the use of 96-well plates. One of these is the possibility of well-to-well contamination during the assay. This was observed in an initial subset of Experiment 4 (data not shown), in which some sites received lyophilised plates with SEB as a positive control. Some of these sites experienced high backgrounds in the negative control wells adjacent to the SEB-containing wells. It was later determined that cross-contamination probably occurred during the initial distribution of the antigens on the plates, and this was compounded by the fact that the donors used were unusually sensitive to SEB stimulation (responses >30% of CD4⁺ and CD8⁺ T cells). When SEB was replaced with CEF as a positive control, no such problems were noted. This experience suggests that the choice and placement of positive control wells on a plate deserves consideration.

The current study was designed to determine inter-lab variability in ICS assays. As such, there were no data "filters" applied to exclude potentially erroneous data or outliers. However, improved precision of ICS results might be obtained if certain acceptance criteria were applied before data were taken as valid. For example, a minimum number of collected events could be specified (sites in this study were asked to collect 10,000–40,000 CD4⁺ or CD8⁺ T cells per sample, or 60,000 CD3⁺ cells). This number of events was designed to yield precision levels that would minimize event number as a factor in inter-lab reproducibility. There could also be acceptance criteria based upon the absolute level of background, or the degree of reproducibility between duplicate samples, if run (the current study did not use duplicate samples).

It is also possible to apply statistics to derive further meaning from numerical results. For example, statistical tests could be used to determine whether a given response can be discriminated from a given background, for a particular number of events collected 2324. This can be given by a power calculation as follows:

N = [2*P_av(1-P_av)(Z_α +Z_β)²]/Δ²

where N is the number of events in each sample needed for significance, P_av is the average proportion (between the background and test samples), and Δ is the difference between these two proportions. The term (Z_α +Z_β)² is referred to as a power index, and varies depending upon the desired power and p value. For example, (Z_α +Z_β)² = 23.9 for 99% power and p < 0.005 23.

In addition, a confidence interval could be derived around the difference of the test result and the negative control 24, in order to allow discrimination of significant differences between various samples. Other statistical methods have also been employed in order to determine cut-off values for positive responses in ICS 2526. No attempt was made in the current study to define which results were positive, as all data were reported objectively, and all donors were known to be CMV seropositive.

Examination of the data from Figures 3, 4, 6, and 7 suggests that samples with a low number of cytokine-positive cells had higher variability than samples with a high number of cytokine-positive cells. The relationship of response level and C.V. is summarized in Table 2 for all assays (CD4 and CD8, whole blood and PBMC) considered together. These data emphasize the difficulty of achieving precise results at response levels of less than 0.1% of CD4 or CD8 T cells. For these samples, collecting even more events than what was suggested would be expected to improve precision, per the discussion above.

The average C.V. across the four experiments is summarized in Table 3. These data are confounded by the fact that different donors and different laboratories participated in the four experiments. However, variability due to individual analysis can be excluded by comparing only centrally analysed data (bottom row of Table 3). Assuming no effect from the other confounding variables, we found that Experiment 4 (cryopreserved PBMC with lyoplates) yielded a significantly lower average C.V. than Experiment 2 (shipped whole blood) (p < 0.05). Also, the average C.V. of centrally analysed data from all experiments was significantly lower than that of individually analysed data (p < 0.0001). This highlights the amount of variability in each experiment that is due to gating differences between sites.

Mitigation of gating variability was achieved in these experiments by centralized analysis with a dynamic gating template (see Figure 2B). The dynamic gating template allowed for more automated, batch analysis of the data. Once such a template was created and optimized (see Materials and Methods section for description), it could also have been provided to individual sites in order to yield the same results. It is further possible that similar results could be achieved by manual analysis, provided it was done by a single operator. Standardization of gating techniques, in the absence of centralized analysis or dynamic gating templates, could also improve precision. The improvement in C.V. made by centralized analysis was most marked in the first experiment, and progressively less in experiments 2 and 3, perhaps because of standardization of gating among sites over time. Experiment 4 included many new sites, and the improvement in C.V. from centralized analysis was again more marked.

Because the C.V. varies as a function of the response level (Table 2), it is possible that differences in mean C.V. between assay formats were due to the number of low versus high responders in each experiment (since different donors were used in the four experiments). Also, the C.V. is highly sensitive to small changes in the mean, when the mean is a very low number. Therefore, an analysis of S.D. versus mean was also performed for the four experiments (Figure 10). This data confirms the data of Table 3, indicating that the three assay formats showed grossly similar reproducibility. However, when analysis variability was removed, cryopreserved PBMC assays appeared to be slightly more reproducible than shipped whole blood assays. This seemed especially apparent in experiment 4, where lyophilised reagents were used.

In addition to differences in reproducibility, the various assay formats have other benefits and drawbacks as well. Cryopreserved PBMC are much more amenable to peptide (and superantigen) stimulation than to whole protein stimulation 9; while whole blood assays are equally amenable to stimulation with either type of antigen. Also, consistently good cryopreservation of PBMC at multiple clinical sites is difficult to achieve, but highly important for achieving reproducible results with PBMC 2728 (DeLaRosa et al., manuscript in preparation). This could become less of a factor if a stabilizing matrix for preserving whole blood or PBMC function during shipping were discovered. All in all, the choice of assay format for a clinical trial will depend not only upon considerations of assay precision, but also upon the type of antigen(s) used and the capabilities of the participating clinical sites.

The use of lyophilised reagent plates appeared to reduce inter-lab variability. This conclusion cannot be drawn with certainty, because different participating laboratories and different donors were used between experiments 3 and 4. However, it is intriguing to note that, when centrally analysed data was compared (to remove gating as a source of variability), the mean C.V.'s of experiment 4 were the lowest of all four experiments (18%, Table 3). This is despite the fact that the donors and stimuli used in experiment 4 resulted in lower mean response levels, which should tend to increase the C.V. This is also borne out by the analysis of Figure 11B, where the results for experiment 4 appeared to be generally closer to the theoretical minimum SD than did the results for the other experiments.

With the possibility of achieving inter-laboratory C.V.'s of less than 20%, even with relatively low responses, ICS compares favourably to ELISPOT, for which interassay C.V.'s of 17–18% for PHA and 55–65% for Candida have been reported 2930. ICS is also comparable to cytokine ELISA, the latter having reported interassay C.V.'s of <25% 3132. Phenotypic staining, such as used for CD4 counting, can achieve higher precision levels than functional assays, and averaged around 10% C.V. in one multisite study 33. For comparison, the inter-lab C.V. of the CD4⁺ or CD8⁺ cell percentages was around 5% in experiment 4 of the present study (data not shown). CD4 counting precision has also been shown to be dependent upon the number of events collected, gating, and use of automated analysis 3334. Since functional assays are subject to more variables than phenotypic staining, the ability to achieve precision levels such as those reported here should be considered favourable. ICS could thus be a viable tool for comparing immune responses even across clinical trials, provided the methodology was standardized.

ICS assays could be performed with inter-laboratory C.V.'s of approximately 20% at response levels of >0.5%, and C.V.'s of approximately 25–30% at response levels of 0.1–0.5%. The C.V. increased further at response levels of =0.1%. A significant portion of inter-laboratory variability could be eliminated by use of centralized analysis and/or a dynamic gating template.

Whole blood and cryopreserved PBMC showed grossly similar levels of reproducibility. However, when analysis variability was removed, cryopreserved PBMC processed with lyophilized reagents showed significantly better reproducibility than shipped whole blood. Shipped whole blood assays were also subject to data loss when samples were not delivered in a timely fashion.

Background cytokine production was mostly =0.05% for both CD4 and CD8 cells. While CD8 backgrounds were lower in cryopreserved PBMC than in whole blood, this could have been due to the use of different donors in the four experiments. With the high viabilities and recoveries obtained for cryopreserved PBMC in this study, there was no obvious relationship between viability/recovery and response.

The use of microtiter plates containing lyophilised reagents simplified the ICS protocol, and appeared to improve assay reproducibility. This format lends itself to international shipping of reagents (because there is no need for refrigeration), and also to larger clinical trials (because of the stability of the lyophilised reagents). It is also a way to reduce the chance of pipetting errors, because the plates are pre-formatted.

The results of this study indicate that ICS assays can be reasonably standardized between sites, but that considerations of sample format and expected response levels can influence the precision of the results. These data should guide comparisons of ICS results between different groups or in different clinical trials.

HTM compiled the data and drafted the manuscript. AR coordinated the study. P.D'S. and J.D. secured logistic and financial support. All authors helped design the study, supervised and/or carried out the experiments, and provided editorial comments and assistance.