If this E-Bulletin does not display correctly, please go to www.IMMUNIZATIONbasics.jsi.com to view it online.
Link to IMMUNIZATIONbasics website Link to USAID website
    IMMUNIZATIONbasics

Snap Shots is designed to keep busy public health professionals abreast of developments in the immunization world. Each issue focuses on an important topic or topics that may affect USAID missions, projects and country partners. We welcome your comments and suggestions.

Coverage Confusion! Trying to Make Sense of It

The Topic

Vaccination coverage rates are the most commonly used indicators of immunization program performance. Have you ever encountered different vaccination coverage estimates for the same country? In this issue of Snap Shots, we explain where these different coverage estimates come from and how they should and should not be used. We also point you to other references that we hope will help to clear up the coverage confusion.

DTP3 and Routine Immunization

Although coverage is calcuated for all vaccines, DTP3 coverage by one year of age is now widely accepted as the performance indicator for routine immunization. This is because DTP vaccine is most often delivered in routine immunization sessions versus campaigns. Also, completion of the DTP series before the first birthday has been a serious problem in some countries, so adoption of the DTP3 indicator is helping to focus global attention on this problem. We use DTP3 coverage throughout this issue to describe coverage trends.

Regional Coverage Trends

After increasing rapidly in the 1980s, vaccination coverage in many countries held steady or decreased during the 1990s. Since 2000, coverage has slowly begun to increase again. Globally, WHO/UNICEF estimate that DTP3 coverage increased from 75% in 2000 to 78% in 2004. The Africa region, where coverage fell sharply after 1990, experienced the most dramatic gains (54% in 2000 to 66% in 2004). Coverage in the Eastern Mediterranean region also improved (81% in 2000 to 86% in 2004), but in Southeast Asia coverage remained low (69% in 2004). The fact that it has not changed significantly since 1995 is also cause for concern.

Figure 1: Global DPT3 Coverage 1980 - 2004; Global Coverage Estimate at 78% in 2004

Common Sources of Coverage Data

WHO/UNICEF's estimates of DTP3 coverage are used in Figure 1 above. Other common sources of coverage data are administrative reports and household surveys. How do these three sources compare?
  • Administrative reports: National coverage rates are usually based on administrative data collected during vaccination sessions. Administrative data are used monthly, quarterly and annually to estimate coverage, calculate left-outs and drop-outs, and monitor trends. Administrative data overestimate coverage(1) when vaccinations given to children after their first birthdays are reported, census figures (coverage denominators) are too low, or reports are falsified. On the other hand, they underestimate coverage when census figures are too high or reporting is incomplete.
  • Household surveys: Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) produce national coverage estimates, but they are conducted infrequently and do not, in most cases, produce district coverage estimates. The CORE Group's Knowledge, Practice and Coverage (KPC) surveys produce reliable coverage estimates and are often used with smaller populations. Immunization Coverage Cluster Surveys, which are appropriate for use at both district and national level, provide a wealth of data not only about immunization coverage but also about service quality.

WHO/UNICEF Joint Reporting Form
Key Performance Indicators

  • Official country estimates of coverage for all EPI antigens
  • WHO/UNICEF estimates of coverage for most EPI antigens
  • DTP1-DTP3 drop-out rates - national rate and % of districts with drop-out >10%
  • Proportion of districts reporting DTP3 coverage of >90%, 80%-89%, 50%-79%, >50%
  • Proportion of districts reporting and with data included in the JRF
  • Line items in the national budget for vaccines, for injection supplies
  • Proportion of vaccine spending funded by Government (routine program only)
  • Proportion of immunization financed by Government
  • WHO/UNICEF estimates of coverage: In April each year, WHO and UNICEF collect coverage estimates and other information from individual countries on a Joint Reporting Form, or JRF. The JRF contains the official country estimates of coverage, which are most often based on administrative reports and census data. WHO and UNICEF then adjust the official country rates to reflect findings of population-based surveys and known data-quality and/or denominator issues.(2) The WHO/UNICEF coverage estimates are now used at global level to track country performance and measure progress toward global and regional immunization goals.

Where to Find Country Coverage Estimates

Both the official country estimates and the WHO/UNICEF estimates of vaccination coverage can be accessed through WHO's Vaccine Preventable Diseases Monitoring System.(3) This interactive web page contains country coverage estimates since 1980. It also makes available many other items of information from the annual JRF submissions that may be useful in assessing country performance (see box to the right).

Interpreting Coverage Estimates

When interpreting immunization coverage estimates, keep the following points in mind...
  1. Coverage rates are averages. High coverage rates almost always mask pockets of much lower coverage. Coverage rates also tell us little on their own about disparities among socioeconomic groups.(4) To truly understand a country's coverage situation, one must have access to subnational coverage data. To determine whether the poor and other marginalized groups are being reached with immunization services, whenever possible, household surveys should be analyzed by wealth quintile.
  2. Like stocks and mutual funds, even very high coverage this year does not guarantee the same coverage next year. In one African country, national DTP3 coverage has been over 80% for all but three of the past 10 years, but only 11% of districts have reported 80% coverage or higher in each of the last three years. This suggests frailty in the immunization system and demonstrates the need to consider more than one year's data at any given time.
  3. A coverage estimate is only reliable if the data behind it are of reasonable quality. GAVI Data Quality Audits (DQA) are helping improve data collection and reporting in many countries. Nonetheless, different sources of immunization data often yield different coverage estimates, and all coverage data have certain limitations. For this reason, one must always look at the source of a coverage estimate to figure out what it really means and how it might be used.

Deciding Which Coverage Estimates to Use

Coverage estimates from different sources are often used for different purposes. Administrative data are most readily available at each level of a health system and, in lieu of more reliable data, they are routinely used by program managers and donors to monitor trends in coverage and pinpoint problems. Surveys generate more reliable estimates of coverage, but they are carried out less frequently and cannot be used to monitor trends on a monthly or even an annual basis. Also, national surveys do not generally permit district-level estimates, which, as mentioned above, are critical to understanding country performance.

Figure 2: Comparing Immunization Coverage Rates from Different Sources
Whichever data source or sources are chosen, it is always important NOT to attempt to compare coverage estimates that are NOT comparable. In Figure 2 above, clearly it would be meaningless to use the DHS data from 1997 and the administrative data from 1999 to describe a coverage trend. A quick list of the questions that program planners and managers should ask themselves before selecting and attempting to compare coverage estimates is shown below.

BEFORE Comparing Coverage Estimates, Ask Yourself...


  • Are the geographic areas and populations the same? Comparing coverage rates from national and district surveys at a single point in time can be interesting. However, national coverage rates should not be used as baseline measures for individual districts because they do not represent the same populations.
  • Are the age groups the same? Some surveys of children 12-23 months of age report vaccinations received only up to 12 months of age (the recommended method when reporting immunization coverage). Others report all vaccinations that children receive up to the time of a survey. This means that vaccinations given to children older than 12 months of age are included and are thereby inflating coverage estimates.
  • Does "vaccinated" mean the same thing? Some surveys report coverage based only on what is documented by vaccination cards, while others include data from caretakers' recall. This has a huge effect on reported coverage. In reality, coverage "by card" and "by card and recall" should both be reported.
  • Are denominators correct? In administrative reporting, coverage may rise suddenly following a census that significantly undercounts the population. Or, it may fall just as suddenly when a new census corrects a faulty denominator.
  • Are the data reliable? Administrative reports often overestimate coverage, but surveys may suffer from data quality and sampling problems as well. Whatever the source, data quality issues should be considered before deciding to use and compare coverage results.
  • What time periods are covered? If comparing data from different sources for what you believe are the same time periods, make sure this is actually the case. Surveys conducted this year generally reflect vaccinations administered 12-23 months ago. Therefore, comparing this year's administrative data to this year's survey findings will not produce comparable results.

What Can USAID Missions and Projects Do to Reduce the Coverage Confusion?

  • Use available coverage rates, but also question them. Look behind the figures to understand how they were calculated and what they mean.
  • Seek information on coverage over time and in smaller geographical areas. This will give you the information you need to better direct scarce resources toward low-performing areas.
  • When commissioning or conducting household surveys, make sure that samples permit coverage estimates by socioeconomic status and other units of interest. This will help in identifying special-needs populations, even in countries and regions with relatively high immunization coverage.
  • Support health systems initiatives to improve the quality of immunization data and teach health personnel to interpret and use their own data to improve program performance.
  • Visit the WHO Vaccine Preventable Disesases Monitoring System web page and become familiar with the information it contains. This site makes both official country estimates and the WHO/UNICEF estimates of immunization coverage since 1980 available in an interactive format.

References:

(1) Murray CJ, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet, 2003 Sep 27;362(9389):1022-7

(2) WHO/UNICEF Estimates of National Immunization Coverage, 1980-2004: Methods

(3) WHO Data, Statistics and Graphics and WHO Vaccine Preventable Diseases Monitoring System web pages

(4) Gwatkin DR."The need for equity-oriented health sector reforms." International Journal of Epidemiology 2001;30:720-723

(5) Immunization Essentials: A Practical Field Guide, USAID, 2003 Chapter 4, pp.80-81.

New Additions to the IMMUNIZATIONbasics Website

The IMMUNIZATIONbasics website contains information about the project and an electronic resource center on routine immunization and immunization financing. Visit www.immunizationbasics.jsi.com today to find the following resources:

In Our Next Issue...

Issue #1 of Snap Shots (Nov. 2005) focused on changes expected during the next phase of GAVI, which begins in 2006. At its biannual meeting in December 2005, the GAVI Executive Board confirmed many of the changes we told you about and added a few more. Our next issue of Snap Shots will focus on the Board’s decisions and how the requirement that all GAVI applicants complete a comprehensive Multi-Year Plan (cMYP) may affect the country or countries where you are working.


Please send any requests or questions regarding Snap Shots to ImmBasics_Info@jsi.com.

If you want to be removed from the Snap Shots mailing list, please send an e-mail to ImmBasics_Info@jsi.com and include "Leaving Snap Shots " in the subject line.

This publication was made possible through support provided by the Office of Health, Infectious Disease and Nutrition, Bureau for Global Health, U.S. Agency for International Development, under the terms of Award No. GHS-A-00-04-00004-00. The IMMUNIZATIONbasics project is managed by JSI Research & Training, Inc., and includes Abt Associates, Inc., the Academy for Educational Development, and The Manoff Group, Inc., as partners. The opinions expressed herein are those of the authors and do not necessarily reflect the views of the U.S. Agency for International Development.