|
In December 2000, the Voter News Service (VNS) Board of Managers contracted with
Research Triangle Institute, a not-for-profit research organization in North Carolina, to review
VNS' data collection procedures, estimation methodology, and other operations. These VNS
operations provide data for the news media to project the outcome of U.S. elections. The purpose
of this review was to provide an independent, scientific assessment of the causes for the mistaken
calls in the Florida presidential vote and to make recommendations for improving the Election
Day forecasting methodology in order to avoid mistaken calls in future elections.
RTI assembled a team of six senior statisticians and survey methodologists to conduct this
review: Drs. Paul Biemer, Ralph Folsom, Richard Kulka, Judith Lessler, and Babu Shah, and Mr.
Mike Weeks. None of these six individuals had any significant prior knowledge of, or connection
with, VNS operations nor any experience with the Election Day voting process. This apparently
was a key requirement of the VNS Board for selecting RTI for the review and ensured an
objective and impartial assessment of VNS operations.
The scope of the review was confined to the following six areas: (1) evaluation of precinct
sampling methods; (2) evaluation of pre-election research methods, particularly with regard to any
evidence of political bias; (3) evaluation of projection models and calculations; (4) evaluation of
data collection procedures, particularly with regard to any evidence of political bias; (5)
evaluation of quality control procedures throughout the system; and (6) audit of information in the
report written by VNS for the VNS Members dated December 8, 2000. The Executive Summary(1)
highlights the major findings in each of these areas and summarizes RTI's recommendations
based upon the review. The details of our evaluation and recommendations are contained in the
full RTI report1.
Our review was based upon essentially three sources of information: (a) documentation
provided by VNS (the list of materials RTI received appears in Appendix A of the full report), (b)
a two-day meeting with VNS staff during which we discussed (1) through (6) above, and (c) ad
hoc interactions between RTI and VNS during the review for clarification purposes. A major
focus of the review was the Election Day system as it performed for Florida. Although the VNS
Board requested that we assess the system in at least five other states, data from state and local
officials for the selected states were not available in time for our review. Therefore, an analysis of
the data for states other than Florida was not possible.
Due to necessary constraints on resources, time, and the scope for this review, we did not
independently verify all of the information in the primary documents and data sources that were
available to us. However, there were several key areas where VNS information was
independently verified, and these are noted in our report.
Unless otherwise noted, the term "bias" used in our report refers to statistical bias, not
political, media or other bias. Statistical bias is the expected difference between the value of an
estimator and the population value it is intended to estimate when these differences are averaged
over many, hypothetical repetitions of the entire election forecasting process. It is caused by
random errors occurring in the data collection, data processing, or estimation processes that are
either uncontrolled by, or beyond the control of, the designer. Political bias refers to deliberate
errors in estimates, which force the election outcomes in the direction of one political party or
another in order to bring about a desired election result. In our review, we found no evidence of a
political bias; however, we did find evidence of statistical biases in the estimates, as we discuss
below.
1. Precinct Sampling Methods
Our investigation revealed that VNS' precinct sampling and associated estimation
methods were well designed for estimating the Election Day vote and generally follow standard
statistical practices. The oversampling of precincts that are designated as high-percent-black
units, which was done to improve the precision of the estimates, is a widely accepted technique
for sampling heterogeneous populations. To account for the higher percentage of blacks in the
sample using this method, the estimate for blacks is given a lower weight in the calculation of the
overall estimates, so that the resulting estimator of the total vote is technically unbiased. We
verified that this was done appropriately in the exit poll estimates.
While VNS' sampling methods for estimating the Election Day vote are well-conceived,
the methods for estimating the total vote including absentee and early voters are subject to
potentially serious statistical bias. We also found that the exit poll samples typified by the 45
precincts drawn for Florida are too small for producing accurate state-level results. A small
sample size not only reduces the precision of the exit poll results, it also increases the statistical
bias in the ratio estimates of the candidate vote shares.
In reviewing the VNS estimation formulas, we had questions about two approximations
VNS had employed in their standard error calculations. Therefore, we recalculated the standard
errors for the Gore percent advantage in Florida based on official vote totals with and without
these approximations and found no important differences in the standard errors of the candidate
vote differentials. Thus, we believe the standard errors for the exit poll and VPA estimates
appropriately account for sampling variation.
2. Pre-election Research Methods
Overall, VNS' pre-election methods for collecting data from sampled precincts are
generally sound, but the ways in which they are used both to perform edit and consistency checks
on the election night system and in formulating decisions could be substantially improved. The
VNS decision processes would benefit from both better information on the absentee/early vote in
each county/state and better use of the available data on past elections from the research. Further,
in spite of an excellent track record, the VNS Research Department could benefit from a general
review of new methods and technologies by which it could implement some of its protocols more
quickly and accurately.
3. Projection Models and Calculations
The exit poll and VPA (Voter Profile Analysis) precinct estimates are based on a well-designed sample survey that provides an excellent source of timely information on the voting
preferences of voters, their characteristics, and their opinions. The Core estimation process
provides timely information on race outcomes and also attempts to incorporate measures of
uncertainty in the estimates, despite the fact that the samples are not random.
However, the measures of uncertainty presented on the decision screens eliminate some
potentially important sources of error in the VNS system and, thus, the true uncertainty in the
estimates may be understated. In close elections, the risk that an election analyst will call an
election erroneously could be substantially higher than the information on the decision screens
indicates, even if the analyst correctly interprets the information. In addition, the information
provided on the decision screens is prone to misinterpretation, and the rules used for election
decision-making are inappropriate, given the continual flow of data into the process. A much-simplified screen that uses a sequential-sampling-type decision rule would better control the error
in forecasting a winner future elections.
4. Data Collection Procedures
VNS' field staffing operations worked as planned for the exit polls and the NETS county
reporting system. The problems were more frequent in the operations for the VPA sample and the
NETS precinct reporting system. Nationally, reports were received from 98% of the exit poll
precincts, 84% of the VPA precincts, 62% of the precincts in the NETS precinct reporting pool,
and 100% of the counties (or county equivalents) in the NETS county reporting system. However,
these rates are similar to those in past presidential elections.
We found training materials and the training process for the exit poll interviewers to be
thorough and appropriate. Based on a sample of interviewer debriefing questionnaires, it appears
that the exit poll data collection protocol was implemented reasonably well in non-problematic
voting places--e.g., those with only one exit, a good interviewer location close to the exit, and a
cooperative polling place official. The follow-up telephone conversations we had with 12 exit
poll interviewers reinforced the data in the questionnaires.
While many of VNS' data collection procedures worked as intended, we have some
concerns about their adequacy to produce data of sufficient precision to call very close elections
like the 2000 presidential election in Florida. We believe the most serious data collection
problem VNS had in the 2000 election was noncoverage of the absentee vote and early voters in
critical states such as Florida.
Another problem area is exit poll nonresponse (refusals and misses). According to VNS,
the average state-level response rate in 2000 was 51%; this compares with 55% in 1996 and 60%
in 1992. A nonresponse rate of this magnitude is a potential source of statistical bias in the model
projections if the voters who respond have voting characteristics that are significantly different
from those of nonrespondents. In our two-day meeting with VNS staff, we were told that the exit
polls more often overrepresented Democrats than they do Republicans. This effect could be the
result of a statistical bias due to nonresponse.
Finally, another area of concern is the absence of any direct quality control check on the
interviewers' data collection activities.
5. Quality Control Procedures
One of the strengths of the exit poll survey quality control system is the repetitive nature of
the reporting process. Another is the use of overlaid precinct estimates of exit poll survey bias.
This indicator, if used appropriately, can provide valuable information regarding the accuracy of
the exit poll results and can sound a warning if there are major problems with the exit poll data.
The NETS (New Election Tabulation System) makes good use of the data from similar past races
to check the counts provided by the county reporters. In addition, the multiple reporting
procedures of the NETS provide a self-correcting feature in the process that can be effective for
correcting previous erroneous reports.
Despite the numerous inspections, verifications, and edit checks that have become part of
the VNS Election Day system design, there are still opportunities for important errors to enter the
system and dramatically change the election results. For example, the number of overlaid
precincts in Florida was too small at the time of the Gore call to be a reliable indicator of the exit
poll bias. In addition, the quality control methods for ensuring that accurate data are received
from precinct and county reporters is inadequate, particularly for close races, and can be
improved.
6. Audit of Information in VNS Report
A strength of the VNS report (dated December 8, 2000) is its broad scope, which provides
a good indication of the difficulty in mounting the extremely complex process that culminates in
the collection, analysis, and reporting of data from diverse and dynamic sources in a single 24-hour period. The authors took a broad view of what could have gone wrong and attempted to
determine if there was an error in the procedures that contributed to the errors in the calls. This
report should be useful to VNS Members for gaining an understanding of why errors were made
in the Florida calls and to formulate ideas as to how the system needs to be changed.
However, the report needs more documentation. Although considerable work was done to
assemble information, there is frequently not a clear description of how this information was
compiled. The procedures and data that were used to prepare the individual reports included in
the December 8 report are largely not discussed. Many statements are made that reference results
that were assembled by VNS staff without showing the supporting data. The report also
inadequately describes the contribution that the modeling and the use of partial samples made to
the error in the early call for Gore. This lack of explanation of the modeling and lack of
documentation on information available for analysis also limit the reader's ability to formulate
ideas as to how the system needs to be changed.
7. Summary
In summary, we believe that the errors that led to the Gore call in Florida and then the late
night shift to Bush were the product of a number of system errors that tended to work in concert at
various points in the evening toward favoring one candidate and then the other. The major
sources of error were: (a) estimation of the early/absentee vote, (b) exit poll ratio estimator bias,
(c) end of night outstanding vote needed estimation, and (d) county-level reports. Stricter quality
controls and quality standards and improved estimation methodology could prevent these errors
from occurring in future elections.
In addition, we believe the measures of uncertainty provided on the decision screens
underestimate the true total error in the estimates. Thus, the risk that an election analyst will call
an election erroneously could be substantially higher than indicated by the information on the
decision screens. The complexity and amount of information provided on the decision screens
increase the risks of misinterpretation of the election results. In addition, we believe the rules
used for election decision-making are inappropriate, given the continual flow of data into the
process. A much simplified screen format that uses a sequential sampling type decision rule
would better control the error in predicting the outcomes of future elections.
Our key recommendations for VNS are as follows:
1. Improve the methodology for estimating the effect of absentee votes on estimates of the
candidate vote differential.
2. Improve the methodology for estimating the outstanding votes needed by candidates to
win an election.
3. Improve the measures of uncertainty for the key election estimators to more fully reflect
the total variation and statistical bias in the measurement process.
4. Improve the quality control systems to quickly and reliably signal the occurrence of error.
5. Integrate the ideas of sequential analysis in the election decision process and develop
better guidelines and decision rules for deciding to either make a call or wait for additional
data.
1. RTI's full report and the Executive Summary were provided to the Subcommittee. These documents are
included as part of this statement by reference.
|
