A summary I received by email follows.
- - - - - - - - - - - - - - - - - - - - - - - - -
Warren Mitofsky <mitofsky@MINDSPRING.COM>
Sent by: Survey Research Methods Section of the ASA <SRMSNET@LISTSERV.UMD.EDU>
01/19/2005 10:20 AM
Please respond to Warren Mitofsky
To: SRMSNET@LISTSERV.UMD.EDU
cc:
Subject: 2004 Exit Poll Evaluation
This morning we made public our complete report evaluating the 2004 exit
polls. It is available on www.exit-poll.net Also available there are
methods statements, questionnaires and completion rates.
The executive summary from the report is posted below.
Thanks for your patience since the election while we put this together. It
took a number of us a fair bit of work to put this together.
Joe Lenski and Warren Mitofsky
Executive Summary
On November 2, 2004, the Election System created by Edison Media Research
and Mitofsky International for the National Election Pool (NEP) produced
election estimates and exit poll data for analysis in 120 races in all 50
states and the District of Columbia. In addition, between January and
March 2004, Edison and Mitofsky conducted exit polls for 23 Democratic
Primaries and Caucuses. For every election, the system delivered on its
main goals: there were no incorrect NEP winner projections, and the exit
poll data produced on election day were used on-air and in print by the six
members of the NEP (AP, ABC, CBS, CNN, FOX and NBC) as well as several
dozen media organizations who subscribed to that data. However, the
estimates produced by the exit poll data on November 2nd were not as
accurate as we have produced with previous exit polls.
Our investigation of the differences between the exit poll estimates and
the actual vote count point to one primary reason: in a number of precincts
a higher than average Within Precinct Error most likely due to Kerry voters
participating in the exit polls at a higher rate than Bush voters. There
have been partisan overstatements in previous elections, more often
overstating the Democrat, but occasionally overstating the
Republican. While the size of the average exit poll error has varied, it
was higher in 2004 than in previous years for which we have data. This
report measures the errors in the exit poll estimates and attempts to
identify the factors that contributed to these errors.
The body of this report contains the details of our analysis of the
performance of the exit polls and the election system. In addition to the
information included in this report, exit poll data from this election is
being archived at the Roper Center at the University of Connecticut and at
the Institute for Social Research at the University of Michigan and is
available there for review and further analysis. This is the procedure
that we have followed for all previous exit polls, which are also available
at the Roper Center and ISR. The description of the methodology of the
exit polls has already been posted on our Web site -
<http://www.exit-poll.net>www.exit-poll.net - along with all
questionnaires used on election day and the completion rates nationally and
by state.
Here is a brief summary of our findings:
1. Exit Poll Estimates
The exit poll estimates in the 2004 general election overstated John
Kerry's share of the vote nationally and in many states. There were 26
states in which the estimates produced by the exit poll data overstated the
vote for John Kerry by more than one standard error, and there were four
states in which the exit poll estimates overstated the vote for George W.
Bush by more than one standard error. The inaccuracies in the exit poll
estimates were not due to the sample selection of the polling locations at
which the exit polls were conducted. We have not discovered any systematic
problem in how the exit poll data were collected and processed. Exit polls
do not support the allegations of fraud due to rigging of voting equipment.
Our analysis of the difference between the vote count and the exit poll at
each polling location in our sample has found no systematic differences for
precincts using touch screen and optical scan voting equipment. We say this
because these differences are similar to the differences for punch card
voting equipment, and less than the difference for mechanical voting equipment.
Our detailed analysis by polling location and by interviewer has identified
several factors that may have contributed to the size of the Within
Precinct Error that led to the inaccuracies in the exit poll
estimates. Some of these factors are within our control while others are not.
It is difficult to pinpoint precisely the reasons that, in general, Kerry
voters were more likely to participate in the exit polls than Bush voters.
There were certainly motivational factors that are impossible to quantify,
but which led to Kerry voters being less likely than Bush voters to refuse
to take the survey. In addition there are interactions between respondents
and interviewers that can contribute to differential non-response
rates. We can identify some factors that appear to have contributed, even
in a small way, to the discrepancy. These include:
* Distance restrictions imposed upon our interviewers by election
officials at the state and local level
* Weather conditions which lowered completion rates at certain polling
locations
* Multiple precincts voting at the same location as the precinct in our
sample
* Polling locations with a large number of total voters where a smaller
portion of voters was selected to be asked to fill out questionnaires
* Interviewer characteristics such as age, which were more often
related to precinct error this year than in past elections
We plan further analysis on the following factors:
* Interviewer training and election day procedures
* Interviewing rate calculations
* Interviewer characteristics
* Precinct characteristics
* Questionnaire length and design
We also suggest the following changes for future exit polls:
* Working to improve cooperation with state and local election officials
* Improvements in interviewing training procedures
* Changes in our procedures for hiring, recruiting and monitoring
interviewers
Even with these improvements, differences in response rates between
Democratic and Republican voters may still occur in future
elections. However, we believe that these steps will help to minimize the
discrepancies.
It is also important to note that the exit poll estimates did not lead to a
single incorrect NEP winner projection on election night. The Election
Night System does not rely solely on exit polls in its computations and
estimates. After voting is completed, reported vote totals are entered
into the system. Edison/Mitofsky and the NEP members do not project the
outcome of close races until a significant number of actual votes are counted.
As in past elections, the final exit poll data used for analysis in 2004
was adjusted to match the actual vote returns by geographic region within
each state. Thus, the discrepancy due to differing response rates was
minimized and did not significantly affect the analysis of the vote. The
exit polls reliably describe the composition of the electorate and how
certain demographic subgroups voted.
2. Survey Weighting
Early in the afternoon on November 2nd, preliminary weightings for the
national exit poll overstated the proportion of women in the
electorate. This problem was caused by a programming error involving the
gender composition that was being used for the absentee/early voter portion
of the national exit poll. This error was discovered after the first two
sets of weighting; subsequent weightings were corrected. This adjustment
was made before NEP members and subscribers used exit poll results on-air
or in print.
After election day, we adjusted the exit poll analysis data in three states
(Tennessee, Texas, and Washington) to more accurately reflect the
proportion of absentee ballots that came from each geographic region in
those states. We have implemented a change to the survey weighting program
to take into account the geographic distribution of the absentee votes in
the future.
3. Technical Performance
While the computer system performed well for most of the night, a database
server problem led to NEP member and subscriber screens "freezing up"
shortly after 10:35 PM ET election night. This problem caused disruptions
in the system until shortly after midnight when we switched to a backup
server for the rest of the night. There was a second occurrence of this
problem at approximately 2:45 AM ET. Details of the data server problems
and other technical issues are outlined in the technical performance report
being distributed to the NEP Technical Committee. We have isolated the
reasons behind the database server problem and list several recommended
technical changes in this report to help avoid a repeat of this problem in
future elections.
MITOFSKY INTERNATIONAL
1776 Broadway, Suite 1708
New York, NY 10019
212 980-3031
212 980-3107 Fax
www.mitofskyinternational.com
mitofsky@mindspring.com
|
|
|
Permalink :: 0 Comments :: Post a Comment
|
In order to post a comment, you must be logged in. If you have a member account, please log in to comment.
If not, you can make an account right here. It's quick and free.