Risk-limiting audit

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

A risk limiting post-election audit is one of several types of election audits. It requires manually checking statistical samples of paper ballots to see if election computers interpreted them correctly.[1] The most common type of risk-limiting audit, ballot comparison, also requires independently counting all computer ballots, not just the sample, to check whether election computers added up totals correctly.[2] If samples and new totals agree with earlier results, there is still a limited risk those results are wrong, depending on the sample size used. If there are substantial discrepancies, the risk-limiting audit, like most but not all other election audits, requires a 100% recount by hand, to identify the correct winner(s) of the audited contest(s). The audit usually only checks one or a few contests from the ballot, and it depends on paper ballots, which therefore need to be stored securely from election day to the audit.

Sample sizes depend on: winning margin, number of ballots voted, confidence level, and type of audit.

Categories of audits[edit]

There are two types of election audits: process audits, which determine whether appropriate procedures were followed, and results audits, which determine whether votes were counted accurately.[3] Risk-limiting audits are one form of a results audit.

There are three types of risk-limiting audits:[4]

  1. Ballot comparison. Know how computers interpreted each ballot ("cast vote record"), compare computer interpretation to manual interpretation in a random sample of ballots; count all "cast vote records" independently of earlier counts;[2][5] report differences in interpretations and total counts
  2. Ballot polling. Know computer total for the election; count a random sample of ballots; report differences between computer and manual percentages for the total election
  3. Batch comparison. Know computer total for each batch of ballots (e.g. precinct); hand-count all ballots in a random sample of batches; sum all batch totals independently of earlier sums; report differences in batch counts and overall sums

Colorado uses method 1 in most counties, and method 2 in a few counties which lack "cast vote records."[6]

Hand-counting ballots identifies bugs and hacks in how election computers interpret each ballot. Independently checking counts and sums identifies bugs and hacks in how election computers calculate totals. Method 2 does not need this independent counting step, since it has a large enough sample to check election totals directly.



The process starts with election officials selecting a ‘risk limit,’ such as 9% in Colorado,[7] meaning that if there are any erroneous winners in the initial results, the audit will catch at least 91% of them and let up to 9% stay undetected and take office.[4][2] Using formulae,[4] [8] [9] [10] officials identify a sample size for each contest they intend to verify. The size of the sample depends primarily on the margin of victory in the targeted contest. They then generate a series of random numbers identifying specific ballots to pull from storage, such as the 23rd, 189th, 338th, 480th ballots in precinct 1, and other random numbers in other precincts.

In method 1, paper ballots in the sample are then manually compared to the "cast vote records" from the election computers for these same ballots. In method 2, the contest is simply counted, not compared. In method 3, a sample of batches is chosen, and all ballots in those batches are counted. In method 1 or 3, auditors independently sum all "cast vote records" or batch totals, respectively, to check earlier totals.[2][5]

When the audit produces the same result as the earlier results, the outcome is confirmed, subject to the risk limit, and the audit is complete. If the audit sample shows enough discrepancies to call the outcome into question, a larger sample is selected and counted. This process can continue until the sample confirms the original winner, or a different winner is determined by hand-counting all ballots.[11][4]

The three approaches are listed in order of increasing numbers of ballots which need to be hand-counted. For example in a jurisdiction with 64,000 ballots, an 8% margin of victory, and allowing 10% of any mistakes to go undetected, method 1, ballot comparison, on average, needs 80 ballots, method 2, ballot polling, needs 700 ballots, and method 3, batch comparison, needs 13,000 ballots (in 26 batches).[4]


Sample sizes rise rapidly for narrower margins, with all methods. Even in a smaller jurisdiction, with 4,000 ballots, method 1, ballot comparison, would need 300 ballots for a contest with a 2% margin of victory and 3,000 ballots for a 0.1% margin of victory. Method 2 or 3, ballot polling or batch comparison, would need a full hand count. Margins under 0.1% occur in one in sixty to one in 460 contests. Large numbers of contests on a ballot raise the chances that these small margins and large samples will occur in a jurisdiction, which is why no place does risk-limiting audits on all contests, and Colorado picks contests with wider margins.[12]

When Maryland evaluated audit methods, it noted that local boards of elections could not budget, or plan staffing, for risk-limiting audits, since the sample "is highly dependent on the margin of victory in any given audited contest... A very close margin of victory could... require days of staff work, possibly compromising the local certification deadline."[13]

Method 1 also requires the ballots to be kept in strict order so one can compare the computer interpretations of sampled ballots with those exact physical ballots.[14] Maryland, like other states, randomizes the order of ballots and cast vote records to protect ballot secrecy, so a risk-limiting audit there would only be able to compare computer interpretations to accompanying computer images of the ballots, not the original ballots.[13]

Colorado says it has a system to count cast vote records independently, but it is not yet publicly documented,[15] so the chance of bugs or hacks affecting it along with the initial counts is unknown.

All the methods, when done for a state-wide election, involve manual work throughout the state, wherever ballots are stored, so the public and candidates need observers at every location to be sure procedures are followed. However in Colorado and most states the law does not require any of the audit work to be done in public.[16]

Costs of actual hand counts in 2004 ranged from $0.36 per ballot to count one race in Washington State, including overhead (training, supervision, space rental), to $3.39 per ballot to count 21 contests on each ballot in Nevada, excluding overhead, or $0.18 per contest per ballot.[17] These involved hand-counting all ballots in a precinct, so they are relevant to 100% hand-counts, when needed. Sample counts have fewer ballots, but extra costs for pulling out the sampled ballots and putting them back, and similar overheads for training and supervision.

Keeping paper ballots secure[edit]

Auditing is done several days after the election, so paper ballots and computer files need to be kept securely. Physical security has its own large challenges. No US state has adequate laws on physical security of the ballots.[18] Security recommendations for elections include: starting audits as soon as possible after the election, preventing access by anyone alone,[19] which would typically require two locks, and no one having keys to both locks, having risks identified by people other than those who design or manage the system, using background checks and tamper-evident seals,[20] [4] although seals can typically be removed and reapplied without damage, especially in the first 48 hours, and detecting subtle tampering requires substantial training.[21] [22] [23]

Experienced testers can usually bypass all physical security systems. Security equipment is vulnerable before and after delivery. Insider threats and the difficulty of following all security procedures are usually under-appreciated, and most organizations do not want to learn their vulnerabilities.[24]

Colorado elections in 2017, sample sizes needed for risk-limiting audits

State variations[edit]

As of early 2017, about half the states require some form of results audit.[16] Typically, these states prescribe audits that check only a small flat percentage, such as 1%, of voting machines. As a result, few jurisdictions conduct audits that are rigorous or timely enough to detect and correct counting errors before election results are declared final.[25][26]

In 2017, Colorado[27] became the first state to implement rigorous risk-limiting audits, auditing one race in each of 50 of its 64 counties.[28] Following the 2018 General Election, Colorado will conduct audits in the 62 of its 64 counties that use automated vote counting equipment (the two remaining counties hand count the ballots).

Rhode Island passed legislation[29] requiring that state's Board of Elections to implement risk-limiting audits beginning in 2018. Individual jurisdictions elsewhere may be using the method on the local election clerks' initiative.


In 2010, the American Statistical Association endorsed methods 1 and 3, ballot comparisons and batch comparisons, as long as there is public verifiability:

  • "A batch is a group of ballots (or even one ballot) for which machine and hand counts can be compared."
  • "Ensuring verifiability... publishing machine counts for all batches prior to their being sampled demonstrates that the numbers that the audit checks are the same as the subtotals included in the machine-count outcome."[5]

In 2008 the Brennan Center for Justice and several election integrity groups endorsed methods 2 and 3, ballot polling and batch comparisons, similarly insisting on public verifiability:

  • "For each selected audit unit, the audit must compare vote count subtotals from the preliminary reported election results for the contest, with hand-to-eye counts of the corresponding paper records... subtotals for each batch must be reported prior to the audit as part of the election results."
  • "The public is given sufficient access to witness and verify the random selection of the audit as well as the manual count."[19]

In 2014, the Presidential Commission on Election Administration recommended the methods in broad terms:

  • "Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected."[30]

By selecting samples of varying sizes dictated by statistical risk, risk-limiting audits eliminate the need to count all the ballots to obtain a rapid test of the outcome (that, is, who won?), while providing some level of statistical confidence.

In 2011, the federal Election Assistance Commission initiated grants for pilot projects to test and demonstrate the method in actual elections.[31]

Professor Phillip Stark of the University of California at Berkeley has posted tools for the conduct of risk-limiting audits on the university's website.[32]

See also[edit]


  1. ^ Cyrus Farivar (2012-07-25). "Saving throw: securing democracy with stats, spreadsheets, and 10-sided dice". Ars Technica. Retrieved 2012-07-27.
  2. ^ a b c d Stark, Philip (2012-03-16). "Gentle Introduction to Risk-limiting Audits" (PDF). IEEE Security and Privacy – via University of California at Berkeley, pages 1, 3.
  3. ^ "Report on Election Auditing by the Election Audits Task Force of the League of Women Voters of the United States" (PDF). League of Women Voters. 2009. Retrieved 2017-04-07.
  4. ^ a b c d e f Lindeman, Mark (executive editor), Jennie Bretschneider, Sean Flaherty, Susannah Goodman, Mark Halvorson, Roger Johnston, Ronald L. Rivest, Pam Smith, Philip B. Stark (2012-10-01). "Risk-Limiting Post-Election Audits: Why and How" (PDF). University of California at Berkeley. pp. 3, 16. Retrieved 2018-04-09.
  5. ^ a b c Statement on Risk-limiting Auditing, American Statistical Association, April 2010. Accessed May 12, 2017
  6. ^ Volkosh, Dan (2018-01-04). "Voting Systems Public". Colorado Secretary of State. Retrieved 2018-04-09.
  7. ^ "Round #1 State Report". www.sos.state.co.us/pubs/elections/auditCenter.html. 2017-11-20. Retrieved 2018-04-09.
  8. ^ Stark, Philip (2010-08-01). "Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits" (PDF). Proceedings of the USENIX Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE), Aug. 2010.
  9. ^ Lindeman, Mark, Philip B. Stark, Vincent S. Yates (2012-08-06). "BRAVO: Ballot-polling Risk-limiting Audits to Verify Outcomes" (PDF). EVT/WOTE.
  10. ^ Wald, A. (1945-06-01). "Sequential Tests of Statistical Hypotheses". The Annals of Mathematical Statistics. 16 (2): 117–186. doi:10.1214/aoms/1177731118. ISSN 0003-4851.
  11. ^ http://electionaudits.org/bp-risklimiting
  12. ^ Colorado Secretary of State (2018-07-05). "Target contests for risk‐limiting audits per Election Rule 25.2.2(i)" (PDF). sos.state.co.us. Retrieved 2019-06-29.
  13. ^ a b Maryland State Board of Elections (2016-10-21). "Post-Election Tabulation Audit Pilot Program Report" (PDF). elections.maryland.gov. Retrieved 2019-06-29.
  14. ^ Lovato, Jerome, Danny Casias, and Jessi Romero. "Colorado Risk-Limiting Audit:Conception to Application" (PDF). Bowen center for Public Affairs and Colorado Secretary of State. Retrieved 2018-04-05.
  15. ^ Lindeman, Mark, Ronald L. Rivest, Philip B. Stark and Neal McBurnett (2018-01-03). "Comments re statistics of auditing the 2018 Colorado elections" (PDF). Colorado Secretary of State. Retrieved 2018-04-09.
  16. ^ a b "State Audit Laws". Verified Voting. 2017-02-10. Retrieved 2018-04-02.
  17. ^ Theisen, Ellen (2004). "Cost Estimate for Hand Counting 2% of the Precincts in the U.S." (PDF). votersunite.org. Retrieved 2018-05-04.
  18. ^ Benaloh, Public Evidence from Secret Ballots; et al. (2017). Electronic voting : second International Joint Conference, E-Vote-ID 2017, Bregenz, Austria, October 24-27, 2017, proceedings (PDF). Cham, Switzerland. p. 122. ISBN 9783319686875. OCLC 1006721597.
  19. ^ a b Lindeman, Mark, Mark Halvorson, Pamela Smith, Lynn Garland, Vittorio Addona, Dan McCrea (2008-09-01). "Best Practices: Chain of Custody and Ballot Accounting, ElectionAudits.org" (PDF). Brennan Center for Justice et al. Retrieved 2018-04-09.
  20. ^ "Chapter 3. PHYSICAL SECURITY" (PDF). US Election Assistance Commission. Retrieved 2018-04-24.
  21. ^ Johnston, Roger G., and Jon S. Warner (2012-07-31). "How to Choose and Use Seals". Army Sustainment. Retrieved 2018-05-04.
  22. ^ Coherent Cyber, Freeman, Craft McGregor Group (2017-08-28). "Security Test Report ES&S Electionware" (PDF): 9 – via California Secretary of State.CS1 maint: Multiple names: authors list (link)
  23. ^ Stauffer, Jacob (2016-11-04). "Vulnerability & Security Assessment Report Election Systems &Software's Unity" (PDF) – via Freeman, Craft, MacGregor Group for California Secretary of State.
  24. ^ Seivold, Garett (2018-04-02). "Physical Security Threats and Vulnerabilities - LPM". losspreventionmedia.com. Retrieved 2018-04-24.
  25. ^ Counting Votes 2012: A State by State Look at Voting Technology Preparedness, Pamela Smith, Verified Voting Foundation; Michelle Mulder, Rutgers School of Law-Newark; Susannah Goodman, Common Cause Education Fund, 2012. Accessed May 12, 2017
  26. ^ Confidence in the Electoral System: Why We Do Auditing, Michael W. Trautgott and Frederick G. Conrad, in Confirming Elections: Creating Confidence and Integrity Through Election Auditing, R. Michael Alvarex, Lonna Rae Atkeson, and Thad E. Hall, eds., Palgrave MacMillan, 2012.
  27. ^ Paul, Jesse (2017-11-22). "Colorado's first-of-its-kind election audit is complete, with all participating counties passing". The Denver Post. Retrieved 2018-04-10.
  28. ^ Casias, Danny (2017-11-08). "Audited contests in each county for 2017 RLA (XLSX)". www.sos.state.co.us/pubs/elections/auditCenter.html. Retrieved 2018-04-15.
  29. ^ "Passing Your Vote Through Security: The Rise of Risk Limiting Audits in Rhode Island". State of Elections. 2018-01-29. Retrieved 2019-06-30.
  30. ^ The American Voting Experience: Report and Recommendations of the Presidential Commission on Election Administration, January 2014
  31. ^ Post-Election Risk-Limiting Audit Pilot Program 2011-2013, Final Report to the United States Election Assistance Commission, California Secretary of State 2013. Accessed May 12, 2017
  32. ^ "Tools for Comparison Risk-Limiting Election Audits". www.stat.berkeley.edu. Retrieved 2017-05-17.