Wednesday, March 18, 2009

SoS hearing reveals audit log unreliable in all GEMS versions

Yesterday I attended a hearing at the California Secretary of State's office in Sacramento to examine the findings of a recent investigation by the agency's staff into voting software security problems discovered in Premier's (formerly Diebold) voting system.

The first problem discovered was that Premier's vote counting software, called GEMS, had miscounted the total number of ballots cast in Humboldt County last November, omitting 197 ballots that had been previously counted by the system. The county and its election transparency volunteers discovered the problem when they conducted an additional post-election audit of all of the county's ballots.

At the hearing yesterday, Secretary of State staff Lowell Finley reported that this problem, referred to as the "deck zero" problem, was known by Diebold as far back as 2004. Finley stated that Premier had at no point since that time attempted to upgrade its formal documentation of the system. What the company did do was develop a "workaround" so that counties using this particular version of GEMS (1.18.19) could avoid having ballots inadvertantly zeroed out. However, due to personnel changes in Humboldt County and a lack of documentation of the problem by the vendor, the workaround was not known or used in that county in 2008, resulting in the ballot counting error which left out 197 ballots.

But that was really just the start. The Secretary of State's investigation into the "deck zero" problem led to the discovery of another security problem with GEMS 1.18.19: there is a "clear" button feature that allows an election official to clear out the audit logs stored in GEMS. As is noted in Secretary of State Debra Bowen's March 2, 2009 report to the federal Election Assistance Commission:

"GEMS version 1.18.19 not only includes "Clear" buttons that permit deletion of these records, it provides no warning to the operator that exercising the "Clear" command will result in permanent deletion of the records in the log, nor does it require the operator to confirm the command before GEMS executes it."

Yet another audit log problem was discovered, this time by Kim Zetter at Wired News, who reported on her investigation in a January 13th story. After interviewing the Humboldt county registrar of voters and requesting copies of the county's audit logs, Zetter found that those logs failed to show instances where the registrar had intentionally deleted sets of ballots. Here's an excerpt from that story (which also provides screen shots of the audit logs), including comments from computer scientist Doug Jones at the University of Iowa:

The audit logs appear to record only limited types of events on the system and provide no comprehensive record that tracks every event performed by an election official.

Premier didn't respond to a query from Threat Level about the logs. But Jones said the Premier/Diebold system, as far as he knows, provides no single log file that chronologically lists all events in the life of an election.

Instead, he says, the system keeps "lots and lots of different logs" that appear to have been "independently designed by people who didn't talk to each other" and that are incomprehensible to anyone except the vendor.

The Secretary of State's EAC report highlighted this problem, but did not say whether it was limited to the 1.18.19 version, or if it was a problem throughout all versions of GEMS.

Yesterday's hearing provided an opportunity for the Secretary of State to get some further answers. Speakiing on behalf of Diebold/Premier was Justin Bales, the company's western regional manager. He read a prepared statement, saying the company supported withdrawal of certification for GEMS 1.18.19. He implied that the Secretary of State and the county of Humboldt were to blame for not keeping themselves informed. He stated that his company wanted the three counties using GEMS 1.18.19 to move to 1.18.24 like the sixteen other counties in California using it, and he assured the panel that this version mitigates the problems being discussed. He touted the familiar voting technology industry line (i.e. "elections are a matter of people, process and technology"). He acknowledged that the company could have been more aggressive in getting its customers to upgrade, but he objected to the characterization by the Secretary of State that the deck zero problem had been "hidden" by Diebold/Premier, and stated that his company had discussed the problem with its clients many times.

After the prepared statement was read, the Secretary of State's staff panelists had a chance to ask questions. Only one staffer, a veteran of the Secretary of State's office, Chris Reynolds, had a question. Reynolds noted that Bales hadn't commented on the audit logs raised in the staff report. Bales responded by addressing the "clear" button, which he assured Reynolds had been removed in the next version of GEMS that was released a few weeks later. He explained it was there because some of their client counties wanted to use old election templates to create new ones rather than rebuilding them entirely.

Reynolds then asked about the date and time stamp issue, and Bales assured him again that that problem had been addressed in later versions. Finally, Reynolds asked the question I had been waiting for: what about the failure to log certain system events? Was this problem addressed in subsequent versions? Bales answer was: not yet, they're working on it, and it's "high priority".

The implications of this revelation are enormous - if the audit log in all versions of GEMS in use in the United States is not a reliable record of all program activity, election officials in many states and counties across the country have lost a valuable election verification tool. In my testimony before the panel, I urged the Secretary of State to expand their investigation and highlighted the importance of the one percent manual tally and the new state regulation requiring an expanded, ten percent tally in close contests. Kim Zetter's Wired article provides more coverage of yesterday's events. Excerpts are below.

SACRAMENTO, California — Premier Election Solutions (formerly Diebold Election Systems) admitted in a state hearing Tuesday that the audit logs produced by its tabulation software miss significant events, including the act of someone deleting votes on election day.

The company acknowledged that the problem exists with every version of its tabulation software.

The revelation confirmed that a problem uncovered by Threat Level in January, and reiterated in a report released two weeks ago by the California secretary of state's office, has widespread implications for election jurisdictions around the country that use any version of the company's Global Election Management System (GEMS) software to tabulate votes. The GEMS software is used to tabulate votes cast on every Premier/Diebold touch-screen or optical-scan machine, and is used in more than 1,400 election districts in 31 states. Maryland and Georgia use Premier/Diebold systems exclusively, therefore the GEMS software counts every vote statewide.

"Today's hearing confirmed one of my worst fears," said Kim Alexander, founder and president of the non-profit California Voter Foundation. "The audit logs have been the top selling point for vendors hawking paperless voting systems. They and the jurisdictions that have used paperless voting machines have repeatedly pointed to the audit logs as the primary security mechanism and 'fail-safe' for any glitch that might occur on machines.

"To discover that the fail-safe itself is unreliable eliminates one of the key selling points for electronic voting security," Alexander said.


When asked by a member of the California secretary of state's staff if the company had done anything to address the problem, Justin Bales, general service manager for Premier/Diebold's western region said, "No, not yet."

Bales went on to say that the GEMS logs have been the same since the software was first created more than a decade ago.
"We never, again, intended for any malicious intent and not to log certain activities," Bales said. "It was just not in the initial program, but now we're taking a serious look at that."

California Secretary of State Debra Bowen called the audit logs "useless" and vowed to investigate the issue further. She told Threat Level after the hearing that an examination of audit logs in other voting systems was also merited in light of these revelations. "Clearly, we're going to have to look at this," Bowen said. "That's one of the obvious next steps."

Tuesday, March 17, 2009

Secretary of State Hearing today - media advisory

I'll be at the Secretary of State's today for a hearing to examine problems with Premier/Diebold voting equipment. Here are the contents to today's news advisory issued by the Secretary of State:

Secretary of State’s Office to Hold Hearing to
Examine Ballot-Count Errors Previously Unknown in California

WHAT: The Secretary of State’s Office will conduct a public hearing to receive reports and take testimony on the “Deck Zero” anomaly in Premier Election Solutions’ Global Election Management System (GEMS) version 1.18.19.

The Deck Zero software error, which can delete the first group of optically scanned ballots under certain circumstances, caused 197 ballots to be inadvertently deleted from Humboldt County’s initial results in the November 4, 2008, General Election. Upon discovery of the software error, Humboldt County subsequently corrected its election results. Two other California counties, San Luis Obispo and Santa Barbara, used the same software for the November 4 election but encountered no similar problems in counting ballots.

Secretary of State Debra Bowen’s office conducted an independent investigation into the Premier GEMS 1.18.19 software errors and uncovered even more information that was previously unknown to county and state elections officials. For more about the investigation and the public hearing, go to

In the days after the hearing, Secretary Bowen will consider what action – including possible withdrawal of state approval – to take on the Premier GEMS voting system.

WHEN: Tuesday, March 17, 2009, 10:00 a.m.
WHERE: Secretary of State’s Building Auditorium, 1500 11th Street, Sacramento

Friday, March 6, 2009

Secretary of State Hearing on March 17 to examine software vote counting flaw

The Humboldt Transparancy Project uncovered a serious flaw in the vote counting software produced by Premier (formerly Diebold). The group found that the software erased 197 votes. California Secretary of State Debra Bowen has sent this report to the U.S. Election Assistance Commission summarizing what happened and providing evidence that the vendor was aware of this flaw for years and did little to inform its customers, the counties of California using it. She is also convening a public hearing, to take place March 17 at the Secretary of State's auditorium in Sacramento.

To me, this all feels a lot like a deja vu. It was just about five years ago that another California Secretary of State, Kevin Shelley, held a series of public hearings to examine the same company's practice of distributing uncertified software to California counties, in violation of California statute. Hundreds of people showed up at the Secretary of State's office. In that case, Diebold was found guilty in court and was fined. Electronic voting machines were decertified.

For more on this episode, see Kim Zetter's Wired article, this synopsis by Mitch Trachtenberg (the Humboldt volunteer who created the software that detected the flaw), and today's Electionline story by Kat Zambon.

Redistricting Reform on Capital Public Radio today

Today on Capital Public Radio (90.9 FM in Sacramento) the program "Insight", hosted by Jeffrey Callison, will take on the topic of redistricting reform. I'll be appearing as a guest on the show, to talk about what's been happening in Sacramento since voters passed Proposition 11, a redistricting reform initiative that shifts the power to draw political district lines from the legislature to a citizens' redistricting commission. The show is on the radio from 2-3 p.m. today and a live webcast is available from the web site. Tim Hodson from the Center for California Studies and Kathay Feng with California Common Cause will also appear as guests.