The first problem discovered was that Premier's vote counting software, called GEMS, had miscounted the total number of ballots cast in Humboldt County last November, omitting 197 ballots that had been previously counted by the system. The county and its election transparency volunteers discovered the problem when they conducted an additional post-election audit of all of the county's ballots.
At the hearing yesterday, Secretary of State staff Lowell Finley reported that this problem, referred to as the "deck zero" problem, was known by Diebold as far back as 2004. Finley stated that Premier had at no point since that time attempted to upgrade its formal documentation of the system. What the company did do was develop a "workaround" so that counties using this particular version of GEMS (1.18.19) could avoid having ballots inadvertantly zeroed out. However, due to personnel changes in Humboldt County and a lack of documentation of the problem by the vendor, the workaround was not known or used in that county in 2008, resulting in the ballot counting error which left out 197 ballots.
But that was really just the start. The Secretary of State's investigation into the "deck zero" problem led to the discovery of another security problem with GEMS 1.18.19: there is a "clear" button feature that allows an election official to clear out the audit logs stored in GEMS. As is noted in Secretary of State Debra Bowen's March 2, 2009 report to the federal Election Assistance Commission:
Excerpt:
"GEMS version 1.18.19 not only includes "Clear" buttons that permit deletion of these records, it provides no warning to the operator that exercising the "Clear" command will result in permanent deletion of the records in the log, nor does it require the operator to confirm the command before GEMS executes it."
Yet another audit log problem was discovered, this time by Kim Zetter at Wired News, who reported on her investigation in a January 13th story. After interviewing the Humboldt county registrar of voters and requesting copies of the county's audit logs, Zetter found that those logs failed to show instances where the registrar had intentionally deleted sets of ballots. Here's an excerpt from that story (which also provides screen shots of the audit logs), including comments from computer scientist Doug Jones at the University of Iowa:
The audit logs appear to record only limited types of events on the system and provide no comprehensive record that tracks every event performed by an election official.
Premier didn't respond to a query from Threat Level about the logs. But Jones said the Premier/Diebold system, as far as he knows, provides no single log file that chronologically lists all events in the life of an election.
Instead, he says, the system keeps "lots and lots of different logs" that appear to have been "independently designed by people who didn't talk to each other" and that are incomprehensible to anyone except the vendor.
The Secretary of State's EAC report highlighted this problem, but did not say whether it was limited to the 1.18.19 version, or if it was a problem throughout all versions of GEMS.
Yesterday's hearing provided an opportunity for the Secretary of State to get some further answers. Speakiing on behalf of Diebold/Premier was Justin Bales, the company's western regional manager. He read a prepared statement, saying the company supported withdrawal of certification for GEMS 1.18.19. He implied that the Secretary of State and the county of Humboldt were to blame for not keeping themselves informed. He stated that his company wanted the three counties using GEMS 1.18.19 to move to 1.18.24 like the sixteen other counties in California using it, and he assured the panel that this version mitigates the problems being discussed. He touted the familiar voting technology industry line (i.e. "elections are a matter of people, process and technology"). He acknowledged that the company could have been more aggressive in getting its customers to upgrade, but he objected to the characterization by the Secretary of State that the deck zero problem had been "hidden" by Diebold/Premier, and stated that his company had discussed the problem with its clients many times.
After the prepared statement was read, the Secretary of State's staff panelists had a chance to ask questions. Only one staffer, a veteran of the Secretary of State's office, Chris Reynolds, had a question. Reynolds noted that Bales hadn't commented on the audit logs raised in the staff report. Bales responded by addressing the "clear" button, which he assured Reynolds had been removed in the next version of GEMS that was released a few weeks later. He explained it was there because some of their client counties wanted to use old election templates to create new ones rather than rebuilding them entirely.
Reynolds then asked about the date and time stamp issue, and Bales assured him again that that problem had been addressed in later versions. Finally, Reynolds asked the question I had been waiting for: what about the failure to log certain system events? Was this problem addressed in subsequent versions? Bales answer was: not yet, they're working on it, and it's "high priority".
The implications of this revelation are enormous - if the audit log in all versions of GEMS in use in the United States is not a reliable record of all program activity, election officials in many states and counties across the country have lost a valuable election verification tool. In my testimony before the panel, I urged the Secretary of State to expand their investigation and highlighted the importance of the one percent manual tally and the new state regulation requiring an expanded, ten percent tally in close contests. Kim Zetter's Wired article provides more coverage of yesterday's events. Excerpts are below.
SACRAMENTO, California — Premier Election Solutions (formerly Diebold Election Systems) admitted in a state hearing Tuesday that the audit logs produced by its tabulation software miss significant events, including the act of someone deleting votes on election day.
The company acknowledged that the problem exists with every version of its tabulation software.
The revelation confirmed that a problem uncovered by Threat Level in January, and reiterated in a report released two weeks ago by the California secretary of state's office, has widespread implications for election jurisdictions around the country that use any version of the company's Global Election Management System (GEMS) software to tabulate votes. The GEMS software is used to tabulate votes cast on every Premier/Diebold touch-screen or optical-scan machine, and is used in more than 1,400 election districts in 31 states. Maryland and Georgia use Premier/Diebold systems exclusively, therefore the GEMS software counts every vote statewide.
"Today's hearing confirmed one of my worst fears," said Kim Alexander, founder and president of the non-profit California Voter Foundation. "The audit logs have been the top selling point for vendors hawking paperless voting systems. They and the jurisdictions that have used paperless voting machines have repeatedly pointed to the audit logs as the primary security mechanism and 'fail-safe' for any glitch that might occur on machines.
"To discover that the fail-safe itself is unreliable eliminates one of the key selling points for electronic voting security," Alexander said.
----
When asked by a member of the California secretary of state's staff if the company had done anything to address the problem, Justin Bales, general service manager for Premier/Diebold's western region said, "No, not yet."
Bales went on to say that the GEMS logs have been the same since the software was first created more than a decade ago.
"We never, again, intended for any malicious intent and not to log certain activities," Bales said. "It was just not in the initial program, but now we're taking a serious look at that."
California Secretary of State Debra Bowen called the audit logs "useless" and vowed to investigate the issue further. She told Threat Level after the hearing that an examination of audit logs in other voting systems was also merited in light of these revelations. "Clearly, we're going to have to look at this," Bowen said. "That's one of the obvious next steps."
