Category: Security

Computer SecurityExpert WitnessInformation & Communication TechnologySecurity

Is New Hampshire the Next Iowa Voting Disaster? Information Technology Expert Analysis

A hastily-developed app and combined with a lack of user testing caused a ruckus in Iowa Caucus voting this week. What’s in store for New Hampshire, Super Tuesday, and beyond?

It has been an exciting week in US politics. We had a State of the Union address and an impeachment vote. A whirlwind week by any standard! Before we could even get to those two events, we started the week with an outrageous technology failure in the Iowa Democratic Caucus. For purposes of this blog post, I’m not going into the differences between a caucus and a primary. Let’s just assume they accomplish the same result: selecting a candidate for political office.

For the Iowa Democratic Party, Monday night was a disaster and then it continued into Tuesday, Wednesday… you get the idea. As I write this blog post on Friday morning, I’m not even sure if they have an official determination of who won. The news stories seem to be conflicting.

So here is what we know about the app (IowaRecorder) failure based on available reports. The Iowa Democratic Party hired a marketing technology company to build an app which would be used, statewide, to report results of local caucus votes (I’m simplifying for purposes of brevity). The app was going to be used to submit voting results. Nobody was actually voting through the app.

This first really good article I read that outlined the technology implementation failure, came from Slate. Here was a good summary from a couple of days ago:

“It’s still unclear what exactly went wrong with the app, but all of these issues appear to have something in common: The Iowa Democratic Party clearly wasn’t prepared for any possible issues with the app and a more involved method of vote reporting introduced this year—and sure enough, it reportedly turns out that the app was never tested on a statewide scale. Shadow, which is run by alumni of the Barack Obama and Hillary Clinton campaigns as well as Google, was paid $60,000 to develop the app, but it had just two months after party officials decided to abandon plans to report results over the phone.”

There’s some updated information on the failure from Motherboard, which was released yesterday (along with the app code). Below, you’ll see that they released an app that was still in beta format:

“And Instead of going through proper app store review processes conducted by Apple and Google, Shadow used beta testing platforms like Apple’s TestFlight to distribute the software so it could meet the Monday deadline. So when it came time for the app to do its most critical role — letting Democratic precinct leaders report results from Iowa on Monday — it failed in every way imaginable.”

Expert Analysis:

As I do when these major stories break, I turn to Experts.com members to get insights. You may recall Dr. Stephen Castell. Dr. Castell, Chartered Information Systems Practitioner and Member of the Expert Witness Institute, is Chairman of CASTELL Consulting. He is an internationally acknowledged Independent Computer Expert who has been involved in a wide range of computer litigation over many years.

Dr. Castell and I wrote a blog post back in 2018, regarding West Virginia’s Blockchain voting program. They are actually expanding this plan, which may necessitate a separate blog post.

Below, please find my questions and Dr. Castell’s answers (Disclaimer: these questions and answers provided on February 5th, 2020):

Nick: From available reporting, it appears the Iowa Democratic Party failed to do a statewide testing of this vote reporting application. What type of tests would have been necessary to identify errors in the system before statewide roll-out?

Dr. Castell: As other ICT professionals comment in the reports, there should be thorough systems testing and QA procedures, including User Acceptance Testing and Pilot Trials, plus scaled-up ‘soak testing’, before contemplating any real-world launch, such as this statewide roll-out. You expect to get errors in systems testing – its main purpose is to identify faults and fix them. Sadly, software systems and Apps these days do seem often to be launched publicly without adequate systems testing, let alone with adequate prior User Testing and Pilot Trials. If such standard professional QA processes were omitted, or truncated, for something as high-profile and important as an App to collect and relay voter data in the Iowa Caucus, that does appear rather astonishing.

Nick: It doesn’t appear that all of the fault lies with the app developer. It seems the Iowa Democratic Party only gave the app developer 2 months to develop and deploy this application. What sort of time-frame would you anticipate to develop, test, and implement a software of this scale?

Dr. Castell: That is difficult to estimate without more knowledge of the actual detailed Customer Requirements Specification that the developer’s App was contracted to meet. On the face of it, an App simply to in-gather voting data, aggregate and transfer it, sounds in principle like fairly straightforward functionality to code, test (at scale) and implement, and 2 months may not have been an unrealistic timescale for development, testing and deployment.

Nick: Would you expect there to be a certain level of user sophistication for those using the app on this scale? Should there have been company representatives available at caucus sites?

Dr. Castell: Reports suggest that there was little prior familiarity, let alone ‘training’, or ‘user sophistication’, with the App on the part of those expected to employ it for real, in the high-pressure, real-time Iowa Caucus conditions. Whatever the state of compliance of the App with its contractual specification – perhaps reasonably well delivered to time, budget, specification, and of suitable quality, ‘fit for purpose’ – if there was no program for adequate user familiarity and training, plus some sort of support and trouble-shooting team from the developer company at caucus sites, that alone could account for the problems encountered in statewide roll-out operation.

Nick: From what I’ve read, it looks like the company was paid $60,000 to build this application. Is there any way to gauge whether this is too little or too much for this type of application development?

Dr. Castell: Again, that is difficult to gauge without more knowledge of the actual detailed Customer Requirements Specification, and thus the likely complexity of the functionality needed, and its associated software design and coding; also, there may have been a tight budget to which the developer company was obliged to work. It is not unusual for software developers to invest in a ‘plum’ assignment such as this high-profile Iowa Caucus project, for the promotional and marketing impact that gives them in securing hopefully more lucrative and profitable development jobs later. In this case, the $60,000 could have been much less than the true cost to the developer company of the analyst, designer, coder, tester, deployer and trainer man-days expended in building and launching the App with a statewide roll-out, against a tough deadline.

Nick: What sort of testing, trials, and quality assurance requirements would you have employed prior to such an implementation?

Dr. Castell: There should ideally have been thorough systems testing and QA procedures, including User Acceptance Testing and Pilot Trials, plus scaled-up ‘soak testing’, well understood by ICT professionals, before the real-world launch of this statewide roll-out. Relevantly, I teach a Course Avoiding IT Disasters – the Expert Way, the principles of which are also covered in my seminal paper “Forensic Systems Analysis: A Methodology for Assessment and Avoidance of IT Disasters and Disputes”, issued as a Cutter Consortium Executive Report, Enterprise Risk Management & Governance Advisory Service series (Vol. 3, No. 2, March 8, 2006).


 

We cannot say that New Hampshire is next. All available information tells us that New Hampshire is not using the same company/app used in the Iowa Caucuses. Furthermore, there was talk of Nevada using the app, but they have claimed they will not move forward with the application.

That’s the end of this particular blog post. Though, we’re already in talks about another post related to voting systems.

Computer ForensicsComputer SecuritySecurity

Tesla Trade Secrets Lawsuit: Investigators & Expert Witnesses

Did you hear about Tesla suing a former employer for stealing trade secrets?

Early last week, Tesla CEO Elon Musk emailed Tesla employees reporting another employee had done some pretty significant sabotage to the company’s manufacturing operations. According to one Ars Technica article, “In the all-hands email to Tesla staff, Musk wrote that the employee had made ‘direct code changes’ to the company’s production systems, as well as exporting ‘large amounts’ of Tesla’s data to unknown third parties.”

In the same article, Ars Technica quotes Musk’s email further, “the alleged saboteur could have been working with short sellers, oil and gas companies—whom he described as ‘sometimes not super nice’—or ‘the multitude of big gas/diesel car company competitors.’ Of this last group, Musk reminded his employees that, since the traditional OEMs have been known to cheat emissions tests, ‘maybe they’re willing to cheat in other ways.'”

Mr. Musk is not subtle in his indication that he believes the saboteur may have been working with others in a coordinated effort of corporate espionage and theft of trade secrets.

Later in the week, Tesla filed suit against a now-former-employee, Martin Tripp. We can only assume this is the employee to whom Mr. Musk referred in the earlier email, given the civil complaint allegations against Mr. Tripp. The civil complaint link is courtesy of Cyrus Farivar of Ars Technica.

The civil complaint alleges Mr. Tripp violated the Defend Trade Secrets Act and the Nevada Uniform Trade Secrets Act. Further, Tesla alleges of breach of contract, breach of fiduciary duty of loyalty, and violating the Nevada Computer Crimes Law.

To me, the interesting part was the “prayer for relief” (an absurdly arcane way of saying “this is what we want!”).  Here’s the summary. Take note of the first item:

tesla-prayer-for-relief

Section A goes hand in hand with Elon Musk’s comments indicating the saboteur may have been acting with unknown third parties. The legal action seems intended to prohibit the use of any stolen trade secrets and preventing any potential financial or competitive damage resulting from corporate espionage.

What experts played a role or may play a role?

The case being brand new (complaint filed last week), I don’t expect we’ll hear about expert witness involvement for some time. However, I do imagine some experts (working for Tesla) were involved in uncovering the alleged sabotage.

Digital Forensics, Computer Security, Corporate Security, Software, Human Resources…

We know from the complaint there are allegations that Mr. Tripp stole trade secrets from Tesla. According to the complaint, Mr. Tripp “has thus far admitted to writing software that hacked Tesla’s manufacturing operating system (MOS) and to transferring several gigabytes of Tesla data to outside entities.” To someone like me, with fairly basic coding experience, it appears Mr. Tripp was quite advanced. He was able to bypass Tesla’s internal security to install hacking software.

Mr. Tripp has not admitted, but Tesla further alleges, “he also wrote computer code to periodically export Tesla’s data off it’s network and into the hands of third parties.” This sentence alone makes me wonder why Tesla did not add potential JOHN DOES to the complaint. Nevertheless, the complaint continues, “his hacking software was operating on three separate computer systems of other individuals at Tesla so that the data would be exported even after he left the company and so that those individuals would be falsely implicated as guilty parties.” Again, I mention this is a pretty advanced thought process because Mr. Tripp had plans to cover his tracks.

Given this information, I presume Tesla’s corporate security in conjunction with their legal department had to investigate Mr. Tripp’s actions for some time before terminating his employment and filing suit.

This investigation was likely to include those with knowledge and experience in digital forensics, computer security, and hacking software. Since the investigation involved an employee, I suppose there’s a chance human resources was included in the investigation as well.

Why was the investigation likely to include this variety of individuals? Tesla had to identify the breach using digital forensics and computer security experts. After recognizing the hacking software in their system, it’s possible they would have reviewed the code to see how it breached their computer security and I assume they would be able to identify the terminals on which the code resided. Since Mr. Tripp had taken precautions to misdirect Tesla, they may have identified the three other employees as responsible parties early in the investigation, causing them to monitor those employees.

Having watched enough spy movies to pretend I know what I’m talking about, I have to imagine Tesla would want to identify the third party entities mentioned in the complaint. So, they probably allowed some data to be exported while they were monitoring the situation in an effort to identify those who may have conspired with Mr. Tripp. With my spy movie knowledge, prognostications, and five bucks, you can get a cup of coffee.

Of this, I am certain. Tesla had to use investigators familiar with protection of intellectual property and digital evidence collection. Experts listed above would have the appropriate specialization to conduct this investigation in preparation for the recent litigation.

So as the litigation develops and if it goes to trial, I will expect to see software, digital forensics, computer security, corporate security, and human resources experts and consultants assisting in discovery and preparing for trial.