Any federal IT practitioner who can fog a mirror is thinking about cybersecurity these days, or should be. In reviewing 15 large software development programs underway in the Defense Department, congressional auditors found that cyber concerns have the potential to stretch out projects or boost costs. The director of the information technology and cybersecurity team at the Government Accountability Office, Kevin Walsh, joined the Federal Drive with Tom Temin with more details.
Tom Temin: Any federal IT practitioner who can fog a mirror is thinking about cybersecurity issues these days, or should be. In reviewing 15 large software development programs underway in the Defense Department, congressional auditors found that cyber concerns do have the potential to stretch out projects or boost costs. Here with highlights, a director of the information technology and cybersecurity team at the Government Accountability Office, Kevin Walsh. Mr. Walsh, good to have you on.
Kevin Walsh: Good to be here Tom, thank you.
Tom Temin: These 15 programs that you reviewed, give us the overview of them. Were they battle related? Were they logistics related or across the board?
Kevin Walsh: Sure. So we looked at the 15 major IT programs that had a formal cost estimate, which is known as an acquisition program baseline. These included things like HR systems, financial management, healthcare and communications. And these are not small dollars. These are big, big programs, they are spending in the order of hundreds of millions, or in some cases, billions of dollars over their lifecycle. So for example, we’re looking at DoD’s modernization of its health care management system, as well as the Navy’s replacement of their networks on their bases, and ships, submarines and the like. So these are very big, very important, but they do not include weapon systems. So to manage expectations, these are enterprise systems and the like.
Tom Temin: Got it. So the things that take them 10, 15 years or so usually, if they get them done at all, I suppose. What did you find generally, in terms of their ability to stay on program schedule goals?
Kevin Walsh: Sure. So this is kind of a good news, bad news story. We had 11 that decreased their costs. So fantastic news for the taxpayer. The other four increased their costs. However, on the bad news side of things, those four massively increased their costs to the tune of two increased more than a billion dollars total. And if you looked at the total change, if you added all the changes up for the 15, they increased more than a billion dollars in total. So that means that the 11 that did good work were outweighed by the four that went over cost.
Tom Temin: Which were those bad boys on cost?
Kevin Walsh: Sure. So the two biggest and most guilty are the Army’s Integrated Personnel and Pay System Increment II as well as the modernization of the Healthcare Management System I mentioned earlier. Those both exceeded their costs by north of a billion dollars.
Tom Temin: Wow. All right, then we’re going to talk about schedule.
Kevin Walsh: Sure. So we looked at cost scheduling, we even looked at some of the performance aspects of these 15. So for schedule, we saw five that had increased or been delayed in their schedule by more than a year. So in one case, five years, two years, two years, three years and one year. The other 10 either had no scheduled delay, or delays on the order of months. So again, good news, bad news. Good news, 10 had no or minimal delays, but the remaining five had more than a year delay.
Tom Temin: And the report does mention the fact, specifically, that cybersecurity concerns can cause delays or cost overruns, or both. And I guess, given the fact that every agency is thinking about cyber these days, what were your findings with respect to how they were able to inculcate cyber without getting out of control on the whole project?
Kevin Walsh: You’re absolutely right Tom, we want to see agencies looking at cybersecurity early, often, and we want it really to be something that is built in from the get go. So to DoD’s credit, their guidance requires that and it requires things like having a cybersecurity strategy, and conducting vulnerability assessments. So each of the 15 that we looked at had a cybersecurity strategy and eight of the 15 had conducted some form of vulnerability assessment. The remaining seven, keep in mind, these are programs that are being developed so they may have been a little bit too early to actually conduct those vulnerability assessments. And one program actually cited the heavy involvement of DoD’s red team, which is the guys pretending to be bad actors, as one of the reasons for their success. So DoD is doing a good job on cyber — for DoD, at least — thinking about it as they as they go. But as seen from the recent SolarWinds breaches, just because you’re thinking about it doesn’t mean you’re bulletproof.
Tom Temin: We’re speaking with Kevin Walsh, a director in the information technology and cybersecurity team at the Government Accountability Office. And what else would cause them to go over schedule and over costs? Were there any common factors that they need to address?
Kevin Walsh: Sure. So as is typical in government programs, we saw that changing requirements resulted in delays both in terms of the schedule, as well as cost overruns. If you’re changing the goalposts as you go, that makes it harder to actually stay on target. But agencies also did a bunch of good things. We found that the 10 who had decreased their costs, basically maintained tight control and monitored their costs, they also cited the government acting as a system integrator as a reason for decreased costs. On the mixed side of things, some cut scope to reduce costs, which can lead to maybe not delivering the product that you initially anticipated. But I think the changing requirements and difficulty integrating software is probably the more common ones that we saw.
Tom Temin: And you also report that most of the projects, I think 11 or 12 out of 15, are using what you might roughly call the agile development methodology, as opposed to the waterfall, may not meet the strict definitions of the agile trade groups. But it was not the old fashioned way.
Kevin Walsh: That’s correct. And that is seen as a good thing moving forward. Because agile and iterative and DevOps and DevSecOps allow you if you’re going to fail, we’d rather these programs fail early so that we can figure it out and cut it off before they spend billions of dollars as many of these are going to do. So incremental allows you to identify problems earlier and even allows you to deliver a working viable product at an earlier point in the lifecycle of software development.
Tom Temin: And I would think if you do things incrementally, then that would avoid some of the change orders and rework that gets so expensive and takes so long. The vendors love it. But it does take things up to the stratosphere.
Kevin Walsh: That is entirely spot on, Tom.
Tom Temin: All right. And this was a report without recommendations. So this was just a look, see, we want you to know we’re watching,
Kevin Walsh: You’re correct. And this is actually going to be in an annual assessment. So you can expect to see one of these roughly every year, the Congress put in an annual mandate for us to do this kind of look. But while it doesn’t have any recommendations, the report did highlight several opportunities for DoD to keep on improving its IT capabilities. And we also highlighted problems with DoD finding the right people. So we found that it was difficult for many of these programs to find the right staff with the right expertise. And this is something we’ve heard over and over again in federal IT. Finding the right people, putting them in place can be difficult for the government.
Tom Temin: So if they read between the lines, they’ll get the idea of what they ought to be doing.
Kevin Walsh: Yes, that is that is absolutely correct.
Tom Temin: All right. Kevin Walsh is a director of the information technology and cybersecurity team at the Government Accountability Office. Thanks so much for joining me.
Kevin Walsh: Thanks, Tom. Great to be on.