DoD Publishes DevSecOps 2.0 Docs For Accelerating Apps

WASHINGTON: In just three years, the Department of Defense has made significant progress in creating a secure software development operations environment, or DevSecOps, to make better code faster. As part of these ongoing efforts, the Pentagon has released a batch of Enterprise DevSecOps v2.0 documents — and one of the leaders behind that initiative has just started working with the Joint Staff J6 on making DevSecOps resources available to Joint All Domain Command and Control.

JADC2, as it’s called, is the future interservice meta-network to link forces across land, sea, air, space, and cyberspace. It’s such a daunting technical challenge that traditional federal procurement processes can’t develop it fast enough — but DevSecOps could.

“DevSecOps is the foundation of the success of JADC2,” said Nicolas Chaillan, the Air Force’s chief software officer, in an interview with Breaking Defense. “Without DevSecOps, you won’t be able to move at the pace” needed, Chaillan told me.

What’s In a Name?

DevOps combines software development and IT operations for the rapid creation and use of apps, with developers and users working side by side to test new software, find improvements, and quickly push out upgrades. DevSecOps brings in cybersecurity experts and practices to DevOps. In DoD’s case, this entails implementing zero-trust security in the DevSecOps environment. Talking to Chaillan for just a few minutes reveals his focus on security, with zero trust frequently mentioned.

Also underlying DoD’s DevSecOps is a software development framework called Agile, which enables teams to continuously improve and rapidly update the underlying code for apps.

The new DevSecOps 2.0 docs include:

A fundamentals guide
A strategy guide
Tools and activities guide
A playbook
A reference guide for Cloud Native Computing Foundation Kubernetes, a cloud technology used for automating tasks, getting apps into the cloud (e.g., to servers), and underlying cloud operations.

From Early Skepticism at DoD to Building a Thriving Community

Chaillan joined DoD after years in the private sector, where he founded 12 companies and also created and sold 190 software products. Beginning at 14, Chaillan became a pioneer of and early contributor to PHP, which has become a widely used programming language for web servers. His background includes software and cybersecurity. “The two go together,” he says, but “at the same time, it’s about the right balance” between the speed of software development and security.

He was initially tapped in 2018 to co-lead, with the DoD CIO, the initiative to bring DevSecOps to the DoD enterprise. A great deal of his time has been spent “breaking silos” to enable collaboration, Chaillan says. He was surprised when some people told him early on he was “wasting his time” on these efforts — but he hasn’t quit.

The result? Today, there are more than 200 teams implementing DevSecOps across the Defense Department, Chaillan tells me, with 650 software containers.

What’s a container? In layperson’s terms, it’s a way to package software so it can be run in the cloud. A container can hold multiple apps, and containerized apps can be deployed on a variety of servers without time-consuming reconfiguration for each one. This simplifies and speeds up the process of getting software to users. It also helps avoid getting “locked in” with a single cloud provider or platform. Containers enable software developers to focus on building apps and the IT team to focus on infrastructure (e.g., servers that host apps), deployment (getting apps into the cloud so people can use them), scaling the needed resources to run apps, and other operations.

Containers are critical to Chaillan’s focus on avoiding so-called “lock in” to a single provider or platform. “This was important,” Chaillan says, “because we wanted to give teams options, while at the same time make sure we’re not building software in a vacuum and that we can reuse software across DoD programs.”

Chaillan describes the approach as the “Lego block concept,” in which teams can share containers among themselves and other teams. This more modular, adaptive, and flexible approach to software, in which a high percentage of code can be shared, contrasts with the software for the F-35, where Chaillan notes only five percent is shared between platforms today.

“It’s very important to us that we do better than that,” Chaillan says. “We don’t want to get locked in, whether a DevSecOps or cloud platform,” referring to Platform One and Cloud One, respectively.

Now, Chaillan says, “We have some of the largest DoD programs on [Platform One].” He says the apps currently entail “a little of everything.” However, he adds, “When I started, I wanted to focus on the war mission, well, because that’s why we’re here. I wanted to demonstrate it’s possible on weapons systems, because if you can do it on weapons systems, you can pretty much do it anywhere. So, we wanted to show the hardest use case first.”

That included F-16 software, which took just 45 days in 2019 to move into the DevSecOps environment. “That was a big win, just showing we can do that on the jet that is 40 years old, using legacy software.” It expanded from there to include the F-35 and B-21, among others, as well as code for the Navy’s AEGIS and code being developed by the Pentagon’s Joint Artificial Intelligence Center.

So far, Platform One has saved, on average, a year and $12.5 million per app it’s been used to launch, according to Chaillan. That adds up across the volume of apps DoD expects to develop going forward.

There are two options: The Party Bus, which is a multi-tenant cloud environment, and Big Bang, which enables people to take Platform One code to be deployed anywhere — on premise or in the cloud across classification levels, as well as at the edge on jets and bombers, for instance.

“It’s been a big enabler,” Chaillan observes. “It allows teams to move to DevSecOps on Day 1 instead of having to spend a year to build [all the prerequisites]. This is a living, breathing organism, if you will, and DevSecOps moves very fast, and that’s part of the challenge.”

Another challenge is training people in DevSecOps. So Chaillan and his team created a self-guided curriculum for would-be users. The goal this year, Chaillan says, is to train 100,000 people in DevSecOps.

Despite the initial success, Chaillan says, “I still struggle with silos, but once you get going, people tend to jump on the train.”

And the success started relatively early for Chaillan and his team. The 1.0 DevSecOps doc, which took about four months to develop and eight months to be approved, focused on building blocks like zero-trust security, behavior detection, continuous monitoring, and Kubernetes. Upon release of the 1.0 doc, Chaillan says it got 500,000 views on LinkedIn alone. “It was pretty amazing,” Chaillan reflects. “Clearly, a lot of people paid attention to it.”

Chaillan says the response was important because his goal has been to build a “very strong public community of practice around DevSecOps.” To do this, he’s created partnerships across the DoD services, federal agencies, the Intelligence Community, and private companies — including about 100 startups. That community of practice has grown to about 1,500 people sharing information and collaborating.

The Automation Effect

DoD’s DevSecOps entails a significant amount of automation, from security to testing. Chaillan says there are “so many pieces that will benefit” from automation. In the case of Platform One, it now releases 31 times a day, which “isn’t Facebook, but it’s pretty good,” Chaillan jokes.

With DevSecOps automation, DoD can “effectively save between 12 to 18 months per program for every five-year cycle of planned time, just by automating” a single process around continuous authority to operate. In addition, DevSecOps condenses the timeline between getting feedback from the warfighters using the software and the development team’s ability to push small, incremental improvements faster. Chaillan says this process can save, on average, six to eight months. So far, Chaillan says, DoD has saved 100 years of planned time by moving these apps into the DevSecOps environment.

“That’s 100 years of planned time that was going to be spent — well, I guess wasted — without that automation,” Chaillan observes. “The fact is, no one is waiting for us to figure this out. Other nations are also rapidly adopting DevOps, so it’s important for us to automate as much as we can.”

The goal is to get DoD to a place where someone can “push a button [and] deploy anywhere” — to the edge, in the cloud, on premise, Chaillan says.

Bringing DevSecOps to JADC2

Chaillan recently announced that, in addition to his current Air Force CSO position, he will be working part time on JADC2 with Lt. Gen. Dennis Crall, the director for Command, Control, Communications, and Computers / Cyber and chief information officer, Joint Staff J6. The J6 has been charged with leading the JADC2 effort.

“The goal is to bring all the great work we’ve done with all these programs and help with the adoption of Platform One in JADC2 as an enterprise service that will be available for teams.” This will include all the component pieces already in place, such as DevSecOps, identity management, and, of course, zero-trust security. “You can’t just connect things and hope for the best,” Chaillan observes. “You have to have that zero-trust enforcement.”

The first step will be to assess where the DoD is in regards to the J6 roadmap for introducing enterprise services into JADC2. “A lot of it is still being discussed,” Chaillan says. “I just started [working on JADC2] two weeks ago. But the vision is to sit down with all the services and collaborate and bring some of these centralized options to enable teams to move faster.”