Supercomputers on demand: Rescale, Parallel Works ramp up for DIU cloud contract

WASHINGTON — High-powered computing (HPC) is in increasingly high demand, for everything from training artificial intelligence to running simulations. But while computers were invented for complex military calculations, today Defense Department agencies struggle to access the latest tech being developed in the private sector.

A recently announced initiative from the Defense Innovation Unit aims to change that, radically, over the next few months. Based on a two-year pilot project with the Defense Department’s High Powered Computing Modernization Program, the DIU — the Pentagon’s official outreach arm to unorthodox innovators outside the traditional defense world — last month picked two companies, Rescale and Parallel Works, to deliver, in essence, virtual supercomputers on demand.

“We’re probably a few months away from being open for business for DoD users,” said Matthew Shaxted, co-founder and CEO of Parallel Works, which just today said it rolled out a top-to-bottom overhaul of its software, driven, in large part, by DoD’s AI needs. The next big step, he told Breaking Defense, is final completion of DoD’s complex cybersecurity approval process and getting official Authority To Operate for “high sensitivity controlled unclassified information,” aka Impact Level IL-5.

That can be a huge hurdle. The firm has done commercial high-powered computing for almost a decade and was already overhauling its original software when DIU first launched the pilot in 2022, but “the difficult part was [Defense Department] security compliance,” he told Breaking Defense: “That took us about two years.”

But once Parallel Works and Rescale complete that laborious process, DoD customers should be able to access their services in minutes. “The DoD project teams don’t need to do the two years,” Shaxted said. “You click a big button and in two minutes you get one of these HPC systems, ready to go.”

“We are a ready-made, off-the-shelf capability,” echoed Rescale executive Kiley Naas. Instead of having to hire a contractor to build a physical supercomputer or cobble together a virtual one from commercial cloud services, the DoD customer can specify the kind of supercomputing task they need and Rescale’s software (or Parallel’s) will automatically assemble the capabilities they need.

“Our secret sauce is basically orchestrating and abstracting all that complexity, on the cloud,” added another Rescale exec, Derek McCoy.

Just as important as the streamlined technology is the streamlined contracting, Naas told Breaking Defense. DIU’s original competition two years ago, where Rescale and Parallel beat out dozens of other companies, many of them much larger, and the subsequent pilot project meet the requirements for a competitive contract award, she explained. That means there’s no need to hold a new competition every time a DoD agency wants cloud-based high-powered computing. Instead, she said, DoD offices will be able to order virtual supercomputers off a single Other Transaction Authority (OTA) contract.

So, if Rescale and Parallel build it, will DoD customers come? Both companies say they’ve already seen a surge of interest in their offerings since the end of June, when DIU officially announced it had awarded them a formal “success memo” based on their performance in the pilot.

“There’s been about a four-fold increase in demand,” said Naas. Parallel didn’t provide even rough figures but agreed they’d seen a major uptick. That indicates that the DIU initiative does address a real unmet need.

I Feel the Need For (Computing) Speed

Historically, the defense sector has long relied on supercomputers for ultra-detailed modeling of physical events, from nuclear explosions and missile aerodynamics to weather patterns. (Parallel’s first big federal contract was with NOAA.) Such simulations have become an even bigger deal with the move to detailed digital design and high-fidelity digital twins of physical systems.

In the last few years, the military’s surge of interest in AI has become another major driver, because training cutting-edge AI requires masses of data and computing power. But a single centralized supercomputing hub is too inflexible for current Pentagon policy, which emphasizes empowering a multitude of bottom-up initiatives to use AI.

The commercial world’s solution to more and more customers wanting more and more computing power has been to offer high-powered computing as a service. In plainer English, you rent the chips. Big cloud service providers such as Amazon AWS, Microsoft Azure, Google, and Oracle offer remote access to the kinds of GPUs and other cutting-edge chips favored by AI developers, along with data storage, labeling, and other necessities

But there’s a problem with these big-box cloud services, Parallel and Rescale argue. They offer so many options, for so many different applications, that it gets overwhelmingly complex for costumers who — like most federal agencies — aren’t already expert in HPC. There are literally hundreds of different HPC “infrastructure” options available from the major cloud providers, said McCoy, with tens of thousands of potential combinations.

It’s the digital equivalent of Home Depot: a huge store that offers every conceivable kind of tool and building material, but requires you to take it home and put it together yourself. In this analogy, Rescale and Parallel instead act like a general contractor, with their software designed to automatically purchase the right kind of computing power, data storage, and so on — from multiple cloud providers, rather than being beholden to only one — and then assembling it into a virtual supercomputer, ready to use.

“We’re like the plumbers,” Shaxted said “They can just come in and turn on the water.”

Rescale and Parallel argue argue that this ease-of-use was a major factor that set them apart from dozens of other companies that competed, including giant cloud providers. In fact, Parallel Works, which is even smaller than Rescale, still has only about two dozen employees. “We’re like 20 to 25 people now, we’re very small, and we’re almost exclusively technical [staff] — engineers, developers,” Shaxted said. “We just started bringing on a few marketing and sales people.”

“Without DIU, it would have been really difficult and even impossible to do this. … They really guided us through,” he said. “That’s the whole point of DIU.”