GAO Needs to Step Up Its Game On Annual Weapons Study

Peter Levine was long the top staffer on the Senate Armed Services Committee and then served as Defense undersecretary for personnel and readiness at the Pentagon. He pressed hard for acquisition reform while on the SASC, along with his partner in crime, Bill Greenwalt who worked on the Republican side of the committee. In this piece, Peter takes the congressional watchdog, the GAO, to task. Why? Read on! The Editor.

For the better part of two decades, the annual Government Accountability Office (GAO) assessment of major defense acquisition programs (MDAPs) has been a gold standard for measuring the successes and failures of the acquisition system.

Although GAO cost figures are not adjusted to account for evolving portfolio composition and other changes over time, these reports have provided a set of readily available and consistently updated benchmarks that have helped set the framework for congressional reform initiatives since 2003.

The first half-dozen reports documented skyrocketing acquisition costs, with unit cost growth approaching 50 percent on major programs, which resulted in total overruns of almost $300 billion by 2008. These reports also demonstrated the failure of the Department of Defense (DoD) to meet its own standards for technological and design maturity, a key factor in the enactment of the Weapon Systems Acquisition Reform Act (WSARA). Subsequent reports documented the implementation of WSARA’s requirements for starting acquisition programs on a sound footing, which showed that increased up-front investments resulted in improved knowledge levels and associated performance improvements (although there are indications that these gains could be endangered by more recent backsliding).

Unfortunately, this year’s report suffers from serious analytic problems that must be addressed to ensure the continuing value of the series.

The headline of the 2019 report is that, after several years of effective cost controls, MDAP portfolio costs have started to grow again. According to GAO, the total cost of DoD’s weapon systems portfolio has grown by $8 billion in the past year even though the total number of programs has declined. GAO seeks to explain the problem, asserting that “one of the primary drivers of this cost growth is that, since 2012, the average age of programs has increased,” that “cost growth has occurred in an environment where awards are often made without full and open competition,” and that close to half of the sole-source contracts for major weapon systems have been awarded to the five largest defense contractors.

GAO notes that even DoD’s newer programs — those initiated after the enactment of WSARA — have started to show cost growth. “We attribute the deteriorating performance of newer programs to the inconsistent implementation of knowledge-based acquisition practices—a condition we highlighted in our 2018 assessment,” the report states. Better DoD leadership, GAO argues, “could help reverse the cost growth we observed and position the military departments to obtain persistent efficiencies in the acquisition programs they manage.”

But GAO does not provide any evidence to support its conjectures about the cost growth. For example, no data are included in the report to demonstrate that there was a higher percentage of sole-source contracts, or of contracts awarded to the five largest contractors, or of contracts entered without low levels of knowledge in the 2019 portfolio than there was in previous portfolios. Similarly, no data are included to show that sole-source contracts, “Big 5” contracts, or low-knowledge contracts were responsible for a disproportionate share of the cost growth identified by GAO.

More importantly, GAO’s own data show that the supposed cost growth at the heart of the analysis does not exist at all. The patient reader will find that the real reason for this “cost growth” is increased purchase quantities. In plain terms, the Pentagon bought more weapons. As GAO explains deep in the report, “A program’s cost can increase because of additional quantities. While that does represent a cost increase, it does not necessarily indicate acquisition problems or a loss of buying power.” In fact, GAO concludes, when DoD’s MDAP portfolio is adjusted for quantity increases, the Pentagon experienced an “overall buying power gain of $3.9 billion” in the last year. (Emphasis added) Newer programs and older programs alike became more efficient and required more funding only because of increased purchase quantities.

In other words, costs went down, not up, and the department spent more only because it chose to buy more. Rather than explaining a deterioration in cost performance that did not exist, GAO should have been assessing how the department was able to deliver yet another year of cost reductions even while it was implementing significant acquisition policy changes and contending with a disruptive reorganization. Was the overall cost decrease solely attributable to larger, more economically efficient purchase quantities, or were there other significant contributing factors? The report cannot tell us, because GAO was looking in the wrong direction.

Like past reports, the 2019 report provides useful information on knowledge levels attained at key acquisition decision points, as measured against GAO benchmarks. For example, GAO continues to assess how early in a program critical technologies are demonstrated in a relevant environment, when preliminary design review is completed, when engineering drawings are released to manufacturing, when critical manufacturing processes are in statistical control, when a production-representative prototype is tested, and when a baseline software capability is delivered. Moreover, GAO goes beyond providing aggregate information on these issues by assessing the attainment of product knowledge on a case-by-case basis for all programs in the Pentagon’s major program portfolio.

Even here, however, the continuing value of the annual report series is at risk because it appears that GAO may decide not to assess knowledge levels of the programs for which such an assessment is most needed.

DoD’s approach to the acquisition of major weapon systems is currently undergoing a major change as it places its emphasis on speed — acquisition “at the speed of relevancy,” as former Defense Secretary James Mattis described it. As the department pushes to acquire major weapon systems faster and faster, it risks rushing into major acquisition commitments on the basis of inadequate knowledge and incurring risks that could lead to cascading costs and program failures. GAO’s knowledge benchmarks could help ensure against such an adverse outcome by providing an early warning sign if the department is taking on excessive risk or if the risk begins to produce unacceptable cost growth.

Unfortunately, it appears that GAO may choose to exclude all or most weapon systems acquired under the new policy from its annual reports. As GAO explains, acquisitions conducted pursuant to the so-called “middle-tier” of acquisition authorized by section 804 of the National Defense Authorization Act for Fiscal 2016—the default tool used by the department for rapid acquisition of weapon systems — “are expressly excluded from the statutory definition of MDAP and therefore will not join the portfolio.” For example, “during the course of our review, six programs that DoD previously classified as pre-MDAPs are now proceeding via the alternate rapid prototyping and rapid fielding acquisition pathways.”

For the last 17 years, the GAO assessment has addressed only major defense acquisition programs covered by DOD Directive (DODD) 5000.01. During that time, all major weapon systems acquisitions have been conducted pursuant to that Directive. But if the Pentagon decides to start acquiring major weapon systems in a different manner, GAO faces a choice: it can limit the scope of its report to traditional acquisition programs that meet the MDAP definition, or it can find a way to apply its benchmarks to the full range of major weapons acquisition programs regardless of the legal authority under which they are conducted.

The traditional approach enables GAO to produce a straightforward, cookie-cutter report. The expansive approach would require more work, but would enable GAO to produce a far more informative analysis. Let’s hope that GAO improves upon its disappointing performance this year and opts for the more informative approach.

Peter Levine, who served on the staff of the Senate Armed Services Committee from August 1996 to February 2015, is now a senior research fellow at the Institute for Defense Analyses, often described as the Pentagon’s thinktank.