Pentagon’s annual weapon test report reverses classification, details major program challenges

WASHINGTON — A year after members of Congress raised transparency concerns about the Defense Department’s independent weapons tester removing previously public information from its annual report, the office of the Director of Operational Test and Evaluation (DOT&E) has reversed course on its restriction of information.

As a result, DOT&E’s annual report, the most recent edition [PDF] of which was released today, once again features in-depth information about the challenges facing America’s most important weapon systems. It does away with the system of issuing both a public report and a second “controlled unclassified information” (CUI) version available only to lawmakers and those inside the Pentagon.

Nickolas Guertin, who leads DOT&E, said the office consulted with Congress to release one public version of the report, even if he defended the use of CUI before.

“Issuing two documents allowed DOT&E to be more transparent with congressional and DOD personnel, while maintaining the integrity of information related to programs under oversight,” Guertin said in the intro to the 2023 edition. “This year, in consultation with Congress, we produced a single, publicly releasable report. Doing so complies with both the spirit and letter of the legislation governing the Annual Report.”

The CUI version of the FY21 DOT&E report raised transparency concerns from oversight organizations and received criticism from Sen. Elizabeth Warren, D-Mass., who cited reporting from Breaking Defense and urged the office to reverse the decision.

“This unjustified restriction of public access will not serve to protect national security information but will instead be abused to avoid disclosure of failures in our major weapons programs,” Warren said in letters [PDF] to Defense Secretary Lloyd Austin and Guertin last year.

Beyond the at least temporary death of the CUI version, this year’s report was notable for the new information it provided on major weapons systems, from delays with Space Force GPS plans to some questions about the utility of hypersonic programs.

Networks: In Need Of ‘Overarching’ Policies

In the world of information systems, the report took aim at several initiatives under the Pentagon’s Digital Modernization Strategy (DMS).

“Many DMS efforts lack an overarching systems integration process, test strategy, and program executive organization to manage cost, drive schedules, and monitor performance factors,” according to the report. “Many DMS initiatives also use commercial cloud environments, but threat-representative cyber survivability testing on the commercial side of cloud environments is not currently being conducted by the DOD per the DOT&E memorandum, Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs, dated April 3, 2018.”

DOT&E lists several recommendations for the DoD CIO, DISA and the Digital Modernization Infrastructure Executive Committee, which is chaired by the CIO, US Cyber Command (CYBERCOM) and the Joint Staff J-6. 

For one key program called Identity, Credential and Access Management (ICAM), intended to create a secure environment for users to access authorization sources while letting DoD know who is on the network, DOT&E recommends designating an operational test agency to support capability testing and developing an overarching test and evaluation strategy. The same recommendation goes for another DMS initiative called the Mission Partner Environment and funding the Joint Interoperability Test Command to fully support DMS initiatives and testing. (The report’s mention of cloud security testing comes just a day after a Pentagon official said his office would have National Security Agency hackers attack civilian cloud providers to test their zero-trust defenses.)

RELATED: NSA red team will attack JWCC providers to test zero trust security

The report also recommends DoD “conduct adequate cyber survivability testing of all DMS enterprise IT programs, projects, and initiatives in accordance with current DOD and DOT&E cyber survivability T&E guidance and policy.”

In a separate effort called the Joint Cyber Warfighting Architecture (JWCA), which CYBERCOM plans to use to support its cyberspace operations and training, DOT&E found that despite the concept continuing to mature, there is no dedicated operational test and evaluation planned, potentially limiting DoD’s ability to understand its effectiveness and survivability. 

“JCWA is designed to collect, fuse and process data and intelligence in order to provide situational awareness and battle management at the strategic, operational and tactical levels while also enabling access to a suite of cyber capabilities needed to rehearse and then act in cyberspace,” according to the report. 

The report notes that there isn’t enough data to determine the resilience of JCWA in a cyber-contested environment and recommends CYBERCOM complete a JCWA-level OT&E. 

The recommendations include establishing “a cadence of test for dedicated OT&E, beginning in FY23, to understand how the capability afforded by JCWA is evolving over time and to ensure it is an effective, suitable, and survivable enabler of cyber operations.”

Army: Rare Praise For FLRAA, Vexation Versus IVAS

After decades of blasting big Army programs — for good reason — DOT&E delivered a rare dollop of praise for one of the service’s most ambitious efforts, the Future Long-Range Assault Aircraft initiative to replace the workhorse UH-60 Black Hawk helicopter with a much faster, longer-ranged, and more expensive tiltrotor. “The Army has been transparent and collaborative while developing the FLRAA acquisition strategy and supporting T&E [Testing & Evaluation] strategy,” the report declares.

True, it is early days for FLRAA, which just picked the Bell V-280 Valor over the rival Sikorsky SB>1, both of which were technology demonstrators and not prototypes. Bell is now on the hook to make an official prototype of the winning design by 2025.  “No operational testing has been conducted to assess its effectiveness, suitability, and survivability,” DOT&E admits. But so far, so very good: “The FLRAA program has set a solid foundation for a successful program if current practices are retained.”

By contrast, DOT&E blasted IVAS, the Individual Visual Augmentation System, an out-of-the-box attempt to turn ruggedized Microsoft HoloLens goggles into a combination targeting scope, heads-up display and training tool. In a mid-2022 test, the report says, the IVAS 1.0 variant “did not demonstrate improvements to deficiencies identified during previous … testing.” Troops using current kit consistently outperformed those wearing IVAS in simulated combat missions and those wearing them found the goggles disorienting.

“Soldiers prefer their current equipment… to IVAS,” DOT&E said. “The majority of soldiers reported at least one symptom of physical impairment to include disorientation, dizziness, eyestrain, headaches, motion sickness and nausea, neck strain and tunnel vision. Soldiers cited IVAS 1.0’s poor low-light performance, display quality, cumbersomeness, poor reliability, inability to distinguish friend from foe, difficulty shooting, physical impairments and limited peripheral vision as reasons for their dissatisfaction.” (The Army and Microsoft recently came to an agreement for the company to produce three variants of the goggles, with IVAS 1.2 hopefully fixing major issues.)

Navy: Open Questions About The Ford, Hypersonics And A New Sonar

The Gerald R. Ford (CVN-78) aircraft carrier remains one of the most highly scrutinized programs under DOT&E’s watch, despite the nearly three months underway the ship touted in fiscal 2022.

While the report dives into several individual systems on the ship, it says the new technologies collectively amount to the biggest obstacle in the ship’s ability to pass through initial operational test and evaluation, a necessary Pentagon milestone for all warships and their associated capabilities. That testing phase began in September 2022 and is scheduled to last through the fourth quarter of fiscal 2024, according to the report.

RELATED: Too much new tech on Ford was a mistake, CNO says

The service’s premiere hypersonic weapon program, Conventional Prompt Strike, achieved a bevy of “Joint Flight Campaign” tests in FY22, but overall it’s too early in its development for DOT&E to provide meaningful assessments about its effectiveness. The report does however reaffirm the service is still working towards integrating CPS onto a Zumwalt-class destroyer in FY25 and a Virginia-class submarine in FY29.

And on the topic of the Zumwalt destroyers, weapons testers report that “due to competing operational commitments and system readiness,” the ship’s initial operational testing will continue through “at least FY24.” The three-ship class already has an extended history dating back prior to 2011, which includes program truncations and mission changes leading to numerous schedule delays.

“Surface Warfare (SUW) operational testing is complete, but all other primary and secondary mission areas require additional testing,” according to DOT&E.

And lastly, on the Constellation-class program, weapons testers weighed in on the Navy’s recently announced swap to integrate the Thales CAPTAS-4 variable depth sonar. They say that although the Navy believes its inclusion is “low risk, DOT&E is unable to assess the integration at this time.”

Air Force: Testers Knock Hypersonic ARRW Program, Relive KC-46 Woes

The DOT&E report looks at 17 Air Force programs, including the Advanced Pilot Training using the new T-7 Redhawk aircraft, the F-15EX Eagle II, the F-22A Raptor, the KC-46A Pegasus tanker, the MH-139A Grey Wolf helicopter and the AGM-183A Air-Launched Rapid Response Weapon (ARRW) hypersonic cruise missile. 

The ARRW program office in particular has attracted some DOT&E ire for beginning testing of the system without bothering to wait for approved testing plans. The report also raises concerns about the missile’s ability to survive cyber attack, and concludes: “The ARRW program has not yet demonstrated the required warfighting capability.”

As for the KC-46, built by Boeing, the DoD testing office reviews the litany of woes, from the well-known issues with refueling boom’s remote cameras to a series of Category One deficiencies with the cargo operations capabilities. “The Air Force is working with Boeing to develop critical upgrades to the remote vision system (RVS), refueling boom, and defensive systems necessary for closure of IOT&E in FY24,” the report notes.

RELATED: After years of KC-46 vision system troubles, Boeing thinks it’s finally cracked the code

Another Boeing program, the Grey Wolf helicopter, also is facing potential delay to its Milestone C decision to allow production to begin, the DOT&E report warns.

“The current program schedule provides only a few months to collect operationally relevant data before the Milestone C decision, currently scheduled in January 2023. The MH-139A program needs to address several additional challenges to mitigate the risk to meeting operational effectiveness, suitability, and survivability requirements.”

The report identifies continued deficiencies in ground and flight testing that “represent a risk to MH-139A meeting operational effectiveness requirements.” It also notes continued problems with the “capability of the cabin layout to support employment of armed tactical response forces” the helicopter is designed to carry.

The Advanced Pilot Training (APT) program is re-baselining its testing schedule to support full rate production in FY26, rather than FY24 as earlier planned. “The top critical issues for the Air Force to address are the T-7A emergency escape system (including the bird strike capability of the canopy), the lack of an Automatic Ground Collision Avoidance System (AGCAS), the On-Board Oxygen Generation System (OBOGS), and cyber survivability.”

F-35 Joint Strike Fighter Fields ‘Immature’ Software

The F-35 Joint Strike Fighter program continues to draw serious criticism from DOT&E, although the report notes that “steady progress” was made in FY22 to enable the Joint Simulation Environment for the 64 tests needed for the program to pass initial operational test and evaluation (IOT&E) and thus officially enter full rate production. The report says the IOT&E trials currently are estimated to wrap up in August, but caveat that DOT&E thinks that date will slip “due to the possibility of further discoveries of deficiencies and potential delays in the verification, validation, and accreditation (VV&A) process.”

The report also chides the F-35 program for continuing “to field immature, deficient, and insufficiently tested Block 4 mission systems software to fielded units. The operational test (OT) teams continue to identify deficiencies that require software corrections and, with them, additional time and resources.“

DOT&E further asserts that the Joint Program Office hasn’t “adequately planned” for operational testing of Technology Refresh 3 (TR-3), the upgraded hardware configuration currently scheduled to be delivered in production Lot 15 aircraft, beginning in the fourth quarter of FY23. In particular, the report notes, required flight test instruments for the current Technology Refresh 2 aircraft as well as the TR-3 jets are “not all on contract and will not be available,” meaning there may not be enough test aircraft to do so before operational employment of the upgraded version.

Lastly, the program’s full transition to the Operational Data Integrated Network, or ODIN, logistics network to replace the F-35’s notorious Autonomic Logistics Information System has been delayed another year, to at least June 2024, following a discovery of a number of safety-imperiling “Category One” deficiencies.

Space Force: Delays Hit C2 And GPS Programs

The DOT&E report reviews two major Space Force programs, both with troubled history: the Space Command and Control System (Space C2); and the Next-Generation Operational Control System (OCX) for the GPS satellite constellation that will allow use of the jam resistant M-Code.

The testing office notes that Space C2 didn’t actually undergo any approved operational testing in FY22 “primarily due to delayed product delivery, understaffed development teams, unclear test team constructs and responsibilities, and development focus on non-critical capabilities.”

The report adds the Space Force has been working to address those issues, having “changed key leadership personnel, restructured development teams, more clearly defined their integrated testing construct, and refocused capability development to only the most crucial capabilities.” 

That said, DOT&E worries that one piece of the sprawling, software intensive Space C2 effort, Palantir’s Warp Core data-as-a-service capability, “was conditionally accepted for operations by the Space Force, and was actually operationally fielded by the Air Force and North American Aerospace Defense Command /Northern Command” without any operational testing.

As for OCX, the report repeats concerns raised by the testing shop last year about continued delays both in development of the OCX system and production of receivers for ground terminals and radios that “increase the risk that U.S. and allied warfighters will not be able to conduct operations with GPS-derived position, navigation, and time (PNT) information.”