Data at the Department of Defense
All of the latest news and announcements from Applied’s government team
The Nexus Newsletter
Welcome back to The Nexus Newsletter. In this edition, due to some recent press, we focus on the Department of Defense’s data practices - or lack thereof - and provide our commentary and recommendations.
Applied’s government team is hiring! Read through to the bottom of the newsletter to see a list of our open roles in D.C..
Finally, we are less than two weeks away from Nexus 23! There is still time to join us in-person at the National Press Club, or virtually.
Data
Data is at the heart of every modernization program currently underway in the Department of Defense, but current collection, management, and triage practices are far from workable. Data is our biggest risk: continued investment in modernization programs without reforming the underlying data practices means that we are investing vast amounts of time and money in programs that are ultimately doomed to fail. In particular, major autonomy or automation programs that require training data now to be successful over the next 3 to 5 years, such as OMFV or CCA, have a severe deficit of data at all. The DOD, instead of investing resources into legacy platforms that currently fly, drive, or sail en masse around the globe to collect, curate, annotate, and make available data for autonomy software development and test, continues to pump resources into these new modernization efforts that have, at best, a handful of prototype platforms collecting data at CONUS ranges for limited periods of time. This is not a great place to be when considering the timeline for potential conflict and program schedule for fielding.
In a recent Congressional hearing, the wise Dr. Josh Lospinoso, CEO of Shift 5, stated the problem simply: “Unfortunately, today the DOD struggles to liberate even the simplest data streams from our weapon systems. These machines are talking, but the DOD is unable to hear them.” Fixing the problem, he continued, “requires taking seriously the difficult, unglamorous work of laying strong foundations.” We’ve said this before, too: In a blog post last fall, we noted that “autonomy teams—including those at the DOD—that under-invest in data infrastructure and management capabilities will face significant hurdles that extend development timelines, inflate costs, and dampen outcomes.” Unlike the commercial autonomous vehicle industry, which spent billions to generate tens of billions of annotated data, with a few notable exceptions the DOD expects its autonomy programs to deliver a miracle. Data isn’t sexy, and legacy systems certainly aren’t sexy; and so the DoD continues to fall victim to its focus on shortcuts to the next sexy new platform.
However, senior leaders in the DOD, Congress, and elsewhere often voice their understanding of the foundational role that data plays in systems deployed today and the next-generation systems that will be deployed - in theory - in the near future. At the same hearing referenced above, Sen. Manchin called data the “DOD’s most crucial resource in AI development.” More recently, CENTCOM’s CTO Schuyler Moore noted that while data is the “limiting factor for maturity and function of a model” because it supplies the “fuel that allows these models to perform in a meaningful way,” “in a lot of the areas that we operate, you don’t even have the data to collect to then fuel a model.” That’s great. Acknowledgment is the first step in the process, so claps for making it the first step. But who will drive the DoD from thoughts and words to actual action?
Let’s first start by reducing the aperture of the problem. “Data” broadly, could mean nearly anything. Data, in the case of autonomy development and test, consists of both the mobility sensor data, the reconnaissance sensor data, and the platform telemetry data for single and collaborative platforms with respect to the design reference mission, or scenario and behaviors of the platform tied to the mission it must accomplish.
Let’s pick a single autonomy program: Collaborative Combat Aircraft. A recent paper from the Mitchell Institute for Aerospace Studies - “Collaborative Combat Aircraft Need Data to Train for Combat” - provides an overview of the landscape and challenges. One section stands out in particular:
“Can CCA developers use existing datasets to train algorithms to fight alongside a manned fighter aircraft, and leverage decades of data collected from highend test and training events on advanced flying ranges and in combat simulators? The answer is no. There are no datasets that represent complex air combat scenarios. This unfortunate lack of airpower-related data is a huge barrier to rapid CCA fielding, yet still, the U.S. Air Force has no data management plan for tactical airpower.”
Left unsaid is that the Air Force will still resource CCA in spite of these issues. The lack of a robust, historical data collection and management regime is frustrating and disappointing, but not surprising. The Army and Navy are not doing any better.
Podcast: Is the DOD struggling with its data management?
We joined NDIA’s Emerging Tech Horizons podcast to discuss the DOD’s current data management practices, why data can be emotional, lessons learned from Project Maven and the Joint AI Center, and lessons the Department can learn from commercial industry.
Nexus 23
We are less than two weeks away from Nexus 23! We are thrilled to bring you an action-packed two-day event featuring the world’s top experts on autonomy and national security, including foreign policy experts, allied delegations, warfighters, technology visionaries, policy leaders, and program managers.
Head to our website to see the most updated roster of speakers and agenda. There’s still time to join us in-person at the National Press Club, or virtually, on May 17th and 18th.
Already registered? Keep an eye out for an email with details for in-person and virtual attendees in the next few days.
Analyzing synthetic datasets
Our team won the Best Presentation award at the SPIE Defense + Commercial Sensing conference last week!
Our presentation, “Analyzing synthetic datasets through the training and inference domain gap,” focused on how to conduct detailed test and evaluation (T&E) of synthetic datasets and the models they feed. Specifically, we demonstrated how creating a digital twin of a real-world dataset allows us to execute a more principled evaluation of the synthetic-to-real domain gaps for training and inference, and use this information to test and evaluate synthetic datasets themselves, rather than the models trained on them.
The full presentation and paper will be available soon.
XPONENTIAL
We’re heading to Colorado for XPONENTIAL next week! On Wednesday, we are speaking on the “Future of Ground Autonomy for the Warfighter” panel alongside LTC Christopher Orlowski, Product Manager for the Army’s Robotic Combat Vehicle, Andrew Dallas from NAMC, and representatives from Secmation, Kodiak Robotics, and Robotic Research. Join the conversation in Room 401/402 at 2:16 PM on Wednesday, May 10.
Stop by booth 1730 to meet our team and learn more about how we’re bringing our commercial autonomy expertise to the Department of Defense.
Redefining mobility summit
We were invited to speak to commercially-proven best practices for verification and validation (V&V) of autonomous vehicle safety and the public policy implications of virtual testing at the Contra Costa Transportation Authority’s Redefining Mobility Summit. We are proud to take a leading role in sharing our perspectives on V&V and autonomous system safety.
News we’re reading
Autonomous systems are gaining momentum in the national security space. Below, we’ve pulled key quotes from recent articles of interest, plus brief commentary from Applied Intuition’s government team:
Andreessen Horowitz | How the U.S. Can Rewire the Pentagon for a New Era
By Porter Smith, Deal Partner at a16z, and David Ulevitch, General Partner at a16z
Key quote: Responding to the paradigm shift requires reengineering the Pentagon’s DNA for a new era. Given that the winner of the next big war will more closely resemble a distributed computer operating at scale — programmatically collecting, sharing, and acting upon data from relatively inexpensive and configurable endpoints, like drones — computing design and organizational principles honed in the tech industry can help provide guidance to meet the challenge. [...]
The challenge is in balancing these processes against the benefits of modern software-enabled platforms that invert the traditional upgrade cadence. Software updates naturally occur on a more continuous basis than rigid hardware timelines. Adjusting to a continuous cycle means re-thinking budgeting rhythms that allocate funding on multi-year schedules guided by hardware updates. Software-first systems provide rapid upgradeability and redeployment, but their benefits are only realized if the testing and certification processes can match their speed.
The DoD should look to startups to help solve this problem. Automating testing is a clear way to cut development and deployment times if quality control thresholds can be maintained. Additionally, the DoD could provide new testing guidelines that delineate between deploying new software and activating embedded capabilities in existing software that should shorten recertification timelines. The procurement process needs a philosophical and operational shift to recognize the value of software not just commensurate with the value of hardware, but exceeding it. From now on, it’s the software that matters, the hardware is often a commodity.
Our take: There is growing recognition of the importance of software-defined capabilities - including autonomous systems - to a high-end fight with a near-peer adversary. Enabling the procurement and effective, efficient development of those systems requires a re-think of the budgeting cycles and contracting mechanisms that underpin the procurement of capabilities. We are proud to pioneer the software acquisition pathway through our engagement with the RCV program: By separating hardware and software development and acquisition, the RCV program will benefit from rapid iterations on software, unhindered by slower development timelines associated with hardware. This approach is the only way to ensure that technologies that support both survivability and lethality remain operationally relevant. In the Army, PEO GCS continues to break new ground on coordinating software procurement within the constraints of the defense procurement process. We’re excited to see this approach spread across other programs and services.
Jake Chapman | Principal / Agent Challenges in the DoD
By Jake Chapman, Managing Director at Army Venture Capital Corporation (AVCC)
Key quote: In any decision there is (1) the person authorized to make a decision, (2) the person held accountable for the results of the decision, (3) the direct beneficiary or object of the decision and (4) the person who must pay the cost of implementing the decision. In an ideal scenario (from the perspective of incentive alignment and efficiency) a single person wears the mantle of all four roles. In most life situations one person or at most two represent all of those roles and incentives are fairly aligned. [...]
Why this rather boring lecture on principle agent issues? Because for major AND minor decisions in the DOD those four roles are often occupied by four different individuals and those individuals are often in different organizations, including Congress or other offices of the executive branch. [...]
The funding decision is being made by people far removed from the battlefields where the rubber will someday meet the road, those who are paying the cost have no voice in the decision, the accountable parties are responsible for balancing dozens of programs and must make horse trades, and the beneficiaries of keeping the old platform alive are vocal and have a concentrated interest in protecting the old platform.
Our take: Warfighters have been kept out of the procurement process for new capabilities for too long, resulting in the procurement of capabilities that do not always map to the needs of the end user. That space allows a number of different priorities to be introduced into the decision making process, obscuring our view of what is truly important: The needs of the men and women tasked with using the system in combat.
C4ISRNET | Goodbye Hoarding? Official sees increased data sharing at Pentagon
By Colin Demarest, Networks & AI Reporter, Defense News
Key quote: “I think what we’re seeing across the department is folks leaning into the need to share,” John Turner, the adviser, said April 20 at the C4ISRNET Conference. “I think the culture of ‘data hoarding’ is absolutely on the way out, and we’re seeing incredible responsiveness as, increasingly, data is being shared across components for various use cases.” [...]
“I think data leadership is standing up across the DoD components, as well as across the national security community, and that’s time and investment in talent that’s well placed,” he said. [...]
Turner said the work of fleshing out and readying the [CDAO] has instilled a feeling of opportunity among people to “align our strategy on how to drive change across the department, how to lead and oversee the adoption of data analytics and AI.” [...]
“The success from the data perspective is the speed at which we can close the loop between the producer and the consumer, and share data faster,” the senior adviser said Wednesday. “And data sharing is a complex, multifaceted issue.”
Our take: We appreciate the optimism and “feeling of opportunity” here, but caution against applauding the DOD and Congress for standing up new “data leadership” positions alone: Often, standing up new offices helps you feel good, but doesn’t necessarily drive change. Ultimately, results are what matters. Growing recognition of data’s role in enabling software-defined capabilities is important, but the fact of the matter is that the Department has a long way to go when it comes to data collection from its vast number of deployed platforms, data management, and sharing with similar programs across the organization.
Defense One | Robot Rescue? Air Force Seeks New Way to Recover Downed Troops
By Audrey Decker, Staff Reporter, Defense One
Key quote: The HH-60W is “not particularly helpful in the Chinese” area of responsibility, Lt. Gen. Richard Moore said at a hearing of the Senate Armed Service airland subcommittee.
That’s largely because the Sikorsky-made aircraft is not expected to be “survivable to the threat environment,” said Lt. Gen James Slife, the service’s deputy chief of staff for operations. “You end up losing more people trying to recover somebody than the person you lost to begin with. And so the challenge we're facing is really how to address the question of how we will do personnel recovery in a contested environment.”
So the Air Force is now considering “non-traditional” methods, such as unmanned platforms, to “fulfill that moral imperative of leaving nobody behind,” Slife said.
Our take: The conversation here is focused on troop evacuation, but the larger point is that the traditional manned platforms - like the HH-60W discussed here - are not survivable in the Indo-Pacific in a fight against a near-peer adversary. As we’ve said before, small numbers of large, exquisite, manned platforms are not the answer in that fight. Instead, large numbers of small, attritable, unmanned, and autonomous systems will prove essential: They enhance situational awareness, act as force multipliers, increase lethality, and present the adversary with paralyzing dilemmas, all while keeping American and allied warfighters out of harm's way.
Join our team
Our government team is hiring! Join our fast-moving, rapidly-growing team in D.C. A few of our open roles are listed below:
Thank you for reading The Nexus Newsletter. Subscribe for more news from the nexus of national security and autonomy.