Katherine Strandburg calls for enhanced certification of software used in criminal cases

Software-based forensic tools—often proprietary products—are prominently featured in criminal cases in American courtrooms. Probabilistic genotyping programs interpret DNA mixtures too complex for human analysts. Cell-site mapping tools place a defendant’s phone near a crime scene. Gunshot detection systems report where shots were fired, and breath-alcohol devices generate the readings behind DUI prosecutions. 

Yet according to new research co-authored by Katherine Strandburg, Pauline Newman Professor of Law, courts almost never examine whether the software behind that evidence actually works as claimed.

In “Adversarial Certification of Evidentiary Software,” published in June in the Texas Law Review, Strandburg, who is faculty director of NYU Law’s Information Law Institute, proposes a remedy for the current insufficient vetting of forensic software. Together with her co-authors—Bryan H. Choi of the University of Colorado Law School; Nicholas Vincent ’20, an associate at Williams & Connolly LLP; New York State-based criminal justice attorney Eli Siems ’19; and Clay Halbert ’23, judicial law clerk for the US District Court for the Eastern District of California—Strandburg argues that software-based forensic tools should have to pass a mandatory, independent certification process before their outputs can be admitted in criminal trials. That process would include adversarial “red-team” testing designed to expose flaws missed by the software’s developers.

According to the authors, the stakes are simple: “Faulty software produces faulty evidence.” The paper extends earlier work by Strandburg, Siems, and Vincent, whose 2022 article in Hastings Law Journal, “Trade Secrecy and Innovation in Forensic Technology,” challenged the claim that trade secret protection for forensic software is needed to encourage innovation. The new paper turns from that critique to a proposal for reform.

“When liberty depends on software outputs, the justice system cannot rely on either judicial indifference to software validation or perfunctory rubber-stamping of vendors’ products,” Strandburg says.

Courts screen scientific evidence under one of two standards. The Frye test asks whether a method is generally accepted in the relevant scientific community. The Daubert test, adopted by federal courts and most states, asks judges to act as gatekeepers assessing scientific validity. According to the paper, both tests focus on two questions: whether the underlying science is sound, and whether it was properly applied in the case at hand.

Software, the authors argue, falls between those questions. An expert witness typically understands the relevant science or knows how to run the program, but is rarely able to say whether the code correctly carries out that science. Two programs that both perform probabilistic genotyping may rely on different algorithms, contain different bugs, and produce different results. Even a routine update can introduce new errors.

To see how courts are handling the problem, the authors reviewed 240 judicial opinions issued between 2002 and late 2025 that addressed the admissibility of software-based evidence. They found that courts routinely skip software validation altogether. Some ignore the question. Others treat software as reliable as long as the science it claims to implement is sound, or defer the issue to cross-examination at trial. Courts have also allowed vendors to invoke trade secrecy to keep source code away from defendants.

The paper points to two New York State court examples. In People v. Wakefield, a concurring opinion noted that the TrueAllele genotyping program had gone through 25 versions since 1999, yet only one validation study identified which version had been tested. In the second case, New York State’s Forensic Statistical Tool, a DNA program used by the New York City’s medical examiner, was found to have been modified after a state oversight body approved it.

The authors call the emerging pattern “common law certification.” Courts admit evidence from a tool largely because earlier courts admitted it, even when those earlier rulings were thinly reasoned or concerned different versions or different programs altogether. Over time, the authors argue, these rulings reinforce one another, making it very difficult for defendants with limited resources to mount a successful challenge.

The authors contend that the same features that make software hard for judges to evaluate make it well suited to formal certification. Forensic software performs a narrow set of tasks repeated across very large numbers of cases. Given identical inputs and settings, it should produce identical outputs. Engineers can also verify exactly which version of the code was used in a particular case. The recommended process for the certification body would proceed as follows:

• A “blue team” would work with the vendor to review its requirements documents, testing plans, and internal validation results. 
• A separate “red team” of software testing experts and representatives of the defense bar would stress-test the software, probing for weaknesses, checking edge cases, and generally looking for ways the software might fail. For example, for gunshot detection, experts would test how the software performs on a sample with a lot of background noise.
• Vendors would have to disclose source code to the certifying body, enabling “white box” testing for full certification. 
• If a vendor insists on keeping its code confidential, the red team would be allowed an unrestricted range of “black box” testing, enabling them to submit unlimited sample data to the software tool and then to analyze the accuracy of the results. Black box testing would be reserved for cases where the software developers could demonstrate legitimate trade secrecy concerns, and resulting certification would be labeled as limited.

Certification would cover only a specific program, version, and defined range of uses. The certifying body would continue to monitor tools after approval, and certification could be revoked. For software-based tools, certification would replace the Frye and Daubert inquiries into foundational validity. Questions about how a tool was used in a given case would still be argued in court.

The design draws on scholarship from NYU Law’s Policing Project on “soft-law certification” of police technologies, which advocates that policing technology products be reviewed by an independent government or nonprofit certification body. Unlike soft law certification, in which participation for software providers is voluntary, the authors’ recommended process would require certification be a mandatory precondition for admissibility in court. 

The authors argue that mandatory certification of software would serve every party in the system. Defense lawyers could pool scarce resources instead of challenging the same software in case after case. Vendors could confine disclosure of trade secrets to a small, defined group. Newer companies would gain a clear route to market, rather than facing what the authors describe as a system that favors the products that courts happened to accept first.

“The Institute of Electrical and Electronics Engineers (IEEE) is currently in the beginning stages of developing a ‘Standard for the Procurement, Verification and Validation, and Life Cycle Management of Forensic Technologies,’” Strandburg adds. “While IEEE standards are not mandatory, such a standard could provide a reference point for the development of a more stringent certification process.”