What to Measure While the FDA Weighs Generative AI in Medtech

MedTech Dive reported this week that patients, providers and medtech firms submitted hundreds of comments to the FDA on how the agency should approach generative AI in medical devices. The publication pulled five of those perspectives into a single piece. The volume of input shows that the people closest to these products do not yet agree on how generative AI should be regulated. Guidance that has to account for hundreds of comments will take time to settle.

For a mid-market device company, the obvious response is to wait for the final word before committing to anything. I think that leaves useful work undone. I would be surprised if any approach the FDA settles on let a company skip showing where generated output enters its process and how people check it. Both of those questions can be answered now, from your own operations, without knowing the final rules.

The review step is where the cost sits

Every system has exceptions, the one-off cases that fall outside the normal path. Everyone knows they exist, but few people know how often they happen. Trying to handle all of them in software is what makes a system slow and expensive. I think this is the most underestimated cost in workflow projects, and the reason to map a process and size its exceptions before building anything.

Generative AI changes the math because the exception path becomes part of the normal path. A generated complaint summary, CAPA narrative or draft customer response is a starting point that someone has to read and often edit. That edit is the control working as intended, and it is also where the labor goes.

That makes the correction rate the most useful number for a generative AI tool in a quality system. Out of every hundred outputs, how many were accepted as written, how many were edited, and how many were thrown out? I suspect few companies could answer that today for the tools their staff already use. Without that number, the quality team cannot say whether review is catching problems, and finance cannot say whether the tool saves anything. A tool whose drafts are mostly rewritten still costs the reviewer's time plus the license.

Where to start counting

Start with the places where generated text lands in a controlled record or reaches someone outside the company. In a device company that usually means complaint intake and investigation summaries, CAPA and nonconformance write-ups, draft labeling and instructions for use, and responses to customer or field inquiries. If your product generates output that a clinician or patient sees, that belongs at the top of the list, since that is the territory the FDA comments addressed.

For each one, capture three things at the point of review. Record whether the output was accepted, edited or rejected. Add a short reason code when it was edited or rejected, and note who reviewed it. Keep the reason codes few and plain (factual error, missing information, wrong format) so reviewers will actually use them. The goal is a rough frequency you can trust, gathered over a few months.

Put the capture inside the workflow people already use. A separate spreadsheet that reviewers fill in after the fact tends to be abandoned within weeks, and the data it does collect will be uneven. A field on the complaint record or CAPA form, filled in at the moment of approval, costs the reviewer a few seconds and produces data you can report on.

The friction you should expect

This is harder than it sounds. Adding a field to an eQMS form or a complaint record goes through change control, and some older systems make even small form changes slow or costly. Reviewers who are already behind will see reason codes as one more box to click, and some of them are skeptical of the tools in the first place. It helps to tell them plainly that the data is meant to judge the tool, and that it is not a measure of their work.

The data will be messy at first, with inconsistent codes and gaps from weeks when people forgot. That is normal in a growing company, and it is worth planning a cleanup pass into the first quarter instead of treating early noise as a reason to stop.

The payoff is a number that serves two audiences. The quality team gets evidence about how well review is working, which it will want however the guidance reads. The CFO usually approves expanding a tool or building it into a larger workflow as capital spend. With the correction rate, the CFO gets an honest input for the payoff side of that proposal, instead of a vendor's estimate of time saved.

Next steps