Google Cloud & MLCommons launch secure MedPerf tests

Google Cloud and MLCommons have launched MedPerf on Google Cloud Confidential Computing, allowing medical AI models to be evaluated on patient data without exposing either the data or the model code.

MedPerf is an open-source benchmarking platform created by MLCommons to standardise how medical AI is assessed across institutions. It now runs in a secure environment that uses confidential computing to isolate workloads while models are tested against real-world clinical data.

The setup addresses a long-running problem in healthcare AI. Developers need access to varied patient datasets to assess whether a model performs reliably across different hospitals and populations, but hospitals and research organisations face strict privacy rules and may also want to protect their own data.

Under the system, AI models are evaluated inside hardware-isolated Trusted Execution Environments using Google Cloud Confidential Space. This means hospitals, research institutions, other participants and Google itself cannot view the patient data or the model code during evaluation.

The environment also encrypts memory while workloads are running and hardens the operating system. Before patient data is made available to a workload, the system provides cryptographic proof that approved code is running on the intended confidential computing hardware.

Medical AI benchmarking often requires significant computing resources, particularly when inference runs on graphics processors. MedPerf is therefore running on Google Cloud A3 machines with NVIDIA H100 GPUs, combining Intel TDX technology on the CPU side with NVIDIA confidential computing on the GPU side.

Research use

The technology is already being used in brain tumour research through the Federated Tumor Segmentation, or FeTS, initiative. The work focuses on validating AI models using private brain MRI scans contributed by institutions in different countries.

This is significant because brain tumours such as glioblastomas are relatively rare, limiting the amount of data available to any one hospital. It also reflects a broader issue in healthcare AI: a model that performs well in one setting may produce weaker results elsewhere because of differences in patient demographics, scanning methods or clinical equipment.

The validation work has shown that performance can vary sharply between sites. In one example, a model reached 95% accuracy at one institution but only 63% at another, underscoring the importance of testing on more representative datasets before clinical use.

The researchers involved include Dr. Spyridon Bakas at Indiana University, Dr. Yury Velichko at Northwestern University and Dr. Amber Simpson at the University of Alberta. Their participation reflects academic and clinical interest in methods that allow cross-institutional evaluation without requiring raw patient data to be pooled in one location.

MLCommons launched MedPerf in 2023 as part of a broader effort to create common benchmarks for medical AI. The organisation, whose members span technology and academia, has promoted federated evaluation as a way to test models across multiple sites while keeping sensitive data under local control.

In practice, the approach may appeal to hospitals and developers that want independent evidence of model performance but remain cautious about handing over datasets or proprietary code. It could also help address concerns from clinicians and regulators about whether benchmark results reflect actual care settings rather than narrow test environments.

Dr. Yury Velichko, Associate Professor of Radiology at Northwestern University, said the cloud-based infrastructure had enabled testing beyond a laboratory setting. "My experience testing federated learning on Google Cloud has shown that the future of medical AI lies in secure, scalable, and collaborative cloud environments," he said. "Moving beyond the controlled lab setting to test these workflows in a production-ready infrastructure provided a unique opportunity to evaluate the performance and security of federated learning in real-world clinical applications."

Alexandros Karargyris, MedPerf Lead at MLCommons, said the combination of benchmarking and confidential computing is intended to strengthen trust in how medical AI is validated. "Medical AI holds enormous promise for patients around the world, but that promise can only be realized if clinicians, researchers, and regulators can trust the benchmarks we use to evaluate it," he said. "By bringing MedPerf onto Google Cloud's Confidential Computing infrastructure, we have taken a major step toward a future where AI models can be rigorously tested on real patient data - without compromising privacy, intellectual property, or benchmark integrity."