Seoul hospital, Harvard replace static AI testing with hospital simulator

They tested AI impact on patient outcomes and hospital operations.

Seoul National University Hospital (SNUH) and Harvard Medical School have developed a virtual hospital simulator to evaluate medical artificial intelligence (AI) in dynamic clinical conditions before deployment in real healthcare settings, according to a press release.

The Clinical Environment Simulator is a system that introduces what the researchers—led by Seong-Eun Kim of SNUH—describe as a shift from static, diagnosis-based evaluation to dynamic evaluation that reflects real-time clinical environments.

The simulator tests medical AI based on large language models within a virtual hospital environment that replicates patient care and hospital operations.

It also evaluates both patient outcomes and hospital operational efficiency through a dual-metric approach.

The “Patient Engine,” one of the two core engines, simulates changes in patient condition over time using large language models, drawing on disease trajectory templates developed by specialists and initial data from electronic medical records.

This generates potential pathways of symptom progression and treatment response.

Simultaneously working with it is the “Hospital Engine,” which simulates hospital workflows using operational time data.

It tracks the availability of beds, staff, and medical equipment, whilst allocating resources in sequence based on task duration and prioritises critically ill patients.

The simulator models the effects of AI-driven clinical decisions on patient outcomes and hospital operations.

Delayed diagnostic decisions can lead to deterioration in patient condition. Allocation of limited resources to emergency cases can increase waiting times for other patients.

The evaluation framework applies a dual-metric composite score, measuring patient prognosis, including survival, treatment timing, and adherence to clinical guidelines.

It also measures hospital efficiency, including length of stay, emergency department throughput, and utilisation of beds and equipment.

The system includes stress testing under conditions such as network failures and simultaneous emergency cases.

It also assesses the impact of AI decisions under constrained operational scenarios.

The researchers state that the simulator provides a preclinical environment for testing medical AI systems without exposure to patients, noting that virtual hospitals cannot replicate all physiological responses of patients.

Research Professor Kim said the system does not fully capture complex biological responses but stated that it provides a step towards evaluating medical AI within dynamic healthcare systems.