Evaluating and Understanding Model Editing for Medical Vision-Language Models

Overview

Medical vision-language models are increasingly used for image-grounded clinical reasoning, but post-deployment failures are inevitable due to distribution shifts, changing protocols, rare findings, and patient-specific variation. Model editing offers a lightweight way to correct targeted mistakes without full retraining. However, existing editing benchmarks are often too simplified for clinical deployment.

M³Bench evaluates whether an edit is not only successful on the target error, but also precise, clinically transferable, robust to multimodal variation, and consistent across related clinical contexts.

Figure 1. Post-deployment workflow of medical VLM model editing with clinician-in-the-loop feedback. Instead of globally retraining a model after every failure, targeted editing attempts to correct specific post-deployment errors while preserving existing model behavior.

Benchmark Design

M³Bench is organized around three core editing desiderata: Reliability, Locality, and Generality. We extend these criteria into clinically meaningful settings involving image variation, text variation, modality and protocol shifts, clinical composition, and longitudinal temporal consistency.

Figure 2. Construction pipeline and task suite of M³Bench. The benchmark contains ten evaluation dimensions spanning target correction, side-effect prevention, semantic transfer, modality shift, compositional reasoning, and temporal consistency.

Task Suite

Dataset Composition

M³Bench is built from public medical VQA and image-text resources, including VQA-RAD, SLAKE, PMC-VQA, and PadChest-GR. The benchmark contains 16,276 questions over 1,398 unique images, covering multiple imaging modalities, anatomical sites, and clinical findings.

Original third-party images and datasets are not redistributed. The release provides processed benchmark metadata and instructions for reconstructing the benchmark after obtaining the source datasets.

Figure 3. Benchmark distribution. Inner ring: modality. Middle ring: anatomical site. Outer ring: lesion or finding category.

Experimental Setup

We evaluate four representative medical VLM backbones and four model editing methods.

We group editing methods into two families: gradient-based editors, such as MEND and LoRA, and memory-based editors, such as GRACE and BalancEdit. The benchmark evaluates both single-edit and sequential-edit settings.

Main Results

Across models and tasks, no single editing method dominates all clinical criteria. Gradient-based editors often achieve strong reliability and generality, but can cause serious locality failures. Memory-based editors better preserve locality, but struggle with compositional transfer and require careful hyperparameter calibration.

Figure 4. Radar plot summarizing single and sequential editing performance across ten tasks and four medical VLM backbones.

Figure 5. Example sequential-editing performance profile on LLaVA-Med.

Key Findings

Analysis

Representation Geometry

We analyze medical VLM latent representations and find that embeddings are highly anisotropic: medical concepts concentrate in a narrow cone. This makes localized editing difficult because nearby clinical concepts can be unintentionally affected by a targeted update.

Figure 6. Cone effect in medical VLM representations. High pairwise cosine similarity indicates that different clinical concepts occupy a tightly concentrated region in representation space.

Non-target Concept Drift

Gradient-based editors tend to induce broader latent-space shifts, which can move non-target clinical concepts and cause locality failures. Memory-based editors restrict activation to a gated region, which improves locality but may limit transfer.

Figure 7. Concept drift analysis. LoRA broadly shifts topic centroids, while BalancEdit applies more spatially restricted changes.

Compositional Failure Modes

Clinical cases often contain multiple co-occurring findings. Memory-based editors can under-transfer from single-finding edits to multi-finding cases because hard activation boundaries may not capture partially overlapping clinical evidence.

Figure 8. Composition probe comparing single-finding and multi-finding cases under editing.

Where to Edit

We further study component and layer sensitivity using a hybrid BELoRA editor. Editing the language model together with the vision-language projector provides useful gains, while expanding to the vision encoder does not consistently improve stability.

Figure 9. Architecture sweep over editable components and layer depths.

Hyperparameter Sensitivity

Memory-based editors depend strongly on the activation radius. A radius that works well for one VLM backbone can under-activate or over-merge edits for another backbone.

Figure 10. Backbone-dependent sensitivity of memory-based editing radius.

Repository Structure

The release is organized into benchmark task files, metadata, editing method implementations, and documentation.

Citation

If you use M³Bench in your research, please cite our paper:

@inproceedings{m3bench2026,
  title     = {Evaluating and Understanding Model Editing for Medical Vision-Language Models},
  author    = {Zhu, Guli and Wu, Chenwei and Shen, Liyue},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}