Surgical Intelligence Leaderboard

Surgical Data Science CollectiveThe University of Chicago Booth School of Business

Can a model understand surgical video?

We benchmark models on whether they can recognize the actions a surgeon performs throughout a 21-minute laparoscopic surgery.

Example

What surgical gesture is being performed in this frame?

Choose one label.

  • aspirate
  • clip
  • coagulate
  • cut
  • dissect
  • grasp
  • irrigate
  • no action
  • remove specimen
  • suture
  1. SDSC MViT Multi-Task685.50
  2. SurgMotion775.26
  3. OmniRAS-2B linear probe974.49
  4. OmniRAS-1B attentive probe974.37
  5. OmniRAS-1B linear probe974.19
  6. OmniRAS-2B attentive probe972.56
  7. LemonFM (linear probe)869.19
  8. Claude Fable 5.167.39
  9. GPT-5.6 Terra66.89
  10. Gemini 4 Argon66.52
  11. Gemini 3.7 Flash64.39
  12. Claude Opus 51262.16
  13. Claude Fable 561.60
  14. Gemini 3.8 Flash61.25
  15. GPT-5.6 Sol1060.91
  16. Qwen3.8 Max 0902 high60.70
  17. Gemini 3 Flash Preview59.92
  18. Gemini 3.1 Pro Preview58.88
  19. Qwen3.8 27B high58.30
  20. GPT-5.456.97
  21. GLM-5.3 Flash high56.80
  22. Claude Sonnet 553.84
  23. Claude Opus 4.648.80
  24. Claude Sonnet 4.643.27
  25. Gemma 3 27B IT standard19.20

Exact frame accuracy (%)

Model — evaluation settingExact frame accuracy
SDSC MViT Multi-Task685.50%
SurgMotion775.26%
OmniRAS-2B linear probe974.49%
OmniRAS-1B attentive probe974.37%
OmniRAS-1B linear probe974.19%
OmniRAS-2B attentive probe972.56%
LemonFM (linear probe)869.19%
Claude Fable 5.167.39%
GPT-5.6 Terra66.89%
Gemini 4 Argon66.52%
Gemini 3.7 Flash64.39%
Claude Opus 51262.16%
Claude Fable 561.60%
Gemini 3.8 Flash61.25%
GPT-5.6 Sol1060.91%
Qwen3.8 Max 0902 high60.70%
Gemini 3 Flash Preview59.92%
Gemini 3.1 Pro Preview58.88%
Qwen3.8 27B high58.30%
GPT-5.456.97%
GLM-5.3 Flash high56.80%
Claude Sonnet 553.84%
Claude Opus 4.648.80%
Claude Sonnet 4.643.27%
Gemma 3 27B IT standard19.20%

One laparoscopic procedure (21:42 of video, 10 gesture classes), annotated by expert surgeons at 15 FPS. The annotations cover 71.4% of the video; frames with overlapping or missing labels are not scored, leaving 13,001 frames.

Kimi K3 uses the documented normalization that maps its clip label to the benchmark's cut label. No confidence intervals are reported for this single-case comparison.

SDSC MViT Multi-Task, SurgMotion, and LemonFM (linear probe) are surgical foundation-model reference rows; the other rows are high-thinking-effort model evaluations.

  1. 6 Fan, H., Xiong, B., Mangalam, K., et al. Multiscale Vision Transformers. Proceedings of ICCV 2021 PyTorch Video MViT documentation and arXiv:2112.01526 (2021).
  2. 7 Wu, J., Holm, F., Chen, C., et al. SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos. arXiv preprint arXiv:2602.05638 (2026).
  3. 8 Che, C., Wang, C., Vercauteren, T., et al. LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings. Proceedings of CVPR 2026 CVPR 2026 (2026).
  4. 9 Borgioli, L., Getty, N., Xiu, W., et al. OmniRAS: Standardizing Foundation Model Training and Evaluation in Robot-Assisted Surgery. arXiv preprint arXiv:2608.31048 (2026).
  5. 10 OpenAI. Previewing GPT-5.6 Sol. OpenAI release notes openai.com (2026).
  6. 11 Moonshot AI. Kimi K3. GitHub repository github.com/MoonshotAI/Kimi-K3 (2026).
  7. 12 Anthropic. Claude Opus 5 System Card. Anthropic anthropic.com/system-cards (2026).

About

If you found this website useful, please cite as:

@misc{skobelev2026comparativestudysurgicalai,
      title={A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling},
      author={Kirill Skobelev and Eric Fithian and Yegor Baranovski and Jack Cook and Sandeep Angara and Shauna Otto and Zhuang-Fang Yi and John Zhu and Neeraj Mainkar and Margaux Masson-Forsythe and Daniel A. Donoho and X. Y. Han},
      year={2026},
      eprint={2603.27341},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.27341},
}