Surgical Intelligence Leaderboard

Surgical Data Science CollectiveThe University of Chicago Booth School of Business

Which anatomical structures and operative entities are visible?

Recognizing anatomy and other annotated entities in the operative field is a prerequisite for scene understanding and downstream decision support.

Example

Which anatomical structures are visible in this laparoscopic frame?

Select every matching label.

  • abdominal wall
  • colon
  • inferior mesenteric artery
  • intestinal veins
  • liver
  • pancreas
  • small intestine
  • spleen
  • stomach
  • ureter
  • uterus
  • vesicular glands
  1. ResNet-5071.60
  2. Gemma 3 27B fine-tuned65.20
  3. LemonFM (linear probe)557.60
  4. GPT-6 Astra54.50
  5. Gemini 3.8 Flash54.40
  6. Gemini 3.7 Flash53.00
  7. Claude Fable 5.152.30
  8. Claude Fable 549.50
  9. Qwen3.8 Max 090248.97
  10. Gemini 3 Flash Preview46.90
  11. GLM-5.3-Flash46.66
  12. Claude Opus 545.90
  13. Gemini 3.1 Pro Preview44.30
  14. GPT-5.6 Sol43.70
  15. Kimi K341.80
  16. Gemma 3 27B-it41.30
  17. GPT-5.6 Terra40.50
  18. GPT-5.439.00
  19. Grok 4.636.69
  20. GPT-5.6 Luna36.60
  21. Claude Opus 4.635.80
  22. Claude Sonnet 535.70
  23. Claude Sonnet 4.631.20
  24. Qwen3.8 27B31.00

Micro-averaged F1 (%)

dashed line: majority-class baseline 6.00%

The plot reports micro-averaged F1 on 12 anatomical structures in the DSAD dataset. Error bars show 95% bootstrap confidence intervals. The dashed line shows the majority-class baseline.
ModelMicro-averaged F195% CI
ResNet-50[huggingface]71.60%70.50–72.80
Gemma 3 27B fine-tuned[huggingface]65.20%63.90–66.50
LemonFM (linear probe)5[huggingface]57.60%56.20–59.10
GPT-6 Astra54.50%52.50–56.50
Gemini 3.8 Flash54.40%52.40–56.50
Gemini 3.7 Flash53.00%51.00–55.10
Claude Fable 5.152.30%50.30–54.30
Claude Fable 549.50%47.30–51.50
Qwen3.8 Max 090248.97%47.17–50.82
Gemini 3 Flash Preview46.90%45.00–48.80
GLM-5.3-Flash46.66%44.58–48.64
Claude Opus 545.90%44.30–47.40
Gemini 3.1 Pro Preview44.30%42.20–46.40
GPT-5.6 Sol43.70%41.90–45.60
Kimi K341.80%39.80–43.80
Gemma 3 27B-it41.30%40.10–42.40
GPT-5.6 Terra40.50%38.60–42.50
GPT-5.439.00%37.20–40.90
Grok 4.636.69%34.72–38.83
GPT-5.6 Luna36.60%34.70–38.60
Claude Opus 4.635.80%34.00–37.60
Claude Sonnet 535.70%33.80–37.70
Claude Sonnet 4.631.20%29.30–33.00
Qwen3.8 27B31.00%29.70–32.30
Majority-class baseline, not a modelMajority-class baseline6.00%—

Structure presence is multi-label, so exact match requires the predicted structure set to equal the ground-truth set, while micro-averaged F1 credits partial overlap.

Local models are scored on all 1,978 validation frames. API models use a seed-42 sample of 1,000 validation frames.

  1. 5 Che, C., Wang, C., Vercauteren, T., et al. LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings. arXiv preprint arXiv:2503.19740 (2025).
  2. 12 Carstens, M., Rinner, F. M., Bodenstedt, S., et al. The Dresden Surgical Anatomy Dataset for Abdominal Organ Segmentation in Surgical Data Science. Scientific Data, 10, 3 doi:10.1038/s41597-022-01719-2 (2023).
  3. 15 Grammatikopoulou, M., et al. CaDIS: Cataract dataset for surgical RGB-image segmentation. Medical Image Analysis, 71, 102053 doi:10.1016/j.media.2021.102053 (2021).
  4. 16 Murali, A., Alapatt, D., Mascagni, P., et al. The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark. arXiv preprint arXiv:2312.12429 (2023).

About

If you found this website useful, please cite as:

@misc{skobelev2026comparativestudysurgicalai,
      title={A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling},
      author={Kirill Skobelev and Eric Fithian and Yegor Baranovski and Jack Cook and Sandeep Angara and Shauna Otto and Zhuang-Fang Yi and John Zhu and Neeraj Mainkar and Margaux Masson-Forsythe and Daniel A. Donoho and X. Y. Han},
      year={2026},
      eprint={2603.27341},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.27341},
}