AdaptOR: Deep Generative Model Challenge for Domain Adaptation in Surgery

+++ AdaptOR challenge re-opened December 2021 +++
The AdaptOR challenge will be further continued and is open to receiving submissions from Dec 6, 2021. The participating teams continue to have a maximum of 3 submissions. The submissions will be evaluated periodically and updated on our leaderboard. Please make note of our revised data access form. The joint challenge publication has not yet been carried out. This publication will be planned by the organisers, once a sufficient number of submissions are received from the participants.

News

• December 6th, 2021 - The AdaptOR challenge is re-opened and is now accepting further submissions. Please take note of the revised data access form. The results of the evaluation will be periodically updated on our leaderboard.
November 8th, 2021 - Find the recent work from the organisers, that proposes further methodological developments on the challenge dataset here.
September 25th, 2021 - Find the submissions of the participants, published in the proceedings of the DGM4Miccai Workshop, LNCS MICCAI 2021 here.
September 20th, 2021 - The final results are computed and will be announced at the DGM4Miccai Workshop on October 1st, 2021. The program can be seen here.
July 16th, 2021 - Find the recent work from the organisers, that proposes methodological developments on the challenge dataset here. Stay tuned for the challenge publication, coming soon.
July 05th, 2021 - The final Docker Container Submission must be submitted to the Evaluation Queue AdaptOR Challenge 2021 (MICCAI) Submission (9614850) by July 07th 2021, as explained in the Submission Tutorial.
June 25th, 2021 - Deadline extended to July 2nd 2021, 11:59 PM Pacific Time
June 22nd, 2021 - Don't forget to choose the AdaptOR Challenge option under "Subject Areas", while creating a submission in the CMT platform on June 25th. Read more about what to submit here and use the opportunity of the Platform Testing Phase while it is still possible.
June 01st, 2021 - The Platform Testing Phase has started. Submit your models to test the submission process to the Evaluation Queue AdaptOR Challenge 2021 (MICCAI) Platform Test (9614808) as explained in the Submission Tutorial. The results of your test submissions will be visible here.
May 27th, 2021 - The last possible registration date is now 06/06/2021. Don't miss your chance and register now!
May 27th, 2021 - Find out how we will evaluate the output of submissions with our evaluation script on the Synapse platform and in the Docker Example GitHub Repo!
May 12th, 2021 - We have now made more information about the Submission Details available. Check out the Submission Tutorial and our Docker Example!
May 11th, 2021 - We are happy to announce that Challenge Awards will be provided for the three top performing teams. We thank Fehling Instruments GmbH & Co. KG for their support!
May 03rd, 2021 - AdaptOR is featured as challenge of the month on Computer Vision News!
April 27th, 2021 - It's now official: The AdaptOR Challenge will be hosted by the DGM4MICCAI Workshop.
April 09th, 2021 - To jump-start our participants' work, helper functions in python can be found on the Synapse platform.
April 01st, 2021 - Training phase starts: Datasets are now available and teams can register on the Synapse platform!

Challenge Abstract

Mitral regurgitation (MR) is the second most frequent indication for valve surgery in Europe and may occur for organic or functional causes [1]. Mitral valve repair, although considerably more difficult, is prefered over mitral valve replacement, since the native tissue of the valve is preserved. It is a complex on-pump heart surgery, often conducted only by a handful of surgeons in high-volume centers. Minimally invasive procedures, which are performed with endoscopic video recordings, became more and more popular in recent years. However, data availability and data privacy concerns are still an issue for the development of automatic scene analysis algorithms. The AdaptOR challenge aims to address these issues by formulating a domain adaptation problem „from simulation to surgery“: We provide a smaller number of datasets from real surgeries, and a larger number of annotated recordings of training and planning sessions from a physical mitral valve simulator. The goal is to reduce the considerable domain gap between simulation and intraoperative cases, e.g. by incorporating generative models, as in [2,3].

The task associated to the domain adaptation itself is to detect a varying number of 2D landmarks per frame [4] in the target domain. The landmarks are defined by the placement of sutures during mitral annuloplasty (entry and exit points into the tissue), which renders useful for surgical skill assessment and detailed intraoperative documentation. The evaluation metrics of this challenge will be related to how well these points could be identified in unseen intraoperative scenes, therefore it is also possible to only come up with a solution to a landmark detection problem in a single domain. More complex methods, however, would leverage data from both domains and adapt them on input-, output-, and/or feature level. Due to the specific clinical motivation of improving the realism of surgical simulation [2,3], the AdaptOR challenge especially aims to provide a framework for comparison of the performance of different image-to-image translation approaches. Such approaches need to learn how to sucessfully transform the images into an intraoperative appearance, thereby not altering already realistic entities of the image (surgical instruments, sutures, needles etc.). While this can be merely assessed visually, and we will show example results during the workshop, we hypothesize that the success of landmark detection may be an indicator for the quality of the transfer with respect to the consistency of sutures in both domains.

More Background

In our surgical training scenario, there is a clinical need for transforming not-so-realistic phantom data into more realistic surgical images [2,3]. Therefore, we encourage the participants to use image-to-image translation approaches, however, this is not mandatory.
In general, we think the underlying detection task could be solved differently:

1. Training of a landmark detection approach only on the intraoperative domain
2. Using the simulation domain for pre-training/dataset fusion
3. Incorporating the simulation domain by using a combination of more advanced input-, output-, feature-level domain adaptation approaches, possibly in an end-to-end training strategy
4. Others.

The authors should detail on their approaches in their submitted LNCS papers.
In case an image-to-image translation task was solved, we will provide visual examples of the generative model’s output for visual comparison. These results are qualitative and will not be considered in the ranking scheme. We hypothesize that the quantitative assessment for landmark detection may be an indicator for the quality of the domain transfer with respect to the consistency of sutures in both domains.

Figure 1. Example of domain transfer done with tempCycleGAN from [2]. Left: Original video from a simulator. Center: Frames transformed into intraoperative appearance. Right: Backtransformed video frames.

Challenge Document

DOI 10.5281/zenodo.4646979 (v2)

Keywords

References

Rules

• Only fully automatic approaches are allowed
• No additional training data and no models pre-trained on other datasets are allowed.
• Each team is only allowed to register once and all submissions must be done from the same account.
• A single participant is only allowed to be part of one team.
• The participants agree to participate in a challenge publication that will be planned by the organisers once a sufficient number of submissions are received.
  Participating teams must submit a LNCS paper with comprehensive descriptions of their methods via CMT one week after docker container submission deadline. Failure of submitting the paper will render the submission invalid.

Prizes

The 1st Prize for the AdaptOR challenge was bagged by team LS_Group comprising of Jiacheng Wang, Haojie Wang, Ruochen Mu, and Liansheng Wang from Xiamen University, China. Their submission titled Cross-Domain Landmarks Detection in Mitral Regurgitation can be found here. The 2nd Prize was won by Team Neko, comprising of Huifeng Yao, Ziyu Guo, Yatao Zhang, and Liansheng Wang from Shangdong University China, and Hongkong University. Their submission titled Improved Heatmap-Based Landmark Detection can be found here.

The two top performing teams received certificates and prize money of 600€ and 400€ for the first and second place respectively. The awards are generously sponsored by Fehling Instruments GmbH & Co. KG.

Potential Future Plans

Our envisioned goal is to extend the dataset with additional cases and potentially establish a recurring AdaptOR event to support progress in this application field.
In 2022, we would like to focus on stereo tasks and method generalization.