As kick-off for the USCT data exchange and collaboration we initiated a USCT data challenge at SPIE Medical Imaging 2017. The aim was to apply existing image reconstruction algorithms to real 2D and 3D USCT data and bring together experts from the USCT community to identify best practices, to establish specifications for interfaces and carry out a first comparison of reconstructed images.

Data challenge description

Long term goals

  • Build up a free and open licensed reference USCT data base, freely available for the whole community.
  • Enabling reproducible comparison of image reconstruction algorithms and USCT systems.
  • Establish user friendly and easy to use interfaces, standards and data formats between the different USCT systems and their reconstruction algorithms, software and data formats.
  • Identify properties of systems, experimental setups, data, and algorithms towards optimal images.

Rules

  • Data and interface software will be provided using a free and open license, i.e. the BSD 2-clause license for code and data, allowing free use and publication of results.
  • The data sets may remain with the participants, can be used for additional research and other purposes. Further publication and results should follow good academic attribution practices and cite the following paper:

“Nicole V. Ruiter, Michael Zapf, Torsten Hopp, Koen van Dongen: USCT data challenge, in Proc. SPIE 10139, Medical Imaging 2017: Ultrasonic Imaging and Tomography, 101391N“

The paper is published with open access and can be downloaded at SPIE Digital Library.

  • Participants are encouraged to a submit field reports, used methods and results: algorithm identification (e.g. citation of paper), and – if available – reconstructed images and performance results.
  • Participants agree that their participation and their results may be published in connection with their names and affiliations, also under a free and open license and open access standards

Data and code access

For this challenge we provided three types of data and code:

  • 3D KIT USCT data with data access interface software.
  • 2D TU Delft USCT data with data access interface software.
  • 2D CSIC/UCM USCT Data with data access interface software.

For details see information on data access.

Collection, evaluation and publication of results

  • Participants are invited to share their results, feedback and suggestions.
  • Possible conclusive paper of results, e.g. Journal of Medical Imaging, inclusion as coauthor(s)

Panel discussion at SPIE Medical Imaging 2017

On 14th February a panel discussion took place. The discussion was moderated by Nicole Ruiter (Karlsruhe Institute of Technology, Germany) and Koen van Dongen (Technical University Delft, Netherlands). As panelists we were glad to welcome

  • Thomas Matthews (Washington University in St. Louis, USA),
  • Mailyn Perez Liva (Universidad Complutense de Madrid, Spain),
  • Ken-ichi Kawabata (Hitachi Ltd., Japan),
  • Roberto Janniel Lavarello Montero (Pontifica Universidad Catolica, Peru),
  • Ivana Balic (SonoView Accoustic Sensing Technologies, Swiss),
  • Neb Duric (Delphinus Medical Technologies).

The panelists and the audience discussed the experiences on applying the currently available datasets and future directions during the two hour session. A detailed summary of the discussion is given below.

Panelists of USCT data challenge panel discussion at SPIE Medical Imaging 2017

Panelists of the USCT data challenge panel discussion at SPIE Medical Imaging 2017. From left to right: Thomas Matthews, Mailyn Perez Liva, Ken-ichi Kawabata, Roberto Janniel Lavarello Montero, Ivana Balic, Neb Duric. Photo taken by SPIE, 2017

In addition to the panel discussion, seven posters were presented at the Wednesday poster session. Participating authors of field reports and posters were acknowledged by certificates provided by SPIE.

Summary of the panel discussion

Interfaces & data formats

All groups who worked with the data found the provided data format and software interfaces convenient. However there is a need for a common set of metadata that should be provided by groups offering experimental data. It might be beneficial to offer software interfaces for open source software as well.

Restrictions on data sets

There was a consent that the number of provided data sets should be in some way limited in order to allow for a fair comparison of algorithms. Furthermore the data set quality should be checked before publishing. While an independent check by a single entity might not be feasible, an option could be a rating system for the data and/or a platform to exchange experiences, data quality, best practices etc.

The discussion also revealed the need for patient data as the complex material composition cannot be represented by phantom data only. Yet the conditions and restrictions to distribute patient data to the public need to be investigated further.

Phantoms

All participants and panelists agreed that a standard phantom would be very beneficial to compare not only image reconstruction algorithms but also the different USCT systems. Yet when comparing algorithms in future challenges, a comparison metric should be established to allow specifying the phantom characteristics in more detail. It might also be necessary to have dedicated phantoms for the different USCT imaging modes. They should have limited complexity in order to calibrate image reconstruction methods.

Furthermore synthetic data would be very much welcomed, preferably in the same data format as the experimental data in order to provide an easy transition from synthetic to experimental data.

A remaining open question is the manufacturing of phantoms, for which there might be opportunities to collaborate with manufacturers and ultrasound phantom task forces.

Intellectual property

A controversial discussion was held on the restrictions of the intellectual property: generally the opinion was that the data should be provided without restrictions. Yet this might currently not be feasible for companies.

Long term perspective

There were also mixed opinions on the maintenance of the data: on the one hand the data should not change over time to allow for a fair comparison of the imaging algorithms. On the other hand, there should be a possibility to correct obvious data errors. A possible solution could be a versioning of datasets and software interfaces (data provenance) which would also guarantee accessibility of revised data for a long time. There was also agreement that the data should be available for several years, yet possibilities to guarantee this need to be investigated.

How should we proceed?

The USCT data challenge was very well received. There was consent in the panel that the data base should be continued and opened for other groups to provide data and participate in applying their image reconstruction algorithms. The data challenge could be continued in SPIE Medical Imaging meetings in future, possibly also with more dedicated challenges for special USCT reconstruction tasks.

Posters presented at SPIE Medical Imaging 2017

poster    poster    poster    poster     poster

Posters are available under the OpenAccess Creative Commons license CC BY.

SPIE Proceedings

A paper on the SPIE Data Challenge 2017 was published in the Proceedings of SPIE Medical Imaging Conference:

Nicole V. Ruiter ; Michael Zapf ; Torsten Hopp ; Hartmut Gemmeke ; Koen W. A. van Dongen USCT data challenge
in Proc. SPIE 10139, Medical Imaging 2017: Ultrasonic Imaging and Tomography, 101391N (March 13, 2017); doi:10.1117/12.2272593, Conference Volume 10139

Released as OpenAccess Creative Commons license CC BY