Publications

2024

1. CVPR Poster

   XFibrosis: Explicit Vessel-Fiber Modeling for Fibrosis Staging from Liver Pathology Images

   C. Yin, S. Liu, F. Lyu, J. Lu, S. Darkner, V. W. Wong, and P. C. Yuen

   In Conference on Computer Vision and Pattern Recognition (CVPR), 2024

   Abstract

   The increasing prevalence of non-alcoholic fatty liver disease (NAFLD) has caused public concern in recent years. Fibrosis staging from liver biopsy images plays a key role in demonstrating the histological progression of NAFLD. Fibrosis mainly involves the deposition of fibers around vessels. Current deep learning-based fibrosis staging methods learn spatial relationships between tissue patches but do not explicitly consider the relationships between vessels and fibers, leading to limited performance and poor interpretability. In this paper, we propose an eXplicit vessel-fiber modeling method for Fibrosis staging from liver biopsy images, namely XFibrosis. Specifically, we transform vessels and fibers into graph-structured representations, where their micro-structures are depicted by vessel-induced primal graphs and fiber-induced dual graphs, respectively. Moreover, the fiber-induced dual graphs also represent the connectivity information between vessels caused by fiber deposition. A primal-dual graph convolution module is designed to facilitate the learning of spatial relationships between vessels and fibers, allowing for the joint exploration and interaction of their micro-structures. Experiments conducted on two datasets have shown that explicitly modeling the relationship between vessels and fibers leads to improved fibrosis staging and enhanced interpretability.

2. SIIMS

   

   Sliding at First Order: Higher-Order Momentum Distributions for Discontinuous Image Registration

   L. Bao, J. Lu, S. Ying, and S. Sommer

   SIAM Journal on Imaging Sciences, 2024

   Abstract arXiv Code

   In this paper, we propose a new approach to deformable image registration that captures sliding motions. The large deformation diffeomorphic metric mapping (LDDMM) registration method faces challenges in representing sliding motion since it per construction generates smooth warps. To address this issue, we extend LDDMM by incorporating both zeroth- and first-order momenta with a non-differentiable kernel. This allows to represent both discontinuous deformation at switching boundaries and diffeomorphic deformation in homogeneous regions. We provide a mathematical analysis of the proposed deformation model from the viewpoint of discontinuous systems. To evaluate our approach, we conduct experiments on both artificial images and the publicly available DIR-Lab 4DCT dataset. Results show the effectiveness of our approach in capturing plausible sliding motion.

2023

1. ISBI Oral

   

   cRedAnno+: Annotation Exploitation in Self-Explanatory Lung Nodule Diagnosis

   J. Lu, C. Yin, K. Erleben, M. B. Nielsen, and S. Darkner

   In IEEE International Symposium on Biomedical Imaging (ISBI), 2023

   Abstract arXiv HTML PDF Code Poster Slides

   Recently, attempts have been made to reduce annotation requirements in feature-based self-explanatory models for lung nodule diagnosis. As a representative, cRedAnno achieves competitive performance with considerably reduced annotation needs by introducing self-supervised contrastive learning to do unsupervised feature extraction. However, it exhibits unstable performance under scarce annotation conditions. To improve the accuracy and robustness of cRedAnno, we propose an annotation exploitation mechanism by conducting semi-supervised active learning in the learned semantically meaningful space to jointly utilise the extracted features, annotations, and unlabelled data. The proposed approach achieves comparable or even higher malignancy prediction accuracy with 10x fewer annotations, meanwhile showing better robustness and nodule attribute prediction accuracy. Our complete code is open-source available: https://github.com/diku-dk/credanno.

2022

1. iMIMIC Oral

   

   Reducing Annotation Need in Self-Explanatory Models for Lung Nodule Diagnosis

   J. Lu, C. Yin, O. Krause, K. Erleben, M. B. Nielsen, and S. Darkner

   In Workshop on Interpretability of Machine Intelligence in Medical Image Computing (iMIMIC) at MICCAI, 2022

   Abstract arXiv HTML PDF Code Poster Slides

   Feature-based self-explanatory methods explain their classification in terms of human-understandable features. In the medical imaging community, this semantic matching of clinical knowledge adds significantly to the trustworthiness of the AI. However, the cost of additional annotation of features remains a pressing issue. We address this problem by proposing cRedAnno, a data-/annotation-efficient self-explanatory approach for lung nodule diagnosis. cRedAnno considerably reduces the annotation need by introducing self-supervised contrastive learning to alleviate the burden of learning most parameters from annotation, replacing end-to-end training with two-stage training. When training with hundreds of nodule samples and only 1% of their annotations, cRedAnno achieves competitive accuracy in predicting malignancy, meanwhile significantly surpassing most previous works in predicting nodule attributes. Visualisation of the learned space further indicates that the correlation between the clustering of malignancy and nodule attributes coincides with clinical knowledge. Our complete code is open-source available: https://github.com/diku-dk/credanno.

2. PLOSONE

   

   Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study

   J. Lu, J. Öfverstedt, J. Lindblad, and N. Sladoje

   PLOS ONE, 2022

   Abstract arXiv HTML Code

   Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of multimodal biomedical image registration. We compare the performance of four Generative Adversarial Network (GAN)-based methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on three publicly available multimodal datasets of increasing difficulty, and compare with the performance of registration by Mutual Information maximisation and one modern data-specific multimodal registration method. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. When less information is shared between the modalities, the I2I translation methods struggle to provide good predictions, which impairs the registration performance. The evaluated representation learning method, which aims to find an in-between representation, manages better, and so does the Mutual Information maximisation approach. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval).

3. SSBA

   Rotationally Equivariant Representation Learning for Multimodal Images

   E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, and N. Sladoje

   In Swedish Society for Automated Image Analysis (SSBA) Symposium on Image Analysis, 2022

2021

1. COMULIS

   

   Image-to-Image Translation in Multimodal Image Registration : How Well does It Work?

   J. Lu, J. Öfverstedt, J. Lindblad, and N. Sladoje

   In COMULIS Conference, 2021

   HTML Poster
2. Dataset

   

   Datasets for Evaluation of Multimodal Image Registration

   J. Lu, J. Öfverstedt, J. Lindblad, and N. Sladoje

   DOI: 10.5281/zenodo.5557568, 2021

   Abstract HTML

   In total, it contains 864 image pairs created from the aerial dataset, 5040 image pairs created from the cytological dataset, 536 image pairs created from the histological dataset, and metadata with scripts to create the 480 volume pairs from the radiological dataset. Each image pair consists of a reference patch I_Ref and its corresponding initial transformed patch I_Init in both modalities, along with the ground-truth transformation parameters to recover it.

3. COMULIS

   Registration of Multimodal Microscopy Images using CoMIR - Learned Structural Image Representations

   E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, and N. Sladoje

   In COMULIS Conference, 2021

   HTML Poster
4. Workshop

   Contrastive Learning for Equivariant Multimodal Image Representations

   E. Wetzer, N. Pielawski, E. Breznik, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, and N. Sladoje

   In The Power of Women in Deep Learning Workshop at the Mathematics of Deep Learning Programme at the Isaac Newton Institute, 2021

   Poster

2020

1. ICIAR Oral

   

   A Deep Learning Based Pipeline for Efficient Oral Cancer Screening on Whole Slide Images

   J. Lu, N. Sladoje, C. Runow Stark, E. Darai Ramqvist, J-M. Hirsch, and J. Lindblad

   In International Conference on Image Analysis and Recognition (ICIAR), 2020

   Abstract arXiv HTML Code

   Oral cancer incidence is rapidly increasing worldwide. The most important determinant factor in cancer survival is early diagnosis. To facilitate large scale screening, we propose a fully automated pipeline for oral cancer detection on whole slide cytology images. The pipeline consists of fully convolutional regression-based nucleus detection, followed by per-cell focus selection, and CNN based classification. Our novel focus selection step provides fast per-cell focus decisions at human-level accuracy. We demonstrate that the pipeline provides efficient cancer classification of whole slide cytology images, improving over previous results both in terms of accuracy and feasibility. The complete source code is made available as open source ( https://github.com/MIDA-group/OralScreen ).

2. NeurIPS

   

   CoMIR: Contrastive Multimodal Image Representation for Registration

   N. Pielawski, E. Wetzer, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, and N. Sladoje

   In Advances in Neural Information Processing Systems (NeurIPS), 2020

   Abstract arXiv HTML PDF Code Poster Slides

   We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.

3. MSc Thesis

   

   Evaluation of Learning-based Methods for Multimodal Biomedical Image Registration

   J. Lu

   Master Programme in Computational Science, 2020

   HTML PDF
4. NEUBIAS

   Cross-modal Representation Learning for Efficient Registration of Multiphoton and Brightfield Microscopy Images of Skin Tissue

   E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lu, J. Lindblad, I. Floroiu, A. Dumitru, M. Costache, R. Hristu, S.G. Stanciu, and N. Sladoje

   In NEUBIAS Conference, 2020