Fully convolutional structured LSTM networks for joint 4D medical image segmentation

Yang Gao, Jeff M. Phillips, Yan Zheng, Renqiang Min, P. Thomas Fletcher, Guido Gerig

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Longitudinal medical image analysis has great potential to reveal developmental trajectories and monitor disease progression. This process relies on consistent and robust joint 4D segmentation. Traditional methods highly depend on the similarity of images over time and either build a template or assume the images could be co-registered. This process may fail when image sequences present major appearance changes. Recently, deep learning (DL) approaches have achieved state-of-the-art results for related challenges in computer vision. These approaches make use of models such as fully convolutional networks (FCNs) for end-to-end pixel-wise segmentation and recurrent neural networks (RNNs) with long short-term memory (LSTM) units for sequence-to-sequence modeling. In this paper, we propose a new DL framework called FCSLSTM for 4D image segmentation with FCNs for the spatial model and LSTM for the temporal model. This is the first DL framework with deep integration of FCNs and LSTM for joint 4D segmentation that could be trained end-to-end. Our approach achieves promising results with the demonstrated application to longitudinal pediatric magnetic resonance imaging (MRI) segmentation.

    Original languageEnglish (US)
    Title of host publication2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018
    PublisherIEEE Computer Society
    Pages1104-1108
    Number of pages5
    Volume2018-April
    ISBN (Electronic)9781538636367
    DOIs
    StatePublished - May 23 2018
    Event15th IEEE International Symposium on Biomedical Imaging, ISBI 2018 - Washington, United States
    Duration: Apr 4 2018Apr 7 2018

    Other

    Other15th IEEE International Symposium on Biomedical Imaging, ISBI 2018
    CountryUnited States
    CityWashington
    Period4/4/184/7/18

    Fingerprint

    Long-Term Memory
    Image segmentation
    Short-Term Memory
    Joints
    Learning
    Pediatrics
    Recurrent neural networks
    Magnetic resonance
    Image analysis
    Computer vision
    Disease Progression
    Pixels
    Trajectories
    Magnetic Resonance Imaging
    Imaging techniques
    Long short-term memory
    Deep learning

    ASJC Scopus subject areas

    • Biomedical Engineering
    • Radiology Nuclear Medicine and imaging

    Cite this

    Gao, Y., Phillips, J. M., Zheng, Y., Min, R., Fletcher, P. T., & Gerig, G. (2018). Fully convolutional structured LSTM networks for joint 4D medical image segmentation. In 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018 (Vol. 2018-April, pp. 1104-1108). IEEE Computer Society. https://doi.org/10.1109/ISBI.2018.8363764

    Fully convolutional structured LSTM networks for joint 4D medical image segmentation. / Gao, Yang; Phillips, Jeff M.; Zheng, Yan; Min, Renqiang; Fletcher, P. Thomas; Gerig, Guido.

    2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April IEEE Computer Society, 2018. p. 1104-1108.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Gao, Y, Phillips, JM, Zheng, Y, Min, R, Fletcher, PT & Gerig, G 2018, Fully convolutional structured LSTM networks for joint 4D medical image segmentation. in 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. vol. 2018-April, IEEE Computer Society, pp. 1104-1108, 15th IEEE International Symposium on Biomedical Imaging, ISBI 2018, Washington, United States, 4/4/18. https://doi.org/10.1109/ISBI.2018.8363764
    Gao Y, Phillips JM, Zheng Y, Min R, Fletcher PT, Gerig G. Fully convolutional structured LSTM networks for joint 4D medical image segmentation. In 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April. IEEE Computer Society. 2018. p. 1104-1108 https://doi.org/10.1109/ISBI.2018.8363764
    Gao, Yang ; Phillips, Jeff M. ; Zheng, Yan ; Min, Renqiang ; Fletcher, P. Thomas ; Gerig, Guido. / Fully convolutional structured LSTM networks for joint 4D medical image segmentation. 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April IEEE Computer Society, 2018. pp. 1104-1108
    @inproceedings{a5afcec20f8e4dadb94bdfc3badee2ea,
    title = "Fully convolutional structured LSTM networks for joint 4D medical image segmentation",
    abstract = "Longitudinal medical image analysis has great potential to reveal developmental trajectories and monitor disease progression. This process relies on consistent and robust joint 4D segmentation. Traditional methods highly depend on the similarity of images over time and either build a template or assume the images could be co-registered. This process may fail when image sequences present major appearance changes. Recently, deep learning (DL) approaches have achieved state-of-the-art results for related challenges in computer vision. These approaches make use of models such as fully convolutional networks (FCNs) for end-to-end pixel-wise segmentation and recurrent neural networks (RNNs) with long short-term memory (LSTM) units for sequence-to-sequence modeling. In this paper, we propose a new DL framework called FCSLSTM for 4D image segmentation with FCNs for the spatial model and LSTM for the temporal model. This is the first DL framework with deep integration of FCNs and LSTM for joint 4D segmentation that could be trained end-to-end. Our approach achieves promising results with the demonstrated application to longitudinal pediatric magnetic resonance imaging (MRI) segmentation.",
    author = "Yang Gao and Phillips, {Jeff M.} and Yan Zheng and Renqiang Min and Fletcher, {P. Thomas} and Guido Gerig",
    year = "2018",
    month = "5",
    day = "23",
    doi = "10.1109/ISBI.2018.8363764",
    language = "English (US)",
    volume = "2018-April",
    pages = "1104--1108",
    booktitle = "2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018",
    publisher = "IEEE Computer Society",

    }

    TY - GEN

    T1 - Fully convolutional structured LSTM networks for joint 4D medical image segmentation

    AU - Gao, Yang

    AU - Phillips, Jeff M.

    AU - Zheng, Yan

    AU - Min, Renqiang

    AU - Fletcher, P. Thomas

    AU - Gerig, Guido

    PY - 2018/5/23

    Y1 - 2018/5/23

    N2 - Longitudinal medical image analysis has great potential to reveal developmental trajectories and monitor disease progression. This process relies on consistent and robust joint 4D segmentation. Traditional methods highly depend on the similarity of images over time and either build a template or assume the images could be co-registered. This process may fail when image sequences present major appearance changes. Recently, deep learning (DL) approaches have achieved state-of-the-art results for related challenges in computer vision. These approaches make use of models such as fully convolutional networks (FCNs) for end-to-end pixel-wise segmentation and recurrent neural networks (RNNs) with long short-term memory (LSTM) units for sequence-to-sequence modeling. In this paper, we propose a new DL framework called FCSLSTM for 4D image segmentation with FCNs for the spatial model and LSTM for the temporal model. This is the first DL framework with deep integration of FCNs and LSTM for joint 4D segmentation that could be trained end-to-end. Our approach achieves promising results with the demonstrated application to longitudinal pediatric magnetic resonance imaging (MRI) segmentation.

    AB - Longitudinal medical image analysis has great potential to reveal developmental trajectories and monitor disease progression. This process relies on consistent and robust joint 4D segmentation. Traditional methods highly depend on the similarity of images over time and either build a template or assume the images could be co-registered. This process may fail when image sequences present major appearance changes. Recently, deep learning (DL) approaches have achieved state-of-the-art results for related challenges in computer vision. These approaches make use of models such as fully convolutional networks (FCNs) for end-to-end pixel-wise segmentation and recurrent neural networks (RNNs) with long short-term memory (LSTM) units for sequence-to-sequence modeling. In this paper, we propose a new DL framework called FCSLSTM for 4D image segmentation with FCNs for the spatial model and LSTM for the temporal model. This is the first DL framework with deep integration of FCNs and LSTM for joint 4D segmentation that could be trained end-to-end. Our approach achieves promising results with the demonstrated application to longitudinal pediatric magnetic resonance imaging (MRI) segmentation.

    UR - http://www.scopus.com/inward/record.url?scp=85048126617&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85048126617&partnerID=8YFLogxK

    U2 - 10.1109/ISBI.2018.8363764

    DO - 10.1109/ISBI.2018.8363764

    M3 - Conference contribution

    VL - 2018-April

    SP - 1104

    EP - 1108

    BT - 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018

    PB - IEEE Computer Society

    ER -