Multimodal Deep Learning for Breast Cancer Neoadjuvant Therapy Outcomes

Breast cancer affects about one in eight women annually, and is the most common type of cancer in all women. Oftentimes prior to invasive surgery, radiologists and oncologists will prescribe patients with neoadjuvant systemic therapy (NST), a treatment they give prior to invasive and long-term treatments like surgery. Because NST can involve harsh chemotherapy and other medications, it is critical to predict whether or not a patient will respond to NST. The primary metric physicians use for this task is the pathological Complete Response (pCR), which is based on a patient's breast histology. In this work, we aim to present a statistical and Deep Learning-based approach for predicting a patient's response to NST. We work with the ISPY-1 dataset's MRI images and clinical datapoints to compose various subnetworks prior to a final fusion classifier. For the unstructured MRI data, we investigate the performance of both a 3D CNN and a CNN-LSTM on imaging slices. For the structured data, we investigate the role of various statistical approaches ranging from random forests to support vector machines (SVM)s. In the end, we show that a combined CNN-LSTM and SVM approach provides the highest performance, with an accuracy of about 84%. This work has real-world implications, because it can help with augmenting radiologist and oncologist decisions on how to present patient prognoses and which treatment options they recommend to patients to ensure positive patient outcomes.