Richard J. Chen

I am a Ph.D. Candidate advised by Faisal Mahmood at Harvard University, and also within Brigham and Women’s Hospital, Dana-Farber Cancer Institute, and the Broad Institute.

Prior to starting my Ph.D., I obtained my B.S/M.S. in Biomedical Engineering and Computer Science at Johns Hopkins University, where I worked with Nicholas Durr and Alan Yuille. I am fortunate to have also spent time working at worked at Apple Inc. in the Health Special Project and Applied Machine Learning Groups (with Belle Tseng and Andrew Trister), Microsoft Research in the BioML Group (with Rahul Gopalkrishnan), and at the National Institutes of Health in NIBIB (with Richard Leapman).

Research Highlights

Multimodal Integration: Multimodal learning has emerged as an interdisciplinary field to solve many core problems in machine perception, human-computer interaction, and recently in biology & medicine, in which there is often an enormous wealth of multimodal data collected in parallel to study the same underlying disease. Since first starting out in research, I have a range of experiences working on multimodal learning for integrating: 1) multimodal sensor streams from the Apple Watch and iPhone Data to predict mild cognitive decline, 2) RGB and depth images for non-polyploidal lesion classification and SLAM in surgical robotics, and 3) pathology images and genomics for cancer prognosis.
Weakly-Supervised & Set-Based Deep Learning: Though deep learning has revolutionized computer vision in many disciplines, gigapixel whole-slide imaging (WSI) in computational pathology is a complex computer vision domain that renders traditional, Convnet-based supervised learning approaches infeasible. To address this issue, I have been working on interpretting large gigapixel images as permutation-invariant sets (or bags in MIL literature), and then developing set-based learning algorithms for weakly-supervised learning on WSIs.
Synthetic Data Generation & Domain Adaption: “What constitutes authenticity, and how would the lack of authenticity shape our perception of reality?” The science fiction American writer Philip K. Dick posited similar questions throughout his literary career and, in particular, in his 1972 essay “How to build a universe that doesn’t fall apart two days later”. I am interested in: using synthetic data for domain adaptation / generalization, developing synthetic environments for simulating challenging scenarios for neural networks, as well as the the policy challenges in training AI-SaMDs with synthetic data,

Please feel free to contact me through email if you have any questions and interest in collaborating!

Recent News

Aug, 2021	Excited to announce that the preprint for PORPOISE, our Pathology-Omic Research Platform for Integrated Survival Estimation, is out (with its associated demo)!
Jul, 2021	Two papers, Patch-GCN and Multimodal Co-Attention Transformers (MCAT), were accepted into MICCAI 2021 and ICCV 2021 respectively.
Jun, 2021	Joined Microsoft Research as an PhD Research Intern, working with Rahul Gopalkrishnan in the BioML Group.
Jun, 2021	Passed my Qualifying Exam, and am now officially a PhD Candidate!

Select Publications

Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams

Richard J. Chen, Filip Jankovic, Nikki Marinsek, Luca Foschini, Lampros Kourtis, Alessio Signorini, Melissa Pugh, Jie Shen, Roy Yaari, Vera Maljkovic, Marc Sunga, Han Hee Song, Hyun Joon Jung, Belle Tseng, and Andrew Trister

In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2019

Oral Presentation & Best Paper Runner-Up

Abs HTML Oral News Cite

The ubiquity and remarkable technological progress of wearable consumer devices and mobile-computing platforms (smart phone, smart watch, tablet), along with the multitude of sensor modalities available, have enabled continuous monitoring of patients and their daily activities. Such rich, longitudinal information can be mined for physiological and behavioral signatures of cognitive impairment and provide new avenues for detecting MCI in a timely and cost-effective manner. In this work, we present a platform for remote and unobtrusive monitoring of symptoms related to cognitive impairment using several consumer-grade smart devices. We demonstrate how the platform has been used to collect a total of 16TB of data during the Lilly Exploratory Digital Assessment Study, a 12-week feasibility study which monitored 31 people with cognitive impairment and 82 without cognitive impairment in free living conditions. We describe how careful data unification, time-alignment, and imputation techniques can handle missing data rates inherent in real-world settings and ultimately show utility of these disparate data in differentiating symptomatics from healthy controls based on features computed purely from device data.
```
@inproceedings{chen2019developing,
  doi = {10.1145/3292500.3330690},
  url = {https://machinelearning.apple.com/research/developing-measures-of-cognitive-impairment-in-the-real-world-from-consumer-grade-multimodal-sensor-streams},
  year = {2019},
  month = jul,
  publisher = {{ACM}},
  author = {Chen, Richard J. and Jankovic, Filip and Marinsek, Nikki and Foschini, Luca and Kourtis, Lampros and Signorini, Alessio and Pugh, Melissa and Shen, Jie and Yaari, Roy and Maljkovic, Vera and Sunga, Marc and Song, Han Hee and Jung, Hyun Joon and Tseng, Belle and Trister, Andrew},
  title = {Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams},
  booktitle = {Proceedings of the 25th {ACM} {SIGKDD} International Conference on Knowledge Discovery {\&} Data Mining},
  honor = {Oral Presentation & Best Paper Runner-Up},
  press = {https://www.technologyreview.com/2019/08/08/102821/your-apple-watch-might-one-day-spot-if-youre-developing-alzheimers/},
  abbr = {chen2019developing.png},
  oral = {https://www.youtube.com/watch?v=H_wTI4LUW7A},
  selected = {true}
}
```
Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Richard J. Chen, Ming Y. Lu, Jingwen Wang, Drew F. K. Williamson, Scott J. Rodig, Neal I. Lindeman, and Faisal Mahmood

IEEE Transactions on Medical Imaging 2020

Top 5 Posters, NVIDIA GTC 2020

Abs arXiv Oral News Code Cite

Cancer diagnosis, prognosis, and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNASeq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gatingbased attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired wholeslide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion.
```
@article{chen2019pathomic,
  doi = {10.1109/tmi.2020.3021387},
  year = {2020},
  publisher = {Institute of Electrical and Electronics Engineers ({IEEE})},
  pages = {1--1},
  author = {Chen, Richard J. and Lu, Ming Y. and Wang, Jingwen and Williamson, Drew F. K. and Rodig, Scott J. and Lindeman, Neal I. and Mahmood, Faisal},
  title = {Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis},
  journal = {{IEEE} Transactions on Medical Imaging},
  arxiv = {1912.08937},
  abbr = {chen2019pathomic.png},
  honor = {Top 5 Posters, NVIDIA GTC 2020},
  oral = {https://www.youtube.com/watch?v=TrjGEUVX5YE},
  code = {https://github.com/mahmoodlab/PathomicFusion},
  press = {https://blogs.nvidia.com/blog/2019/11/07/harvard-pathology-lab-data-fusion-ai-cancer/},
  selected = {true}
}
```
Synthetic Data in Machine Learning for Medicine and Healthcare

Richard J. Chen, Ming Y. Lu, Tiffany Y. Chen, Drew F. K. Williamson, and Faisal Mahmood

Nature Biomedical Engineering 2021

Abs HTML Cite

The proliferation of synthetic data in artificial intelligence for medicine and healthcare raises concerns about the vulnerabilities of the software and the challenges of current policy.
```
@article{chen2021synthetic,
  doi = {10.1038/s41551-021-00751-8},
  url = {https://doi.org/10.1038/s41551-021-00751-8},
  year = {2021},
  month = jun,
  publisher = {Springer Science and Business Media {LLC}},
  volume = {5},
  number = {6},
  pages = {493--497},
  author = {Chen, Richard J. and Lu, Ming Y. and Chen, Tiffany Y. and Williamson, Drew F. K. and Mahmood, Faisal},
  title = {Synthetic Data in Machine Learning for Medicine and Healthcare},
  journal = {Nature Biomedical Engineering},
  abbr = {chen2021synthetic.png},
  selected = {true}
}
```
Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning

Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, Zahra Noor, and Faisal Mahmood

arXiv preprint arXiv:2108.02278 2021

Abs arXiv Code Demo Cite

The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq abundance, copy number variation, and mutation data from 5,720 patients across 14 major cancer types. Our interpretable, weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities for predicting outcomes and discover prognostic features from these modalities that corroborate with poor and favorable outcomes via multimodal interpretability. We compared our model with unimodal deep learning models trained on histology slides and molecular profiles alone, and demonstrate performance increase in risk stratification on 9 out of 14 cancers. In addition, we analyze morphologic and molecular markers responsible for prognostic predictions across all cancer types. All analyzed data, including morphological and molecular correlates of patient prognosis across the 14 cancer types at a disease and patient level are presented in an interactive open-access database (http://pancancer.mahmoodlab.org) to allow for further exploration and prognostic biomarker discovery. To validate that these model explanations are prognostic, we further analyzed high attention morphological regions in WSIs, which indicates that tumor-infiltrating lymphocyte presence corroborates with favorable cancer prognosis on 9 out of 14 cancer types studied.
```
@article{chen2021pan,
  title = {Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning},
  author = {Chen, Richard J and Lu, Ming Y and Williamson, Drew FK and Chen, Tiffany Y and Lipkova, Jana and Shaban, Muhammad and Shady, Maha and Williams, Mane and Joo, Bumjin and Noor, Zahra and Mahmood, Faisal},
  journal = {arXiv preprint arXiv:2108.02278},
  year = {2021},
  arxiv = {2108.02278},
  code = {https://github.com/mahmoodlab/PORPOISE},
  demo = {http://pancancer.mahmoodlab.org},
  selected = {true},
  abbr = {chen2021pan.png}
}
```