Convex Clustering and Its Application in Identification of Molecular Subtypes of Cancers

        Recently introduced convex clustering approach formulates clustering as a convex optimization problem and ensures a globally optimal solution. However, the state-of-the-art convex clustering algorithms require large computation and memory space. In this dissertation, we first developed an efficient convex clustering algorithm named Sproga. Computer simulations and real data analysis demonstrated that Sproga outperformed several traditional clustering algorithms. Comparing with the state-of-the-art convex clustering algorithms, Sproga offered similar performance but improved the speed and memory usage both by at least one order of magnitude.  We then applied the idea of convex clustering to formulate two optimization problems for clustering with Pearson correlation and cosine dissimilarity, respectively, and accordingly developed two algorithms, named ADMMP and ADMMC, to solve these optimization problems. Computer simulations and real data analysis showed superior performance of ADMMP and ADMMC over several widely used clustering algorithms. As an application of the clustering algorithms, we used the Sprog algorithm to perform clustering analysis on the gene expression data of 11,069 human tumors across 33 different types of cancers in the database of The Cancer Genome Atlas (TCGA). The clustering analysis resulted in 22 major clusters of cancers and demonstrated that the cell-of-origin patterns played a dominate role in molecular classification of cancers. Further clustering of several groups of cancers and several individual cancers revealed a number of immune subtypes for each of these cancer groups or individual cancers with distinct survival characteristics, which may be helpful to the design of personalized treatment of cancers.