Abraham Apfel of the Department of Biostatistics defends his dissertation on "A Stability Analysis of Sparse K-means"
Graduate faculty of the University and all other interested parties are invited to attend.
Sparse K-Means clustering is an established method of simultaneously excluding uninformative features and clustering the observations. This is particularly useful in a high dimensional setting such as micro-array. However, the subsets of features selected is often inaccurate when there are overlapping clusters, which adversely affects the clustering results. The current method also tends to be inconsistent, yielding high variability in the number of features selected.
We propose to combine a stability analysis with Sparse K-Means via performing Sparse K-Means on subsamples of the original data to yield accurate and consistent feature selection. After reducing the dimensions to an accurate, small subset of features, the standard K-Means clustering procedure is performed to yield accurate clustering results. Our method demonstrates improvement in accuracy and reduction in variability providing consistent feature selection as well as a reduction in the clustering error rate (CER) from the previously established Sparse K-Means clustering methodology. Our method continues to perform well in situations with strong cluster overlap where the previous methods were unsuccessful.
Public health significance: Clustering analysis on transcriptomic data has shown success in disease phenotyping and subgroup discovery. However, with current methodology, there is a lack of confidence in terms of the accuracy and reliability of the results, as they can be highly variable. With our methodology, we hope to allow the researcher to use cluster analysis to achieve disease phenotyping and subgroup discovery with confidence that they are uncovering accurate and stable results.