Cancer genetic diagnosis using six types of microarray data, representative of horizontal data
Shuichi Niimura, Professor Emeritus, Faculty of Economics, Seikei University
According to Professor Golub et al. of Harvard Medical School, "cancer genetic diagnosis using microarray data, which is a representative example of horizontal data" has been conducted for over 30 years, but no clear conclusion has been reached (Problem 5 of discriminant analysis). Many statisticians point out problems that cannot be addressed statistically.
1. Small n and large p problem, that is, typical horizontal data with p>>n. For example, when performing discrimination or regression of 100 cancer and normal patients with around 10,000 data, variable selection, etc. is an NP-hard problem.
2. Cancer genes that are useful for discrimination are hidden in the remaining high-dimensional noise and cannot be separated well.
And so on.
After graduating from university in 1971, the author spent four years at the Osaka Prefectural Adult Disease Center developing the logic of an automatic electrocardiogram diagnosis system using Fisher's LDF, but it did not match the branching logic of doctors. This experience led him to believe that statistical discrimination theory is inappropriate for medical diagnosis, and he developed an optimal linear discriminant function based on the Minimum Number of Misclassification (MNM) criterion using mathematical programming, as well as the "100-fold cross-validation method for small samples (Method 1)" and the "Matryoshka Feature Selection Method (Method 2)," and solved five problems in discriminant analysis. Problem 5, which had taken 30 years to solve, was solved in just 54 days with Method 2. Microarray data can be completely discriminated between normal and abnormal with a set of genes far fewer than the number of data items, n (100), and can be completely separated into an exclusive union of several dozen such sets of genes (Small Matryoshka, SM) and the remaining high-dimensional noise space. These were published by Springer. We then analyzed all the SMs obtained using JMP's one-way analysis of variance, cluster analysis, PCA, and statistical discriminant functions, and report the world's first successful creation of a cancer malignancy index. These results have been published in an Amazon Kindle edition (From Cancer Gene Analysis to Cancer Gene Diagnosis, 1,200 yen).
