Linear methods of dependence estimation from empirical data

The commonly adopted approach to inferring, from a set of empirical data, how a hidden characteristic of real-world objects of a certain kind depends on another characteristic, which is observable, are based on the compactness hypothesis – if two objects are close to each in the sense of some accepted quantitative judgment on similarity or dissimilarity, the values of the hidden characteristic are expected to be close to each other, too. In their turn, the most developed methods of algorithmic realization of the compactness hypothesis follow from the assumption that the objects under consideration are elements of a linear space, in which dissimilarity between them is formally expressed as a natural metric. Methods of dependence estimation, that imply embedding the set of real-world objects into a linear space, are said here to be linear methods. A unified mathematical idea is formulated, which covers all the known linear methods of dependence estimation from empirical data, i.e., from a training set. In particular, we consider the unity and distinction of such popular methods of pattern recognition learning as Support Vector Machine (SVM) and Logistic Regression.