Manifold alignment is a class of machine learning algorithms that produce projections between sets of data, given that the original data sets lie on a common manifold.
The concept was first introduced as such by Ham, Lee, and Saul in 2003,[1][non-primary source needed] adding a manifold constraint to the general problem of correlating sets of high-dimensional vectors.
[2] Manifold alignment assumes that disparate data sets produced by similar generating processes will share a similar underlying manifold representation.
By learning projections from each original space to the shared manifold, correspondences are recovered and knowledge from one domain can be transferred to another.
Most manifold alignment techniques consider only two data sets, but the concept extends to arbitrarily many initial data sets.
Consider the case of aligning two data sets,
Manifold alignment algorithms attempt to project both
into a new d-dimensional space such that the projections both minimize distance between corresponding points and preserve the local manifold structure of the original data.
The projection functions are denoted:
represent the binary correspondence matrix between points in
represent pointwise similarities within data sets.
This is usually encoded as the heat kernel of the adjacency matrix of a k-nearest neighbor graph.
Finally, introduce a coefficient
, which can be tuned to adjust the weight of the 'preserve manifold structure' goal, versus the 'minimize corresponding point distances' goal.
With these definitions in place, the loss function for manifold alignment can be written:
Solving this optimization problem is equivalent to solving a generalized eigenvalue problem using the graph laplacian[3] of the joint matrix, G:
{\displaystyle G=\left[{\begin{array}{cc}\mu S_{X}&\left(1-\mu \right)W\\\left(1-\mu \right)W^{T}&\mu S_{Y}\end{array}}\right]}
The algorithm described above requires full pairwise correspondence information between input data sets; a supervised learning paradigm.
However, this information is usually difficult or impossible to obtain in real world applications.
Recent work has extended the core manifold alignment algorithm to semi-supervised [4] , unsupervised [5] , and multiple-instance [6] settings.
The algorithm described above performs a "one-step" alignment, finding embeddings for both data sets at the same time.
A similar effect can also be achieved with "two-step" alignments [7] [8] , following a slightly modified procedure: Manifold alignment can be used to find linear (feature-level) projections, or nonlinear (instance-level) embeddings.
While the instance-level version generally produces more accurate alignments, it sacrifices a great degree of flexibility as the learned embedding is often difficult to parameterize.
Feature-level projections allow any new instances to be easily embedded in the manifold space, and projections may be combined to form direct mappings between the original data representations.
These properties are especially important for knowledge-transfer applications.
Manifold alignment is suited to problems with several corpora that lie on a shared manifold, even when each corpus is of a different dimensionality.
Many real-world problems fit this description, but traditional techniques are not able to take advantage of all corpora at the same time.
Manifold alignment also facilitates transfer learning, in which knowledge of one domain is used to jump-start learning in correlated domains.
Applications of manifold alignment include: