Some readers may remember a post series I have written in this blog about the reconstruction of human faces with a CNN-based Autoencoder. I could show that the information in the latent space of the Autoencoder is given in form of a core of a Multivariate Normal Distribution [MVN].

This did not surprise too much as there are good reasons to assume that facial features on average, but in particular across rather symmetric celebrity faces follow Gaussian distributions. Hundreds of encoded features together form a MVN-distribution in a latent space of hundreds of dimensions. The Encoder part of a CNN-based Autoencoder is a pattern extraction machine – and there is no simpler pattern in multiple dimensions than a (off-center) MVN! A MVN’s multidimensional and concentric contour surfaces are ellipsoids, which have an algebraic description in form of quadratic forms. In case of the MVN defined by the inverse of the covariance matrix.

During the named series, I have extensively used that fact that the projections of a multidimensional MVN down to a coordinate planes result in 2-dimensional bivariate MVNs. The elements of the (2×2)-covariance matrix of the various 2-dimensional projected distributions could simply be picked from (nxn) covariance matrix of the original MVN – by a simple selection process. I had taken this procedure as granted, as it had been claimed in some publications. And it worked very well … See e.g.

• Autoencoders and latent space fragmentation – X – a method to create suitable latent vectors for the generation of human face images

and links therein to other posts. The projection of course affects the (n-1)-dimensional ellipsoidal and concentric contour surfaces of a MVN and maps them onto (p-1)-dimensional contour ellipses of the projected 2-dimensional MVNs. For respective images see this post:

• Autoencoders and latent space fragmentation – VIII – approximation of the latent vector distribution by a multivariate normal distribution and ellipses

Last weeks I looked a bit deeper into the mathematics of orthogonal projections of multidimensional ellipsoids onto sub-spaces of the ℝⁿ. It came a bit of a surprise to me that the math behind the projections of figures controlled by quadratic forms is relatively complicated. In the general case of the projection to a p-dimensional sub-space, the quadratic form matrix for the ellipsoidal hull of the projection image is a so called Schur complement of the original ellipsoid’s quadratic form matrix.

Fortunately, the relation between the inverse matrices of the quadratic forms for the ellipsoids could be established in a way that is fully consistent with the mapping of covariance matrices of MVNs and and related matrices of their projection images. However and in contrast to other publications, I found that a solid proof requires some Linear Algebra around Schur complements.

Readers interested in MVNs and their mathematical properties for statistical analysis e.g. in Machine Learning contexts may find detailed information in the following articles of mine:

Orthogonal projections of multidimensional ellipsoids

• Post I: Points on the ellipsoid that give us the surface points of the projection – with illustrations
• Post II: The surface of the projection image is a lower dimensional ellipsoid
• Post III: Arbitrary vectors for the projection direction and cuts of the unit sphere
• Post IV: The relation to a Schur complement of the quadratic form matrix
• Post V: Relation between the inverse matrices of the involved quadratic forms and of respective covariance matrices

However, basic Linear Algebra knowledge is required! The articles should also be interesting for physicists.