Fixed Rank Matrices

\(\operatorname{FixedRank}(n,k,r)\) is the manifold of matrices of rank equal to \(r\), for a given \(r \leq \min\{n, k\}\):

\[\operatorname{FixedRank}(n,k,r) = \{X \in \mathbb{R}^{n\times k}\:\mid\:\operatorname{rank}(X) = r\}\]

It is realized via an SVD-like factorization:

\[\begin{split}\begin{align*} \pi \colon \operatorname{St}(n,r) \times \mathbb{R}^r \times \operatorname{St}(k, r) &\to \operatorname{FixedRank}(n,k,r) \\ (U, \Sigma, V) &\mapsto Uf(\Sigma)V^\intercal \end{align*}\end{split}\]

where we have identified the vector \(\Sigma\) with a diagonal matrix in \(\mathbb{R}^{r \times r}\). The function \(f\colon \mathbb{R} \to (0, \infty)\) is applied element-wise to the diagonal. By default, the softmax function is used

\[\begin{split}\begin{align*} \operatorname{softmax} \colon \mathbb{R} &\to (0, \infty) \\ x &\mapsto \log(1+\exp(x)) + \varepsilon \end{align*}\end{split}\]

where we use a small \(\varepsilon > 0\) for numerical stability.

Note

For practical applications, it will be almost always more convenient to use the class LowRank, as it is less restrictive, and most of the times it will converge to a max-rank solution anyway.

class geotorch.FixedRank(size, rank, f='softplus', triv='expm')[source]

Manifold of non-square matrices of rank equal to rank

Parameters

  • size (torch.size) – Size of the tensor to be parametrized

  • rank (int) – Rank of the matrices. It has to be less or equal to \(\min(\texttt{size}[-1], \texttt{size}[-2])\)

  • f (str or callable or pair of callables) –

    Optional. Either:

    • "softplus"

    • A callable that maps real numbers to the interval \((0, \infty)\)

    • A pair of callables such that the first maps the real numbers onto \((0, \infty)\) and the second is a (right) inverse of the first

    Default: "softplus"

  • triv (str or callable) – Optional. A map that maps skew-symmetric matrices onto the orthogonal matrices surjectively. This is used to optimize the \(U\) and \(V\) in the SVD. It can be one of ["expm", "cayley"] or a custom callable. Default: "expm"

sample(init_=<function xavier_normal_>, eps=5e-06, factorized=False)[source]

Returns a randomly sampled matrix on the manifold by sampling a matrix according to init_ and projecting it onto the manifold.

If the sampled matrix has more than self.rank small singular values, the smallest ones are clamped to be at least eps in absolute value.

The output of this method can be used to initialize a parametrized tensor that has been parametrized with this or any other manifold as:

>>> layer = nn.Linear(20, 20)
>>> M = FixedRank(layer.weight.size(), rank=6)
>>> geotorch.register_parametrization(layer, "weight", M)
>>> layer.weight = M.sample()
Parameters

  • init_ (callable) – Optional. A function that takes a tensor and fills it in place according to some distribution. See torch.init. Default: torch.nn.init.xavier_normal_

  • eps (float) – Optional. Minimum singular value of the sampled matrix. Default: 5e-6

in_manifold(X, eps=1e-05)

Checks that a given matrix is in the manifold.

Parameters

  • X (torch.Tensor or tuple) – The input matrix or matrices of shape (*, n, k).

  • eps (float) – Optional. Threshold at which the singular values are considered to be zero Default: 1e-5