Almost Orthogonal Matrices

\(\operatorname{AlmostOrthogonal}(n,k,\lambda)\) is the manifold matrices with singular values in the interval \((1-\lambda, 1+\lambda)\) for a \(\lambda \in [0,1]\).

\[\operatorname{AlmostOrthogonal}(n,k,\lambda) = \{X \in \mathbb{R}^{n\times k}\:\mid\:\left|1-\sigma_i(X)\right| < \lambda,\ i=1, \dots, k\}\]

It is realized via an SVD-like factorization:

\[\begin{split}\begin{align*} \pi \colon \operatorname{St}(n,k) \times \mathbb{R}^k \times \operatorname{SO}(k) &\to \operatorname{AlmostOrthogonal}(n,k,\lambda) \\ (U, \Sigma, V) &\mapsto Uf_\lambda(\Sigma) V^\intercal \end{align*}\end{split}\]

where we have identified the vector \(\Sigma\) with a diagonal matrix in \(\mathbb{R}^{k \times k}\). The function \(f_\lambda\colon \mathbb{R} \to (1-\lambda, 1+\lambda)\) takes a function \(f\colon \mathbb{R} \to (-1, +1)\) and rescales it to be a function on \((1-\lambda, 1+\lambda)\) as

\[f_\lambda(x) = 1+\lambda f(x).\]

The function \(f_\lambda\) is then applied element-wise to the diagonal of \(\Sigma\).

If \(\lambda = 1\) is chosen, the resulting space is not a manifold, although this should not hurt optimization in practice.

Warning

In the limit \(\lambda = 0\), the resulting manifold is exactly Special Orthogonal Group. For this reason, we discourage the use of small values of \(\lambda\) as the algorithm in this class becomes numerically unstable for very small \(\lambda\). We recommend to use geotorch.SO rather than this one in this scenario.

Note

There are no restrictions in place for the image of the function \(f\). For a function \(f\) with image \([a,b]\), the function \(f_\lambda\) will take values in \([\lambda (1+a), \lambda (1+b)]\). As such, rescaling the function \(f\), one may use this class to perform optimization with singular values constrained to any prescribed interval of \(\mathbb{R}_{\geq 0}\).

class geotorch.AlmostOrthogonal(size, lam, f='sin', triv='expm')

Manifold of matrices with singular values in the interval \((1-\lambda, 1+\lambda)\).

The possible default maps are the \(\sin,\,\tanh\) functions and a scaled sigmoid. The sigmoid is scaled as \(\operatorname{scaled\_sigmoid}(x) = 2\sigma(x) - 1\) where \(\sigma\) is the usual sigmoid function. This is done so that the image of the scaled sigmoid is \((-1, 1)\).

Parameters

• size (torch.size) – Size of the tensor to be parametrized

• lam (float) – Radius of the interval. A float in the interval \((0, 1]\)

• f (str or callable or pair of callables) –

  Optional. Either:

  • One of ["scaled_sigmoid", "tanh", "sin"]

  • A callable that maps real numbers to the interval \((-1, 1)\)

  • A pair of callables such that the first maps the real numbers to \((-1, 1)\) and the second is a (right) inverse of the first

  Default: "sin"

• triv (str or callable) – Optional. A map that maps skew-symmetric matrices onto the orthogonal matrices surjectively. This is used to optimize the \(U\) and \(V\) in the SVD. It can be one of ["expm", "cayley"] or a custom callable. Default: "expm"

sample(distribution='uniform', init_=None)

Returns a randomly sampled orthogonal matrix according to the specified distribution. The options are:

• "uniform": Samples a tensor distributed according to the Haar measure on \(\operatorname{SO}(n)\)

• "torus": Samples a block-diagonal skew-symmetric matrix. The blocks are of the form \(\begin{pmatrix} 0 & b \\ -b & 0\end{pmatrix}\) where \(b\) is distributed according to init_. This matrix will be then projected onto \(\operatorname{SO}(n)\) using self.triv

The output of this method can be used to initialize a parametrized tensor that has been parametrized with this or any other manifold as:

>>> layer = nn.Linear(20, 20)
>>> M = AlmostOrthogonal(layer.weight.size(), lam=0.5)
>>> geotorch.register_parametrization(layer, "weight", M)
>>> layer.weight = M.sample()

Parameters

• distribution (string) – Optional. One of ["uniform", "torus"]. Default: "uniform"

• init_ (callable) – Optional. To be used with the "torus" option. A function that takes a tensor and fills it in place according to some distribution. See torch.init. Default: \(\operatorname{Uniform}(-\pi, \pi)\)

in_manifold(X, eps=1e-05)

Checks that a given matrix is in the manifold.

Parameters

• X (torch.Tensor or tuple) – The input matrix or matrices of shape (*, n, k).

• eps (float) – Optional. Threshold at which the singular values are considered to be zero Default: 1e-5