Almost Orthogonal Matrices
\(\operatorname{AlmostOrthogonal}(n,k,\lambda)\) is the manifold matrices with singular values in the interval \((1-\lambda, 1+\lambda)\) for a \(\lambda \in [0,1]\).
It is realized via an SVD-like factorization:
where we have identified the vector \(\Sigma\) with a diagonal matrix in \(\mathbb{R}^{k \times k}\). The function \(f_\lambda\colon \mathbb{R} \to (1-\lambda, 1+\lambda)\) takes a function \(f\colon \mathbb{R} \to (-1, +1)\) and rescales it to be a function on \((1-\lambda, 1+\lambda)\) as
The function \(f_\lambda\) is then applied element-wise to the diagonal of \(\Sigma\).
If \(\lambda = 1\) is chosen, the resulting space is not a manifold, although this should not hurt optimization in practice.
Warning
In the limit \(\lambda = 0\), the resulting manifold is exactly Special Orthogonal Group. For this reason, we discourage the use of small values of \(\lambda\) as the algorithm in this class becomes numerically unstable for very small \(\lambda\). We recommend to use geotorch.SO
rather than this one in this scenario.
Note
There are no restrictions in place for the image of the function \(f\). For a function \(f\) with image \([a,b]\), the function \(f_\lambda\) will take values in \([\lambda (1+a), \lambda (1+b)]\). As such, rescaling the function \(f\), one may use this class to perform optimization with singular values constrained to any prescribed interval of \(\mathbb{R}_{\geq 0}\).
- class geotorch.AlmostOrthogonal(size, lam, f='sin', triv='expm')[source]
Manifold of matrices with singular values in the interval \((1-\lambda, 1+\lambda)\).
The possible default maps are the \(\sin,\,\tanh\) functions and a scaled sigmoid. The sigmoid is scaled as \(\operatorname{scaled\_sigmoid}(x) = 2\sigma(x) - 1\) where \(\sigma\) is the usual sigmoid function. This is done so that the image of the scaled sigmoid is \((-1, 1)\).
- Parameters
size (torch.size) – Size of the tensor to be parametrized
lam (float) – Radius of the interval. A float in the interval \((0, 1]\)
f (str or callable or pair of callables) –
Optional. Either:
One of
["scaled_sigmoid", "tanh", "sin"]
A callable that maps real numbers to the interval \((-1, 1)\)
A pair of callables such that the first maps the real numbers to \((-1, 1)\) and the second is a (right) inverse of the first
Default:
"sin"
triv (str or callable) – Optional. A map that maps skew-symmetric matrices onto the orthogonal matrices surjectively. This is used to optimize the \(U\) and \(V\) in the SVD. It can be one of
["expm", "cayley"]
or a custom callable. Default:"expm"
- sample(distribution='uniform', init_=None)[source]
Returns a randomly sampled orthogonal matrix according to the specified
distribution
. The options are:"uniform"
: Samples a tensor distributed according to the Haar measure on \(\operatorname{SO}(n)\)"torus"
: Samples a block-diagonal skew-symmetric matrix. The blocks are of the form \(\begin{pmatrix} 0 & b \\ -b & 0\end{pmatrix}\) where \(b\) is distributed according toinit_
. This matrix will be then projected onto \(\operatorname{SO}(n)\) usingself.triv
The output of this method can be used to initialize a parametrized tensor that has been parametrized with this or any other manifold as:
>>> layer = nn.Linear(20, 20) >>> M = AlmostOrthogonal(layer.weight.size(), lam=0.5) >>> geotorch.register_parametrization(layer, "weight", M) >>> layer.weight = M.sample()
- Parameters
distribution (string) – Optional. One of
["uniform", "torus"]
. Default:"uniform"
init_ (callable) – Optional. To be used with the
"torus"
option. A function that takes a tensor and fills it in place according to some distribution. See torch.init. Default: \(\operatorname{Uniform}(-\pi, \pi)\)
- in_manifold(X, eps=1e-05)
Checks that a given matrix is in the manifold.
- Parameters
X (torch.Tensor or tuple) – The input matrix or matrices of shape
(*, n, k)
.eps (float) – Optional. Threshold at which the singular values are considered to be zero Default:
1e-5