xnn.hybrid.models.spookynet.orthogonal_random_features#

xnn.hybrid.models.spookynet.orthogonal_random_features(n_features, dim, generator=None)[source]#

Projection matrix of positive orthogonal random features (FAVOR+).

Blocks of dim mutually orthogonal unit vectors (QR of Gaussian matrices) are stacked to n_features rows and each row is rescaled to the length of an independent Gaussian vector, so the rows are marginally N(0, I) but orthogonal within a block (Choromanski et al., ICLR 2021).

Parameters:

n_features (int) – Number of random features m.
dim (int) – Dimension d of the queries and keys.
generator (torch.Generator, optional) – Source of randomness.

Returns:

(dim, n_features), float64.

Return type:

Tensor