**Orthogonal Regularization** is a regularization technique for convolutional neural networks, introduced with generative modelling as the task in mind. Orthogonality is argued to be a desirable quality in ConvNet filters, partially because multiplication by an orthogonal matrix leaves the norm of the original matrix unchanged. This property is valuable in deep or recurrent networks, where repeated matrix multiplication can result in signals vanishing or exploding. To try to maintain orthogonality throughout training, Orthogonal Regularization encourages weights to be orthogonal by pushing them towards the nearest orthogonal manifold. The objective function is augmented with the cost:

$$ \mathcal{L}\_{ortho} = \sum\left(|WW^{T} − I|\right) $$

Where $\sum$ indicates a sum across all filter banks, $W$ is a filter bank, and $I$ is the identity matrix

**Seesaw Loss** is a loss function for long-tailed instance segmentation. It dynamically re-balances the gradients of positive and negative samples on a tail class with two complementary factors: mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. The synergy of the two factors enables Seesaw Loss to mitigate the overwhelming punishments on tail classes as well as compensate for the risk of misclassification caused by diminished penalties.

$$ L\_{seesaw}\left(\mathbf{x}\right) = - \sum^{C}\_{i=1}y\_{i}\log\left(\hat{\sigma}\_{i}\right) $$

$$ \text{with } \hat{\sigma\_{i}} = \frac{e^{z\_{i}}}{- \sum^{C}\_{j\neq{1}}\mathcal{S}\_{ij}e^{z\_{j}}+e^{z\_{i}} } $$

Here $\mathcal{S}\_{ij}$ works as a tunable balancing factor between different classes. By a careful design of $\mathcal{S}\_{ij}$, Seesaw loss adjusts the punishments on class j from positive samples of class $i$. Seesaw loss determines $\mathcal{S}\_{ij}$ by a mitigation factor and a compensation factor, as:

$$ \mathcal{S}\_{ij} =\mathcal{M}\_{ij} · \mathcal{C}\_{ij}  $$

The mitigation factor $\mathcal{M}\_{ij}$ decreases the penalty on tail class $j$ according to a ratio of instance numbers between tail class $j$ and head class $i$. The compensation factor $\mathcal{C}\_{ij}$ increases the penalty on class $j$ whenever an instance of class $i$ is misclassified to class $j$.

Seesaw Loss

Seesaw Loss for Long-Tailed Instance Segmentation

Orthogonal Regularization

Neural Photo Editing with Introspective Adversarial Networks

**WaveNet** is an audio generative model based on the [PixelCNN](https://paperswithcode.com/method/pixelcnn) architecture. In order to deal with long-range temporal dependencies needed for raw audio generation, architectures are developed based on dilated causal convolutions, which exhibit very large receptive fields.

The joint probability of a waveform $\vec{x} = \{ x_1, \dots, x_T \}$ is factorised as a product of conditional probabilities as follows:

$$p\left(\vec{x}\right) = \prod_{t=1}^{T} p\left(x_t \mid x_1, \dots ,x_{t-1}\right)$$

Each audio sample $x_t$ is therefore conditioned on the samples at all previous timesteps.

Source	Neural Photo Editing with Introspective Adversarial Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com