**SimpleNet** is a convolutional neural network with 13 layers. The network employs a homogeneous design utilizing 3 × 3 kernels for convolutional layer and 2 × 2 kernels for pooling operations. The only layers which do not use 3 × 3 kernels are 11th and 12th layers, these layers, utilize 1 × 1 convolutional kernels. Feature-map down-sampling is carried out using nonoverlaping 2 × 2 max-pooling. In order to cope with the problem of vanishing gradient and also over-fitting, SimpleNet also uses batch-normalization with moving average fraction of 0.95 before any [ReLU](https://paperswithcode.com/method/relu) non-linearity.

**Siren**, or **Sinusoidal Representation Network**, is a periodic activation function for implicit neural representations. Specifically it uses the sine as a periodic activation function:

$$ \Phi\left(x\right) = \textbf{W}\_{n}\left(\phi\_{n-1} \circ \phi\_{n-2} \circ \dots \circ \phi\_{0} \right) $$

Siren

Implicit Neural Representations with Periodic Activation Functions

SimpleNet

Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

The **MelGAN Residual Block** is a convolutional [residual block](https://paperswithcode.com/method/residual-block) used in the [MelGAN](https://paperswithcode.com/method/melgan) generative audio architecture. It employs residual connections with dilated convolutions. Dilations are used so that temporally far output activations of each subsequent layer has significant overlapping inputs. Receptive field of a stack of [dilated convolution](https://paperswithcode.com/method/dilated-convolution) layers increases exponentially with the number of layers. Incorporating these into the MelGAN generator allows us to efficiently increase the induced receptive fields of each output time-step. This effectively implies larger overlap in the induced receptive field of far apart time-steps, leading to better long range correlation.

Source	Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com