Network On Network (NON) is practical tabular data classification model based on deep neural network to provide accurate predictions. Various deep methods have been proposed and promising progress has been made. However, most of them use operations like neural network and factorization machines to fuse the embeddings of different features directly, and linearly combine the outputs of those operations to get the final prediction. As a result, the intra-field information and the non-linear interactions between those operations (e.g. neural network and factorization machines) are ignored. Intra-field information is the information that features inside each field belong to the same field. NON is proposed to take full advantage of intra-field information and non-linear interactions. It consists of three components: field-wise network at the bottom to capture the intra-field information, across field network in the middle to choose suitable operations data-drivenly, and operation fusion network on the top to fuse outputs of the chosen operations deeply

**AMSGrad** is a stochastic optimization method that seeks to fix a convergence issue with [Adam](https://paperswithcode.com/method/adam) based optimizers. AMSGrad uses the maximum of past squared gradients 
$v\_{t}$ rather than the exponential average to update the parameters:

$$m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)g\_{t} $$

$$v\_{t} = \beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g\_{t}^{2}$$

$$ \hat{v}\_{t} = \max\left(\hat{v}\_{t-1}, v\_{t}\right) $$

$$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{\hat{v}_{t}} + \epsilon}m\_{t}$$

AMSGrad

On the Convergence of Adam and Beyond

Network On Network for Tabular Data Classification in Real-world Applications

**WaveGAN** is a generative adversarial network for unsupervised synthesis of raw-waveform audio (as opposed to image-like spectrograms). 

The WaveGAN architecture is based off [DCGAN](https://paperswithcode.com/method/dcgan). The DCGAN generator uses the [transposed convolution](https://paperswithcode.com/method/transposed-convolution) operation to iteratively upsample low-resolution feature maps into a high-resolution image. WaveGAN modifies this transposed [convolution](https://paperswithcode.com/method/convolution) operation to widen its receptive field, using a longer one-dimensional filters of length 25 instead of two-dimensional filters of size 5x5, and upsampling by a factor of 4 instead of 2 at each layer. The discriminator is modified in a similar way, using length-25 filters in one dimension and increasing stride
from 2 to 4. These changes result in WaveGAN having the same number of parameters, numerical
operations, and output dimensionality as DCGAN. An additional layer is added afterwards to allow for more audio samples. Further changes include:

1. Flattening 2D convolutions into 1D (e.g. 5x5 2D conv becomes length-25 1D).
2. Increasing the stride factor for all convolutions (e.g. stride 2x2 becomes stride 4).
3. Removing [batch normalization](https://paperswithcode.com/method/batch-normalization) from the generator and discriminator.
4. Training using the [WGAN](https://paperswithcode.com/method/wgan)-GP strategy.

Source	Network On Network for Tabular Data Classification in Real-world Applications
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com