Stefano Guarnieri, Francesco Piazza, and Aurelio Uncini 
"Multilayer Feedforward Networks with Adaptive Spline Activation Function," 
 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 3, MAY 1999

Abstract — In this paper, a new adaptive spline activation function neural network (ASNN) is presented. Due to the ASNN’s high representation capabilities, networks with a small number of interconnections can be trained to solve both pattern recognition and data processing real-time problems. The main idea is to use a Catmull–Rom cubic spline as the neuron’s activation function, which ensures a simple structure suitable for both software and hardware implementation. Experimental results demonstrate improvements in terms of generalization capability
and of learning speed in both pattern recognition and data processing tasks.
Index Terms— Adaptive activation functions, function shape autotuning, generalization, generalized sigmoidal functions, multilayer
perceptron, neural networks, spline neural networks.

**FastSpeech 2s** is a text-to-speech model that abandons mel-spectrograms as intermediate output completely and directly generates speech waveform from text during inference. In other words there is no cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder). FastSpeech 2s generates waveform conditioning on intermediate hidden, which makes it more compact in inference by discarding the mel-spectrogram decoder.

Two main design changes are made to the waveform decoder. 

First, considering that the phase information is difficult to predict using a variance predictor, [adversarial training](https://paperswithcode.com/methods/category/adversarial-training)  is used in the waveform decoder to force it to implicitly recover the phase information by itself. 

Secondly, the mel-spectrogram decoder of [FastSpeech 2](https://paperswithcode.com/method/fastspeech-2) is leveraged, which is trained on the full text sequence to help on the text feature extraction. As shown in the Figure, the waveform decoder is based on the structure of [WaveNet](https://paperswithcode.com/method/wavenet) including non-causal convolutions and gated activation. The waveform decoder takes a sliced hidden sequence corresponding to a short audio clip as input and upsamples it with transposed 1D-convolution to match the length of audio clip. The discriminator in the adversarial training adopts the same structure in Parallel WaveGAN, which consists of ten layers of non-causal [dilated 1-D convolutions](https://paperswithcode.com/method/dilated-convolution) with [leaky ReLU](https://paperswithcode.com/method/leaky-relu) activation function. The waveform decoder is optimized by the multi-resolution STFT loss and the [LSGAN discriminator](https://paperswithcode.com/method/lsgan) loss following Parallel WaveGAN. 

In inference, the mel-spectrogram decoder is discarded and only the waveform decoder is used to synthesize speech audio.

FastSpeech 2s

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

ASAF

Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

While [standard convolution](https://paperswithcode.com/method/convolution) performs the channelwise and spatial-wise computation in one step, **Depthwise Separable Convolution**  splits the computation into two steps: [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) applies a single convolutional filter per each input channel and [pointwise convolution](https://paperswithcode.com/method/pointwise-convolution) is used to create a linear combination of the output of the depthwise convolution. The comparison of standard convolution and depthwise separable convolution is shown to the right.

Credit: [Depthwise Convolution Is All You Need for Learning Multiple Visual Domains](https://paperswithcode.com/paper/depthwise-convolution-is-all-you-need-for)

Source	Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com