[Neural architecture search](https://paperswithcode.com/method/neural-architecture-search) (NAS) has witnessed prevailing success in image classification and (very recently) segmentation tasks. In this paper, we present the first preliminary study on introducing the NAS algorithm to generative adversarial networks (GANs), dubbed AutoGAN. The marriage of NAS and GANs faces its unique challenges. We define the search space for the generator architectural variations and use an RNN controller to guide the search, with parameter sharing and dynamic-resetting to accelerate the process. Inception score is adopted as the reward, and a multi-level search strategy is introduced to perform NAS in a progressive way.

**SimAdapter** is a module for explicitly learning knowledge from adapters. SimAdapter aims to learn the similarities between the source and target languages during fine-tuning using the adapters, and the similarity is based on an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1). 

The detailed composition of the SimAdapter is shown in the Figure. By taking the language-agnostic representations from the backbone model as the query, and the language-specific outputs from multiple adapter as the keys and values, the final output for SimAdapter over attention are computed as (For notation simplicity, we omit the layer index $l$ below):

$$
\operatorname{SimAdapter}\left(\mathbf{z}, \mathbf{a}\_{\left$S\_{1}, S\_{2}, \ldots, S\_{N}\right$}\right)=\sum_{i=1}^{N} \operatorname{Attn}\left(\mathbf{z}, \mathbf{a}\_{S\_{i}}\right) \cdot\left(\mathbf{a}\_{S\_{i}} \mathbf{W}\_{V}\right)
$$

where SimAdapter $(\cdot)$ and $\operatorname{Attn}(\cdot)$ denotes the SimAdapter and attention operations, respectively. Specifically, the attention operation is computed as:

$$
\operatorname{Attn}(\mathbf{z}, \mathbf{a})=\operatorname{Softmax}\left(\frac{\left(\mathbf{z} \mathbf{W}\_{Q}\right)\left(\mathbf{a} \mathbf{W}\_{K}\right)^{\top}}{\tau}\right)
$$

where $\tau$ is the temperature coefficient, $\mathbf{W}\_{Q}, \mathbf{W}\_{K}, \mathbf{W}\_{V}$ are attention matrices. Note that while $\mathbf{W}\_{Q}, \mathbf{W}\_{K}$ are initialized randomly, $\mathbf{W}\_{V}$ is initialized with a diagonal of ones and the rest of the matrix with small weights $(1 e-6)$ to retain the adapter representations. Furthermore, a regularization term is introduced to avoid drastic feature changes:

$$
\mathcal{L}\_{\mathrm{reg}}=\sum\_{i, j}\left(\left(\mathbf{I}\_{V}\right)\_{i, j}-\left(\mathbf{W}\_{V}\right)_{i, j}\right)^{2}
$$

where $\mathbf{I}\_{V}$ is the identity matrix with the same size as $\mathbf{W}\_{V}$

SimAdapter

Exploiting Adapters for Cross-lingual Low-resource Speech Recognition

AutoGAN

AutoGAN: Neural Architecture Search for Generative Adversarial Networks

The **Simple Neural Attention Meta-Learner**, or **SNAIL**, combines the benefits of temporal convolutions and attention to solve meta-learning tasks. They introduce positional dependence through temporal convolutions to make the model applicable to reinforcement tasks - where the observations, actions, and rewards are intrinsically sequential. They also introduce attention in order to provide pinpoint access over an infinitely large context. SNAIL is constructing by combining the two: we use temporal convolutions to produce the context over which we use a causal attention operation.

Source	AutoGAN: Neural Architecture Search for Generative Adversarial Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com