**ProxylessNet-Mobile** is a convolutional neural architecture learnt with the [ProxylessNAS](https://paperswithcode.com/method/proxylessnas) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) algorithm that is optimized for mobile devices. It uses inverted residual blocks (MBConvs) from [MobileNetV2](https://paperswithcode.com/method/mobilenetv2) as its basic building block.

Please enter a description about the method here

Apollo

Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

ProxylessNet-Mobile

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

The **Adaptive Attention Span Transformer** is a Transformer that utilises an improvement to the self-attention layer called [adaptive masking](https://paperswithcode.com/method/adaptive-masking) that allows the model to choose its own context size. This results in a network where each attention layer gathers information on their own context. This allows for scaling to input sequences of more than 8k tokens.

Their proposals are based on the observation that, with the dense attention of a traditional [Transformer](https://paperswithcode.com/method/transformer), each attention head shares the same attention span $S$ (attending over the full context). But many attention heads can specialize to more local context (others look at the longer sequence). This motivates the need for a variant of self-attention that allows the model to choose its own context size (adaptive masking - see components).

Source	ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com