**nnFormer**, or **not-another transFormer**, is a semantic segmentation model with an interleaved architecture based on empirical combination of self-attention and [convolution](https://paperswithcode.com/method/convolution). Firstly, a light-weight convolutional embedding layer ahead is used ahead of [transformer](https://paperswithcode.com/method/transformer) blocks. In comparison to directly flattening raw pixels and applying 1D pre-processing, the convolutional embedding layer encodes precise (i.e., pixel-level) spatial information and provide low-level yet high-resolution 3D features. After the embedding block, transformer and convolutional down-sampling blocks are interleaved to fully entangle long-term dependencies with high-level and hierarchical object concepts at various scales, which helps improve the generalization ability and robustness of learned representations.

**BIMAN**, or **Bot Identification by commit Message, commit Association, and author Name**, is a technique to detect bots that commit code. It is comprised of three methods that consider independent aspects of the commits made by a particular author: 1) Commit Message: Identify if commit messages are being generated from templates; 2) Commit Association: Predict if an author is a bot using a random forest model, with features related to files and projects associated with the commits as predictors; and 3) Author Name: Match author’s name and email to common bot patterns.

BIMAN

Detecting and Characterizing Bots that Commit Code

nnFormer

nnFormer: Interleaved Transformer for Volumetric Segmentation

**Locally-Grouped Self-Attention**, or **LSA**, is a local attention mechanism used in the [Twins-SVT](https://paperswithcode.com/method/twins-svt) architecture. Locally-grouped self-attention (LSA). Motivated by the group design in depthwise convolutions for efficient inference, we first equally divide the 2D feature maps into sub-windows, making self-attention communications only happen within each sub-window. This design also resonates with the multi-head design in self-attention, where the communications only occur within the channels of the same head. To be specific, the feature maps are divided into $m \times n$ sub-windows. Without loss of generality, we assume $H \% m=0$ and $W \% n=0$. Each group contains $\frac{H W}{m n}$ elements, and thus the computation cost of the self-attention in this window is $\mathcal{O}\left(\frac{H^{2} W^{2}}{m^{2} n^{2}} d\right)$, and the total cost is $\mathcal{O}\left(\frac{H^{2} W^{2}}{m n} d\right)$. If we let $k\_{1}=\frac{H}{n}$ and $k\_{2}=\frac{W}{n}$, the cost can be computed as $\mathcal{O}\left(k\_{1} k\_{2} H W d\right)$, which is significantly more efficient when $k\_{1} \ll H$ and $k\_{2} \ll W$ and grows linearly with $H W$ if $k\_{1}$ and $k\_{2}$ are fixed.

Although the locally-grouped self-attention mechanism is computation friendly, the image is divided into non-overlapping sub-windows. Thus, we need a mechanism to communicate between different sub-windows, as in Swin. Otherwise, the information would be limited to be processed locally, which makes the receptive field small and significantly degrades the performance as shown in our experiments. This resembles the fact that we cannot replace all standard convolutions by depth-wise convolutions in CNNs.

Source	nnFormer: Interleaved Transformer for Volumetric Segmentation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com