**FixRes** is an image scaling strategy that seeks to optimize classifier performance. It is motivated by the observation that data augmentations induce a significant discrepancy between the size of the objects seen by the classifier at train and test time: in fact, a lower train resolution improves the classification at test time! FixRes is a simple strategy to optimize the classifier performance, that employs different train and test resolutions. The calibrations are: (a) calibrating the object sizes by adjusting the crop size and (b) adjusting statistics before spatial pooling.

**Channel-wise Cross Attention** is a module for semantic segmentation used in the [UCTransNet](https://paperswithcode.com/method/uctransnet) architecture. It is used to fuse features of inconsistent semantics between the Channel [Transformer](https://paperswithcode.com/method/transformer) and [U-Net](https://paperswithcode.com/method/u-net) decoder. It guides the channel and information filtration of the Transformer features and eliminates the ambiguity with the decoder features.

Mathematically, we take the $i$-th level Transformer output $\mathbf{O\_{i}} \in \mathbb{R}^{C×H×W}$ and i-th level decoder feature map $\mathbf{D\_{i}} \in \mathbb{R}^{C×H×W}$ as the inputs of Channel-wise Cross Attention. Spatial squeeze is performed by a [global average pooling](https://paperswithcode.com/method/global-average-pooling) (GAP) layer, producing vector $\mathcal{G}\left(\mathbf{X}\right) \in \mathbb{R}^{C×1×1}$ with its $k$th channel $\mathcal{G}\left(\mathbf{X}\right) = \frac{1}{H×W}\sum^{H}\_{i=1}\sum^{W}\_{j=1}\mathbf{X}^{k}\left(i, j\right)$. We use this operation to embed the global spatial information and then generate the attention mask:

$$ \mathbf{M}\_{i} = \mathbf{L}\_{1} \cdot \mathcal{G}\left(\mathbf{O\_{i}}\right) + \mathbf{L}\_{2} \cdot \mathcal{G}\left(\mathbf{D}\_{i}\right) $$

where $\mathbf{L}\_{1} \in \mathbb{R}^{C×C}$ and $\mathbf{L}\_{2} \in \mathbb{R}^{C×C}$ and being weights of two Linear layers and the [ReLU](https://paperswithcode.com/method/relu) operator $\delta\left(\cdot\right)$. This operation in the equation above encodes the channel-wise dependencies. Following [ECA-Net](https://paperswithcode.com/method/eca-net) which empirically showed avoiding dimensionality reduction is important for learning channel attention, the authors use a single [Linear layer](https://paperswithcode.com/method/linear-layer) and sigmoid function to build the channel attention map. The resultant vector is used to recalibrate or excite $\mathbf{O\_{i}}$ to $\mathbf{\bar{O}\_{i}} = \sigma\left(\mathbf{M\_{i}}\right) \cdot \mathbf{O\_{i}}$, where the activation $\sigma\left(\mathbf{M\_{i}}\right)$ indicates the importance of each channel. Finally, the masked $\mathbf{\bar{O}}\_{i}$ is concatenated with the up-sampled features of the $i$-th level decoder.

Channel-wise Cross Attention

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

FixRes

Fixing the train-test resolution discrepancy

**MAML**, or **Model-Agnostic Meta-Learning**, is a model and task-agnostic algorithm for meta-learning that trains a model’s parameters such that a small number of gradient updates will lead to fast learning on a new task.

Consider a model represented by a parametrized function $f\_{\theta}$ with parameters $\theta$. When adapting to a new task $\mathcal{T}\_{i}$, the model’s parameters $\theta$ become $\theta'\_{i}$. With MAML, the updated parameter vector $\theta'\_{i}$ is computed using one or more gradient descent updates on task $\mathcal{T}\_{i}$. For example, when using one gradient update,

$$ \theta'\_{i} = \theta - \alpha\nabla\_{\theta}\mathcal{L}\_{\mathcal{T}\_{i}}\left(f\_{\theta}\right) $$

The step size $\alpha$ may be fixed as a hyperparameter or metalearned. The model parameters are trained by optimizing for the performance of $f\_{\theta'\_{i}}$ with respect to $\theta$ across tasks sampled from $p\left(\mathcal{T}\_{i}\right)$. More concretely the meta-objective is as follows:

$$ \min\_{\theta} \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta'\_{i}}\right) = \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta - \alpha\nabla\_{\theta}\mathcal{L}\_{\mathcal{T}\_{i}}\left(f\_{\theta}\right)}\right) $$

Note that the meta-optimization is performed over the model parameters $\theta$, whereas the objective is computed using the updated model parameters $\theta'$. In effect MAML aims to optimize the model parameters such that one or a small number of gradient steps on a new task will produce maximally effective behavior on that task. The meta-optimization across tasks is performed via stochastic gradient descent ([SGD](https://paperswithcode.com/method/sgd)), such that the model parameters $\theta$ are updated as follows:

$$ \theta \leftarrow \theta - \beta\nabla\_{\theta} \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta'\_{i}}\right)$$

where $\beta$ is the meta step size.

Source	Fixing the train-test resolution discrepancy
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com