A **Global Context Block** is an image model block for global context modeling. The aim is to have both the benefits of the simplified [non-local block](https://paperswithcode.com/method/non-local-block) with effective modeling of long-range dependencies, and the [squeeze-excitation block](https://paperswithcode.com/method/squeeze-and-excitation-block) with lightweight computation. 

In the Global Context framework, we have (a) global attention pooling, which adopts a [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) $W_{k}$ and [softmax](https://paperswithcode.com/method/softmax) function to obtain the attention weights, and then performs the attention pooling to obtain the global context features, (b) feature transform via a 1x1 [convolution](https://paperswithcode.com/method/convolution) $W\_{v}$; (c) feature aggregation, which employs addition to aggregate the global context features to the features of each position. Taken as a whole, the GC block is proposed as a lightweight way to achieve global context modeling.

Most attention mechanisms learn where to focus using only weak supervisory signals from class labels, which inspired Linsley et al. to investigate how explicit human supervision can affect the performance and interpretability of attention models. As a proof of concept, Linsley et al. proposed the global-and-local attention (GALA) module, which extends an SE block with a spatial attention mechanism.

Given the input feature map $X$, GALA uses an attention mask that combines global and local attention to tell the network where and on what to focus. As in SE blocks, global attention aggregates global information by global average pooling and then produces a channel-wise attention weight vector using a multilayer perceptron. In local attention, two consecutive $1\times 1$ convolutions are conducted on the input to produce a positional weight map. The outputs of the local and global pathways are combined by addition and multiplication. Formally, GALA can be represented as:
\begin{align}
    s_g &= W_{2} \delta (W_{1}\text{GAP}(x))
\end{align}

\begin{align}
    s_l &= Conv_2^{1\times 1} (\delta(Conv_1^{1\times1}(X)))
\end{align}

\begin{align}
    s_g^* &= \text{Expand}(s_g)
\end{align}

\begin{align}
    s_l^* &= \text{Expand}(s_l) 
\end{align}

\begin{align}
    s &= \tanh(a(s_g^\* + s_l^\*) +m \cdot (s_g^\* s_l^\*) )
\end{align}

\begin{align}
    Y &= sX
\end{align}

where $a,m \in \mathbb{R}^{C}$ are learnable parameters representing channel-wise weight vectors. 

Supervised by human-provided feature importance maps, GALA has significantly improved representational power and can be combined with any CNN backbone.

GALA

Learning what and where to attend

Global Context Block

GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond

The **Softmax** output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification.  Given an input vector $x$ and a weighting vector $w$ we have:

$$ P(y=j \mid{x}) = \frac{e^{x^{T}w_{j}}}{\sum^{K}_{k=1}e^{x^{T}wk}} $$

Source	GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com