**Source Hypothesis Transfer**, or **SHOT**, is a representation learning framework for unsupervised domain adaptation. SHOT freezes the classifier module (hypothesis) of the source model and learns the target-specific feature extraction module by exploiting both information maximization and self-supervised pseudo-labeling to implicitly align representations from the target domains to the source hypothesis.

**A Spatial Attention-Guided Mask** is a module for [instance segmentation](https://paperswithcode.com/task/instance-segmentation) that predicts a segmentation mask on each detected box with a spatial attention map that helps to focus on informative pixels and suppress noise. The goal is to guide the mask head for spotlighting meaningful pixels and repressing uninformative ones. 

Once features inside the predicted RoIs are extracted by [RoIAlign](https://paperswithcode.com/method/roi-align) with 14×14 resolution, those features are fed into four conv layers and the [spatial attention module](https://paperswithcode.com/method/spatial-attention-module) (SAM) sequentially. To exploit the spatial attention map $A\_{sag}\left(X\_{i}\right) \in \mathcal{R}^{1\times{W}\times{H}}$ as a feature descriptor given input feature map $X\_{i} \in \mathcal{R}^{C×W×H}$, the SAM first generates pooled features $P\_{avg}, P\_{max} \in \mathcal{R}^{1\times{W}\times{H}}$ by both average and [max pooling](https://paperswithcode.com/method/max-pooling) operations respectively along the channel axis and aggregates them via concatenation. Then it is followed by a 3 × 3 conv layer and normalized by the sigmoid function. The computation process
is summarized as follow:

$$
A\_{sag}\left(X\_{i}\right) = \sigma\left(F\_{3\times{3}}(P\_{max} \cdot P\_{avg})\right)
$$

where $\sigma$ denotes the sigmoid function, $F\_{3\times{3}}$ is 3 × 3 conv layer and $\cdot$ represents the concatenate operation. Finally, the attention guided feature map $X\_{sag} ∈ \mathcal{R}^{C\times{W}\times{H}}$ is computed as:

$$
X\_{sag} = A\_{sag}\left(X\_{i}\right) \otimes X\_{i}
$$

where ⊗ denotes element-wise multiplication. After then, a 2 × 2 deconv upsamples the spatially attended feature map to 28 × 28 resolution. Lastly, a 1 × 1 conv is applied for predicting class-specific masks.

Spatial Attention-Guided Mask

CenterMask : Real-Time Anchor-Free Instance Segmentation

Source Hypothesis Transfer

Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation

**TL;DR: GAP-Layer is a GNN Layer which is able to rewire a graph in an inductive an parameter-free way optimizing the spectral gap (minimizing or maximizing the bottleneck size), learning a differentiable way to compute the Fiedler vector and the Fiedler value of the graph.**

## Summary
 **GAP-Layer** is a rewiring layer based on minimizing or maximizing the spectral gap (or graph bottleneck size) in an inductive way. Depending on the mining task we want to perform in our graph, we would like to maximize or minimize the size of the bottleneck, aiming to more connected or more separated communities. 

## GAP-Layer: Spectral Gap Rewiring

#### Loss and derivatives using $\mathbf{L}$ or $\mathbf{\cal L}$
For this explanation, we are going to suppose we want to minimize the spectral gap, i.e. make the graph bottleneck size smaller. For minimizing the spectral GAP we minimize this loss:

$$
L\_{Fiedler} = \|\tilde{\mathbf{A}}-\mathbf{A}\| \_F + \alpha(\lambda\_2)^2
$$

The gradients of this cost function w.r.t each element of $\mathbf{A}$ are not trivial. Depending on if we use the Laplacian, $\mathbf{L}$, or the normalized Laplacian, $\cal L$, the derivatives are going to be different. For the former case ($\mathbf{L}$), we will use the derivatives presented in Kang et al. 2019. In the latter scenario ($\cal L$), we present the **Spectral Gradients**: derivatives from the spectral gap w.r.t. the Normalized Laplacian. However, whatever option we choose, $\lambda_2$ can seen as a function of  $\tilde{\mathbf{A}}$ and , hence, $\nabla\_{\tilde{\mathbf{A}}}\lambda\_2$, the gradient of $\lambda\_2$ wrt each component of $\tilde{\mathbf{A}}$ (*how does the bottleneck change with each change in our graph?*),  comes from the chain rule of the matrix derivative $Tr\left[\left(\nabla\_{\tilde{\mathbf{L}}}\lambda\_2\right)^T\cdot\nabla\_{\tilde{\mathbf{A}}}\tilde{\mathbf{L}}\right]$ if using the Laplacian or $Tr\left[\left(\nabla\_{\tilde{\mathbf{\cal L}}}\lambda\_2\right)^T\cdot\nabla\_{\tilde{\mathbf{A}}}\tilde{\mathbf{\cal L}}\right]$ if using the normalized Laplacian. Both of this derivatives, relies on the Fiedler vector (2nd eigenvector: $\mathbf{f}\_2$ if we use $\mathbf{L}$ and $\mathbf{g}\_2$ if using $\mathbf{\cal L}$ instead). For more details on those derivatives, and for the sake of simplicity in this blog explanation, I suggest go to the original paper.

#### Differentiable approximation of $\mathbf{f}_2$ and $\lambda_2$
Once we have those derivatives, the problem is still not that trivial. Note that our cost function $L\_{Fiedler}$, relies on an eigenvalue $\lambda\_2$. In addition, the derivatives also depends on the Fiedler vector $\mathbf{f}\_2$ or $\mathbf{g}\_2$, which is the eigenvector corresponding to the aforementioned eigenvalue. However, we **DO NOT COMPUTE IT SPECTRALLY**, as its computation has a complexity of $O(n^3)$ and would need to be computed in every learning iteration. Instead, **we learn an approximation of $\mathbf{f}\_2$ and use its Dirichlet energy ${\cal E}(\mathbf{f}\_2)$ to approximate the $\lambda_2$**. 
$$
\mathbf{f}\_2(u) =  \begin{array}{cl}
       +1/\sqrt{n}  & \text{if}\;\; u\;\; \text{belongs to the first cluster} \\
       -1/\sqrt{n}  & \text{if}\;\; u\;\; \text{belongs to the second cluster} 
\end{array} 
$$
In addition, if using $\mathbf{\cal L}$, since $\mathbf{g}\_2=\mathbf{D}^{1/2}\mathbf{f}_2$, we first approximate $\mathbf{g}_2$ and then approximate $\lambda_2$ from ${\cal E}(\mathbf{g}\_2)$. With this approximation, we can easily compute the node belonging to each cluster with a simple MLP. In addition, such as the Fiedler value must satisfy orthogonality and normality, restrictions must be added to that MLP Clustering.

### GAP-Layer
To sum up, **GAP-Layer** can be defined as the following. Given the matrix $\mathbf{X}\_{n\times F}$ encoding the features of the nodes after any message passing (MP) layer, $\mathbf{S}\_{n\times 2}=\textrm{Softmax}(\textrm{MLP}(\mathbf{X}))$ learns the association $\mathbf{X}\rightarrow \mathbf{S}$ while $\mathbf{S}$ is optimized according to the loss:
  
$$
L\_{Cut} = -\frac{Tr[\mathbf{S}^T\mathbf{A}\mathbf{S}]}{Tr[\mathbf{S}^T\mathbf{D}\mathbf{S}]} + \left\|\frac{\mathbf{S}^T\mathbf{S}}{\|\mathbf{S}^T\mathbf{S}\|\_F} - \frac{\mathbf{I}\_n}{\sqrt{2}}\right\|\_F
$$
Then, the $\mathbf{f}\_2$ is approximated from $\mathbf{S}$ using $\mathbf{f}\_2(u)$ equation. Once calculated  $\mathbf{f}\_2$ and  $\lambda\_2$ we consider the loss:

$$
L\_{Fiedler} = \|\tilde{\mathbf{A}}-\mathbf{A}\|\_F + \alpha(\lambda\_2)^2
$$
$$\mathbf{\tilde{A}} = \mathbf{A} - \mu \nabla_\mathbf{\tilde{A}}\lambda\_2$$
returning $\tilde{\mathbf{A}}$. Then the GAP diffusion $\mathbf{T}^{GAP} = \tilde{\mathbf{A}}(\mathbf{S}) \odot \mathbf{A}$ results from minimizing 

$$L_{GAP}= L\_{Cut} + L\_{Fiedler}$$


**References**
(Kang et al. 2019) Kang, J., & Tong, H. (2019, November). N2n: Network derivative mining. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (pp. 861-870).

Source	Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com