**Spatial Group-wise Enhance** is a module for convolutional neural networks that can adjust the
importance of each sub-feature by generating an attention factor for each spatial location in each semantic group, so that every individual group can autonomously enhance its learnt expression and suppress possible noise

Inside each feature group, we model a spatial enhance mechanism inside each feature group, by scaling the feature vectors over all the locations with an attention mask. This attention mask is designed to suppress the possible noise and highlight the correct semantic feature regions. Different from other popular attention methods, it utilises the similarity between the global statistical feature and the local ones of each location as the source of generation for the attention masks.

**LLaMA** is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were subsequently proposed. The main difference with the original architecture are listed below.

- RMSNorm normalizing function is used to improve the training stability, by normalizing the input of each transformer sub-layer, instead of normalizing the output.
- The ReLU non-linearity is replaced by the SwiGLU activation function to improve performance.
- Absolute positional embeddings are removed and instead rotary positional embeddings (RoPE) are added at each layer of the network.

LLaMA

LLaMA: Open and Efficient Foundation Language Models

Spatial Group-wise Enhance

Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks

**Collaborative Distillation** is a new knowledge distillation method (named Collaborative Distillation) for encoder-decoder based neural style transfer to reduce the number of convolutional filters. The main idea is underpinned by a finding that the encoder-decoder pairs construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for style transfer models.

Source	Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com