**UCTransNet** is an end-to-end deep learning network for semantic segmentation that takes [U-Net](https://paperswithcode.com/method/u-net) as the main structure of the network. The original skip connections of U-Net are replaced by CTrans consisting of two components: [Channel-wise Cross fusion Transformer](https://paperswithcode.com/method/channel-wise-cross-fusion-transformer) ([CCT](https://paperswithcode.com/method/cct)) and [Channel-wise Cross Attention](https://paperswithcode.com/method/channel-wise-cross-attention) (CCA) to guide the fused multi-Scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity.

**Gaussian Gated Linear Network**, or **G-GLN**, is a multi-variate extension to the recently proposed [GLN](https://paperswithcode.com/method/gln) family of deep neural networks by reformulating the GLN neuron as a gated product of Gaussians. This Gaussian Gated Linear Network (G-GLN) formulation exploits the fact that exponential family densities are closed under multiplication, a property that has seen much use in [Gaussian Process](https://paperswithcode.com/method/gaussian-process) and related literature. Similar to the Bernoulli GLN, every neuron in the G-GLN directly predicts the target distribution.  

Precisely, a G-GLN is a feed-forward network of data-dependent distributions. Each neuron calculates the sufficient statistics $\left(\mu, \sigma\_{2}\right)$ for its associated PDF using its active weights, given those emitted by neurons in the preceding layer. It consists of consists of $L+1$ layers indexed by $i \in\{0, \ldots, L\}$ with $K\_{i}$ neurons in each layer. The weight space for a neuron in layer $i$ is denoted by $\mathcal{W}\_{i}$; the subscript is needed since the dimension of the weight space depends on $K_{i-1}$. Each neuron/distribution is indexed by its position in the network when laid out on a grid; for example, $f\_{i k}$ refers to the family of PDFs defined by the $k$ th neuron in the $i$ th layer. Similarly, $c\_{i k}$ refers to the context function associated with each neuron in layers $i \geq 1$, and $\mu\_{i k}$ and $\sigma\_{i k}^{2}$ (or $\Sigma\_{i k}$ in the multivariate case) referring to the sufficient statistics for each Gaussian PDF.

There are two types of input to neurons in the network. The first is the side information, which can be thought of as the input features, and is used to determine the weights used by each neuron via half-space gating. The second is the input to the neuron, which is the PDFs output by the previous layer, or in the case of layer 0, some provided base models. To apply a G-GLN in a supervised learning setting, we need to map the sequence of input-label pairs $\left(x\_{t}, y\_{t}\right)$ for $t=1,2, \ldots$ onto a sequence of (side information, base Gaussian PDFs, label) triplets $\left(z\_{t},\left$f\_{0 i}\right$\_{i}, y\_{t}\right)$. The side information $z\_{t}$ is set to the (potentially normalized) input features $x\_{t}$. The Gaussian PDFs for layer 0 will generally include the necessary base Gaussian PDFs to span the target range, and optionally some base prediction PDFs that capture domain-specific knowledge.

G-GLN

Gaussian Gated Linear Networks

UCTransNet

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

**Powerpropagation** is a weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, it gives rise to weight updates exhibiting a “rich get richer” dynamic, leaving low-magnitude parameters largely unaffected by learning.In other words, parameters with larger magnitudes are allowed to adapt faster in order to represent the required features to solve the task, while smaller magnitude parameters are restricted, making it more likely that they will be irrelevant in representing the learned solution.  Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely.

Source	UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com