A **Switch FFN** is a sparse layer that operates independently on tokens within an input sequence. It is shown in the blue block in the figure. We diagram two tokens ($x\_{1}$ = “More” and $x\_{2}$ = “Parameters” below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).

**Self-Cure Network**, or **SCN**, is a method for suppressing uncertainties for large-scale facial expression recognition, prventing deep networks from overfitting uncertain facial images. Specifically, SCN suppresses the uncertainty from two different aspects: 1) a self-attention mechanism over mini-batch to weight each training sample with a ranking regularization, and 2) a careful relabeling mechanism to modify the labels of these samples in the lowest-ranked group.

Suppressing Uncertainties for Large-Scale Facial Expression Recognition

Switch FFN

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

In recent years, there has been a growing interest on deep learning-based pansharpening.
Research has mainly focused on architectures.
However, lacking a ground truth, model training is also a major issue.
A popular approach is to train networks in a reduced resolution domain, using the original data as ground truths.
The trained networks are then used on full resolution data, relying on an implicit scale invariance hypothesis.
Results are generally good at reduced resolution, but more questionable at full resolution.

Here, we propose a full-resolution training framework for deep learning-based pansharpening.
Training takes place in the high resolution domain, relying only on the original data, with no loss of information.
To ensure spectral and spatial fidelity, suitable losses are defined,
which force the pansharpened output to be consistent with the available panchromatic and multispectral input.
Experiments carried out on WorldView-3, WorldView-2, and GeoEye-1 images show that methods trained with the proposed framework
guarantee an excellent performance in terms of both full-resolution numerical indexes and visual quality.
The framework is fully general, and can be used to train and fine-tune any deep learning-based pansharpening network.

Source	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com