**I-BERT** is a quantized version of [BERT](https://paperswithcode.com/method/bert) that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer only approximation methods for nonlinear operations, e.g., [GELU](https://paperswithcode.com/method/gelu), [Softmax](https://paperswithcode.com/method/softmax), and [Layer Normalization](https://paperswithcode.com/method/layer-normalization), it performs an end-to-end integer-only [BERT](https://paperswithcode.com/method/bert) inference without any floating point calculation.

In particular, GELU and Softmax are approximated with lightweight second-order polynomials, which can be evaluated with integer-only arithmetic. For LayerNorm, integer-only computation is performed by leveraging a known algorithm for integer calculation of
square root.

**Hybrid Firefly and Particle Swarm Optimization (HFPSO)** is a metaheuristic optimization algorithm that combines strong points of firefly and particle swarm optimization. HFPSO tries to determine the start of the local search process properly by checking the previous global best fitness values.

[Click Here for the Paper](https://www.sciencedirect.com/science/article/abs/pii/S156849461830084X)

[Codes (MATLAB)](https://www.mathworks.com/matlabcentral/fileexchange/67768-a-hybrid-firefly-and-particle-swarm-optimization-hfpso)

HFPSO

I-BERT

I-BERT: Integer-only BERT Quantization

**DropPathway** randomly drops an audio pathway during training as a regularization technique for audiovisual recognition models.  Specifically, at each training iteration, we drop the Audio pathway altogether with probability $P\_{d}$. This way, we slow down the learning of the Audio pathway and make its learning dynamics more compatible with its visual counterpart. When dropping the audio pathway, we sum zero tensors with the visual pathways.

Note that DropPathway is different from simply setting different learning rates for the audio/visual pathways in that it 1) ensures the audio pathway has fewer parameter updates, 2) hinders the visual pathway to 'shortcut' training by memorizing audio information, and 3) provides extra regularization as different audio clips are dropped in each epoch.

Source	I-BERT: Integer-only BERT Quantization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com