**Weight Decay**, or **$L_{2}$ Regularization**, is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the $L\_{2}$ Norm of the weights:

$$L\_{new}\left(w\right) = L\_{original}\left(w\right) + \lambda{w^{T}w}$$

where $\lambda$ is a value determining the strength of the penalty (encouraging smaller weights). 

Weight decay can be incorporated directly into the weight update rule, rather than just implicitly by defining it through to objective function. Often weight decay refers to the implementation where we specify it directly in the weight update rule (whereas L2 regularization is usually the implementation which is specified in the objective function).

Image Source: Deep Learning, Goodfellow et al

**ComiRec** is a multi-interest framework for sequential recommendation. The multi-interest module captures multiple interests from user behavior sequences, which can be exploited for retrieving candidate items from the large-scale item pool. These items are then fed into an aggregation module to obtain the overall recommendation. The aggregation module leverages a controllable factor to balance the recommendation accuracy and diversity.

ComiRec

Controllable Multi-Interest Framework for Recommendation

Weight Decay

**Hardtanh** is an activation function used for neural networks:

$$ f\left(x\right) = -1 \text{ if } x < - 1 $$
$$ f\left(x\right) = x \text{ if } -1 \leq x \leq 1 $$
$$ f\left(x\right) = 1 \text{ if } x > 1 $$

It is a cheaper and more computationally efficient version of the [tanh activation](https://paperswithcode.com/method/tanh-activation).

Image Source: [Zhuan Lan](https://zhuanlan.zhihu.com/p/30385380)

Year	1943
Data Source	CC BY-SA - https://paperswithcode.com