We present a novel classifier network called STEP, to classify perceived human emotion from gaits, based on a Spatial Temporal Graph Convolutional Network (ST-[GCN](https://paperswithcode.com/method/gcn)) architecture. Given an RGB video of an individual walking, our formulation implicitly exploits the gait features to classify the perceived emotion of the human into one of four emotions: happy, sad, angry, or neutral. We train STEP on annotated real-world gait videos, augmented with annotated synthetic gaits generated using a novel generative network called STEP-Gen, built on an ST-GCN based Conditional Variational Autoencoder (CVAE). We incorporate a novel push-pull regularization loss in the CVAE formulation of STEP-Gen to generate realistic gaits and improve the classification accuracy of STEP.
We also release a novel dataset (E-Gait), which consists of 4,227 human gaits annotated with perceived emotions along with thousands of synthetic gaits. In practice, STEP can learn the affective features and exhibits classification accuracy of 88\% on E-Gait, which is 14--30\% more accurate over prior methods.

**MPNet** is a pre-training method for language models that combines masked language modeling (MLM) and permuted language modeling (PLM) in one view. It takes the dependency among the predicted tokens into consideration through permuted language modeling and thus avoids the issue of [BERT](https://paperswithcode.com/method/bert). On the other hand, it takes position information of all tokens as input to make the model see the position information of all the tokens and thus alleviates the position discrepancy of [XLNet](https://paperswithcode.com/method/xlnet).

The training objective of MPNet is:

$$ \mathbb{E}\_{z\in{\mathcal{Z}\_{n}}} \sum^{n}\_{t=c+1}\log{P}\left(x\_{z\_{t}}\mid{x\_{z\_{<t}}}, M\_{z\_{{>}{c}}}; \theta\right) $$

As can be seen, MPNet conditions on ${x\_{z\_{<t}}}$ (the tokens preceding the current predicted token $x\_{z\_{t}}$) rather than only the non-predicted tokens ${x\_{z\_{<=c}}}$ in MLM; comparing with PLM, MPNet takes more information (i.e., the mask symbol $[M]$ in position $z\_{>c}$) as inputs. Although the objective seems simple, it is challenging to implement the model efficiently. For details, see the paper.

MPNet

MPNet: Masked and Permuted Pre-training for Language Understanding

STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits

A Gsop block has a squeeze module and an excitation module, and uses a second-order pooling to model high-order statistics while gathering global information.
In the squeeze module, a GSoP block firstly reduces the number of channels from $c$ to $c'$ ($c' < c$) using a $1 \times 1$ convolution,  then  computes a $c' \times c'$ covariance matrix for the different channels to obtain their correlation.  Next, row-wise normalization is performed on the covariance matrix.  Each $(i, j)$ in the normalized covariance matrix explicitly relates channel $i$ to channel $j$. 

In the excitation module, a GSoP block performs row-wise convolution to  maintain structural information and output a vector. Then a fully-connected layer and a sigmoid function are applied  to get a $c$-dimensional attention vector. Finally, it multiplies the input features by the attention vector, as in an SE block. A GSoP block can be formulated as:
\begin{align}
    s = F_\text{gsop}(X, \theta) & = \sigma (W \text{RC}(\text{Cov}(\text{Conv}(X))))
\end{align}
\begin{align}
    Y & = s  X
\end{align}
Here, $\text{Conv}(\cdot)$ reduces the number of channels,
$\text{Cov}(\cdot)$ computes the covariance matrix and
$\text{RC}(\cdot)$ means row-wise convolution.

Source	STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com