**ENIGMA** is an evaluation framework for dialog systems based on Pearson and Spearman's rank correlations between the estimated rewards and the true rewards.  ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data (see details in Section 2), which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors.

VERtex Similarity Embeddings (VERSE) is a simple, versatile, and memory-efficient method that derives graph embeddings explicitly calibrated to preserve the distributions of a selected vertex-to-vertex similarity measure. VERSE learns such embeddings by training a single-layer neural network.

Source: [Tsitsulin et al.](https://arxiv.org/pdf/1803.04742v1.pdf)

Image source: [Tsitsulin et al.](https://arxiv.org/pdf/1803.04742v1.pdf)

VERSE

VERSE: Versatile Graph Embeddings from Similarity Measures

ENIGMA

Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

**Voxel RoI Pooling** is a RoI feature extractor extracts RoI features directly from voxel features for further refinement. It starts by dividing a region proposal into $G \times G \times G$ regular sub-voxels. The center point is taken as the grid point of the corresponding sub-voxel. Since $3 D$ feature volumes are extremely sparse (non-empty voxels account for $<3 \%$ spaces), we cannot directly utilize max pooling over features of each sub-voxel. Instead, features are integrated from neighboring voxels into the grid points for feature extraction. Specifically, given a grid point $g\_{i}$, we first exploit voxel query to group a set of neighboring voxels $\Gamma\_{i}=\left$\mathbf{v}\_{i}^{1}, \mathbf{v}\_{i}^{2}, \cdots, \mathbf{v}\_{i}^{K}\right$ .$ Then, we aggregate the neighboring voxel features with a [PointNet](https://paperswithcode.com/method/pointnet) module $\mathrm{a}$ as:

$$
\mathbf{\eta}\_{i}=\max _{k=1,2, \cdots, K}\left$\Psi\left(\left[\mathbf{v}\_{i}^{k}-\mathbf{g}\_{i} ; \mathbf{\phi}\_{i}^{k}\right]\right)\right$
$$

where $\mathbf{v}\_{i}-\mathbf{g}\_{i}$ represents the relative coordinates, $\mathbf{\phi}\_{i}^{k}$ is the voxel feature of $\mathbf{v}\_{i}^{k}$, and $\Psi(\cdot)$ indicates an MLP. The [max pooling](https://paperswithcode.com/method/max-pooling) operation $\max (\cdot)$ is performed along the channels to obtain the aggregated feature vector $\eta_{i} .$ Particularly, Voxel RoI pooling is exploited to extract voxel features from the 3D feature volumes out of the last two stages in the $3 \mathrm{D}$ backbone network. And for each stage, two Manhattan distance thresholds are set to group voxels with multiple scales. Then, we concatenate the aggregated features pooled from different stages and scales to obtain the RoI features.

Source	Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com