Repository: Freie Universität Berlin, Math Department

VarGrad: A Low-Variance Gradient Estimator for Variational Inference

Richter, Lorenz and Boustati, Ayman and Nüsken, Nikolas and Ruiz, Francisco J. R. and Akyildiz, Ömer Deniz (2020) VarGrad: A Low-Variance Gradient Estimator for Variational Inference. Advances in Neural Information Processing Systems 2020 . pp. 1-25. (Submitted)


Official URL:


We analyse the properties of an unbiased gradient estimator of the evidence lowerbound (ELBO) for variational inference, based on the score function method withleave-one-out control variates. We show that this gradient estimator can be obtainedusing a new loss, defined as the variance of the log-ratio between the exact posteriorand the variational approximation, which we call thelog-variance loss. Undercertain conditions, the gradient of the log-variance loss equals the gradient of the(negative)ELBO. We show theoretically that this gradient estimator, which we callVarGraddue to its connection to the log-variance loss, exhibits lower variance thanthe score function method in certain settings, and that the leave-one-out controlvariate coefficients are close to the optimal ones. We empirically demonstrate thatVarGrad offers a favourable variance versus computation trade-off compared toother state-of-the-art estimators on a discrete variational autoencoder (VAE)

Item Type:Article
Subjects:Mathematical and Computer Sciences > Mathematics > Applied Mathematics
Divisions:Department of Mathematics and Computer Science > Institute of Mathematics
ID Code:2477
Deposited By: Monika Drueck
Deposited On:18 Nov 2020 15:38
Last Modified:18 Nov 2020 15:40

Repository Staff Only: item control page