Richter, Lorenz and Boustati, Ayman and Nüsken, Nikolas and Ruiz, Francisco J. R. and Akyildiz, Ömer Deniz (2020) VarGrad: A LowVariance Gradient Estimator for Variational Inference. Advances in Neural Information Processing Systems 2020 . pp. 125. (Submitted)

PDF
960kB 
Official URL: https://arxiv.org/abs/2010.10436
Abstract
We analyse the properties of an unbiased gradient estimator of the evidence lowerbound (ELBO) for variational inference, based on the score function method withleaveoneout control variates. We show that this gradient estimator can be obtainedusing a new loss, defined as the variance of the logratio between the exact posteriorand the variational approximation, which we call thelogvariance loss. Undercertain conditions, the gradient of the logvariance loss equals the gradient of the(negative)ELBO. We show theoretically that this gradient estimator, which we callVarGraddue to its connection to the logvariance loss, exhibits lower variance thanthe score function method in certain settings, and that the leaveoneout controlvariate coefficients are close to the optimal ones. We empirically demonstrate thatVarGrad offers a favourable variance versus computation tradeoff compared toother stateoftheart estimators on a discrete variational autoencoder (VAE)
Item Type:  Article 

Subjects:  Mathematical and Computer Sciences > Mathematics > Applied Mathematics 
Divisions:  Department of Mathematics and Computer Science > Institute of Mathematics 
ID Code:  2477 
Deposited By:  Monika Drueck 
Deposited On:  18 Nov 2020 15:38 
Last Modified:  18 Nov 2020 15:40 
Repository Staff Only: item control page