Pytorch: Gumbel noise

Created on 30 Jan 2017 · 3Comments · Source: pytorch/pytorch

Hi,

I've been trying to add Gumbel noise as in here, but no success.
Looks like several modules are still missing (e.g., basic variable functions such as nn.Uniform()), or am I wrong? How would you implement in pytorch e.g. the following rows:

``` -- Create noise ε sample module
local noiseModule = nn.Sequential()
noiseModule:add(nn.Uniform(0, 1)) -- Sample from U(0, 1)
-- Transform uniform sample to Gumbel sample
noiseModule:add(nn.AddConstant(1e-9, true)) -- Improve numerical stability
noiseModule:add(nn.Log())
noiseModule:add(nn.MulConstant(-1, true))
noiseModule:add(nn.AddConstant(1e-9, true)) -- Improve numerical stability
noiseModule:add(nn.Log())
noiseModule:add(nn.MulConstant(-1, true))

-- Create sampler q(z) = G(z) = softmax((log(π) + ε)/τ) (reparametrization trick)
local sampler = nn.Sequential()
local samplerInternal = nn.ConcatTable()
samplerInternal:add(nn.Identity()) -- Unnormalised log probabilities log(π)
samplerInternal:add(noiseModule) -- Create noise ε
sampler:add(samplerInternal)
sampler:add(nn.CAddTable())
self.temperature = nn.MulConstant(1 / self.tau, true) -- Temperature τ for softmax
sampler:add(self.temperature)
sampler:add(nn.View(-1, self.k)) -- Resize to work over k
sampler:add(nn.SoftMax())
sampler:add(nn.View(-1, self.N * self.k)) -- Resize back
```

Source

eliabruni

Most helpful comment

Here you go. Much more readable and no modules required:

import torch.nn.functional as F
from torch.autograd import Variable

def sampler(input, tau, temperature):
    noise = torch.rand(input.size())
    noise.add_(1e-9).log_().neg_()
    noise.add_(1e-9).log_().neg_()
    noise = Variable(noise)
    x = (input + noise) / tau + temperature
    x = F.softmax(x.view(input.size(0), -1))
    return x.view_as(input)

We're using GitHub for bug reports only, if you have questions please post the on our forums.

apaszke on 30 Jan 2017

👍7

All 3 comments

Here you go. Much more readable and no modules required:

import torch.nn.functional as F
from torch.autograd import Variable

def sampler(input, tau, temperature):
    noise = torch.rand(input.size())
    noise.add_(1e-9).log_().neg_()
    noise.add_(1e-9).log_().neg_()
    noise = Variable(noise)
    x = (input + noise) / tau + temperature
    x = F.softmax(x.view(input.size(0), -1))
    return x.view_as(input)

We're using GitHub for bug reports only, if you have questions please post the on our forums.

apaszke on 30 Jan 2017

👍7

Thanks for the prompt answer!, and sure, will write on the forum next time.

eliabruni on 30 Jan 2017

Thanks for the answer!