python - output.grad None even after loss.backward()

Question

I'm confused...

My model output: tensor([[0.0000,0.1537],...],grad_fn=<ReluBackward0>)

If I use print(output.grad) it gives me None but even after gradient computation with loss.backward() I get the same result, which again is None...

Even with with torch.set_grad_enabled(True): added, still the same.

I've tried now with multiple model variants, always the same.

I was achieving good result with my model there seemed no problem, now I see this and I'm not sure any more if there not might be a major flaw I didn't recognized so far. But my model is learning, it improves so I guess it has to?

Why do I get None instead of an actual value?

Answer 1

You are getting None because the gradient is only stored on the .grad property for leaf tensors. Those are tensors that don't have parents in the computational graph.

You can check whether a tensor is a leaf or not with is_leaf:

>>> x = torch.FloatTensor([1,2,3])
>>> x.requires_grad = True
>>> x.sum().backward() # backward pass

>>> x.is_leaf
True
>>> x.grad
tensor([1., 1., 1.])

The tensor you printed shows grad_fn=<ReluBackward0> indicating it's the result of a ReLU layer, and therefore not a leaf tensor.

Here is an example of a non-leaf tensor:

>>> x = torch.FloatTensor([1,2,3])
>>> x.requires_grad=True
>>> z = x.sum()
>>> z.backward()

>>> z.is_leaf
False
>>> z.grad
None

Notice that z will show as tensor(6., grad_fn=<SumBackward0>).

---

Actually accessing .grad will give a warning:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.

And if you want to access the gradient on non-leaf tensors, following the warning message:

>>> z.retain_grad()
>>> z = x.sum()
>>> z.retain_grad()
>>> z.backward()

>>> z.is_leaf
False
>>> z.grad
tensor(1.)