blender python script building a node tree works in command line and within django, but fails behind Apache (under www-data user)

Question

I have a python Blender script that contains the following lines that build denoising compositor at the end of rendering. This has been working both in command line and within blender GUI. I then wrote a django web backend api that uses subprocess to invoke the command line and generate image output. The django app/api rendering works well at django's runserver (http://localhost:8000). Now I'm exposing the api to the outside so it's sitting behind apache2 as a wsgi application. The rest of the api that does not deal with blender all works as well as before. But I'm getting an error that seems to indicate the blender background subprocess run by www-data (the apache user) may be missing some key data. If so, what is it and how do I fix that?

# part of the blender python script: scene.py
        ...
        node_tree = self.scene.node_tree
        node_render_layers = node_tree.nodes['Render Layers']
        node_composite = node_tree.nodes['Composite']
        node_denoise = node_tree.nodes.new(type="CompositorNodeDenoise")

        # link up the nodes
        node_tree.links.new(node_render_layers.outputs['Noisy Image'], node_denoise.inputs['Image'])
        node_tree.links.new(node_render_layers.outputs['Denoising Normal'], node_denoise.inputs['Normal'])
        ...

Apache2's error log - note that node name keys are there, but node's output/input keys are not found.

Traceback (most recent call last):
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/scene.py", line 67, in <module>
    image_path = build_and_render_scene(asset, output_dir=output_dir)
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/scene.py", line 50, in build_and_render_scene
    auto_blender.render_prep(samples=16, displacement_force_off=asset['displacement']==0)
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/auto.py", line 245, in render_prep
    node_tree.links.new(node_render_layers.outputs['Noisy Image'], node_denoise.inputs['Image'])
KeyError: 'bpy_prop_collection[key]: key "Noisy Image" not found'

In essence, this script works when it's run by me, a regular user:

$ /home/wr200m/software/blender-2.91.2-linux64/blender --background --python /home/wr200m/workspace/art/src/py/main/blender/blender/scene.py -- arg1 arg2

... SUCCESS ...

But it would fail when run as www-data (or as root):

$ sudo -u www-data /home/wr200m/software/blender-2.91.2-linux64/blender --background --python /home/wr200m/workspace/art/src/py/main/blender/blender/scene.py -- arg1 arg2

Blender 2.91.2 (hash 5be9ef417703 built 2021-01-19 16:16:34)
found bundled python: /home/wr200m/software/blender-2.91.2-linux64/2.91/python
Info: Deleted 3 object(s)
Info: Deleted 3 object(s)
Info: Deleted 0 object(s)
Info: Deleted 0 object(s)
Bricks001 1.3 0.01
Traceback (most recent call last):
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/scene.py", line 67, in <module> 
    image_path = build_and_render_scene(asset, output_dir=output_dir)
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/scene.py", line 50, in build_and_render_scene
    auto_blender.render_ep(samples=16, displacement_force_off=asset['displacement']==0)
  File "/home/wr200m/workspace/art/src/py/main/blender/blender/auto.py", line 245, in render_ep
    node_tree.links.new(node_render_layers.outputs['Noisy Image'], node_denoise.inputs['Image'])
KeyError: 'bpy_op_collection[key]: key "Noisy Image" not found'

question from:https://stackoverflow.com/questions/65909435/blender-python-script-building-a-node-tree-works-in-command-line-and-within-djan

Answer 1

I realized that the blender background process did not read my blender preferences as normally it does when I launch as myself for background or for GUI:

# normally the following will spit out
Read prefs: /home/wr200m/.config/blender/2.91/config/userpref.blend

So I suppose when I sudo as any id outside of myself, blender does not know where to get my special setups. So I passed the $HOME env variable to sudo like this and voila.

$ sudo --preserve-env=HOME -u www-data /home/wr200m/software/blender-2.91.2-linux64/blender --background --python /home/wr200m/workspace/art/src/py/main/blender/blender/scene.py -- arg1 arg2
Blender 2.91.2 (hash 5be9ef417703 built 2021-01-19 16:16:34)
Read prefs: /home/wr200m/.config/blender/2.91/config/userpref.blend
found bundled python: /home/wr200m/software/blender-2.91.2-linux64/2.91/python

... SUCCESS ...