python 3.x - How to call function in parallel from another parallel fucntion Python3?

Question

I am not an expert in using threads and etc in python. Nevertheless, at this moment I'm curious hot to make code parallel in the following situation:

for instance, I have a dictionary that stores such information for each blocks: {name_of_block: 'template':name_of_template, 'blocks':[range of numbers for this name_of block]}. There are many of name_of_block that can be used in function g(name_of_block), in which I want to call f('number' for name_of_block). I want to make it as faster as it can be. The first attemp that I have and it works:

from multiprocessing import Pool, TimeoutError

def f(number):
   global d
   print('AAA'+str(int(number)*2)+'-'+d[name_of_block])

def g(name_of_block):
   global d
   for number in d[name_of_block]['blocks']: 
       f(number)



d={'262,263,264,265': {'template': 'template.yaml', 'blocks': ['262', '263', '264', '265']}, '264': {'template': 'template1.yaml', 'blocks': ['264']}, '265-266': {'template': 'template2.yaml', 'blocks': ['265', '266']}, '267,2509_3': {'template': 'template15.yaml', 'blocks': ['267', '2509_3']}, '2509_4,2509_5,2509_8': {'template': 'template4.yaml', 'blocks': ['2509_4', '2509_5', '2509_8']}, '2509_5-2599_6': {'template': 'template234134.yaml', 'blocks': []}, '2509_7-2509_23': {'template': 'template555.yaml', 'blocks': ['2509_7', '2509_8', '2509_9', '2509_10', '2509_11', '2509_12', '2509_13', '2509_14', '2509_15', '2509_16', '2509_17', '2509_18', '2509_19', '2509_20', '2509_21', '2509_22', '2509_23']}}

name_of_blocks = d.keys()



with Pool(processes=10) as pool:
    pool.map(g, name_of_blocks)

BUT it is possible to avoid problem AssertionError: daemonic processes are not allowed to have children that I face if I try to do:

from multiprocessing import Pool, TimeoutError

def f(number):
   global d
   print('AAA'+str(int(number)*2)+'-'+d[name_of_block])

def g(name_of_block):
   global d
   with Pool(processes=5) as subpool:
       subpool.map(f, d[name_of_block]['blocks'])



d={'262,263,264,265': {'template': 'template.yaml', 'blocks': ['262', '263', '264', '265']}, '264': {'template': 'template1.yaml', 'blocks': ['264']}, '265-266': {'template': 'template2.yaml', 'blocks': ['265', '266']}, '267,2509_3': {'template': 'template15.yaml', 'blocks': ['267', '2509_3']}, '2509_4,2509_5,2509_8': {'template': 'template4.yaml', 'blocks': ['2509_4', '2509_5', '2509_8']}, '2509_5-2599_6': {'template': 'template234134.yaml', 'blocks': []}, '2509_7-2509_23': {'template': 'template555.yaml', 'blocks': ['2509_7', '2509_8', '2509_9', '2509_10', '2509_11', '2509_12', '2509_13', '2509_14', '2509_15', '2509_16', '2509_17', '2509_18', '2509_19', '2509_20', '2509_21', '2509_22', '2509_23']}}

name_of_blocks = d.keys()



with Pool(processes=10) as pool:
    pool.map(g, name_of_blocks)

This code just for play. But I'll be very happy if you can help me to understand how to make more parallel code in this case.

question from:https://stackoverflow.com/questions/65848889/how-to-call-function-in-parallel-from-another-parallel-fucntion-python3

Answer 1

Waitting for answers