python - Use multiple directories for flow_from_directory in Keras

Question

My scenario is that we have multiple peers with their own data, located in different directories, with the same sub-directory structure. I want to train the model using those data, but if I copy all of them to one folder, I can't keep track of which data is from whose (the new data is also created occasionally so it's not suitable to keep copy the files every time) My data is now stored like this:

-user01
-user02
-user03
...

(all of them have similar sub-directory structure)

I have searched for solution, but I only found the multi-input case in here and here, which they concatenate multiple input into 1 single parallel input, which is not my case.

I know that the flow_from_directory() can only be fed by 1 directory at a time, so how can I build a custom one that can be fed by multiple directory at a time?

If my question is low-quality, please give advice on how to improve it, I have searched also on the github of keras but didn't find anything that I can adapt.

Thank you.

Answer 1

The Keras ImageDataGenerator flow_from_directory method has a follow_links parameter.

Maybe you can create one directory which is populated with symlinks to files in all the other directories.

This stack question discusses using symlinks with Keras ImageDataGenerator: Understanding 'follow_links' argument in Keras's ImageDataGenerator?