Python & Concurrency

Question

Hi everyone I am trying to increase the speed of my auto cloud uploading script in python 3.7 . annoyingly I'm finding a bit of delay with the uploads. Currently I have 40 tiny files being created every 1 second uploading to cloud 2 locations. so 800 uploads every 1 second which really isn't much.

Am I using concurrent futures correctly?

def watch(name, config):
    i = inotify.adapters.InotifyTree(config["WATCHPATH"])
    with concurrent.futures.ProcessPoolExecutor(max_workers=PROCESSES) as executor:
    #with concurrent.futures.ThreadPoolExecutor(max_workers=PROCESSES) as executor:
        try:
            for event in i.event_gen(yield_nones=False):
                date_time = datetime.now().strftime("%m/%d/%Y, %H:%M:%S")
                (_, type_names, filepath, filename) = event
                typename = type_names[0]
                if typename in ["IN_MODIFY", "IN_CREATE"]:
                    if typename == "IN_MODIFY":
                        print(f"[{date_time}] file modified event for {filepath}/{filename}")
                    if typename == "IN_CREATE":
                        print(f"[{date_time}] file created event for {filepath}/{filename}")
                        executor.submit(CheckAndUpload(filename, filepath, config))

question from:https://stackoverflow.com/questions/65844872/python-concurrency

Answer 1

Waitting for answers