.net - How to implement and run the collection of tasks sequentially + in parallel?

Question

I have a list of Tasks, where each element represents async Task method call. When I construct this list - I make decision on whether this Task needs to be run after the previous one has completed or in parallel with it (based on configuration settings). However, this does not work as expected - the part from ContinueWith lives its own life, no matter what options I use for it (please see piece of source code below).

Log - now I have setting saying the tasks need to be run sequentially (IsSynchronousStart == true):

2021-02-05 13:20:53.616 +02:00 [INF] BatchMasterModule, InitAsync started

2021-02-05 13:20:55.258 +02:00 [INF] SqlBatchJobModule, RunAsync started

2021-02-05 13:22:11.711 +02:00 [INF] SqlBatchJobModule, RunAsync ended

2021-02-05 13:22:11.713 +02:00 [INF] FileBatchJobModule, RunAsync started

2021-02-05 13:22:11.713 +02:00 [INF] Batch tasks status is RanToCompletion

2021-02-05 13:22:14.644 +02:00 [INF] BatchMasterModule, InitAsync started

2021-02-05 13:22:16.224 +02:00 [INF] SqlBatchJobModule, RunAsync started

2021-02-05 13:22:20.638 +02:00 [INF] FileBatchJobModule, RunAsync ended (!!!)

There are two tasks here: batchJobTasks[0] matches SqlBatchJobModule, batchJobTasks[1] matches FileBatchJobModule.

"FileBatchJobModule, RunAsync started" - this had to be called after "SqlBatchJobModule, RunAsync ended" (this is SOMETIMES run as expected, sometimes - after "Batch tasks status is RanToCompletion")

"FileBatchJobModule, RunAsync ended" - this had to be called before "Batch tasks status is RanToCompletion" (is NEVER OK now)

"BatchMasterModule, InitAsync started" - this represents a scheduled invocation of the same method in time

UPDATE

1. the source code of InitAsync (reduced - the part related to Task list creation retained):

    public class BatchMasterModule : IBatchMasterModule
    {
        public async Task InitAsync([DisableConcurrentExecutionKeyAttribute] FullBatchKeyDto batchKey)
        {
            ...
   
            var batchJobs = await _batchJobAppService.GetListForServerAsync(batchKey);
   
            ...
   
            var batchJobTasks = new List<Task>();
   
            BatchJobDto prevJob = null;
   
            foreach (var batchJob in batchJobs)
            {
                if (_batchJobModuleFactory.TryGet(batchJob.Module?.Runtime, out IBatchJobModule module))
                {
                    ...
                    var context = new BatchJobContext
                    (
                        _serviceProvider,
                        batchKey.TenantId,
                        batchJob
                    );
   
                    ...
   
                    if (prevJob == null || prevJob.IsSynchronousStart.Value == false)
                    {
                        batchJobTasks.Add(module.RunAsync(context));
                    }
                    else
                    {
                        var previousTask = batchJobTasks.Last();
                        previousTask = previousTask.ContinueWith(result => module.RunAsync(context));
                    }
   
                    prevJob = batchJob;
                }
                else
                {
                    throw new ArgumentNullException(nameof(batchJob.Module.Runtime));
                }
            }
   
            await Task
                .WhenAll(batchJobTasks)
                .ContinueWith(result =>
                {
                    if (batch.PrintReport.HasValue && batch.PrintReport.Value == true)
                    {
                        //TODO: perform reporting only if successful job execution?
                        Log.Information($"Batch tasks status is {result.Status}. OK, will print report at {DateTime.Now}");
                    }
                });
          }
     }
   

2. FileBatchJobModule - which is not called as expected - instead of being called when the previous module - SqlBatchModule - is completed it is called in some random moments and completed after WhenAll completes, not before (as I would expect too):

    public class FileBatchJobModule : IBatchJobModule
    {
        public Func<IBatchJobContext, Task> OnCompletedAsync { get; set; }
   
        public async Task RunAsync(IBatchJobContext context)
        {
            //TODO
            Log.Information($"FileBatchJobModule, RunAsync started at {DateTime.Now}");
            const int defaultDelay = 15000;
            await Task.Delay(
                context.Parameters.TryGetValue("delay", out string delayString)
                ? (int.TryParse(delayString, out int delay) ? delay : defaultDelay)
                : defaultDelay);
            byte[] encodedText = Encoding.Unicode.GetBytes($"Updating text from async call at {DateTime.Now}{Environment.NewLine}");
            using (var sourceStream = new FileStream(
                @"C:CentralToolsFilesfilebatchjobmodule.txt",
                FileMode.Append,
                FileAccess.Write,
                FileShare.None,
                bufferSize: 4096,
                useAsync: true))
            {
                await sourceStream.WriteAsync(encodedText, 0, encodedText.Length);
            };
            if (OnCompletedAsync != null)
            {
                await OnCompletedAsync.Invoke(context);
            }
            Log.Information($"FileBatchJobModule, RunAsync ended at {DateTime.Now}");
        }
    }
   

question from:https://stackoverflow.com/questions/66062234/how-to-implement-and-run-the-collection-of-tasks-sequentially-in-parallel

Answer 1

I resolved my issue by switching from List to List<Func>. I felt it could be the case...

                if (prevJob == null || prevJob.IsSynchronousStart.Value == false)
                {
                    batchJobTasks.Add(async () => await module.RunAsync(context));
                }
                else
                {
                    var previousTask = batchJobTasks.Last();
                    batchJobTasks.RemoveAt(batchJobTasks.Count - 1);
                    batchJobTasks.Add(async () => { await previousTask(); await module.RunAsync(context); });
                }