I have a process that gets a large list of data. Then the list passes to a sequential subprocess and split into batches.
Than the bathes pas in a sequential manner into a subprocess.
Than another subprocess gets that batch, split it and processed the data in parallel manner. Everything work fine until some of pice of data fail in the parallel subprocess. And I can’t delete or modify the process. Every action that I try is going to terminate the process.
Lats suppose we have list of 100 elements. I split it into batches of 10 lists with 10 elements.
Then I pass the first list in the sequential subprocess. It fails on the second element. As a result, I can’t delete this fail task because it is terminate the whole process and over 90 elements are not processed