Hello,
Apologies if I butcher some concepts below, we are learning Camunda as we go.
Here is a test bpmn, also attached:
Three items are passed to outer subprocess.
Each outer task has a 500ms sleep.
Create inner collection creates three items per outer item and passes to inner subprocess.
Each inner task has a 500ms sleep.
~
I would like each item in the passed outer collection to be processed concurrently by the outer subprocess tasks.
i.e. Assuming I am passing ten items in the outer collection, and job-execution.core-pool-size: 10, âOuter subprocess task 1â is started ten times, at the same time (or as close to).
Further, Iâd like the same to happen for the inner subprocess.
In other words, I would like each subprocess instance that is created to be processed by a separate Job Executor.
Unfortunately I donât understand the job execution engine and async/exclusive flags well enough to achieve this.
Here are some tests showing order of execution.
Outer items as numbers (1, 2, 3), inner items as letters (a, b, c) with associated outer item (1a etc.):
- No async flags anywhere
Everything executed in âmainâ
1, 1a, 1b, 1c, 2, 2a, 2b, 2c, 3, 3a, 3b, 3c
- Both subprocesses Multi instance asynchronous before + Multi instance exclusive
Everything executed in âaTaskExecutor-1â
1, 2, 3, 1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 3c
- Outer subprocess task 1 async before + exclusive, Inner subprocess task A async before + exclusive
Same as above.
Are these two setups essentially the same?
- Both subprocesses Multi instance asynchronous before, only Outer subprocess NOT exclusive
aTaskExecutor 1-6 used.
Outer subprocess tasks started and executed concurrently on aTaskExecutor 1-3
Inner subprocess tasks executed sequentially on aTaskExecutor 4-6 in whatever order as outer tasks were started.
No OptimisticLockingExceptions.
- Both subprocesses Multi instance asynchronous before, only Inner subprocess NOT exclusive
Many task executors get used, i.e. aTaskExecutor-1, 2, 3 etc.
Tasks seems to be executed concurrently in a random order.
Many OptimisticLockingExceptions.
- Both subprocesses Multi instance asynchronous before, but NOT exclusive
Same as above.
- Removing the inner subprocess. Putting the tasks in the main process, started via runtimeService.correlateStartMessage() from Outer subprocess task 2.
Unfortunately I cannot attach two pictures as a new user :C
aTaskExecutor 1-6 used.
Outer subprocess tasks started and executed concurrently on aTaskExecutor 1-3
Inner subprocess tasks executed sequentially on aTaskExecutor 4-6 in whatever order as outer tasks were started.
A few OptimisticLockingExceptions.
~
We will have many inbound items to process, which do not interact with one another, so each should just be processed in itâs own âinstanceâ.
-
We want to maximise throughput - is concurrency the way to go, and is the behaviour described at the beginning of this post the best way to go?
-
If yes, whatâs the best practice way to implement it in Camunda?
-
Why are we getting OptimisticLockingExceptions? I assume itâs internal to Camunda, to do with the job executors fighting for tasks? If so, can we assume all work gets completed anyway (which testing seems to support)?
Appreciate any help 
