runtimeService.startProcessInstanceByKey does not use all threads at once

I have code which looks like
for( about 150 iteration) {
runtimeService.startProcessInstanceByKey(KEY,createUniqueBusinessKey(shipment.getSscc()),variables);
}
when it executes it does not use all 10 thread of camunda, it is doing this sequentially. In logs I can see that it is using different threads but only one at given time. Each startProcessInstanceByKey takes about 3 sec so in Total It takes 3 * 150 seconds. What is strange that exact same code in Test env works normal . it use all 10 threads(so total time is about 3 * 150 / 10). and even in prod sometimes it works fine.

We use camunda with spring boot , here is configs

camunda:
bpm:
authorization:
enabled: true
metrics:
enabled: true
history-level: auto
job-execution:
core-pool-size: 10

Starting of the processes is made from within a loop which is of course executed in just one thread, like any other loop in Java. Other threads get involved after the started process has reached the first async point. After that, the job executor takes over and executes the jobs in the threads fetched from the pool. But the initial piece of the process (before the first async point) is executed synchronously on the thread that starts the processes.

This might be the reason why starting a process takes so long – because it’s not just creating a process instance (which should be very quick) but also executing the process till the first async point.

Thank you very much for reply. I see you point but the first java delegate is async, (please see attached image), and what is strange that exact same code works in test env. Do you see any problem in BPMN model?!

In prod logs I see this picture.

Camunda-thread-1 doning some work
Camunda-thread-1 doning some work
Camunda-thread-1 doning some work
Camunda-thread-1 doning some work
Camunda-thread-2 doning some work
Camunda-thread-2 doning some work
Camunda-thread-2 doning some work
Camunda-thread-2 doning some work
Camunda-thread-3 doning some work
Camunda-thread-3 doning some work
Camunda-thread-3 doning some work
Camunda-thread-3 doning some work

in test env logs I see this (they are working in parralel)
Camunda-thread-1 doning some work
Camunda-thread-2 doning some work
Camunda-thread-3 doning some work
Camunda-thread-1 doning some work
Camunda-thread-2 doning some work
Camunda-thread-3 doning some work
Camunda-thread-1 doning some work
Camunda-thread-2 doning some work
Camunda-thread-3 doning some work

So starting from first delegate all code is executed in camunda thread, so the tomcate thread shoud be free to do runtimeService.startProcessInstanceByKey for next element, but I don’t know why it waits 3 sec.

Could it be that the transaction (in the camunda DB) takes very long (for some reasons)?

Do you have idea about reasons which can lead transaction to take that long.
?

No, I have no idea. If the problem does not disapperar by itself, you’ll probably have to do some profiling to find out where the time is spent.