Job executor performance issues during high load

Camunda 7 Topics Discussion & Questions
1

We are seeing some performance issues in Camunda during load testing. Gaps between jobs been picked up is high and that’s even in minutes. We are using the below job executor configuration.

camunda.bpm.job-execution.core-pool-size=50
camunda.bpm.job-execution.max-pool-size=100
camunda.bpm.job-execution.queue-capacity=10
camunda.bpm.job-execution.max-jobs-per-acquisition=10

We have around 15 async continuations in the workflow. Is it high? There are no parallel executions.

One more thing is we enabled this flag to evenly distribute delays to every instance camunda.bpm.generic-properties.properties.jobExecutorAcquireByDueDate=true. But it seems some of the instances are still taking lot of time to finish.

Can anyone of you please help us optimize our app to get better performance?

2

Is there enough jobs for execution based on queue size? You can check it in act_ru_job table. If there isn’t enough job for execution, camunda job executor wait based on waitTimeInMillis value.
On other guess I make is you set the jobExecutorAcquireByPriority and forget to make appropriate an index in job table.

3

There are lot of jobs in act_ru_job table as we are doing load testing. We don’t have priorities in place for the jobs, so jobExecutorAcquireByPriority won’t make any impact right?

4

If you set jobExecutorAcquireByPriority=true, then jobs with higher priority will be executed first rather than job created time or order.

5

Hi, are you running a single node or a cluster?
Regards
Rob

6

  • Choosing async continuations is as per your business modelling, there’s neither high nor low. Things you have to consider while modeling is not to get caught into antipatterns.

  • Coming to the performance point, you can scale the process applications based on the network traffic/storage/cpu utilizations/IOPS, etc.

  • You can setup homogeneous cluster setup(running multiple process applications, each process applications will have same delegates and listeners deployed), in that case you’ll get enough threads to process the request by camunda job executors.

  • Make sure database isolation level is in READ_COMMITTED to gain performance and to avoid deadlock.

  • If you enable jobExecutorAcquireByDueDate=true, then those jobs are queued for execution once the due date is elapsed. You can configure jobExecutor to acquire jobs based on priority, so in this case based on priority tasks will be queued for execution and it will start executing the jobs immediately.

  • Database replication is another option to improve performance(read & write replica nodes) , in which read operation can be performed independently from write node, thus will allow enough threads to process job execution.

  • For long running processes you can implement external task pattern, so the thread and resource utilization will be less in the process engine.

7

Thank you for your suggestions.

When I use jobExecutorAcquireByDueDate=true, DueDate is populated same as create date. So the jobs should be picked up fast. But what I am seeing is there are long delays even in minutes between jobs getting picked up.

Did anyone face any performance issues with Camunda job scheduler?

8