I have conduct a load test for create 20000 process instances and external task distribution for 20000 process instances last week and here are the result.
Attached please find our process setup. It have 3 service task call task 1 to 3.
Create 20000 process instances (1/s per instance creation by calling api)
Can create 20000 instances but after that Camunda occurred issue. (Error Log: After create 20000 instance.txt )
External Task Distribution( fetch and lock & complete task API ) for 20000 process instances (100ms per api request)
Return status 500 during External Task Distribution and stopped (Please see attached photo)
Create 20000 process instances (Multi-threading , at most 64 thread at the same time)
After create 30000 a few hours, the Camunda server return status 500, cannot call API.
Do you have any info on Camunda stability issue and know the root cause ?
We are using Camunda and Mysql database.
If it have better stability on enterprise version?
Here are some log:
05-Nov-2021 01:23:40.848 SEVERE [http-nio-8080-exec-6] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.
2021-11-05T09:23:40.855185903+08:00 stderr F org.camunda.bpm.engine.ProcessEngineException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.
05-Nov-2021 09:43:10.466 WARNING [http-nio-8080-exec-6] org.camunda.commons.logging.BaseLogger.logWarn ENGINE-REST-HTTP500 org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of ‘DELETE ExternalTaskEntity[0013ddef-3d76-11ec-bc00-8aa5d74cd2ea]’ failed. Entity was updated by another transaction concurrently.
2021-11-05T17:43:10.467059640+08:00 stderr F at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:136)
2021-11-05T17:43:10.467059640+08:00 stderr F at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleConcurrentModification(DbEntityManager.java:413)