Deadlock When completing tasks of different instances in 7.7.0

I am facing deadlock when completing tasks of different process instances simultaneously.

Could this be due to isolation level which i have not configured as READ_COMMITTED??

18-01-18 00:05:30.260 TID= ERROR [ConnectorMessageProcessor-7] o.c.b.e.context [BaseLogger.java:156] ENGINE-16004 Exception while closing command context: ENGINE-03004 Exception while executing Database Operation ‘DELETE ByteArrayEntity[33addebb-fbe3-11e7-830b-000f530e1f1c]’ with message ’

Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

The error may involve org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteByteArray-Inline

The error occurred while setting parameters

SQL: delete from ACT_GE_BYTEARRAY where ID_ = ? and REV_ = ?

Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

'. Flush summary:
[
INSERT ExecutionEntity[4922ba74-fbe3-11e7-830b-000f530e1f1c]
INSERT ByteArrayEntity[49230899-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[491cc700-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4922ba76-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4922ba77-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4922ba78-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4923089a-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4923089b-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4923089c-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4923089d-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addeb8-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addeb9-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addeba-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addebc-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addebd-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addebe-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addebf-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[33addec0-fbe3-11e7-830b-000f530e1f1c]
DELETE ByteArrayEntity[33addebb-fbe3-11e7-830b-000f530e1f1c]
UPDATE ExecutionEntity[33ad1b3d-fbe3-11e7-830b-000f530e1f1c]
DELETE ExecutionEntity[33adb7a6-fbe3-11e7-830b-000f530e1f1c]
]
org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation ‘DELETE ByteArrayEntity[33addebb-fbe3-11e7-830b-000f530e1f1c]’ with message ’

Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

This could certainly be the problem. The Camunda engine very likely does not work as expected on other isolation levels.

Cheers,
Thorben

Hi @thorben,

I executed the same with isolation level READ_COMMITTED. We never faced such issue with 7.5 and recently we migrated to 7.7 and started facing this issue.

Also this has happened when completing tasks from two different workflow instances.

Attaching the full exception here.

18-01-18 00:05:34.886 TID= ERROR [ConnectorMessageProcessor-8] o.c.b.e.context [BaseLogger.java:156] ENGINE-16004 Exception while closing command context: ENGINE-03004 Exception while executing Database Operation ‘DELETE ByteArrayEntity[27a56c35-fbe3-11e7-830b-000f530e1f1c]’ with message ’

Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

The error may involve org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteByteArray-Inline

The error occurred while setting parameters

SQL: delete from ACT_GE_BYTEARRAY where ID_ = ? and REV_ = ?

Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

'. Flush summary:
[
INSERT ExecutionEntity[4bb48925-fbe3-11e7-830b-000f530e1f1c]
INSERT ByteArrayEntity[4bb4d74a-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb01c54-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48927-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48928-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48929-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4d74b-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5c-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5d-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5e-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c32-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c33-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c34-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c36-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c37-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c38-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c39-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c3a-fbe3-11e7-830b-000f530e1f1c]
DELETE ByteArrayEntity[27a56c35-fbe3-11e7-830b-000f530e1f1c]
UPDATE ExecutionEntity[27a4d0c7-fbe3-11e7-830b-000f530e1f1c]
DELETE ExecutionEntity[27a56c30-fbe3-11e7-830b-000f530e1f1c]
]
org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation ‘DELETE ByteArrayEntity[27a56c35-fbe3-11e7-830b-000f530e1f1c]’ with message ’

Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

The error may involve org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteByteArray-Inline

The error occurred while setting parameters

SQL: delete from ACT_GE_BYTEARRAY where ID_ = ? and REV_ = ?

Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

'. Flush summary:
[
INSERT ExecutionEntity[4bb48925-fbe3-11e7-830b-000f530e1f1c]
INSERT ByteArrayEntity[4bb4d74a-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb01c54-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48927-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48928-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb48929-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4d74b-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5c-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5d-fbe3-11e7-830b-000f530e1f1c]
INSERT VariableInstanceEntity[4bb4fe5e-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c32-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c33-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c34-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c36-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c37-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c38-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c39-fbe3-11e7-830b-000f530e1f1c]
DELETE VariableInstanceEntity[27a56c3a-fbe3-11e7-830b-000f530e1f1c]
DELETE ByteArrayEntity[27a56c35-fbe3-11e7-830b-000f530e1f1c]
UPDATE ExecutionEntity[27a4d0c7-fbe3-11e7-830b-000f530e1f1c]
DELETE ExecutionEntity[27a56c30-fbe3-11e7-830b-000f530e1f1c]
]
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:115)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:305)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:281)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:203)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:132)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:42)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:40)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.RuntimeServiceImpl.signal(RuntimeServiceImpl.java:382)
at com.phoenix.workflow.service.impl.WorkflowEngineServiceImpl.completeTask(WorkflowEngineServiceImpl.java:106)
at com.phoenix.workflow.service.impl.WorkflowEngineServiceImpl$$FastClassBySpringCGLIB$$c93d5a02.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.phoenix.workflow.utils.ProcessEngineInterceptor.processServicePointcut(ProcessEngineInterceptor.java:69)
at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.phoenix.workflow.service.impl.WorkflowEngineServiceImpl$$EnhancerBySpringCGLIB$$440f4319.completeTask()
at com.phoenix.workflow.service.impl.WorkflowIntegrationServiceImpl.processAsyncConnectorResponse(WorkflowIntegrationServiceImpl.java:143)
at com.phoenix.workflow.service.impl.WorkflowIntegrationServiceImpl$$FastClassBySpringCGLIB$$51495632.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.phoenix.workflow.service.impl.WorkflowIntegrationServiceImpl$$EnhancerBySpringCGLIB$$e42d35af.processAsyncConnectorResponse()
at com.phoenix.workflow.integration.ConnectorResponseProcessor$1.call(ConnectorResponseProcessor.java:47)
at com.phoenix.workflow.integration.ConnectorResponseProcessor$1.call(ConnectorResponseProcessor.java:43)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException:

Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

The error may involve org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteByteArray-Inline

The error occurred while setting parameters

SQL: delete from ACT_GE_BYTEARRAY where ID_ = ? and REV_ = ?

Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
at org.apache.ibatis.session.defaults.DefaultSqlSession.delete(DefaultSqlSession.java:165)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeDelete(DbSqlSession.java:188)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.deleteEntity(DbSqlSession.java:173)
at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:45)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:303)
... 55 common frames omitted

Caused by: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3685)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1376)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.execute(NewProxyPreparedStatement.java:911)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
... 60 common frames omitted

Does that mean the issue persists after you changed the isolation level?

If yes, please double check that your database schema contains all of the indexes defined in the Camunda create SQL scripts for your database.