Why does a deadlock occur on completing an external task? And how to retry it?

Camunda 7 Topics Discussion & Questions
1

We are using Camunda 7.8 (Spring Boot 1.5.8) with PostgreSQL 10.4.0

On heavy system load some external tasks cannot be completed and result in a deadlock.

This is how the external task is modelled:
image

On completion of the task additional variables are attached to the process instance.

The database logs something like this:

Process 781 waits for ShareLock on transaction 8165578; blocked by process 650.
Process 781: delete from ACT_RU_EXECUTION where ID_ = $1 and REV_ = $2
Process 650: delete from ACT_RU_EXECUTION where ID_ = $1 and REV_ = $2
CONTEXT:  while deleting tuple (274,23) in relation "act_ru_execution"
ERROR:  deadlock detected

I’m wondering why two database processes try to delete the same execution in this context. Is there a reason that I can try to understand or could this be a hint that we are doing something wrong?

As an additional question (if everything is fine with this behavior) I would like to know which exception to handle in order to configure our retry strategy. We already retry on OptimisticLockingExceptions but there seems to be no dedicated exception class for this kind.

2

We use the Spring Retry logic to have several attempts on OptimisticLockingExceptions and deadlocks before a failure is returned.

Since deadlocks in the database are not thrown as specific exceptions, I figured out to ‘catch’ them within a custom RetryPolicy:

public class NextRetryPolicy extends SimpleRetryPolicy {

	private final static NextLogger LOGGER = NextLogger.RETRY_LOGGER_INSTANCE;

	public NextRetryPolicy(int maxAttempts) {
		// Instantiate super class with OptimisticLockingExceptions as retryable and for counting attempts
		super(maxAttempts, Map.ofEntries(new AbstractMap.SimpleEntry<Class<? extends Throwable>, Boolean>(OptimisticLockingException.class, true)), true);
	}

	@Override
	public boolean canRetry(RetryContext context) {
		// Log first actual retry
		boolean simpleRetry = super.canRetry(context);
		if (simpleRetry) {
			if (context.getLastThrowable() != null) {
				LOGGER.debug("Retry operation on exception [ {} ] with attempt count [ {} ]", context.getLastThrowable().getClass().getName(), context.getRetryCount());
			} else if (context.getRetryCount() > 0) {
				LOGGER.debug("Retry operation with attempt count [ {} ]", context.getRetryCount());
			}
			return true;
		}
		if (context.getRetryCount() >= this.getMaxAttempts()) {
			return false;
		}

		// Check for deadlock exception and allow retry in these cases, too
		int indexOfDbException = ExceptionUtils.indexOfType(context.getLastThrowable(), BatchUpdateException.class);
		if (indexOfDbException > -1) {
			Throwable dbUpdateException = ExceptionUtils.getThrowableList(context.getLastThrowable()).get(indexOfDbException);
			if (dbUpdateException.getMessage().contains("deadlock detected")) {
				LOGGER.debug("Retry operation on deadlock (last exception [ {} ]) with attempt count [ {} ]", context.getLastThrowable().getClass().getName(), context.getRetryCount());
				return true;
			}
		}
		return false;
	}
}
3