OptimisticLockingException and endless executed Tasks

i have a problem, that a REST request has been executed multiple times by my camunda workflow (instead of 1 time)

This is a simplified view of the flow (the red part has been executed multiple times):

after log inspection i found the following exception logging

2020-03-04 11:51:52.738 [WARN ] [pool-2-thread-1] o.c.b.e.jobexecutor.logWarn:146 [] ENGINE-14006 Exception while executing job 51559: OptimisticLockingException. To see the full stacktrace set logging level to DEBUG.
...
2020-03-04 11:54:55.360 [WARN ] [pool-2-thread-3] o.c.b.e.jobexecutor.logWarn:146 [] ENGINE-14006 Exception while executing job 51559: OptimisticLockingException. To see the full stacktrace set logging level to DEBUG.
...
2020-03-04 12:02:10.155 [WARN ] [pool-2-thread-2] o.c.b.e.jobexecutor.logWarn:146 [] ENGINE-14006 Exception while executing job 51559: OptimisticLockingException. To see the full stacktrace set logging level to DEBUG.

→ this is continued for more than one hour. After that the flow is back on “normal”

the excecuted java class is doing variable changes like this

    public void process(final DelegateExecution execution) throws Exception {

    try {
        LOGGER.debug("service task started - {}", execution.getVariables());

        // init var(s)
        execution.setVariable(WorkflowConstants.CR_CREATED, 0);

Problem 1:
for me it is unclear why the exception is happening. my assumption is in the moment: The postgress DB was not available for some time and therefore the state could not be updated by camunda inside the DB.
but i dont know how to check if this is a correct assumption?! or is it an error in the flow?

Problem 2
The red part in the flow executes a code, that makes a REST call. This part should never be called more than 1 time. how do i make sure, that this never is repeated? i would prefer to have an abortion of the flow (with an incident or similar…)

the problem happened a second time and i can rule out problems on the DB side.
other ideas:

in the best practice article it is suggested to flag all service tasks with “async before”: https://camunda.com/best-practices/dealing-with-problems-and-exceptions/#additional-save-points
maybe i try this and hope that it helps…

the only other solution would be to write a own implementation: Every service task executed is stored in a separate DB and checked if a double execution happens. but this sounds like a lot of overhead for something that the workflow engine should do for me?!

i have an idea what could lead to random problems.
in our code we clear sometimes the ThreadContext. And i recognized that the new camunda version uses the threadcontext too (we upgraded last week to 7.12…)
maybe there are problems created by this in the transaction handling?

what be nice to her something form a camunda developer about this…

our code:

@Override
  public void execute(final DelegateExecution execution) throws Exception {

      try {
          ThreadContext.put("PID", execution.getProcessInstanceId());

          /*
           * call process method
           */
          process(execution);

      } catch (final Exception ex) {
          sendErrorNotification(execution);
          throw ex;
      } finally {
          ThreadContext.clearAll();
      }
  }

@SirMad based on your bpmn image in the first post, where is your looping occurring that is causing the red task to be called more than once?

Did you manage to solve this problem? I have similar issue. We have one node with bpmn engine and even thouh we sametimes have the

OptimisticLockingException. To see the full stacktrace set logging level to DEBUG

exception. I suspect that it might be caused by long running task.

I faced similar issue with version 7.13? Did any one mange to resolve this?

I had a similar issue with the same WARNING message. What happened to me is that my timer kicked out but hanging. Then my process instance just stuck. Any ideas?

BTW, how to set the logging level to DEBUG at runtime? My version is 7.17.0-alpha4.

Thanks a lot!