"Strange" behaviour of job executor

Hi folks,
I have a little bit strange behaviour of job executor. I will try to explain what happens.

I’m using Spring Boot 1.4.0, MySQL 5.7.21 and embedded Camunda Engine. In early morning hours I have a load in MySQL database from external source. That load is heavy and it’s importing a lot of data and it lasts for 2-3 hours.

After the load is finished my daily workflow process is started by executing query to fetch all candidates for which process need to be started (it’s a service task). Sometimes it happens that query lasts longer then default execution time and incident occur (after few retries). After the incident occurred, I’m doing a retry in Camunda Cockpit and the token waits in service task. In ACT_RU_JOB table I’m seeing the job that is locked with time in future (which is ok I guess). But it happens that engine delays that job for few times (I was waiting almost for two hours for job to be picked and executed).

Anyone has an idea?

Can you show what your job executor settings are?

Yes, of course.

Here is the configuration:

camunda:
  bpm:
    id-generator: simple
    auto-deployment-enabled: true
    application:
      deploy-changed-only: true
    admin-user:
      id: admin
      password: admin
      first-name: admin
    filter:
      create: All tasks
    history-level: "custom-mode"
    database:
      schema-update: true
    job-execution:
        default-number-of-retries: 6

and also:

job:
  executor:
    lock-time-in-millis: 900000

It’s a custom history, nothing special. On variable update I’m checking if variable is type of specific kind of object, and if it is, I’m not going to save that object in history tables.

That’s the whole configuration. All other settings are default.

About hardware:
4 CPU
8GB RAM
CentOS 7.9
Java 1.8
MySQL 5.7.21
Spring Boot 1.4.0

@Niall any ideas?