In our environment, we run camunda as springboot containers in a kubernetes, and sometimes k8s move containers from one node to another. In our microservices, its safe because the spring graceful shutdown is enough, and the remaining pods have resources enough to sustain our load while the new pod is starting.
But in our camunda springboot projects i can see that every now and then, the container shutdown but we still have some jobs locked to it, wich will be unlocked only after LOCK_EXP_TIME_. While i’m writting this, i’m with 3 jobs locked in this situation and i have to set LOCK_OWNER and LOCK_EXP_TIME to null, so it can be executed by another pod. I can see other jobs stuck in this situation too but with LOCK_OWNER and LOCK_EXP_TIME null in ACT_RU_JOB table, maybe there is a lock somewhere else?
Whats the best way to implement a graceful shutdown for camunda projects? maybe a way to disable job acquisition, but keep the job executor awake until the job queue is empty?