Some jobs not picked by job executor

nrpatel · January 5, 2021, 1:25pm

in my case

ten jobs jobs1 to jobs10 available for execute.

but some jobs not picked by job executor
for example :
jobs2 and jobs3 never picked by job executor. all other jobs work fine but jobs2 and jobs3 never picked by executor.

jobs2 and jobs3 details :

LOCK_EXP_TIME : null
LOCK_OWNER : null
RETRIES_ : 3
EXCEPTION_STACK_ID_ : null
EXCEPTION_MSG_ : null
SUSPENSION_STATE_ : 1
REV_ : 1

configure with embedded process engine.
camunda version : 7.8

this is happen for some time.

any solution for it?

nrpatel · January 5, 2021, 1:43pm

Jobs deployment using :

DeploymentWithDefinitions deployment = repositoryService.createDeployment().tenantId(“user”).addModelInstance(“Rule_2343”+ “.bpmn”, modelInstance).name(“Rule_2343”).deployWithResult();

Create process instance using :

ProcessInstantiationBuilder process=runtimeService.createProcessInstanceByKey(“Rule_2343”);
process.setVariables(paramMap).execute();

nrpatel · January 5, 2021, 2:04pm

Any solution for it?
why this happen?

nrpatel · January 6, 2021, 9:34am

please any solution for it or any more details?

Ingo_Richtsmeier · January 8, 2021, 11:12am

Hi @nrpatel,

what are the jobs doing? What are the next tasks of the process?

Have a look at this thread for further information: All processes are stuck

Hope this helps, Ingo

nrpatel · January 8, 2021, 11:42am

Hi @Ingo_Richtsmeier,

Simple rest call in jobs.
All jobs are service task.
Next tasks of process is also service task.

in my case not all process stuck only some jobs are not picked.

Thanks.

Ingo_Richtsmeier · January 8, 2021, 1:13pm

Hi @nrpatel,

how did you implement the service tasks?

Java Delegates, external service or connector?

Which platform do you use?

Embedded process engine with Spring Boot, Embedded Engine deployed as war, Shared Engine with Tomcat or CamundaBPM Run?

nrpatel · January 8, 2021, 1:55pm

Hi @Ingo_Richtsmeier,

Service tasks implement using java delegates

Embedded process engine with spring boot
Embedded Engine deployed as jar
Share Engine with tomcat

Thanks.

Ingo_Richtsmeier · January 11, 2021, 11:48am

Hi @nrpatel,

could you elaborate on this:

Do you refer to the tomcat server, spawned by spring boot?

nrpatel · January 12, 2021, 6:48am

Hi @Ingo_Richtsmeier,

Configuration of process engine with spring boot like
https://docs.camunda.org/get-started/spring/embedded-process-engine/
default configuration.

and deployment and create process instance like.

Jobs deployment using :

DeploymentWithDefinitions deployment = repositoryService.createDeployment().tenantId(“user”).addModelInstance(“Rule_2343”+ “.bpmn”, modelInstance).name(“Rule_2343”).deployWithResult();

Create process instance using :

ProcessInstantiationBuilder process=runtimeService.createProcessInstanceByKey(“Rule_2343”);
process.setVariables(paramMap).execute();

Ingo_Richtsmeier · January 12, 2021, 9:17am

Hi @nrpatel,

in this case, you should start to have a look at the Job execution from the database level. Have a look at the docs, which log categories to use: https://docs.camunda.org/manual/7.14/user-guide/logging/#diagnosing-job-execution

Hope this helps, Ingo

nrpatel · January 12, 2021, 9:22am

Hi @Ingo_Richtsmeier ,

Thanks for quick replied.

No any database log for this particular jobs.

thanks

Ingo_Richtsmeier · January 12, 2021, 9:28am

Hi @nrpatel,

what about the acquistion statements? Which where-clauses are responsible that the jobs are left untouched?

nrpatel · January 12, 2021, 9:33am

Hi @Ingo_Richtsmeier ,

This particular jobs no any acquistion statements.

this jobs are untouched not for acquistion or execution.

This case happened some times not any particular case.

Thanks.

Ingo_Richtsmeier · January 13, 2021, 9:00am

Hi @nrpatel,

could you please post the SQL statement for aqcuisition with all parameters and the rows of ACT_RU_JOB that were not picked up?

nrpatel · January 13, 2021, 10:22am

Hi @Ingo_Richtsmeier ,

ACT_RU_JOB table row.

ID_	REV_	TYPE_	LOCK_EXP_TIME_	LOCK_OWNER_	EXCLUSIVE_	EXECUTION_ID_	PROCESS_INSTANCE_ID_	PROCESS_DEF_ID_	PROCESS_DEF_KEY_	RETRIES_	EXCEPTION_STACK_ID_	EXCEPTION_MSG_	DUEDATE_	REPEAT_	HANDLER_TYPE_	HANDLER_CFG_	DEPLOYMENT_ID_	SUSPENSION_STATE_	PRIORITY_	JOB_DEF_ID_	SEQUENCE_COUNTER_	TENANT_ID_
9805de63-3a2e-11eb-b9af-001dd8b71caa	1	message	NULL	NULL	1	98019892-3a2e-11eb-b9af-001dd8b71caa	98019892-3a2e-11eb-b9af-001dd8b71caa	68a1b88e-2f50-11eb-b9af-001dd8b71caa	Rule_2343	3	NULL	NULL	09-12-2020 14:55:36	NULL	async-continuation	transition-create-scope	689f95ac-2f50-11eb-b9af-001dd8b71caa	1	0	68a1b88f-2f50-11eb-b9af-001dd8b71caa	1	NULL
a9348f53-3a0a-11eb-b9af-001dd8b71caa	1	message	NULL	NULL	1	a9344122-3a0a-11eb-b9af-001dd8b71caa	a9344122-3a0a-11eb-b9af-001dd8b71caa	68a1b88e-2f50-11eb-b9af-001dd8b71caa	Rule_2344	3	NULL	NULL	09-12-2020 10:38:23	NULL	async-continuation	transition-create-scope	689f95ac-2f50-11eb-b9af-001dd8b71caa	1	0	68a1b88f-2f50-11eb-b9af-001dd8b71caa	1	NULL

no any sql statement for this particular jobs.

Thanks.

Ingo_Richtsmeier · January 14, 2021, 1:09pm

Hi @nrpatel,

I added this to my application.yml:

logging:
  level:
    '[org.camunda.bpm.engine.jobexecutor]': DEBUG
    '[org.camunda.bpm.engine.impl.persistence.entity.JobEntity]': DEBUG
    '[org.camunda.bpm.engine.cmd]': INFO

and found this output on my console:

2021-01-14 10:59:47.113 DEBUG 5048 --- [ingJobExecutor]] org.camunda.bpm.engine.jobexecutor       : ENGINE-14012 Job acquisition thread woke up
2021-01-14 10:59:47.114 DEBUG 5048 --- [ingJobExecutor]] o.c.b.e.i.p.e.J.selectNextJobsToExecute  : ==>  Preparing: select RES.ID_, RES.REV_, RES.DUEDATE_, RES.PROCESS_INSTANCE_ID_, RES.EXCLUSIVE_ from ACT_RU_JOB RES where (RES.RETRIES_ > 0) and ( RES.DUEDATE_ is null or RES.DUEDATE_ <= ? ) and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < ?) and RES.SUSPENSION_STATE_ = 1 and ( ( RES.EXCLUSIVE_ = 1 and not exists( select J2.ID_ from ACT_RU_JOB J2 where J2.PROCESS_INSTANCE_ID_ = RES.PROCESS_INSTANCE_ID_ -- from the same proc. inst. and (J2.EXCLUSIVE_ = 1) -- also exclusive and (J2.LOCK_OWNER_ is not null and J2.LOCK_EXP_TIME_ >= ?) -- in progress ) ) or RES.EXCLUSIVE_ = 0 ) LIMIT ? OFFSET ? 
2021-01-14 10:59:47.115 DEBUG 5048 --- [ingJobExecutor]] o.c.b.e.i.p.e.J.selectNextJobsToExecute  : ==> Parameters: 2021-01-14 10:59:47.113(Timestamp), 2021-01-14 10:59:47.113(Timestamp), 2021-01-14 10:59:47.113(Timestamp), 3(Integer), 0(Integer)
2021-01-14 10:59:47.116 DEBUG 5048 --- [ingJobExecutor]] o.c.b.e.i.p.e.J.selectNextJobsToExecute  : <==      Total: 0
2021-01-14 10:59:47.116 DEBUG 5048 --- [ingJobExecutor]] org.camunda.bpm.engine.jobexecutor       : ENGINE-14022 Acquired 0 jobs for process engine 'default': []

After reformating my acquition statement looks like this:

select 
    RES.ID_, 
    RES.REV_, 
    RES.DUEDATE_, 
    RES.PROCESS_INSTANCE_ID_, 
    RES.EXCLUSIVE_ 
from ACT_RU_JOB RES where 
(RES.RETRIES_ > 0) 
and ( RES.DUEDATE_ is null or RES.DUEDATE_ <= ? ) 
and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < ?) 
and RES.SUSPENSION_STATE_ = 1 
and 
( ( RES.EXCLUSIVE_ = 1 and not exists( 
    select 
      J2.ID_ from ACT_RU_JOB J2 
    where 
      J2.PROCESS_INSTANCE_ID_ = RES.PROCESS_INSTANCE_ID_ -- from the same proc. inst. 
      and (J2.EXCLUSIVE_ = 1) -- also exclusive 
      and (J2.LOCK_OWNER_ is not null 
      and J2.LOCK_EXP_TIME_ >= ?) -- in progress 
) ) or RES.EXCLUSIVE_ = 0 ) LIMIT ? OFFSET ?

Having a look into the SQL statement and comparing the conditions your your job entries I could not find a reason while they are not picked up.

How does the acquistion statement look like in your application?

Hope this helps, Ingo

nrpatel · January 18, 2021, 4:47am

Hi @Ingo_Richtsmeier,

Same SQL statement for job acquistion in my application. any particular reason why this happening?

Thanks for replied.

nrpatel · January 21, 2021, 4:43am

Hi @Ingo_Richtsmeier,

job executor not stuck in this case because of all other jobs picked fine.
after restart service working fine for all jobs.
but some times happening for some jobs. any have some particular reason?

Thanks.

system · January 30, 2024, 12:49pm