Long polling for fetchAndLock of external tasks


#1

Hi,
When using external tasks I need to poll the topics to pick up work to be done. This can be done by using the REST interface for external task/fetchAndLock.

Is it common practice to poll every seconds, or is it also possible to apply some sort of long polling for fetchAndLock? For example, would it be possible to supply something similar to the maxWait parameter in the “from push to pull”-article. The solution should also work when multiple servers are pulling in work to be done.

Is such a long polling feature possible with the current API/REST interface and/or are there plans on the roadmap to add this functionality? Or is this not the recommended approach for pulling in work?

Btw. What I am actually looking for is a way to be able to define a parallel execution and have multiple services called without waiting for one another to be completed. And when all service calls are complete to proceed with the remainder of the process.
As I suppose this is not possible using Java based Service tasks; I am investigating whether external tasks could help in achieving this. Therefore I look for an efficient way to pull external tasks so I can execute them concurrently and then complete this task per invoked service in the given process instance.


#2

Hi Patrik,

external tasks aren’t the best solution for your use case. Please have a look at asynchronous continuation which allows you to execute activities (e.g. service tasks) asynchronous by the job executor.

Best regards,
Philipp


#3

Thanks for your answer. Unfortunately I seem not be able to get this working in a true parallel way.

What I tried is asyncBefore and asyncAfter on both the Service Tasks as well as on the parallel gateways in front and after the service tasks that I want in parallel. But either the services are executed sequentially or I get an OptimisticLockException.

Can you please explain how to achieve the goal of parallel execution?

I used the Docker image to have a clean setup and use a Thread.sleep to similate the (blocking) wait for an external service invocation.
Perhaps I need to apply non standard settings in the engine to get this working.


#4

Hi,

Ive attached a very simple process using groovy script to demo actual parallel execution.

The ‘tricks’ are;
Use a fork gate to split the flow.
For each parallel path, mark it as async before and non exlusive
Use a join gate and mark the join gate as async before.

In this demo, the paths use a different sleep so the execution interleaves…

regards

RobParallel_Test.bpmn (13.8 KB)


#5

Hi,

Lots of thanks for your quick reply. I tried it and it executes in parallel as it should.

It made me notice however, that I was not complete in my question. What I consider a typical use case, is that I use the service tasks to retrieve data from external systems and store them as process variables, so I can process them after the joining point and continue processing based on those process variables.

I extended your example to assign a result to variable per ’ lane’ . So a resultA and a resultB variable. Despite the fact that these are distinct variables, setting them introduces the OptimisticLockException for flow B; as this has the higher sleep-value.
In this case, the B-part is rolled back and executed again as a whole, because the async boundary on node Task B. So in the end it is still sequential.

My conclusion for now is that it is possible to execute in parallel and also to invoke other services in parallel. However, it seems not be possible to set process variables when service tasks are executed in parallel. Even if the variables are not shared.

Can you help me by explaining how to achieve keeping the retrieved data at hand, preferably via process variables? Or could it be that my use case is not supported in this version of the platform?

Kind regards,
Patrick

UPDATE 2016-11-18: Rob had reformulated my question here. To me it is fine if the discussion continues in that thread.


#6

Long Polling for external tasks is provided with Camunda 7.9+


#7

Hello guys.

FetchAndLock does not blocks if asyncResponseTimeout is specified. I use Spring Boot
<camunda.version>7.9.0</camunda.version>
<camundaSpringBoot.version>3.0.0</camundaSpringBoot.version>
<springBoot.version>2.0.2.RELEASE</springBoot.version>
Do you have plans to fix it for SpringBoot?


#8

Hi Andrey,

I should work.
Could you please provide your pom.xml and specify which rest endpoint do you use.

Best regards,
Yana


#9

Thank you for cooperation!

pom.xml (5.9 KB)

POST http://localhost:8080/rest/engine/default/external-task/fetchAndLock
{
“asyncResponseTimeout”: 100000,
“workerId”:“worker”,
“maxTasks”:1,
“usePriority”:true,
“topics”: [{“topicName”: “worker_inbound-audit”, “lockDuration”: 5000}]
}


#10

Hi Andrey,

It should work.
Could you please upload a sample of your project on github where I can try to reproduce the issue.

Best regards,
Yana


#11

The source code may be found here: https://www.dropbox.com/s/1ny71tvp75xd20m/camunda.zip?dl=0

Eclipse:
New Maven project
camunda-archetype-spring-boot 7.9.0
Updated the process.bpmn to change service to External Task with topic: topic
maven install
CamundaApplication.java -> Run As -> Java Application

Camunda WEb console (demo / demo)
Tasklist -> Start process -> camunda
http://localhost:8080/rest/engine/default/external-task/fetchAndLock
{
“asyncResponseTimeout”: 10000,
“workerId”:“worker”,
“maxTasks”:1,
“usePriority”:true,
“topics”: [{“topicName”: “topic”, “lockDuration”: 50000}]
}
It returns data (which is correct), but consecutive fetchAndLock API calls do not block for asyncResponseTimeout: 10000 and return [] immediately


#12

Hi Andrey,

I was able to reproduce it just like you say and I will create a bug report for it.
The problem is that you are using camunda-bpm-spring-boot-starter-webapp which contains dependency to camunda-engine-rest which does not support long polling and it is also mapped to the same endpoint.

(I don’t know if it is worth to try to exchange the both dependency camunda-bpm-spring-boot-starter-webapp and camunda-bpm-spring-boot-starter-rest in your pom file so that the starter-rest is before the webapp.
This could help to avoid the mapping to the camunda-engine-rest endpoint however it is really not sure.)

Best regards,
Yana


#13

Hi Andrey,

Thank you for reporting the problem,
here is the created issue CAM-9141 [1]

Best regards,
Yana

[1] https://app.camunda.com/jira/browse/CAM-9141


#14

Thank you!

Swapping camunda-bpm-spring-boot-starter-webapp and camunda-bpm-spring-boot-starter-rest worked!

Best regards, Andrey.


#15

Be aware that is not permanent solution and could happen that the error will occur again.