Long polling for fetchAndLock of external tasks

Hi,
When using external tasks I need to poll the topics to pick up work to be done. This can be done by using the REST interface for external task/fetchAndLock.

Is it common practice to poll every seconds, or is it also possible to apply some sort of long polling for fetchAndLock? For example, would it be possible to supply something similar to the maxWait parameter in the “from push to pull”-article. The solution should also work when multiple servers are pulling in work to be done.

Is such a long polling feature possible with the current API/REST interface and/or are there plans on the roadmap to add this functionality? Or is this not the recommended approach for pulling in work?

Btw. What I am actually looking for is a way to be able to define a parallel execution and have multiple services called without waiting for one another to be completed. And when all service calls are complete to proceed with the remainder of the process.
As I suppose this is not possible using Java based Service tasks; I am investigating whether external tasks could help in achieving this. Therefore I look for an efficient way to pull external tasks so I can execute them concurrently and then complete this task per invoked service in the given process instance.

Hi Patrik,

external tasks aren’t the best solution for your use case. Please have a look at asynchronous continuation which allows you to execute activities (e.g. service tasks) asynchronous by the job executor.

Best regards,
Philipp

Thanks for your answer. Unfortunately I seem not be able to get this working in a true parallel way.

What I tried is asyncBefore and asyncAfter on both the Service Tasks as well as on the parallel gateways in front and after the service tasks that I want in parallel. But either the services are executed sequentially or I get an OptimisticLockException.

Can you please explain how to achieve the goal of parallel execution?

I used the Docker image to have a clean setup and use a Thread.sleep to similate the (blocking) wait for an external service invocation.
Perhaps I need to apply non standard settings in the engine to get this working.

Hi,

Ive attached a very simple process using groovy script to demo actual parallel execution.

The ‘tricks’ are;
Use a fork gate to split the flow.
For each parallel path, mark it as async before and non exlusive
Use a join gate and mark the join gate as async before.

In this demo, the paths use a different sleep so the execution interleaves…

regards

RobParallel_Test.bpmn (13.8 KB)

1 Like

Hi,

Lots of thanks for your quick reply. I tried it and it executes in parallel as it should.

It made me notice however, that I was not complete in my question. What I consider a typical use case, is that I use the service tasks to retrieve data from external systems and store them as process variables, so I can process them after the joining point and continue processing based on those process variables.

I extended your example to assign a result to variable per ’ lane’ . So a resultA and a resultB variable. Despite the fact that these are distinct variables, setting them introduces the OptimisticLockException for flow B; as this has the higher sleep-value.
In this case, the B-part is rolled back and executed again as a whole, because the async boundary on node Task B. So in the end it is still sequential.

My conclusion for now is that it is possible to execute in parallel and also to invoke other services in parallel. However, it seems not be possible to set process variables when service tasks are executed in parallel. Even if the variables are not shared.

Can you help me by explaining how to achieve keeping the retrieved data at hand, preferably via process variables? Or could it be that my use case is not supported in this version of the platform?

Kind regards,
Patrick

UPDATE 2016-11-18: Rob had reformulated my question here. To me it is fine if the discussion continues in that thread.

Long Polling for external tasks is provided with Camunda 7.9+

Hello guys.

FetchAndLock does not blocks if asyncResponseTimeout is specified. I use Spring Boot
<camunda.version>7.9.0</camunda.version>
<camundaSpringBoot.version>3.0.0</camundaSpringBoot.version>
<springBoot.version>2.0.2.RELEASE</springBoot.version>
Do you have plans to fix it for SpringBoot?

Hi Andrey,

I should work.
Could you please provide your pom.xml and specify which rest endpoint do you use.

Best regards,
Yana

Thank you for cooperation!

pom.xml (5.9 KB)

POST http://localhost:8080/rest/engine/default/external-task/fetchAndLock
{
“asyncResponseTimeout”: 100000,
“workerId”:“worker”,
“maxTasks”:1,
“usePriority”:true,
“topics”: [{“topicName”: “worker_inbound-audit”, “lockDuration”: 5000}]
}

Hi Andrey,

It should work.
Could you please upload a sample of your project on github where I can try to reproduce the issue.

Best regards,
Yana

The source code may be found here: https://www.dropbox.com/s/1ny71tvp75xd20m/camunda.zip?dl=0

Eclipse:
New Maven project
camunda-archetype-spring-boot 7.9.0
Updated the process.bpmn to change service to External Task with topic: topic
maven install
CamundaApplication.java -> Run As -> Java Application

Camunda WEb console (demo / demo)
Tasklist -> Start process -> camunda
http://localhost:8080/rest/engine/default/external-task/fetchAndLock
{
“asyncResponseTimeout”: 10000,
“workerId”:“worker”,
“maxTasks”:1,
“usePriority”:true,
“topics”: [{“topicName”: “topic”, “lockDuration”: 50000}]
}
It returns data (which is correct), but consecutive fetchAndLock API calls do not block for asyncResponseTimeout: 10000 and return [] immediately

Hi Andrey,

I was able to reproduce it just like you say and I will create a bug report for it.
The problem is that you are using camunda-bpm-spring-boot-starter-webapp which contains dependency to camunda-engine-rest which does not support long polling and it is also mapped to the same endpoint.

(I don’t know if it is worth to try to exchange the both dependency camunda-bpm-spring-boot-starter-webapp and camunda-bpm-spring-boot-starter-rest in your pom file so that the starter-rest is before the webapp.
This could help to avoid the mapping to the camunda-engine-rest endpoint however it is really not sure.)

Best regards,
Yana

Hi Andrey,

Thank you for reporting the problem,
here is the created issue CAM-9141 [1]

Best regards,
Yana

[1] https://app.camunda.com/jira/browse/CAM-9141

Thank you!

Swapping camunda-bpm-spring-boot-starter-webapp and camunda-bpm-spring-boot-starter-rest worked!

Best regards, Andrey.

Be aware that is not permanent solution and could happen that the error will occur again.

1 Like

I had the same problem and could solve it by changing the webapp dependency to

		<dependency>
			<groupId>org.camunda.bpm.springboot</groupId>
			<artifactId>camunda-bpm-spring-boot-starter-webapp</artifactId>
			<version>${camunda-bpm-spring-boot-starter.version}</version>
			<exclusions>
				<exclusion>
					<groupId>org.camunda.bpm</groupId>
					<artifactId>camunda-engine-rest-core</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

This seems more stable to me than changing the order of the dependencies.

Of course it would be even nicer if the dependency in the webapp spring boot starter was updated to use the camunda-engine-rest-jaxrs2 jar.

I’m just pushing this because I landed here with a similar problem after digging for a solution for a while.

When using spring boot, there is still a problem with concurring dependencies!
Long polling will just work if the “camunda-engine-rest-core” from “camunda-bpm-spring-boot-start-rest” is used (either by dependency order or by excluding it from “camunda-bpm-spring-boot-starter-webapp” and your custom camunda-webapp-webjar if used).

(Using v7.13)

The bug still exits! After 3 years!

If the bug doesn’t get fixed at least the order of the dependencies in the archetypes could be fixed.

This is really annoying, cost me almost a day to find the problem and the solution, which is still the same:

(rest has to be BEFORE webapp)

Sorry to hear you had a hard time with this, but i’m glad you found the solution.
Can you perhaps comment on the JIRA ticket that Yana linked?

Hi Niall, thanks for the response. I’m sorry but I don’t have the permission to write comments there. Maybe our project manager can talk with our Camunda contact person. I’m just the developer.