** This question is long, but this issue has been kicking my b$tt ,so your time is appreciated **
-
I want to implement an Asynchronous Response scheme in a Camunda flow where I loop thru a list of “suspects”, trigger a subflow, and have the subflow return an Asynchronous Response so the calling flow is not waiting for Subflow to complete to trigger the next call. There would be a small delay between calls to the subflow.
-
I have attempted to use the “Asynchronous Before” or “Asynchronous After” features built into Camunda but have experienced a VERY NASTY side effect. The Asynchronous Before/After feature created a “restart” point in the flow where any uncaught exception (including “Broken Pipe” exceptions) and timeouts will cause the flow to “retry” from the “restart” point. This can balloon very quickly causing threads to be consumed and processing to become hopelessly backed up.
-
I have searched for a way to turn off retries/timeouts but none of the suggestions I have found in the Camunda documentation have worked (I am working in a SPRING BOOT Environment). I really don’t want to go down this road (unless there is something REALLY simple I have missed).
-
Question is…. How do I implement a REST Asynchronous response scenario as described above? I have done this many times before using SOAP, but need to implement a similar scenario using Camunda and REST.
-
Can I use Java to send responses? Is there a way to implement 202 (Accepted) REST Response codes?
|- Once an Async Response has been returned, how to I keep the subflow from trying to return a Synchronous response but not finding it (because the calling flow has terminated) and issuing a “Broken Pipe” exception?
Hi Steven.
Thanks for the detailed question. I still try to understand your exact use case. Do you maybe have a process model showing it?
This is what I understood - but it is not clear to me if you want to wait for the respone and where - and what exactly you want to make async here:
Thanks
Bernd
This is a better representation. To call the subflow, I am using a call using a Java HTTP connector (rather than the built-in Camunda Call Activity. I want to get a message back at the START of the subflow so the main looping flow can trigger the next subflow.
To call the subflow, I am using a call using a Java HTTP connector (rather than the built-in Camunda Call Activity)
That make sense, as you don’t want to wait for completion of the sub process. I wonder, why you use REST and not simply the Java API if you are in Spring boot? But this is actually a minor thing.
I want to get a message back at the START of the subflow so the main looping flow can trigger the next subflow.
Still don’t understand this. You want to have your thread back, so it should not block until the long running thing is completed?
Why exactly would a “asyncBefore=true” on the start event of the subflow not work for you?
And I try to understand if your main process needs to know when all subflows are finished?
Still don’t understand this. You want to have your thread back, so it should not block until the long running thing is completed?
Why exactly would a “asyncBefore=true” on the start event of the subflow not work for you?
Yes, the goal is to not have a thread consumed while the long running subflow is active and allow the Main flow to move on to the next call since I am processing thousands of suspects in a loop.
I have tried “asyncBefore=true” in the flow (just past the Start Event) but the execution engine creates a “restart” point where the async is set. Since this is a long running flow I believe it is timing out and triggering a retry from the “restart”/async point. This creates a real mess of multiple flows running. I have never been able to get the retry settings correct to not exhibit this behavior.
I wonder, why you use REST and not simply the Java API if you are in Spring boot? But this is actually a minor thing
This is an interesting question. My team uses REST calls to the Flows Process Endpoint, but it might make more sense to use the Java API type call to trigger the Process Instance.
Do you think this would have an advantage over the REST call? If yes, what are the advantages?
Ok - we are getting somewhere 
Do you think this would have an advantage over the REST call? If yes, what are the advantages?
Architecture wise not - but it would probably just be simpler in the code. But it will not change much for this problem.
I have tried “asyncBefore=true” in the flow (just past the Start Event) but the execution engine creates a “restart” point where the async is set. Since this is a long running flow I believe it is timing out and triggering a retry from the “restart”/async point.
This is exactly the behavior. We are basically relying on some transaction timeouts of the infrastructure on how long we can keep a transaction open. While this can be tweaked, I would not do anything in that area.
But this sounds that long running is really long - not seconds but minutes or even longer?
In this case you should switch to External Tasks: External Tasks | docs.camunda.org. Then you don’t invoke that long running execution in your thread/tx, but instead, you can provide your own code and poll for work. There you can set the timeout yourself. We have customers using this for transcoding videos for hours. You can even extend the timeout regularly to send some kind of “still alive ping”: Camunda Automation Platform 7.21.0-SNAPSHOT REST API
If you use External Tasks there is no need for async any more.
The External Task is interesting since it decouples the long running Subflow from my Main Looping flow. I need to test it to see if it will really meet my needs.
My gut is telling me that it solves the problem, but the root of the problem was trying to use Camunda for this effort in the first place. Since the “Worker” flow is pulling from a Queue created by the Main Looping flow, I should probably just scrap using Camunda for the Looping flow and just write Java that will read the database directly and process within Java only. Involving the Camunda Process Engine is what is causing my problem.
just write Java that will read the database directly and process within Java only. Involving the Camunda Process Engine is what is causing my problem.
I would definitely rephrase it A workflow engine does not solve your problem of having long running executions that exceed a transaction timeout. Then I could agree.
It could solve the problem if you need to wait for all spanned executions to finish, this is what would work very well and make things very easy (e.g. using a multi instance / external task).
Why did you introduce Camunda in the first place?
Camunda is our preferred platform for various reasons (ease of coding; Production Support; logging in Cockput; etc). My team and I have successfully created dozens of solutions is Camunda that work great. The problem I have detailed in this thread is trying to implement a process in Camunda that does not quite fit (round peg in square hole). The combination of looping thru thousands of suspects; trying to create asynchronous responses for long running processes; and trying to manage fallout successfully is conflicting with what Camunda does best.
Your suggested solution is a good one (e.g. using a multi instance / external task), but I keep finding more problems. I am going to take what I learned from this exchange and move forward the best I can. I will try to update when I have more info.
I wonder, why you use REST and not simply the Java API if you are in Spring boot? But this is actually a minor thing
Hello Team ,
I have usecase were i want to make async rest call to the other micro service.
Wanted inputs on how we can handle the async response/failures in bpmn.
Thank you
