External task and multi-instance subprocess, best practice?

Hi Camunda Community,

I would love to have some feedbacks on this subject. I would like to implement an external task in assocation with a multi-instance subprocess.

In details, the context is a notification sender, the multi-instance iterates over receivers of the notification, within the multi-instance subprocess I generate the json containing the body of the notification, then an external worker sends the json object to a Kafka message queuer.

Option1: Putting the external worker within the loop (probably not the best option since the loop is stopped while the worker finishes it’s process)

Option2: Putting the external worker outside of the loop and triggered with a signal, so that process instances are stacked, then the external worker subscribing to the topic processes all instances present in that topic.

Question: What is the best practice? Is there another more suitable option?

Bonus question :slight_smile: : How camnuda handles memory usage in option2? RAM? Camunda Database?

Thanks in advance for your feedback on this! cheers,

Ch.

Hi @ChrisB,

your question is quite difficult to follow, because you are mixing a few topics that usually didn’t belong together.

But let me sort out the details:

You want to use an external task worker to send data into a kafka queue.
You have a process around, that generates the data, based on some input data.

An external task worker is an external program running outside the process engine (https://docs.camunda.org/manual/7.12/user-guide/process-engine/external-tasks/#the-external-task-pattern). The process contains an external service task to make this happen.

If you use the external task client for Java, it will run in an endless loop until you stop it and constanly poll (in a configurable interval) for new tasks and invoke the handler for the task. (https://docs.camunda.org/manual/7.12/user-guide/ext-client/#bootstrapping-the-client)

Why do you name your task “external task worker”? Do you want to control the worker with this process and avoid the endless loop? Or is the process only about generating and sending the data and you named the task misleadlingly?

Hope this helps, Ingo

Hi @Ingo_Richtsmeier,

Sorry for not being that clear. My “external task worker” should indeed be renamed “external service task” . Yes indeed, I want to use an external task worker to send data into a kafka queue. I guess that putting the “service task” inside the loop (option 1) will not stop the loop after each iteration right? But it will stop the execution of the process once the handler is invoked (configurable interval) and wait for all tasks to be completed by the external worker to resume to the next iteration.

I would prefer not to wait for the external worker to complete to resume to the next iteration, so I guess option 2 suits better to my needs right?

Thanks for the answer and sorry for the earlier confusion.

Ch