Scaling Heterogeneous Cluster With Deployment Aware Job Executor

Hello Folks,

I have been reviewing the docs and forums but haven yet found a clear answer to my question regarding the effect of the “jobExecutorActivate” setting in a Heterogeneous Cluster involving more than two servers.

My Spring project contains source code including delegates for two Tenants, the configuration for what should be created by spring is handled using a package scan:
<context:component-scan base-package="com.pcorbett.app.tenantA" />
or
<context:component-scan base-package="com.pcorbett.app.tenantB" />

This means that a single war File will be configured by server Environment Varibables to create either an Instance for TenantA or TenantB.

My spring Application also creates an Embedded Process Engine with the following config for each Application Type (TenantA and TenantB):

<bean id="processEngineConfiguration" class="org.camunda.bpm.engine.spring.SpringProcessEngineConfiguration">
    <property name="processEngineName" value="default" />
    <property name="dataSource" ref="camundaDataSource" />
    <property name="transactionManager" ref="transactionManager" />
    <property name="history" value="audit" />
    <property name="databaseSchemaUpdate" value="${camunda_databaseSchemaUpdate}" />

    <!--
    	the jobExecutorActivate must be actoivtaed when dealing with wait events, see 
    	https://docs.camunda.org/manual/7.6/user-guide/process-engine/the-job-executor/ 
    -->
    <property name="jobExecutorActivate" value="true" />
    
    <!-- get jobs by "ascending due date" so timers are fired earlier -->
    <property name="jobExecutorAcquireByPriority" value="false" />
    <property name="jobExecutorPreferTimerJobs" value="true" />
    <property name="jobExecutorAcquireByDueDate" value="true" />
    
    <!--
     Disable batch processing for oracle 10, 11 and in this case 12 - 
     https://forum.camunda.io/t/fresh-installation-fails-on-startup-oracle-12c/6259/3 
    -->
    <property name="jdbcBatchProcessing" value="false" />

    <property name="deploymentResources" value="classpath:${tenant}/*.bpmn" />
    <property name="deploymentTenantId" value="${tenant}" />
</bean>

<bean id="processEngine" class="org.camunda.bpm.engine.spring.ProcessEngineFactoryBean">
    <property name="processEngineConfiguration" ref="processEngineConfiguration" />
</bean>

This all works as expected and i have the database up and running the Deployments are also defined as being for a specific tenant and most of the time this works well.

In the last few days i have noticed ClassNotFound issues relating to my JavaDelegates similar to This Thread.

The issue is that as i have two deployments and they are not identical (Heterogeneous):

  1. Application for TenantA with the delegates & bpmn files for TenantA
  2. Application for TenantB with the delegates & bpmn files for TenantB

I understand that the Job Executor can run into issues if the Job Executor from TenantA Node gets hold of a job that should be handled by the TenantB Node.

To solve this i plan on using <property name="jobExecutorDeploymentAware" value="true" /> in my engine configuration, however how does this affect the scalablity of my application?.

Right now i only have two nodes one of each Application type, what happens when i have two instances of each application like this:

Does the jobExecutorDeploymentAware option bind the Job to a specific Node or to the Deployment → all Nodes where the deployement is available?

I just want to make sure that if a Job is started in Application B Node 3 and that if this Node for whatever reason is not available that the Job will also be picked up and handled by Application B Node 4.

The configuration seems to work well; based on some initial tests with three server nodes and a single database

  1. Node 1 - TenantA
  2. Node 2 - TenantB
  3. Node 3 - TenantB

Each of the Nodes process engine configuration was modified to include the following configuration entry.

<!-- 
   	Fix for Keeping Jobs tied to the respective Process Engine & Tenant
	https://docs.camunda.org/manual/7.8/user-guide/process-engine/the-job-executor/#deployment-aware-job-executor
-->
<property name="jobExecutorDeploymentAware" value="true" />

I made sure that the latest deployment of the bpmn files was included in the War file as we use spring auto deployment.

I Started a new Process for Tenant B and then paused the Process at a specific part of my workflow using Remote Debugging in a Java Delegate. I then stopped the paused server -> the job executor on the second Tenant B instance then picked up the job up and completed it!

Perhaps someone has some experience with a similar setup?
Can anyone see any drawbacks to this setup?

1 Like

Hi @patrick.corbett,

your setup is the one to go for your use case.

Cheers, Ingo

Hi @Ingo_Richtsmeier,
In order for Heterogeneous Cluster to work, besides setting Deployment Aware to true, we also need to ensure different applications, which contains different BPMN files, are set with different tenant IDs. Is it correct?

Hi @run,

it depends… Multi tenancy is used to separate the data that no tenant can access the data of another tenant.

If you have this requirement from your customers, then yes. If all process instances belong to the same company, then no.

Hope this helps, Ingo

Hi @Ingo_Richtsmeier,
I had this doubt is because setting Deployment Aware alone seems not working. I have two Spring Boot Camunda process applications, each with different application name, different BPMN, different classes but connecting to the same DB. But one would pick up jobs created by the other and resulted error. Only after setting different tenantId, this error does not happen again.

Hi @run,

Abhishek went through all the pain points to setup a heterogenous cluster in Spring Boot. Here you can watch all his findings: https://www.youtube.com/watch?v=Nx4I8lNMUs0

Hope this helps, Ingo

Hi @Ingo_Richtsmeier,
Thanks for the link. But this video does not really solve the Heterogeneous cluster issue. It’s mainly using asyc continuity to pass the job to the main process. I used the same asyc continuity approach. Unfortunately, it does not solve the scenario I described earlier.

Hi @run, like @Ingo_Richtsmeier said the usage of tenant ids is not specially required based on your described use case.

You have already activated the “jobExecutorDeploymentAware” option, this alone is responsible for the binding of the running Job executor instance to only registered processes see docs about job-execution-in-heterogeneous-clusters

If you say that the incorrect node is still picking up jobs from the other node then it could be related to how the transactions in the process if any are setup exist. Which engine/application is responsible for starting the execution thread?

Another issue could be related to using the SpringAutoDeployment feature in multiple applications with a shared Database. The default logic for SpringAutoDeployment deploys the processes using the deployment name “SpringAutoDeployment”. After a restart of the Applications only the last deployment with this name will be restarted meaning the jobs could potentially be retrieved by the wrong engine.

      Map<String, ResourceEntity> existingResources = commandContext
      .getResourceManager()
      .findLatestResourcesByDeploymentName(
          candidateDeployment.getName(),
          candidateResources.keySet(),
          source,
          candidateDeployment.getTenantId());

Using the EnableProcessApplication annotation on your Application class can help solve this issue as the deployment name can be easily defined.

@EnableProcessApplication("APP2-ProcessApp")

Its difficult to say with certainty why your having issues without more detailed information about your setup and processes but maybe these ideas help.

Hi
I want to run heterogeneous cluster as follows
Node 1, Node 2: Process Application A (sales Process)
Node 3, Node 4: Process Application B (marketing process)
(Multiple Engines | docs.camunda.org)
when I need to access the respective process-engines, do I have to set up two domains for it
or run on two port?
rest: http://localhost:8080/engine-rest
rest: http://localhost:1234/engine-rest

If I choose heterogeneous cluster architecture, will my nodes be able to configure autoScale, like can it run on k8s? I will install replicaset and HPA (hpa autoscaling kubernetes)
when That data has an problem ClassNotFoundException with job doing not when which scale many pod same running on a worker node of k8s

regards,

Hi @run,

Can you tell me how and where this tenant configuration You’re talking about can be kept to ensure camunda service A do not acquire jobs started by camunda service B apart from declaring property jobExecutorDeploymentAware to be true in the processes.xml file.

Thanks!!

Hi @patrick.corbett,

I’m still facing the issue of jobs getting picked by wrong node even after using deployment aware property in process.xml file. Can you tell what else can be done to prevent this. I’m also using @EnableProcessApplication annotation over the main method of spring boot application. Challenge with me is, there is already node 1 running with it’s instances and I’m trying to deploy node 2 with different bpmn files so any changes related to workflows can be done only in bpmn files of node 2. From the video shared by @Ingo_Richtsmeier, I’ve also checked asynchronous flag as true for my newly created java delegates but nothing is turning out to be working for me. Do you have any suggestions for me to solve this problem.

Thanks for your response !!