Hi,
Trying to implement Camunda 8 in our ecosystem, Have been trying to use the openshift Helm Charts provided for Camunda 8 installation.
Has been getting the foloowing error
024-02-28 15:59:03.669 [grpc-executor-0] DEBUG
io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/ActivateJobs as the authorized tenants could not be retrieved from Identity. Error message: Thread limit exceeded replacing blocked worker
2024-02-28 15:59:03.675 [grpc-executor-0] DEBUG
io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/ActivateJobs as the authorized tenants could not be retrieved from Identity. Error message: Thread limit exceeded replacing blocked worker
2024-02-28 15:59:03.764 [grpc-executor-0] DEBUG
io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/ActivateJobs as the authorized tenants could not be retrieved from Identity. Error message: Thread limit exceeded replacing blocked worker
2024-02-28 15:59:03.767 [grpc-executor-0] DEBUG
io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/ActivateJobs as the authorized tenants could not be retrieved from Identity. Error message: Thread
Operate and TaskList are to trying to get information. But getting this error
Hi @pavan_Kumar - while I’ve not seen this particular error myself yet, my immediate guess is that it has to do with available resources for the broker/gateway (“Thread limit exceeded”). I’ve asked our engineers for some more detail; it is a debug log, so perhaps it’s being overly verbose and is safe to ignore.
Can you share your values.yaml file? (With secrets and sensitive details redacted of course!)
Hi Nathan,
Thanks for the response, I am in a kind of strange situation, We have Air Gapped environments and our openshift env is restricted with policies and security.
So inorder to access the Identity, Operate and Tasklist outside of openshift, I need to create ReEncrypted routes and we have to install the service with tls annotations.
so in order to make service TLS enabled, I have marked the application Identity, Operate and TaskList as server.ssl.enabled: true
When the Modules are marked SSL Enabled, the Inter POD communication is dirupted and the above error is one of them
Standalone Gateway is trying to connect Identity service, since the Multitenancy is enabled and Identity is exposing HTTPS url which cannot be accessed by.
Gateway is not able to connect to Identity and get the tenant information
If I mark the modules Identity, Operate and Tasklist as SSL disabled. The interpod communication works fine, I do not see the error anymore.
But the same webapplication are not accessible via routes since they expect the service to be HTTPS enabled.
I wish if Camunda web application can also expose 8443 port similar to Keycloak. where Keycloak exposes both 8080 and 8443.
I am able to access keycloak outside of Openshift which is deployed with Camunda 8 modules.
Ingress is not allowed here. Am I missing something ? Please suggests
Hi @pavan_Kumar - that is an environment, with a lot of special cases, that I am not familiar with. Do you have an enterprise agreement with Camunda? I think this type of complex architecture is something that our consulting and support teams excel at; if you have an enterprise agreement, I would strongly recommend reaching out them!
Hi Nathan,
we have 7 enterprise edition which is already implemented, I am working on Camunda 8 POC for future implementation. I badly need a way to expose the services outside openshift… with HTTPS enabling each service is creating its own Certs and inter pod communication is throwing PKIX error. not sure if I am missing something here…
Yes, this is the issue. I know this. I am discussing with our Infra team as well. I am just checking this If I have missed any option Camunda is providing regarding this.
Similar to KeyCloak, If Camunda Web apps also have the option of providing 8080 and 8443 ports, then my problem would be resolved in 2 mins .
@pavan_Kumar - Camunda doesn’t do anything with certificates; that is all part of your infrastructure. Regarding the ports, you can expose whatever ports you want, they can be overridden in the Helm values. (It also depends on how you have your ingress configured, of course.)
@pavan_Kumar - I also discussed the error log you shared with the Zeebe engineers. It means that no multi-tenancy data was found in the gRPC request, but something might possibly be expecting that data. For instance, you may have multi-tenancy enabled in Zeebe but not Operate, or your may have a version mismatch where one version isn’t multi-tenancy aware.
What version of Camunda do you have deployed? Are you using multi-tenancy?
(edit: I gave some wrong information accidentally; comment fixed!)
I am trying to implement Camunda 8.4, However I am trying to enable MultiTenancy which is critical feature for future implementation . I am facing the following error in connectors module
2024-02-29T20:22:58.381Z WARN 1 — [ult-executor-25] io.camunda.zeebe.client.job.poller : Failed to activate jobs for worker AWS Lambda and job type io.camunda:aws-lambda:1
30369
30370io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Expected to handle gRPC request ActivateJobs with tenant identifier ‘’, but tenant could not be retrieved from the request context
The following are the variables I added for conenctor env
ZEEBE_CLIENT_DEFAULT-TENANT-ID - "<default>"
ZEEBE_CLIENT_DEFAULT-JOB-WORKER-TENANT-IDS - "<default>"
ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS - camunda-zeebe-gateway:26500
@pavan_Kumar - can you provide your values.yaml file? You can also send it to me in a private message if needed. Without seeing how you’ve configured the deployment, it’s quite difficult to provide next steps.
Hi @pavan_Kumar - there’s a LOT of configuration there! I looked through, and the first thing I noticed was that authentication.mode does not seem to set. When using multi-tenancy, it should be set to identity (see “Note” here). (You also need to set the Identity baseUrl but looks like you have that set already.)
Perhaps try making that change and see what changes.
I have managed to get it working by installing the certs in all the containers and now they are able to communicate with each other without issue. When MutlTenancy is enabled.
The hard dependency of Identity is causing lot of issue and once Identity is available and conencted, the other applications are fine…
.