Camunda Sping boot out of memory

Hi ,

we are using below versions:

Spring-Boot: (v1.5.15.RELEASE)
Camunda BPM: (v7.6.0)
Camunda BPM Spring Boot Starter: (v2.1.2)

Exception in thread “JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

2019-07-25 15:51:36 [http-nio-8020-exec-10] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread: org.apache.catalina.connector.RequestFacade@34b611d5

java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-exec-2”

Exception in thread “http-nio-8020-exec-3” Exception in thread “http-nio-8020-Acceptor-0” java.lang.OutOfMemoryError: Java heap space

Exception in thread “RxComputationScheduler-1” java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread.

Caused by: java.lang.OutOfMemoryError: Java heap space

2019-07-25 16:23:15 [http-nio-8020-exec-1] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread: org.apache.catalina.connector.RequestFacade@42ecccc9

java.lang.OutOfMemoryError: Java heap space

What exactly do you do before seeing this error?

this is production logs… i see continously below

2019-07-29 06:53:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

2019-07-29 06:53:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14011 Job acquisition thread sleeping for 60000 millis

2019-07-29 06:54:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

2019-07-25 15:38:32 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14011 Job acquisition thread sleeping for 60000 millis

2019-07-25 15:41:59 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

Exception in thread “http-nio-8020-ClientPoller-1”

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-ClientPoller-0”

Exception in thread “http-nio-8020-exec-5” Exception in thread “http-nio-8020-exec-2”

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-exec-4”

java.lang.OutOfMemoryError: Java heap space

at java.util.HashMap$KeySet.iterator(HashMap.java:917)

at java.util.HashSet.iterator(HashSet.java:173)

at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1039)

at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)

at org.apache.tomcat.util.net.NioEndpoint$Poller.timeout(NioEndpoint.java:1018)

at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:838)

at java.lang.Thread.run(Thread.java:748)

Exception in thread “JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

2019-07-25 15:51:36 [http-nio-8020-exec-10] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread:

	SpringProcessEngineConfiguration config = new SpringProcessEngineConfiguration();
	config.setDataSource(dataSource());
	config.setTransactionManager(transactionManager());
	config.setIdGenerator(getIdGenerator());
	config.setDatabaseSchemaUpdate(ProcessEngineConfiguration.DB_SCHEMA_UPDATE_TRUE);
	config.setHistory("full");
	config.setJobExecutorActivate(false);
	config.setMetricsEnabled(false);

	List<ProcessEnginePlugin> processEnginePlugins = new ArrayList<ProcessEnginePlugin>();
	processEnginePlugins.add(getSpinPlugin());
	config.setProcessEnginePlugins(processEnginePlugins);
	return config;

Though i have set jobExecutor as false, still it is working how it works… is that causing out of memory

It’s not job executor issue. Might be you configured some task need to be executed by job executor, so that task seems to be utilized the memory.

Can you check the server RAM configuration, DB storage and any resources been used and didn’t closed properly?

Hi, In my process i have service task and user task, few tasks are pending at manual task

What’s the storage space and RAM size for your servers and database?

How many active process instance running?

active process as now 30,

remaining info will update

some thread dumps

Locked ownable synchronizers:
- <0x00000000e696f068> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

“PollingServerListUpdater-0” #67 daemon prio=5 os_prio=0 tid=0x00007f5ccc3a8000 nid=0x4e waiting for monitor entry [0x00007f5ce9bf8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
at java.util.concurrent.ConcurrentHashMap.(ConcurrentHashMap.java:852)
at com.netflix.loadbalancer.LoadBalancerStats.updateZoneServerMapping(LoadBalancerStats.java:232)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.setServerListForZones(DynamicServerListLoadBalancer.java:180)
at com.netflix.loadbalancer.ZoneAwareLoadBalancer.setServerListForZones(ZoneAwareLoadBalancer.java:91)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.setServersList(DynamicServerListLoadBalancer.java:174)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.updateAllServerList(DynamicServerListLoadBalancer.java:259)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.updateListOfServers(DynamicServerListLoadBalancer.java:242)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer$1.doUpdate(DynamicServerListLoadBalancer.java:62)
at com.netflix.loadbalancer.PollingServerListUpdater$1.run(PollingServerListUpdater.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000e6976128> (a java.util.concurrent.ThreadPoolExecutor$Worker)

“DestroyJavaVM” #62 prio=5 os_prio=0 tid=0x00007f5d4c010800 nid=0xc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” #61 prio=5 os_prio=0 tid=0x00007f5d4db08800 nid=0x4a runnable [0x00007f5ce9cf9000]
java.lang.Thread.State: RUNNABLE

“dd-weak-ref-cleaner” #6 daemon prio=1 os_prio=0 tid=0x00007f5d4c2b9800 nid=0x15 runnable [0x00007f5d384a6000]
java.lang.Thread.State: RUNNABLE
at com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap.getKey(WeakConcurrentMap.java:122)
at com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap.get(WeakConcurrentMap.java:98)
at datadog.trace.agent.tooling.WeakMapSuppliers$WeakConcurrent$Adapter.get(WeakMapSuppliers.java:166)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.mapGet(FieldBackedProvider.java:834)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.realGet(FieldBackedProvider.java)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.get(FieldBackedProvider.java:778)
at datadog.trace.instrumentation.java.concurrent.AdviceUtils.startTaskScope(AdviceUtils.java:20)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000e0f23fe8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

“Signal Dispatcher” #4 daemon prio=9 os_prio=0 tid=0x00007f5d4c1a0000 nid=0x14 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f5d4c16a800 nid=0x13 in Object.wait() [0x00007f5d38905000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e0f26428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Locked ownable synchronizers:
- None

“Reference Handler” #2 daemon prio=10 os_prio=0 tid=0x00007f5d4c167800 nid=0x12 in Object.wait() [0x00007f5d38a06000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e0e1df78> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Locked ownable synchronizers:
- None

“VM Thread” os_prio=0 tid=0x00007f5d4c15d000 nid=0x11 runnable

“GC task thread#0 (ParallelGC)” os_prio=0 tid=0x00007f5d4c033800 nid=0xd runnable

“GC task thread#1 (ParallelGC)” os_prio=0 tid=0x00007f5d4c036000 nid=0xe runnable

“GC task thread#2 (ParallelGC)” os_prio=0 tid=0x00007f5d4c037800 nid=0xf runnable

“GC task thread#3 (ParallelGC)” os_prio=0 tid=0x00007f5d4c039800 nid=0x10 runnable

“VM Periodic Task Thread” os_prio=0 tid=0x00007f5d4c83a800 nid=0x1b waiting on condition

JNI global references: 261

here u can find bpmn file

Hi Niall, can you help me with this

Hello!
We also have a problem in the docker. After some time (~ 2-3 days), the service killed from memory or cpu. With bpmn and dmn (!).
In version 7.5, a similar error was already corrected (https://app.camunda.com/jira/browse/CAM-5073)… but, apparently, not everywhere.
We have 7.9

Need help