Possible Memory Leak in Spring Boot Camunda


#1

We are using Spring Boot Starter version 1.5.10 and Camunda Spring Boot Starter version 2.3.0.

After running our application for a few days, we are experiencing high heap memory usage and eventually our application will hit the out of memory exception.

Looking at the heap dump, we can see that:

Camunda takes up a lot of heap memory and keeps growing. Is this a known issue?


#2

Hi @firedragon852,

There is a known issue before Spring boot 2.0.2 but it seems that this is something else.


Could you collaborate further to share steps to reproduce?

Best regards,
Yana


#3

Hi @Yana

We are able to consistently reproduce this memory leak issue by re-deploying our spring boot/web application in tomcat. Then we can see that the camunda/ibatis classes will eat up the heap memory.

We are not sure if it is because of our tomcat not being able to clean up the jdbc threads. We keep seeing this warning in catalina.log:

11-Jun-2018 18:02:01.419 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)       com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)

#4

We are getting NPE when trying to perform the Spring application context close:

@SpringBootApplication
@EnableProcessApplication
public class TestApplication {       

private static ConfigurableApplicationContext m_context = null;

public static void main(String[] args) {
    m_context = SpringApplication.run(TestApplication.class, args);
}

@EventListener
public void onPreUndeploy(PreUndeployEvent event) {
    System.out.println("context close start");
    try {
        m_context.close();
    }
    catch (Exception e) {
        e.printStackTrace();
    }
    System.out.println("context close end");
}

}

java.lang.NullPointerException
    at org.abc.TestApplication.onPreUndeploy(TestApplication.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:256)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360)
    at org.camunda.bpm.spring.boot.starter.SpringBootProcessApplication.onPreUndeploy(SpringBootProcessApplication.java:90)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.camunda.bpm.container.impl.deployment.PreUndeployInvocationStep.performOperationStep(PreUndeployInvocationStep.java:63)

This may explain why camunda resources are not getting removed.


#5

Hi @firedragon852,

It seems that you are missing the @Autowired annotation.
Have a look at SimpleApplication.

Best regards,
Yana


#6

Hi @Yana

I autowired the ConfigurableApplicationContext and did not get the NPE, however after undeploying the application in tomcat, I could still see references to the camunda classes:



#7

Have you try to close it as it is shown in the example?


#8

Yes, I did try to close the context and there’s no NPE after I autowired the ConfigurableApplicationContext.


#9

Hi @firedragon852,

Could you please try to switch to Camunda spring boot 3.0.0 and Spring boot 2.0.2 and share if the problem still persist.

Best regards,
Yana