Process engine persistence exception Docker Container

Camunda 7 Topics Discussion & Questions
1

Hi,

We are running the Camunda container with tag 7.8.0 in Azure. Ever 2 hours or so we get an Process engine persistence exception. This results in us deleting our container and redeploying it.

Here are our the logs we get from the container:

11-Apr-2018 11:40:45.045 SEVERE [Camunda Metrics Reporter] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: ENGINE-03004 Exception while executing Database Operation ‘INS
ERT MeterLogEntity[2bd5fb33-3d7d-11e8-a0d6-0a580af47805]’ with message ’

Error updating database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.p

ostgresql.util.PSQLException: This connection has been closed.

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.postgresql.util.PSQLExcepti

on: This connection has been closed.’. Flush summary:
[
INSERT MeterLogEntity[2bd5fb33-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb34-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb35-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb36-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb37-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb38-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb39-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3a-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3b-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3c-3d7d-11e8-a0d6-0a580af47805]
]
org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation ‘INSERT MeterLogEntity[2bd5fb33-3d7d-11e8-a0d6-0a580af47805]’ with message ’

Error updating database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.p

ostgresql.util.PSQLException: This connection has been closed.

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.postgresql.util.PSQLExcepti

on: This connection has been closed.’. Flush summary:
[
INSERT MeterLogEntity[2bd5fb33-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb34-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb35-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb36-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb37-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb38-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb39-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3a-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3b-3d7d-11e8-a0d6-0a580af47805]
INSERT MeterLogEntity[2bd5fb3c-3d7d-11e8-a0d6-0a580af47805]
]
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:120)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:331)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:314)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:286)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:203)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:132)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.metrics.reporter.MetricsCollectionTask.collectMetrics(MetricsCollectionTask.java:72)
at org.camunda.bpm.engine.impl.metrics.reporter.MetricsCollectionTask.run(MetricsCollectionTask.java:49)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.ibatis.exceptions.PersistenceException:

Error updating database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.p

ostgresql.util.PSQLException: This connection has been closed.

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.postgresql.util.PSQLExcepti

on: This connection has been closed.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:200)
at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:185)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeInsertEntity(DbSqlSession.java:157)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.insertEntity(DbSqlSession.java:149)
at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:41)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:329)
… 11 more
Caused by: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.postgresql.util.PSQLExcepti
on: This connection has been closed.
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:107)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:142)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:60)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
at org.apache.ibatis.executor.BatchExecutor.doUpdate(BatchExecutor.java:69)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:198)
… 16 more
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843)
at org.postgresql.jdbc2.AbstractJdbc2Connection.getAutoCommit(AbstractJdbc2Connection.java:804)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:79)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
at com.sun.proxy.$Proxy4.getAutoCommit(Unknown Source)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:98)
… 23 more

2

Hi @Nadya_Essebbar,

Could you please give more details of you db - db version, driver version?
I think this post could be helpful.
Also please check your isolation level configuration.

Best regards,
Yana

3

Driver version is the one which is set in the dockerfile. We are using Camunda BPM Platform Docker Image 7.8.0
Db version is PostgreSQL 9.6. This is a managed postgresdb in Azure.

Btw, Our container instance is an Azure Container Instance.

Our DB Isolation level is READ_COMMITTED

4

Hi,

Did you check the configuration from the post I mentioned?

Best regards,
Yana

5

Hi,

I m a colleague of Nadya,
We did indeed check and update the configuration from the post you mentioned but we still get a persistence exception, could you please advice?

Best regards,
Daryl

6

Hi Daryl,

Could you please share your connection pool configuration and your latest server log file.

Best regards,
Yana

7

Hi Yana,

This is our current connection configuration:
<Resource name="jdbc/ProcessEngine" auth="Container" type="javax.sql.DataSource" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" uniqueResourceName="process-engine" driverClassName="org.postgresql.Driver" url="jdbc:postgresql://sqldb-cp-madagascar-test.postgres.database.azure.com:5432/process-engine" defaultTransactionIsolation="READ_COMMITTED" username="camundaAdmin@sqldb-cp-madagascar-test" password="" initialSize="5" maxActive="55" maxIdle="21" minIdle="13" timeBetweenEvictionRunsMillis="34000" minEvictableIdleTimeMillis="55000" validationQuery="SELECT 1" validationInterval="34" testOnBorrow="true" removeAbandoned="true" removeAbandonedTimeout="15" />

And his is our latest container log:
container-logs.txt (89.3 KB)

Kind regards,
Daryl

8

Hello guys,

It seems that the server is not able to connect to the db. Please verify the url, etc.

Best regards,
Yana

9

Hi Yana,

We have connection for a while, but then it seems like the connection pool is empty.

10

I still remember that i had the exact same issue as you some time ago with postgres.

But i don’t remember the exact fix - i think it was either disabling db batch processing or the upgrade to postgres 10.x or a more recent postgres jdbc driver - you could just try those.

11

Hi,
did you solve the issue? I’ve got a similar issue using docker with 7.9.0 on wildfly 11 and postgres 9.6. This is running on a private centos server.

Could anyone please point me to the correct documentation or config-files etc. as I am new to (java)/wildfly/camunda?

Best regards,
Jörg

12

It seems there is a configuration issue with the camunda-wildfly.sh script.

As far as I understand, the h2 datasource configuration read from the standalone.xml gets modified by the script. Driver, connection string and user credentials get modified.

The problem seems to be that there is no connection-pool configuration in the h2 datasource configuration but the postgres driver needs such a pool.

Does anyone agree?

13

My solution is to make a new docker image and copy the adjusted standalone.xml / wildfly config file into it. One could also “mount” it into the image.
Like so (Dockerfile):

FROM camunda/camunda-bpm-platform:wildfly11-7.9.0
COPY standalone.xml /camunda/standalone/configuration/

The datasource part of the standalone.xml looks like that:

<datasource jta="true" jndi-name="java:jboss/datasources/ProcessEngine" pool-name="ProcessEngine" enabled="true" use-java-context="true" use-ccm="true">
	<connection-url>${env.DB_URL}</connection-url>
	<driver>postgresql</driver>
	<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
	<security>
		<user-name>${env.DB_USERNAME}</user-name>
		<password>${env.DB_PASSWORD}</password>
	</security>
	<pool>
		<min-pool-size>4</min-pool-size>
		<initial-pool-size>4</initial-pool-size>
		<max-pool-size>64</max-pool-size>
		<prefill>true</prefill>
	</pool>
	<validation>
		<check-valid-connection-sql>select 1</check-valid-connection-sql>
		<validate-on-match>false</validate-on-match>
		<background-validation>true</background-validation>
		<background-validation-millis>10000</background-validation-millis>
	</validation>
</datasource>

For me that’s okay as I also needed to configure LDAP in the standalone.xml.

14