Problems with persistance: Could not enlist in transaction on entering meta-aware object

egil · November 13, 2017, 7:30pm

Hi guys! I’m working for a client where we want to run Camunda as a process engine embedded in a web app for Wildfly. I have successfully managed to embed both rest api, the engine and the web applications (cockpit, admin and tasklist) and all seemed to be working like a charm until I deployed my first process and tried to view it in the cockpit. Everything seems to be working out nicely at first glance, but then I get an error message on screen with a persistance error. When I look into the log I find some issues which seems to be from Statistics.xml when getting from org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionStatisticsEntity.selectActivityStatistics.

We run on Wildfly with Java 8 and MariaDB, the code I use to (naively as a POC) start the engine is included below. Any help would be grately appreciated as I’ve delved into this the last 4 hours with seemingly no progress. Thanks a lot guys!

First the exception:

### Error querying database.  Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@530a0e94[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 connection handles=0 lastReturned=1510586788435 lastValidated=1510586768322 lastCheckedOut=1510586788993 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2fd5b8ad mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@24cdc17e[pool=AipDS] xaResource=XAResourceWrapperImpl@63d8e128[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 pad=false overrideRmValue=null productName=MySQL productVersion=5.5.56-MariaDB jndiName=java:jboss/datasources/AipDS] txSync=null]
### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Statistics.xml
### The error may involve org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionStatisticsEntity.selectActivityStatistics
### The error occurred while executing a query
### Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@530a0e94[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 connection handles=0 lastReturned=1510586788435 lastValidated=1510586768322 lastCheckedOut=1510586788993 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2fd5b8ad mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@24cdc17e[pool=AipDS] xaResource=XAResourceWrapperImpl@63d8e128[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 pad=false overrideRmValue=null productName=MySQL productVersion=5.5.56-MariaDB jndiName=java:jboss/datasources/AipDS] txSync=null]
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:88)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:168)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:160)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:152)
	at org.camunda.bpm.engine.impl.persistence.entity.StatisticsManager.getStatisticsGroupedByActivity(StatisticsManager.java:54)
	at org.camunda.bpm.engine.impl.ActivityStatisticsQueryImpl.executeList(ActivityStatisticsQueryImpl.java:58)
	at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:186)
	at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:163)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
	... 77 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@530a0e94[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 connection handles=0 lastReturned=1510586788435 lastValidated=1510586768322 lastCheckedOut=1510586788993 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2fd5b8ad mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@24cdc17e[pool=AipDS] xaResource=XAResourceWrapperImpl@63d8e128[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 pad=false overrideRmValue=null productName=MySQL productVersion=5.5.56-MariaDB jndiName=java:jboss/datasources/AipDS] txSync=null]
	at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
	at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
	at org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:87)
	at org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:61)
	at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)
	at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:72)
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:59)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)
	... 88 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@530a0e94[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 connection handles=0 lastReturned=1510586788435 lastValidated=1510586768322 lastCheckedOut=1510586788993 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2fd5b8ad mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@24cdc17e[pool=AipDS] xaResource=XAResourceWrapperImpl@63d8e128[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@7d1847e0 pad=false overrideRmValue=null productName=MySQL productVersion=5.5.56-MariaDB jndiName=java:jboss/datasources/AipDS] txSync=null]
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:933)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:750)
	at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
	... 99 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
	at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:560)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:928)
	... 101 more
Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffffac1f0004:-66e32a91:5a09b97a:65 status: ActionStatus.ABORT_ONLY >
	at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:942)
	at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:394)
	at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:553)
	... 102 more

The code:

@Startup
@Singleton
@ApplicationPath("api")
public class ComProtoApplication extends Application {

	public static final String PROCESS_ENGINE_NAME = "omnipotent";
	
	protected final Logger log = Logger.getLogger(getClass());
	
	@Resource(mappedName = "java:/TransactionManager")
	private TransactionManager transactionManager;
	
	@PostConstruct
	public void init() throws SystemException {
		

		JtaProcessEngineConfiguration config = new JtaProcessEngineConfiguration();
		
		config.setDataSourceJndiName("java:jboss/datasources/AipDS");
		//config.setTransactionManagerJndiName("java:/TransactionManager");
		transactionManager.setTransactionTimeout(5000);

		config.setTransactionManager(transactionManager);
		config.setProcessEngineName(PROCESS_ENGINE_NAME);
		ProcessEngine engine = config.buildProcessEngine();
		log.info("Process Engine created with name " + engine.getName());
		
		DeploymentBuilder builder = engine.getRepositoryService().createDeployment();
		
		builder.addInputStream("test.bpmn", getClass().getResourceAsStream("processes/prototype1.bpmn"));
		
		builder.deploy();
		
	}
	
	@Override
	public Set<Class<?>> getClasses() {
		Set<Class<?>> classes = new HashSet<>();
		
		classes.add(ComProtoService.class);
		classes.addAll(CamundaRestResources.getResourceClasses());
		classes.addAll(CamundaRestResources.getConfigurationClasses());
		return classes;
	}
	
	@PreDestroy
	public void done() {
		log.debug("C000");
	}

}

egil · November 14, 2017, 9:22am

I actually found a workaround for this. After a lot of digging around this seems to be a bug/feature in mysql with transactions running outside the global transaction. The root error is discussed here: https://developer.jboss.org/thread/145321.

What worked for me was simply adding:
<no-tx-separate-pools />

To the xa-pool attribute in my datasource in wildfly. This was a relatively obscure error, so hopefully this can help others if they bump into the same issue.

egil · November 15, 2017, 10:10am

I was able to reproduce this using the standalone-server provided by Camunda, so I created a bug report for it:
https://app.camunda.com/jira/browse/CAM-8426

sdorokhova · November 22, 2017, 10:13am

Hi @egil,

I did not look in the problem in details so far, but I 'm guessing, is there any reason for using MySQL JDBC Driver rather than MariaDB driver?

egil · November 22, 2017, 10:53am

It was just how the server was set up here. It didn’t help with the no-tx-separate-pools issue, but it fixed the other issue I had with instances not being refreshed. Thanks a million

sdorokhova · November 22, 2017, 11:05am

Glad to hear! But do you mean, that the issue described in this thread and in CAM issue you created is still valid? Can you may be update the CAM issue with your new configuration file?

egil · November 22, 2017, 11:48am

Yes, they are still valid. Will update the config file

philipp.hehnle · October 12, 2020, 8:18pm

Although, this thread is very old, I do not want to withhold a workaround that worked for me:

I added the following in the wildfly standalone.xml

	 <system-properties>
        <property name="com.arjuna.ats.arjuna.allowMultipleLastResources" value="true"/>
    </system-properties>

aravindhrs · October 13, 2020, 4:16am

@philipp.hehnle Thanks for sharing the solution

system · January 30, 2024, 11:01am