Connection error when trying to use a re-created datasource


#1

So we have the following usecase. We have integrated camunda in our system and we use wildfly datasources to connecto to the database. We have runtime tenant management which means that a datasource could be created, dropped and then created again with the same name and configurations. Now here comes the problem. We create the tenant, deploy a process engine for it that uses the datasource, everything is ok, we drop the tenant and undeploy the process engine, still ok. This is how we undeploy it btw:

		serviceContainer
			.createUndeploymentOperation(operationName)
				.addAttachment(Attachments.PROCESS_APPLICATION, processApplication)
				.addStep(new PreUndeployInvocationStep())
				.addStep(new UndeployProcessArchivesStep())
				.addStep(new ProcessesXmlStopProcessEnginesStep())
				.addStep(new StopProcessApplicationServiceStep())
				.addStep(new NotifyPostProcessApplicationUndeployedStep())
				.execute();

However, when we create the same tenant again with the same datasource, the process engine starts throwing exceptions that ibatis is trying to use a connection that has already been closed.

### Error updating database.  Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/l2l.ok_camunda
### Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/l2l.ok_camunda
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:120)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:260)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:256)
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:419)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd$1.call(DeployCmd.java:130)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd$1.call(DeployCmd.java:127)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:509)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.doExecute(DeployCmd.java:127)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:96)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:77)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
	... 213 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/l2l.ok_camunda
	at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
	at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67)
	at org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:87)
	at org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:61)
	at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)
	at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:72)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:47)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 226 more
Caused by: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/l2l.ok_camunda
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:725)
	at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
	... 235 more

Ibatis is apparently trying to use the old datasource and the problem occurs in the ManagedTransaction class where it’s stored. So iis there a way to clear the datasource?


#2

Are you reloading WildFly after you drop the datasource? When manipulating datasources in the WildFly command GUI, you must reload WildFly for such changes to take effect. Perhaps that’s what’s needed.


#3

I’m not and we really shouldn’t. This should all happen runtime. For instance, hibernate had the same problem, it also cached the datasources for each tenant and it was throwing the same exception after recreating the same datasource. But you can actually write your own connection provider for hibernate and plug it in, so we wrote our own that clears the datasource when the tenant is deleted. We need to do the same for ibatis. Well… we need to the either clear the datasource in the managedtransaction or somehow clear the managed transaction in camunda.

Otherwise yeah, after a reload everything works as expected but that’s not our requirement.


#4

Is it even possible to do this in WildFly? I can only go by what I experience through the GUI console, but perhaps you can dynamically create the datasource programmatically. I found this StackOverflow article: https://stackoverflow.com/questions/28494959/is-it-possible-to-configure-jboss-wildfly-datasources-to-use-changing-client-cre

I know that in Tomcat you can do this, or at least through the Apache Geronimo distribution.

Sorry I couldn’t be more help.

Michael


#5

Yeah it’s possible and we do use the wildfly’s cli api. The problem is not in wildfly tho. Its in the persistence frameworks that cache those datasources. As i already mentioned we fixed it for hibernate but currently looking for an ibatis solution (Or more precisely something that can be done through the camunda process engine since that’s the only thing that uses ibatis in our system) :).


#6

Hi @Nikolai_Velkov,

Do you use the Camunda Wildfly subsystem to manage the process engines?

Cheers,
Christian


#7

Nope we deploy/undeploy a process engine per tenant and just specify the datasource


#8

So you use the embedded process engine scenario? How do you make sure that the process engine is closed? Do you see Process Engine xxx closed. in the logs?

Cheers,
Christian


#9

Yeah we do. I posted how we are closing the engine in my first post. And yes i see the Process engine closed log. I also see process application undeployed.

11:43:42,931 INFO  [org.camunda.bpm.engine] (default task-29) ENGINE-00007 Process Engine ppp_ppp closed
11:43:42,932 INFO  [org.camunda.bpm.container] (default task-29) ENGINE-08051 Process application ppp.ppp undeployed

#10

So um… no hints on this one eh ?


#11

Do you believe that Camunda keeps a reference to the old process engine and therefore the old datasource (and then reuses the engine when you redeploy) or that mybatis keeps a reference to the old datasource internally and somehow reuses it?

If the former, can you identify why Camunda keeps a reference the old process engine or old data source although you close it?
If the latter, can you identify where Mybatis keeps the reference and what (if any) the correct Mybatis API way is to discard the datasource?

Cheers,
Thorben


#12

I’ve debugged it a little bit and i believe that when the process engine is initialized (ProcessEngineConfigurationImpl#initSqlSessionFactory), the datasource that is being set in the Environment is somehow cached in the ProcessEngineConfiguration. I tried ProcessEngineConfiguration().setDataSource(null); when undeploying the process engine but that doesn’t seem to work. Perhaps it uses the idGeneratorDataSource, which is also saved in the ProcessEngineConfigurationImpl.


#13

Hi,

Do you use the same ProcessEngineConfiguration to build more than one process engine? If yes, then please avoid that and use a new configuration object every time. If no, do you use the engine configuration property isUseSharedSqlSessionFactory?

Cheers,
Thorben


#14

No and No. On each tenant creation we init a new process engine via the MBeanServiceContainer.

                serviceContainer
			.createDeploymentOperation(operationName)
				.addAttachment(Attachments.PROCESS_APPLICATION, processApplication)
				.addStep(new TenantInitializationStep(securityContextManager, camundaDbProvisioning))
				.addStep(new RuntimeProcessesStep(engineName))
				.addStep(new ProcessesXmlStartProcessEnginesStep())
				.addStep(new DeployProcessArchivesStep())
				.addStep(new StartProcessApplicationServiceStep())
				.addStep(new PostDeployInvocationStep())
				.execute();

The RuntimeProcessesStep is a custom step where we init the processArchiveXml and we set our process engine configuration that doesn’t do much tbh.

processEngineXml = new ProcessEngineXmlImpl();
	processEngineXml.setName(engineName);
	processEngineXml.setConfigurationClass(SepProcessEngineConfigurator.class.getName());

And this is our process engine configuration

public class SepProcessEngineConfigurator extends JtaProcessEngineConfiguration{
...
		CamundaConfiguration configurations = ProgrammaticBeanLookup.lookup(CamundaConfiguration.class);
	setHistory(HISTORY_FULL);
	setDatabaseSchemaUpdate(DB_SCHEMA_UPDATE_TRUE);
	setDataSourceJndiName("java:jboss/datasources/" + configurations.getDatasourceName().get());
	setTransactionManagerJndiName("java:jboss/TransactionManager");
	setBpmnStacktraceVerbose(true);
	setCmmnEnabled(true);
	setDmnEnabled(true);
	setJobExecutor(new DefaultJobExecutor());
	setJobExecutorActivate(false);

#15

When you let the JBoss subsystem start the engine, then it creates a new process engine configuration object instantiating your SepProcessEngineConfigurator class (via debugging you can verify that you have a new object every time). Furthermore, these objects do not share anything by default (unless you use isUseSharedSqlSessionFactory which you said you don’t). I believe you’ll have to debug some more to identify where the datasource reference is kept.