Oracle SOA DB Poller Error Caused by BINDING.JCA-11608

Hi All,

Just wanted to highlight a common mistake most developer and most testing teams do. When a need arises to change any setting including connection factory or connection pool setting in Oracle Weblogic Server

The Mistake:

1. Make changes to the weblogic server data source / Connection factory.
2. Which leads to a DB adaptor redeploy.
3. Once the DB adaptor is redeployed all the pollers using that DB adaptor incidences will fail with the following error:

Query name: [xxxxxxxxxxxxxx], Descriptor name: [xxxxxxxxxxxxx.xxxxxxx]. Polling the database for events failed on this iteration.

Caused by BINDING.JCA-11608

Connection Already Closed Exception.

This [javax.resource.cci.Connection] is already closed.

This should not happen when running within a packaged application like BPEL or ESB but may occur when the adapter is used standalone.

..

.

  This exception is considered not retriable, likely due to a modelling mistake.  This polling process will shut down, unless the fault is related to processing a particular row, in which case polling will continue but the row will be rejected (faulted).

       at oracle.tip.adapter.db.exceptions.DBResourceException.createNonRetriableException(DBResourceException.java:690)

       at oracle.tip.adapter.db.exceptions.DBResourceException.createEISException(DBResourceException.java:656)

       at oracle.tip.adapter.db.exceptions.DBResourceException.inboundReadException(DBResourceException.java:491)

       at oracle.tip.adapter.db.InboundWork.handleException(InboundWork.java:1001)

       at oracle.tip.adapter.db.InboundWork.runOnce(InboundWork.java:896)

       at oracle.tip.adapter.db.InboundWork.run(InboundWork.java:627)

       at oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.java:43)

       at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)

       at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)

       at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

Caused by: BINDING.JCA-11608

Connection Already Closed Exception.

This [javax.resource.cci.Connection] is already closed.

Note: When the above error occurs, the Poller state is still active, which deadly in a production environment. 

The Corrective Measure:

1. Retire all DB Pollers which have encountered this error.
2. Activate all DB  Pollers.
3. The issue will be fixed.

The Best Practice:

1. Retire all DB Pollers. (No Polling will happen during this time).
2. Make changes to the weblogic server data source / Connection factory.
3. Which leads to a DB adaptor redeploy / Restart.
4. Once the DB adaptor is redeployed / Restarted.
5. Activate all DB Pollers.

How to Trace or Debug Database for issues / How to setup Tracing in database.

often during integration testing we encounter ORA Data base errors,  In this case we are debugging ORA- 02074 error.

here is how we can get more info about the tracing.

You need to set the command as below to you DB:

alter system set events '2074 trace name errorstack level 3';

If you would like to know more about the alter system set events command please

Oracle event trace types - by Burleson Consulting

When you set the above command and reproduce the issue, what ever that might be, look into the Alert_xxx.log file for statements such as below

Mon Apr 04 10:31:34 2011
Errors in file <Path>/trace/<InstanceName>_ora_20926.trc:
ORA-02074: cannot SET NLS in a distributed transaction
Mon Apr 04 10:31:36 2011
Errors in file <Path>/trace/<InstanceName>_ora_20950.trc:
ORA-02074: cannot SET NLS in a distributed transaction

This file gives the trace file name which contains the session dumps. In order to read the trace file, I believe you need extensive DBA skills. But do give it a shot.