BEA-001112 java.sql.SQLRecoverableException: IO Error: Connection reset in weblogic

Error:


<Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "VPDDataSource" failed with exception: "java.sql.SQLRecoverableException: IO Error: Connection reset".>


Cause:

There are two scenarios that can result in this error:

1. If it is single instance database:
The DB instance that the jdbc datasource is configured for, was unavailable either due to shutdown or some other issues.

2. If it is RAC databsae:
Both A & B below should be true to receive this error in weblogic.
A. One of the database instance of your cluster/RAC database, that your jdbc session was hooked on to, either died or it was shutdown, and 

B. The JDBC pool is configured as generic datasource, and not as gridlink datasource, 

 

Resolution:

Use RAC databsae to make database highly available, and at also ensure that JDBC datasource is configuerd as gridlink , and not as generic.

BEA-150000 Error when starting Admin Server

Are you writing scripts to start & stop the admin server?


Are you getting below error when trying to start the Admin Server using scripts?

 <Error> <Configuration Management> <xxxxx.na.lzb.hq> <AdminServer> <[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1541099907869> <BEA-150000> <An error occurred while establishing a connection back to the Adminstration Server t3://10.1.2.79:7001 during startup. Since bootstrap succeeded, check that t3://1xx.yy.zz.rr:7001 uniquely identifies the Administration Server.

Here the catch:

If you have set this variable any where in your scripts or property file, you are bound to get the error.

ADMIN_URL=<HOST>:<PORT>

Resolution: ==> Do not set the variable ADMIN_URL anywhere in you startup script or property file.

Repeat the above statement 4 more times again.

SOA 11g cluster: Unable to create Queue producer due to JMSException .

Problem: 
- SOA server with integration involving JMS queue throws following errors in the .out log.

Important Error Messages:
JMSMessageProducer_produce:[default destination = jms/LZBSOMInternalMessageQueue]:ERRJMS_ERR_CR_QUEUE_PROD.
ERRJMS_ERR_CR_QUEUE_PROD.
Unable to create Queue producer due to JMSException.
..
Caused by: weblogic.jms.common.JMSException: [JMSExceptions:045103]While trying to find a topic or a queue we could not find the specific JMSServer requested. The linked exception may contain more information about the reason for failure



Relevant log excerpt from soa server .out file:
<Oct 12, 2018 7:32:37 PM EDT> <Error> <oracle.soa.adapter> <BEA-000000> <JMS Adapter  SOMSalesOrder:EnqueueStartAckMessage [ Produce_Message_ptt::Produce_Message(body) ]  JMSMessageProducer_produce:[default destination = jms/LZBSOMInternalMessageQueue]:ERRJMS_ERR_CR_QUEUE_PROD.
ERRJMS_ERR_CR_QUEUE_PROD.
Unable to create Queue producer due to JMSException.
Please examine the log file to determine the problem.
>
<Oct 12, 2018 7:32:37 PM EDT> <Error> <oracle.soa.adapter> <BEA-000000> <JMS Adapter  SOMSalesOrder:EnqueueStartAckMessage [ Produce_Message_ptt::Produce_Message(body) ]  JmsProducer_execute:[default destination = jms/LZBSOMInternalMessageQueue]: A resource exception occured while producing message>
<Oct 12, 2018 7:32:37 PM EDT> <Error> <oracle.soa.adapter> <BEA-000000> <JMS Adapter  SOMSalesOrder:EnqueueStartAckMessage [ Produce_Message_ptt::Produce_Message(body) ]
BINDING.JCA-12137
ERRJMS_ERR_CR_QUEUE_PROD.
ERRJMS_ERR_CR_QUEUE_PROD.
Unable to create Queue producer due to JMSException.
Please examine the log file to determine the problem.
....
....
....
Caused by: weblogic.jms.common.JMSException: [JMSExceptions:045103]While trying to find a topic or a queue we could not find the specific JMSServer requested. The linked exception may contain more information about the reason for failure.
        at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:110)
        at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:61)
        at weblogic.jms.client.JMSSession.createDestination(JMSSession.java:3192)
        at weblogic.jms.client.JMSSession.createQueue(JMSSession.java:2577)
        at weblogic.jms.client.WLSessionImpl.createQueue(WLSessionImpl.java:938)
        at oracle.tip.adapter.jms.JMS.JMSDestination.getQueue(JMSDestination.java:70)
        at oracle.tip.adapter.jms.JMS.JMSConnection.createProducer(JMSConnection.java:696)
        ... 88 more
Caused by: weblogic.jms.common.JMSException: [JMSExceptions:045103]While trying to find a topic or a queue we could not find the specific JMSServer requested. The linked exception may contain more information about the reason for failure.
        at weblogic.jms.dispatcher.Request.handleThrowable(Request.java:87)
        at weblogic.jms.dispatcher.Request.getResult(Request.java:52)
        at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:1124)
        at weblogic.messaging.dispatcher.DispatcherImpl.syncRequest(DispatcherImpl.java:185)
        at weblogic.messaging.dispatcher.DispatcherImpl.dispatchSyncNoTran(DispatcherImpl.java:288)
        at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
        ... 93 more
Caused by: weblogic.jms.common.JMSException: [JMSExceptions:045103]While trying to find a topic or a queue we could not find the specific JMSServer requested. The linked exception may contain more information about the reason for failure.
        at weblogic.jms.frontend.FEManager.destinationCreate(FEManager.java:294)
        at weblogic.jms.frontend.FEManager.invoke(FEManager.java:572)
        at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:961)
        ... 96 more

Root Cause Analyses:

On further investigation it was found that the subdeployment to which the jms queue is assigned, was targetted on only node 2 of the SOA cluster, and the node was shutdown for maintenance. 

Fortunately, we had a planned maintenance on node 2, else we'd have never identified that the jms configuration are not targetted evenly on the cluster.

Resolution:
Target the subdeployment evenly, i.e. on the relevant JMS servers from both nodes.

GUI tests on Windows RDP session stops working after minimizing or disconnecting

Many automation tests require the GUI based application or scripts to keep running on a windows session.
In a standard IT industry scenario, we use RDP session (with mstsc) to connect to a windows server.

However, the challenge comes when we need to leave these applications unattended. In most scenarios, once you minimize the RDP session window, all the GUI operation get stuck, and the application or script is unable to proceed normally.

The situation worsens when we have even longer running tests/scripts, as its not possible for end user to continuously monitor the RDP session.

The trick to deal with both these situations are mentioned below:

1. Minimizing the RDP window halts the GUI test/scripts:

To prevent the minimized RDP sessino from stopping your work, make the following changes

a. Launch registry editor using regedit
b. Navigate to the below two following registries & create an entry as shown below:

HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client

HKEY_LOCAL_MACHINE\Software\Microsoft\Terminal Server Client

Create an entry of type DWORD, with name RemoteDesktop_SuppressWhenMinimized and set its value to 2(decimal value).


2. Disconnecting the RDP session totally stop the execution of GUI activities:

Sometimes is impractical to keep the session continuously monitored. In my case, I had setup a script that needs to runs every 10 minutes, 24x7, and it didn't need any user inputs.

To achieve successful execution of GUI based tests or scripts, with "unconnected*" [*notice that I avoided the term disconnected], use the below commands for closing the RDP session

a. Create a bat file with below contents:

for /f "skip=1 tokens=3" %%s in ('query user %USERNAME%') do ( %windir%\System32\tscon.exe %%s /dest:console )

a. Run the above created file with admin privilege:
- Please note, the script won't work unless you run it as administrator.

With this, your session will show disconnected, however, the user will in reality will be still logged in, and GUI tasks will go on as expected.

Credits:
Thanks to some of the blogs on web:

https://superuser.com/questions/1288469/remote-windows-server-virtual-machine-always-keep-desktop-open-simulate-rdp-se
https://support.smartbear.com/testcomplete/docs/testing-with/running/via-rdp/in-minimized-window.html

Oracle HTTP server 11g quick commands

1. How to get HTTP version 

export ORACLE_HOME=/u01/app/oracle/product/fmw/Oracle_WT1
export LD_LIBRARY_PATH=$ORACLE_HOME/ohs/lib:$ORACLE_HOME/opmn/lib:$ORACLE_HOME/lib: $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/ohs/lib:$ORACLE_HOME/opmn/lib:$ORACLE_HOME/lib:$LD_LIBRARY_PATH

 ./httpd -version

[oracle@dvohs1 bin]$ ./httpd -version
Server version: Oracle-HTTP-Server/2.2.22 (Unix)
Server built:   Jun 29 2017 22:55:25

Server label:   APACHE_11.1.1.9.0_LINUX.X64_RELEASE

2. Work in Progress

JVM thread dump heap dump process count thread count

Thread Dumps:

$JAVA_HOME/bin/jstack -l <pid>  > /tmp/threaddump_pd_runtime1_$(date).txt

Heap Dumps:

$JAVA_HOME/bin/jmap -dump:live,file=/tmp/file_name <PID>

JMAP:

$JAVA_HOME/bin/jmap -heap <pid> > /tmp/heapdump_pd_runtime1_$(date).txt

Getting the open file count:

/usr/sbin/lsof -u oracle | wc –l


Getting process count:
ps -eLf | grep oracle | wc -l

SOA Composite incorrectly appear in failed state in EM Console

SOA Composite incorrectly appear in failed state in EM Console

You just restarted SOA servers, and verified the SOA platform is ready and accepting requests.

However when you look back at "Deployed Composite" tab under SOA folder in EM console, all the composites appear to be in failed state, and the page looks different from how it looks during healthy SOA status.

The below screen shows how the EM console appears in this case. The red dots with white cross encircled do not appears normally.

This usually happens when you continue to use the same browser session that was open before you shutdown and restarted SOA servers. Do not worry, mostly there is no issue with your  SOA servers.

To fix this, Simply log out from SOA EM Console and log back in again, the the EM console page should appear normal.

OAMSSA-20027 oracle.igf.ids.EntityNotFoundException

Sometimes we see the below error in OAM logs:


<Error> <oracle.oam.user.identity.provider> <OAMSSA-20027> <Could not get user : YXXX, idstore: Active Directory, with exception: oracle.igf.ids.EntityNotFoundException: Entity not found for the search filter (&(objectclass=user)(sAMAccountName=YXXX))..>


In our case, we have enabled chained authentication, which authenticates user first in AD, and then in OUD.

The above error is logged if the user is not found in AD. Hence this can be safely ignored.


If you have chained authentication, and not sure if the user was able to login after the above error or not, look at the log file content below "OAMSSA-20027..Entity not found.. " error.

If the issue couldn't further be found in 2nd LDAP of chained authentication, you'd see another error.


### Failed Login Sample ####
PFB the samlpe error wherein I supplied incorrect OUD credentials. Hence there are 2 errors 1) From AD Entity not found and 2) form OUD .. i.e. Inavlid credentials.

<Jan 8, 2018 10:53:19 AM EST> <Error> <oracle.oam.user.identity.provider> <OAMSSA-20023> <Authentication Failure for user : cn=XXX YYY,cn=users,dc=Corp,dc=hq, for idstore OUDIDStore with exception oracle.igf.ids.AuthenticationException: Authentication failed for user cn=XXX YYY,cn=users,dc=Corp,dc=hq. AdditionalInfo: LDAP Error 49 : [LDAP: error code 49 - Invalid Credentials] with primary error message LDAP Error 49 : [LDAP: error code 49 - Invalid Credentials]>


<Jan 8, 2018 10:53:19 AM EST> <Warning> <oracle.ods.virtualization.engine.backend.jndi.adapter1> <LIBOVD-40082> <Could not modify entry.
javax.naming.directory.SchemaViolationException: [LDAP: error code 65 - Entry cn=XXX YYY ,cn=users,dc=Corp,dc=hq cannot not be modified because the resulting entry would have violated the server schema:  Entry cn=XXX YYY,cn=users,dc=Corp,dc=hq violates the Directory Server schema configuration because it includes attribute oblogintrycount which is not allowed by any of the objectclasses defined in that entry]; remaining name 'cn=XXX YYY,cn=users,dc=Corp,dc=hq'
        at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3118)


### Successful Login Sample ####
<Error> <oracle.oam.user.identity.provider> <OAMSSA-20027> <Could not get user : YXXX, idstore: Active Directory, with exception: oracle.igf.ids.EntityNotFoundException: Entity not found for the search filter (&(objectclass=user)(sAMAccountName=YXXX))..>
<Jan 3, 2018 4:12:56 PM EST> <Warning> <oracle.dms.context> <DMS-57008> <The execution context put in place at the start of the request, 005OUMhIiYp6eLWFLz3V8A0000PI004xXg,0:1:1:2, is not the execution context in place at the end of the request, ef9531de2c28bbbd:748b1134:1603f332b01:-8000-00000000006764fa,0. The request is as follows: Request URI:
  /oam/server/auth_cred_submit
Request URL:
  http://XXXidmpdlogin.la-z-boy.com:80/oam/server/auth_cred_submit