Troubleshoot : Oracle HTTP Server (OHS) marked down in #OEM12c or Fusion Middleware Control
I recently configured monitoring of Oracle Fusion Middleware (including Oracle HTTP Server – OHS) using Oracle Enterprise Manager (OEM) 12c here, here and here
In this post I am going to cover issue encountered while managing Oracle HTTP Server using OEM 12c .
Issue : Oracle HTTP Server (OHS) marked as DOWN where as in actual OHS is running .
If you encounter issue like this then check
- OEM Agent on host running OHS server is running and last upload is successful ($OEM_AGENT_INSTANCE_HOME/bin/emctl status agent)
- Check that OHS status in Fusion Middleware Control and ensure that STATUS is displayed correctly (em is a J2EE application deployed on WebLogic Admin Server and OHS is registered with WebLogic Domain using opmnctl registerinstance)
Note: In my case OHS status in Fusion Middleware Control was also marked as DOWN (hence problem is at Fusion Middleware Control side and not at OEM 12c side )
- Next is to check communication from Fusion Middleware Control to Oracle HTTP Server — Enterprise Manager Fusion Middleware (deployed on WebLogic Admin Server) communicates to OHS Server on OHS Admin port (defined in $ORACLE_INSTANCE/config/OHS/ohs[n]/admin.conf defined by Listen parameter). Ensure that OHS Administration port is open across firewall from machine hosting WebLogic Admin Server (where EM is deployed) to machine hosting OHS server.
.
If this works then check logs in
- Check logs in Fusion Middleware Control $DOMAIN_HOME/sysman/log/emoms.trc
- WebLogic Admin Server hosting Fusion Middleware Control $DOMAIN_HOME/servers/AdiminServer/log/AdminServer-diagnostic.log
_LOGS_FROM_EMOMS.TRC__
2013-07-30 11:59:41,861 [[ACTIVE] ExecuteThread: ‘6’ for queue: ‘weblogic.kernel.Default (self-tuning)’] ERROR emas.ohs logp.251 – ServerPerformanceModel instantiated
___
__LOGS_FROM_AdminServer-diagnostic.log___
[2013-07-30T10:28:19.820+00:00] [AdminServer] [ERROR] [] [oracle.ohs] [tid: [ACTIVE].ExecuteThread: ’36’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: <WLS Kernel>] [ecid: 004sfWHLEQm7u105Nzl3ie0004Wz0000Y9,0:1] [APP: NonJ2EEManagement#11.1.1] [URI: /em/faces/as/topology/wlviewer] ReadFile failed Received fatal alert: bad_certificate
[2013-07-30T10:28:19.877+00:00] [AdminServer] [ERROR] [] [oracle.ohs] [tid: [ACTIVE].ExecuteThread: ’36’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: weblogic_idm] [ecid: 004sfWHLEQm7u105Nzl3ie0004Wz0000Y9,0:1] [APP: NonJ2EEManagement#11.1.1] [URI: /em/faces/as/topology/wlviewer] ReadFile failed Received fatal alert: bad_certificate
[2013-07-30T11:48:05.484+00:00] [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: [ACTIVE].ExecuteThread: ’14’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: weblogic_idm] [ecid: 004sf^iJDsf7u105Nzl3ie0004Wz0000xD,0:1] [APP: NonJ2EEManagement#11.1.1] [URI: /em/faces/as/ohs/ohsHome] OPMN.queryProcessStatus failed with the following exception[[ oracle.as.management. opmn. optic.OpticBad ConnectException: Error flushing write connection to OPMN
at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:774)
at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:330)
at oracle.as.management.mbeans.opmn.Opmn.queryProcessStatus(Opmn.java:328)
at oracle.as.management.mbeans.opmn.Opmn.getStatus(Opmn.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
… 4 more
____
Note: EM deployed on WebLogic Admin Server communicates to OHS Server on HTTPS (OHS Admin Port) and CERTS are generated during OHS registration with WebLogic Server.
- Check OHS server is registered with WebLogic Admin Server server and can communicate to WebLogic Admin Server (Host running OHS server can communicate to host running WebLogic Server on WebLogic Admin Server port)
Location of WebLogic Server Administration Server registered with OHS server is specified in $ORACLE_INSTANCE/config/OPMN/opmn/instance.properties
_____
adminPort=7001
adminHost=weblogicAdminHost
registered=true
***
***
_____
Fix: In my case registration entry was wrong (instance.properties) so un-register OHS instance and register again as opmnctl unregisterinstance; opmnctl registerinstance . After registering OHS again with WebLogic Administration Server fixed the issue
Related/References
- 1458381.1 Fusion Middleware Control Fails With “Load failed with IOException” When Trying To Administer OHS
- 1440083.1 FMW Console Fails To Start Webtier : Error flushing write connection to OPMN