This post is related to Oracle HTTP Server (OHS) performance issue from our Oracle Fusion Middleware Training (next batch starts on 30th Aug, 2015) where we cover OHS on Day2 (Installation, Configuration, High Availability, Troubleshooting, integrating OHS with WebLogic as proxy etc) .
One of the trainee from our previous batch asked about the common issues that Oracle Fusion Middleware Administrator encounters. We recently implemented Oracle Fusion Middleware including OHS for one of our customer where file system on OHS server presented from SAN as NFS mount.
Error message while accessing the OHS Instance on Enterprise Manager “Failed to invoke operation Load on MBean”
After checking the ohs log file ($ORCALE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1), I got the exact error behind the issue that was ” apr_proc_mutex_lock failed. Attempting to shutdown process gracefully”
[2015-05-06T08:18:23.8610+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22680] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8628+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22679] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8863+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22681] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8894+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22678] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:24.8024+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22867] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:24.8506+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22872] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
Root Cause : http.lock file (located in default location at $ORACLE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1 ) was under file system on NFS mount causing performance issue which in turn causing OHS to fail. It is recommended to keep OHS lock file under Local Disk and not on the shared storage. If the $ORACLE_INSTANCE is present on NFS mount then the above error will occur as any delay in response from OHS will force OPMN process to restart OHS.
Solution: Change the http.lock file location to Local_Disk (like /tmp or any other place on local disk) which is not NFS mounted and update that location under mpm_prefork_module and mpm_worker_module in httpd.conf (located in $ORACLE_INSTANCE/ config/ OHS/ ohs1) file as below
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
AcceptMutex fcntl
LockFile “<LOCAL_DISK_PATH>”
</IfModule>
<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
AcceptMutex fcntl
LockFile ” <LOCAL DISK_PATH>”
</IfModule>
Reference:
- My Oracle Support Note 1460851.1 (NFS Locking Issues)
If you want to learn more about Oracle Fusion Middleware products, register for our Oracle Fusion Middleware Training (Register before 24th August and Get Discount of 100 USD Apply code: F1OFF at time of checkout)
Note: We provide 100% Money back guarantee so in unlikely case of you being not happy after 2 sessions, just drop us a mail before third session and We’ll refund FULL money.
