Deploying OHS on NFS ? Think again : Locking/Performance Issue

performance

This post is related to Oracle HTTP Server (OHS) performance issue from our Oracle Fusion Middleware Training (next batch starts on 30th Aug, 2015) where we cover OHS on Day2 (Installation, Configuration, High Availability, Troubleshooting, integrating OHS with WebLogic as proxy etc) .

One of the trainee from our previous batch asked about the common issues that Oracle Fusion Middleware Administrator encounters. We recently implemented Oracle Fusion Middleware including OHS for one of our customer where file system on OHS server presented from SAN as NFS mount.

Error message while accessing the OHS Instance on Enterprise Manager  “Failed to invoke operation Load on MBean”

error1

After checking the ohs log file ($ORCALE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1),  I got the exact error behind the issue that was ” apr_proc_mutex_lock failed. Attempting to shutdown process gracefully”

[2015-05-06T08:18:23.8610+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22680] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

[2015-05-06T08:18:23.8628+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22679] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

[2015-05-06T08:18:23.8863+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22681] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

[2015-05-06T08:18:23.8894+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22678] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

[2015-05-06T08:18:24.8024+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22867] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

[2015-05-06T08:18:24.8506+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22872] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error:  apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

 

Root Cause : http.lock file (located in default location at $ORACLE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1 ) was under file system on NFS mount causing performance issue which in turn causing OHS to fail. It is recommended to keep OHS lock file under Local Disk and not on the shared storage. If the $ORACLE_INSTANCE is present on NFS mount then the above error will occur as any delay in response from OHS will force OPMN process to restart OHS.

 

Solution: Change the http.lock file location to Local_Disk (like /tmp or any other place on local disk) which is not NFS mounted and update that location under mpm_prefork_module and mpm_worker_module in httpd.conf (located in $ORACLE_INSTANCE/ config/ OHS/ ohs1) file as below

<IfModule mpm_prefork_module>
StartServers         5
MinSpareServers      5
MaxSpareServers     10
MaxClients         150
MaxRequestsPerChild  0
AcceptMutex fcntl
LockFile “<LOCAL_DISK_PATH>”
</IfModule>

<IfModule mpm_worker_module>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
AcceptMutex fcntl
LockFile ” <LOCAL DISK_PATH>”
</IfModule>

 

Reference:

  • My Oracle Support Note 1460851.1 (NFS Locking Issues)

 

Oracle Fusion Middleware
Interview Questions

Test your knowledge on Oracle Fusion Middleware or Use them to find your dream Job as Oracle Administrator

Click here to download Oracle FMW Interview Questions

 

If you want to learn more about Oracle Fusion Middleware products, register for our Oracle Fusion Middleware Training (Register before 24th August and Get Discount of 100 USD Apply code: F1OFF at time of checkout)

Note: We provide 100% Money back guarantee so in unlikely case of you being not happy after 2 sessions, just drop us a mail before third session and We’ll refund FULL money.

Share This Post with Your Friends over Social Media!

About the Author Shweta Jain

Leave a Comment:

2 comments
Anil says April 7, 2016

Thank you for this blog post. It was very helpful as I encountered “Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully” issue and it was resolved by moving http_lock file to /tmp dir instead of NFS.

Reply
Add Your Reply

[i]
[i]