OAM server gets shut down automatically : fork: Resource temporarily unavailable

Hi All,

Today I ‘m very happy to share some information regarding a show stopper issue which I faced few days back in production box.

The issue was  when the load gets increased in the server , OAM server gets shut down Automatically.

For more information, Please go through below :

Problem Description:

We are using 2 single sign on solutions in which OAM is the centralized authentication server for tracks like portal,Siebel,OBIZ etc. .

  1. OAS SSO 10.1.4.3
  2. OAM 10.1.4.3

The problem was when the load gets increased in the server , OAM server gets shut down automatically.

Error details:

bash-3.00$ /u01apps/oaminstall/access/oblix/apps/common/bin/access_server_watchdog: fork: Resource temporarily unavailable

/u01apps/oaminstall/identity/oblix/apps/common/bin/ois_server_watchdog: fork: Resource temporarily unavailable

Access Server has stopped running. Starting a new instance.

OIS Server has stopped running. Starting a new instance.

/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server

/u01apps/oaminstall/access/oblix/apps/common/bin/start_access_server: fork: Resource temporarily unavailable

/u01apps/oaminstall/identity/oblix/apps/common/bin/ois_server_watchdog: fork: Resource temporarily unavailable

Could not start the Access Server. The watchdog is stopping.

/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server: fork: Resource temporarily unavailable

/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server: fork: Resource temporarily unavailable

Error: Could not determine your OS. Aborting.

Access Server has stopped running. Starting a new instance.

Using Linux Threading Library.

For Past 2 weeks , I was breaking my head to fix up showstopper issue. Lots of workaround & alternate solution approach was buzzing up in my mind  and I was cracking in all possible ways whether the Issue might have happened  because of  Hardware sizing or memory space issue /Loading or Version Problem or Connection Reached to maximum Limits but I never thought that a single change in the server will end the showstopper issue easily J

 I would like to share  the point  which I covered during analyze.

  1. Verified soft and hard limits for file descriptors and number of processes available to a users.

             Result: Limit value looks pretty okay and less possibility for occurrence of issue

     2.   As a rule of thumb,  verified on the firewall timeout the Maximum Client Session Time should be slightly less than firewall inactivity connection timeout value.

             Result: Changed Accordingly , but still the problem persists

      3.  As a rule of thumb, verified on the OAM component version, Access Server and Identity Server need to have version greater than or equal than webgate version.

              Result: Applied with latest version BP13 but same problem persists.

       4.  Verified the resource policies for all tracks.

              Result: Configuration looks pretty okay and less possibility for occurrence of issue

I was wondering & didn’t get any clue where the issue persist . Few days before when I was reading oracle trouble shooting document (http://docs.oracle.com/cd/E15217_01/doc.1014/e12489/trblsht.htm .)

Immediately something strike in my mind , I got some clue and suspected that the issue might have happened  due to F.10 NPTL Requirements and Post-Installation Tasks

Here by the solution :

Solution: The native libraries support something called NPTL which stands for “Native POSIX Thread Library. This library allows Linux to run multithreaded program much faster regardless. the program is 32 bit or 64 bit. The Linux 5 OS comes with this library by default.

Start identity Server using start_ois_server_nptl instead of start_ois_server

Start Access Server using start_access_server_nptl instead of start_access_server

Note: We were used to start the server without NPTL.

These NPTL scripts use the new multithreaded NPTL capable libraries, resulting better performance.

Hope this post would help you if you are facing same kind of issue. 🙂

Share This Post with Your Friends over Social Media!

About the Author sarath

An Oracle Identity and Access Management professional, having working on Oracle Access Manager Single Sign-On implementations, Installation/Configuration of Identity Server, Web Pass, Web Gate, Access Gate, Policy Manager, Access Server, Policy Domains, Authentication /Authorization schemes, Single Sign-On (single and multi-domain), OIM, OVD, OID, OAAM, OIF, High Availability/Failover/ SSL deployment.

Leave a Comment:

3 comments
Dinesh K says March 27, 2013

Hello Sarath,

Time and again you have proved that you are master of debugging issues and fixing it with super cool solution. The usage of NPTL has improved the performance of OAM significantly and we r looking forward for more such useful performance tuning posts from you!!!!

Cheers,
Dinesh.

Reply
sarath says March 28, 2013

Thanks mahi for your comments.
Yes, you are right but the same fix is done for two different issues.
1. when we were Unable to start the OIS server. The reason behind ,32 bit software installed on 64 bit OS which was sorted out in 3 ways , one of those is using nptl .
2. Here the problem was related with performance ,( Previously we didn’t face any kind of issue which is mentioned above ) where as during load time , NPTL will help us to improve the performance .

-Sarath

Reply
Add Your Reply