OAM server gets shut down automatically : fork: Resource temporarily unavailable
Hi All,
Today I ‘m very happy to share some information regarding a show stopper issue which I faced few days back in production box.
The issue was when the load gets increased in the server , OAM server gets shut down Automatically.
For more information, Please go through below :
Problem Description:
We are using 2 single sign on solutions in which OAM is the centralized authentication server for tracks like portal,Siebel,OBIZ etc. .
- OAS SSO 10.1.4.3
- OAM 10.1.4.3
The problem was when the load gets increased in the server , OAM server gets shut down automatically.
Error details:
–bash-3.00$ /u01apps/oaminstall/access/oblix/apps/common/bin/access_server_watchdog: fork: Resource temporarily unavailable
/u01apps/oaminstall/identity/oblix/apps/common/bin/ois_server_watchdog: fork: Resource temporarily unavailable
Access Server has stopped running. Starting a new instance.
OIS Server has stopped running. Starting a new instance.
/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server
/u01apps/oaminstall/access/oblix/apps/common/bin/start_access_server: fork: Resource temporarily unavailable
/u01apps/oaminstall/identity/oblix/apps/common/bin/ois_server_watchdog: fork: Resource temporarily unavailable
Could not start the Access Server. The watchdog is stopping.
/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server: fork: Resource temporarily unavailable
/u01apps/oaminstall/identity/oblix/apps/common/bin/start_ois_server: fork: Resource temporarily unavailable
Error: Could not determine your OS. Aborting.
Access Server has stopped running. Starting a new instance.
Using Linux Threading Library.
For Past 2 weeks , I was breaking my head to fix up showstopper issue. Lots of workaround & alternate solution approach was buzzing up in my mind and I was cracking in all possible ways whether the Issue might have happened because of Hardware sizing or memory space issue /Loading or Version Problem or Connection Reached to maximum Limits but I never thought that a single change in the server will end the showstopper issue easily J
I would like to share the point which I covered during analyze.
- Verified soft and hard limits for file descriptors and number of processes available to a users.
Result: Limit value looks pretty okay and less possibility for occurrence of issue
2. As a rule of thumb, verified on the firewall timeout the Maximum Client Session Time should be slightly less than firewall inactivity connection timeout value.
Result: Changed Accordingly , but still the problem persists
3. As a rule of thumb, verified on the OAM component version, Access Server and Identity Server need to have version greater than or equal than webgate version.
Result: Applied with latest version BP13 but same problem persists.
4. Verified the resource policies for all tracks.
Result: Configuration looks pretty okay and less possibility for occurrence of issue
I was wondering & didn’t get any clue where the issue persist . Few days before when I was reading oracle trouble shooting document (http://docs.oracle.com/cd/E15217_01/doc.1014/e12489/trblsht.htm .)
Immediately something strike in my mind , I got some clue and suspected that the issue might have happened due to F.10 NPTL Requirements and Post-Installation Tasks
Here by the solution :
Solution: The native libraries support something called NPTL which stands for “Native POSIX Thread Library. This library allows Linux to run multithreaded program much faster regardless. the program is 32 bit or 64 bit. The Linux 5 OS comes with this library by default.
Start identity Server using start_ois_server_nptl instead of start_ois_server
Start Access Server using start_access_server_nptl instead of start_access_server
Note: We were used to start the server without NPTL.
These NPTL scripts use the new multithreaded NPTL capable libraries, resulting better performance.
Hope this post would help you if you are facing same kind of issue. 🙂
About the Author sarath
An Oracle Identity and Access Management professional, having working on Oracle Access Manager Single Sign-On implementations, Installation/Configuration of Identity Server, Web Pass, Web Gate, Access Gate, Policy Manager, Access Server, Policy Domains, Authentication /Authorization schemes, Single Sign-On (single and multi-domain), OIM, OVD, OID, OAAM, OIF, High Availability/Failover/ SSL deployment.