Hi All,
Today I would like to cover firewall timings management in access environment.
Here, we have OAM 10g , two SSO servers(OID/OSSO 10g-10.1.4.3.0), oracle portal 11g, OBIZ, SOA, Seibel and etc.
Integration details, Portal having own SSO server(OID/OSSO 10g-10.1.4.3.0) and portal application registered as a partner application in portal OSSO. Portal OSSO server is integrated with OAM for centralized authentication because in our project more than 10 application integrated with OAM10g(10.1.4.3.0) for SSO.
All applications are integrated with OAM 10g. We are accessing all applications through Oracle Portal(11g) application by clicking links under portal workspace.
We were tested application performance by using load runner. During run our OAM server went down automatically and I have noticed some error in logs.
ERROR —/usr/abuild/Oblix/coreid1014/palantir/webgate2/src/web_gate.cpp:143: Error: Exception re-thrown from ObWebGate::Init( <NULL> = <NULL>, <NULL> = <NULL> )%0a../obthread.cpp:748: Error: Exception re-thrown in ObThread::Start%0a../obthread.cpp:726: Error: Create thread failed, error message is Resource temporarily unavailable%0a
/…/oblix/apps/common/bin/start_access_server: fork: Resource temporarily unavailable
Could not start the Access Server. The watchdog is stopping.
This usually means you have hit the limit on the number of running processes, and cannot start any more.
So I thought it was happened due to heavy load in OAM, then I tuned all OAM components as mentioned in this link
http://docs.oracle.com/cd/B28196_01/idmanage.1014/b25344/perform.htm#sthref24
But no luck, again OAM server went down automatically but this time OAM handled more time compare to previous time failure. Again finger crossed :(.
After long investigation, the real root cause has been find out. The real problem due to firewall timeout.
Here, I have specified my workarounds.
- Validate any firewall exist between Access Server and LDAP ? (is it OID ? or active server ).
- Validate any firewall exist between Access Server and webgate ?
- ran immediately once OAM server went down
netstat -a > connections.txt
- In connection.txt I have noticed 11k connections in TIME_WAIT state. It is clear that server restart due number of big connections , and most probable firewall is involved here.
- Implemented doc id : 1253194.1
“Configuration Recommendations
The overall recommendation is simply to keep firewall connection timeout limits higher than the Access Clients’ maximum connection time….”
The problem is vanished and our performance test went smoothly without any issues. Hope, this post helped you. 🙂
About the Author sarath
An Oracle Identity and Access Management professional, having working on Oracle Access Manager Single Sign-On implementations, Installation/Configuration of Identity Server, Web Pass, Web Gate, Access Gate, Policy Manager, Access Server, Policy Domains, Authentication /Authorization schemes, Single Sign-On (single and multi-domain), OIM, OVD, OID, OAAM, OIF, High Availability/Failover/ SSL deployment.
