If you are upgrading your system (or planning to upgrade) to ATG roll up 4, 5 or 6 or using R12 (12.0.X to 12.1.X); beware of below common intermittent issues around Web Server (Apache/Jserv/OC4J/JVM) for 11.5.10 and R12
– Responsibility not visible to users
–NoCalssDef found error in JVM logs
–Intermittent hang in JVM (Jserv/Apache)
–Intermittent page not found while accessing/login apps ( 500 Internal Server Error)
Most (if not all) of these issues are around Distributed Java Cache feature of JVM in Oracle Apps 11i/R12, before going further let me ask you basic questions around Java Cache
Q. What is significance of wrapper.bin.parameters=-DLONG_RUNNING_JVM=FALSE in jserv.properties ?
Q. Why you need s_java_object_cache_port & s_fnd_cache_port_rangein Context File ($SID_hostname.xml) ?
Q. What is $APPLRGF/javacache.log ?
Good starting point to understand Java Caching in 11i/R12 is to go through Mike Shaw’s post here
Lets now understand basics of Java Object Cache (JOC) or Distributed Java Cache which was introduced in apps from ATG RUP 3.
JVM – Java Virtual Machine : to execute Java Code in apps; JServ (for 11i) and OC4J (for R12) .
Caching in JVM: To improve performance, JVM’s cache data they extract from Database before passing it on to User.
Database Invalidation: This feature ensures that when the data in a database is updated, the corresponding cached data in the mid-tier is also updated. This is useful when the updates to the database bypass the JVMS for example updates through forms or sqlplus.
Distributed Java Caching: In a multiple JVM environment (multiple JVM on one or more middle tier or Single JVM on multiple Middle tier ), Distributed caching allows one JVM to inform all the other JVMs about updates to any cached data. This ensures consistency of the cached data across all the JVMs.
The updates and invalidations of the objects in a distributed Component Cache are propagated to all the other JVMs that are running against the same database.
.
How data synchronization and invalidations handled in Distributed Java caching : In multi JVM environment;
a) First JVM to come online acts as Master Cache Controllerand rest other JVMs as client JVMs .
b) Upon startup, each JVM auto-registers the host it is running on in the database ( table name JTF_PREFAB_WSH_POES_B).
c) The Master cache controller listens for cache invalidation in the port specified by s_java_object_cache_port(default is 12345) and the client JVMs will open client ports taken from the range established by s_fnd_cache_port_range (when specified – Default not set).
d)When an object gets invalidated or replaced in the cache, then invalidation message is sent to all other JVMs via Master Cache Controller.
If this Master JVM (acting as Master Cache Controller) dies gracefully or abnormally, then one of client JVM takes role of Master cache Controller (There are couple of Bugs around this issue)
.
Data Partition: This feature allows the cached data to be partitioned based on any partition keys. This is useful for JVMS in hosting environments running against a Virtual Private Database (VPD). The caching framework ensures that the cache data is partitioned based on security group id.
.
How to disable distributed caching or enable local caching ?
Set s_long_running_jvm to false in context (xml) file and run autoconfig . <jms_cache_oa_var=”s_long_running_jvm”>false</jms_cache>
This will update wrapper.bin.parameters=-DLONG_RUNNING_JVM to FALSE in jserv.properties
or
set wrapper.bin.parameters=-DCACHEMODE=LOCAL in jserv.properties (default is Distributed)
.
Checking Java Cache from front end
login to URL http://<machinename>:<port> /OA_HTML/jtflogin.jsp
sysadmin/<sysadmin_passwd> Performance -> Components
For Cloned Instances
Nodes registered for Java Caching and the cache configuration details are stoerd in table JTF_PREFAB_WSH_POES_B, its good idea to truncate this table on cloned instance (If entry contains source instance nodes) and restart Apache to re-register target instance nodes in table JTF_PREFAB_WSH_POES_B
.
Common issues around Java Cache
Issue 1 :
Error message in $IAS_ORACLE_HOME/Apache/Jserv/logs/mod_jserv.log or oacorestd.err in R12
Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. The Distributed Caching System failed to initialize on port: 12345. The list of hosts i
n the distributed caching system is: XXXXXXXXXXXXXXXXXXX . The port 12345 should be free on each host running the JVMs. The default port 12345 can be overridden using -Doracle.apps.jtf.cache.IASCacheProvide
rcacheProvider.port=<port number>
at oracle.apps.jtf.cache. IASCacheProvider. init(IASCacheProvider.java:220)java.lang.NoClassDefFoundError
at oracle.apps.fnd.framework .webui.OAPageContextImp l.getMACValidationLev
el(OAPageContextImpl.java)
at oracle.apps.fnd.framework. webui.OAPageBean.isMac Enabled(OAPageBean.ja
va:7260)
at oracle.apps.fnd .framework.webui. OAPageBean.prepare Page(OAPageBean.jav
a:668)
at oracle.apps.fnd.frame work.webui.OA PageBean.prepare Page(OAPageBean.jav
a:509)
.
Issue 2:
Error message in $APPLRGF/javacache.log
[2009-01-05 Tue 04:51:45.603 PM BST] 022 [Grp] Timeout while waiting for coordinator: [10.24.3.83:46463, pos=0, uid=0, pri=0] to remove member: [XX.XX.XX.XX:46464, pos=1, uid=1, pri=0]. Elapsed time: 300000 msec.
[2009-01-05 Tue 04:58:26.793 PM BST] 021 [Grp] Haven’t got PING message and only 0 recent packets. Declare Coordinator Dead at [XX.XX.XX.XX:46463, pos=0, uid=0, pri=0]
.
Issue 3:
Unable to See Responsibilities after End Date Is Removed
.
Issue 4:
Newly added Responsibilities not appearing in user Home Page
.
Issue 5:
Error in browser while accessing apps 11i/R12
500 Internal Server Error
java.lang.NoClassDefFoundError
at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126)
at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170)
.
References :
- 275879.1 Oracle Applications Java Caching Framework Developer’s Guide Release 11i (11.5.10)
- 455194.1 Diagnosing database invalidation issues with Java Cache for eBusiness Suite
- 374309.1 Newly added responsibility is not displayed until Apache is bounced, after applying ATG RUP3
- 295365.1 JVM is not Responding – Apache/Jserv Is Not Serving Any JSP
- 372832.1 Responsibility Changes Do Not Take Immediate Effect
- 399661.1 NoClassDefFoundError Error received trying to access iStore Store Front
- 336431.1 Getting IAS Cache Initialization Failed While Login to Applications
- Java Object Cache in 10.1.3 for R12
- Mike Shaw’s post – Java Caching for Oracle Applications 11i on Steven Chan’s blog
