JVM Tuning (Garbage Collection) in Oracle Apps 11i

There are some good notes on JVM tuning from Mike Shaw on Steven Chan’s blog  here , here and here and some good Metalink notes at end of this post.
                     Important thing missing from all these notes (for some one like me who is new to Java) is basics of Garbage Collection, Generation and how to read GC output.
In this post I’ll start with basics of JVM GC (Garbage Collection) and then in next post apply this theory for real time performance issues w.r.t. JVM (11i Java Virtual Machine) .

Garbage – Java object is considered garbage when it can no longer be reached from any pointer in the running program.

Generations – Memory in JVM is managed in terms of generation i.e. Young generation and tenured generation. Memory pool holding object of different ages like young, tenured. If a particular generation fills up, garbage collection occurs.

A. Young generation – Objects are initially allocated in Young generation (most of objects die here). When Young generation fills up, it causes Minor Garbage Collection. Any objects survived after Minor GC (Garbage Collection) are moved to Tenured Generation.  Minor Garbage collection is quick as compared to Full/Major GC.

B. Tenured generation – Surviving objects (from young generation) after minor garbage collection are moved to area called tenured generation, When tenured generation fills up it causes major collection (aka Full GC or Full Garbage Collection). Major collection is slow as it involves all live objects.

Garbage Collection (GC) – is program which clears garbage(dead java objects). Garbage Collection work on fundamental principle that majority of java objects die young (quickly after arriving in JVM). There are two kind of Garbage Collection Minor Garbage Collection and Major Garbage Collection (aka Full GC)

Example of Minor GC –  3824.164: [GC 196725K->141181K(209864K), 0.3295949 secs] Example of Minor GC –  3841.051: [Full GC 150466K->87061K(217032K), 3.2626248 secs]

Pauses: is the time when application becomes unresponsive because garbage collection is occurring.

.

Understanding JVM parameter for 11i
Sizing the generation is very important in tuning JVM GC. Before jumping to Sizing generation (Young and Tenured) lets look at default 11i JVM parameters

In context file($APPL_TOP/ admin/ $CONTEXT_NAME.xml) default entry for JVM is like

<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2-XX:+PrintGCTimeStamps -XX:+UseTLAB </jvm_options>

1. Above line represents JVM (OACoreGroup) size in 11i
2. -Xms128M, means start with 128MB heap size
3. -Xmx512M, means grow JVM heap size upto max size of 512 MB
4. -XX:NewRatio=2 is to control young generation i.e. ratio between young and tenured generation is 1:2 (i.e. if size of young generation is 50 MB then size of tenured generation should be approx. 100MB)
5. -XX:MaxPermSize=128M limit the permanent generation to 128M (permanent generation is part/area in tenured generation)
6. -XX:+UseTLAB represents to use thread-local object allocation
7. There are two more parameters (11i JVM uses default values) -XX:MinHeapFreeRatio=<minimum> & -XX:MaxHeapFreeRatio=<maximum> with default value of 40 & 70 resp. (for Solaris)

If percentage of free space in generation falls below 40%, size of generation will expand and if percentage of free space exceeds 70%, the size of generation will shrunk.

.
Various type of Garbage Collector
From JDK 1.4.2 there are total 4 type of collectors (prior to 1.4.2 it was just one collector i.e. default collector)

1. Default Collector: JDK prior to 1.4.2 uses default collector. If you don’t specify any parameter with JVM default is default collector.

2. ThroughPut Collector : This collector uses parallel version of young generation collector but Tenrured generation is collected in normal way. To set throughput collector use –XX:+UseParallelGC  so change

<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB </jvm_options>
to
<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC</jvm_options>

3. Concurrent Low Pause Collector : Concurrent Collector is used to collect tenured generation collection concurrently with execution of application. Parallel version of collector is used for young generation. To set Concurrent Low Pause Collector use –XX:+UseConcMarkSweepGC
like
<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseConcMarkSweepGC</jvm_options>

4. Incremental low pause collector : This collector collects just portion of tenured generation at each minor garbage collection. To use Incremental low pause collector use
-Xincgc

If you are on JDK 1.4.2 with multi CPU try setting Concurrent Low Pause Collectoras Garbage Collector.

Thumb rule for Grabage Collection/ JVM tuning w.r.t. 11i
1.
Stay on latest JVM/JDK version where ever possible (latest certified with 11i is JRE 6, you should be at-least 1.4.2 and higher)
2. For OACoreGroup consider no more than 100 active users per JVM
3. There should NOT be more than 1 active JVM per CPU
4. Try to reduce GC (Garbage Collection) frequency (specially Major/Full GC). Play with various JVM parameters like (-Xmx, -Xms, -XX:MaxPermSize, -XX:NewRatio, -XX:+UseParallelGC/ -XX:+UseConcMarkSweepGC)
5. If you are on JDK 1.4.2 with multiple CPU middle tier, use Concurrent Low Pause Garbage Collector  by setting -XX:+UseConcMarkSweepGC with JVM
6. If you are using Oracle Configurator, assign dedicated JVM for configurator requests
7. Try setting JVM max size NOTgreater than 1 GB, (use multiple JVM’s of 512MB or 1024 MB), this is to reduce GC time (more heap size means more time in GC)
8. Minor GC should be occurring at interval long enough to allow many objects to die young (i.e. lot of objects should die between two minor GC).
9. Throughput (which is time NOT spent on GC) is inversely proportion to amount of memory. Higher the memory for JVM, more time for GC meaning low throughput.
10. Unless you have problems with pauses (time when application becomes unresponsive because garbage collection is occurring), try granting as much memory as possible to VM (128 to 512 is good start and fine tune as per load testing results)
.

How to find JDK version used by Apache/Jserv (JVM) in 11i ?

In context file search for parameter like s_jdktop

<JDK_TOP oa_var=”s_jdktop”>/oracle/apps/11i/vis11icomn/util/java/1.4/j2sdk1.4.2_04</JDK_TOP>

Where is JVM log location in 11i ?
$IAS_ORACLE_HOME/ Apache/ Jserv/ logs/ jvm/ OACoreGroup.0.stdout  (GC output)
$IAS_ORACLE_HOME/ Apache/ Jserv/ logs/ jvm/ OACoreGroup.0.stderr  (JVM Error)

.

How to read GC (JVM stdout) file ?

Example of JVM out file to understand Garbage Collection in 11i

3824.164: [GC 196725K->141181K(209864K), 0.3295949 secs] 3840.734: [GC 207741K->150466K(217032K), 0.3168890 secs] 3841.051: [Full GC 150466K->87061K(217032K), 3.2626248 secs] 3854.717: [GC 155413K->97857K(215568K), 0.2732267 secs] 3874.714: [GC 166209K->109946K(215568K), 0.3498301 secs]

1. Line 1,2 4 and 5 are example of Minor Collection
2. Line 3 (Full GC) is example of Major Collection
3. First entry in each line is time in seconds since JVM started, To find out time between two GC (Garbage Collection) just subtract second entry from first i.e. (3840.734 – 3824.164 = 16.57 seconds)
4. 196725K->141181K in first line indicates combined size of live objects before and after Garbage Collection (GC)
5. (209864K) in first line in parenthesis, represents object after minor collection that aren’t necessarily alive but can’t be reclaimed, either because they are directly alive, or because they are referenced from objects in tenured generation.
6. 0.3295949 secs in first line represents time taken to run minor collection.
7. Full GC in line three represents Full Garbage Collection or Major Collection


References

  • 362851.1  Guidelines to setup the JVM in Apps E-Business Suite 11i and R12
  • 370583.1  Basic troubleshooting of JVM consuming cpu or too many JDBC connections in Apps 11i
  • 567647.1  Using Various Garbage Collection Methods For JVM Tuning
  • 390031.1  Performance Tuning Forms Listener Servlet In Oracle Applications
  • Garbage Collection in Sun JDK
  • GC Example

More on tuning Apache and JVM in 11i/R12 coming soon ….

Share This Post with Your Friends over Social Media!

About the Author Atul Kumar

Oracle ACE, Author, Speaker and Founder of K21 Technologies & K21 Academy : Specialising in Design, Implement, and Trainings.

follow me on:

Leave a Comment:

6 comments
BISWAJITPR says January 9, 2009

Hi,
In our production we have weblogic 10.2 running on windows platform.
In our weblogic domain we have 1 admin and 1 managed server.An enterprise application has been deployed on managed server i.e only targed on managed server.
I have allocated 512m to JVM for both the server.Then we got “out of memory” error while our server trying to serve the request for the request.I have increased the JVM to 1.5gb.After then this problem reduced,however while users are trying to fetch the larger data or upload the data it comes up again.
My JVM memory set as follows:
:noResetJavaVM
if not “%MEM_ARGS%” == “” goto noResetMemArgs
set MEM_ARGS=-Xms1536m -Xmx1536m -Xgc:gencon -Xgcreport -Xgcpause -Xverbose:gc

:noResetMemArgs
Using JRockit SDK 1.5.0_11
Some time my gc also not working.

Can any one have any valuable suggestion..

Reply
Atul Kumar says January 9, 2009

I suppose you are running custom application on this domain.

Possibly you need to tune either not to allow large data or break data in to smaller part and work (Redesign application)

Use jconsole for any memory leaks or Garbage Collection tuning

Reply
PankajMishra says February 25, 2010

We are getting “Out of Memory” error on our JVMs connected to the Configurator. Ver 11.5.10. We see that the Permanent area gets filled up to 99% and the JVMfails. What could be causing the Permanent area to fill up? The other heap areas are only 30-50% full.

Reply
Atul Kumar says February 25, 2010

@ Pankaj,
How many JVM’s you have for configurator and in total ?

How many users connect to your apps concurrently ?

Did you consider increasing number of JVM’s or Size of JVM heap ?

Reply
siva prakash says May 17, 2010

Thanks for this article atul .
learnt about jvm tuning .

Reply
DigJVM says December 16, 2012

Very nice explanation about garbage collectors. I have been using UseParallelGC but my GC’s have been always quicker and enough apart between 400 to 600 secs. It was the FGC’s which is pausing the users and I was under impression that the above setting would address it. I have changed it to UseConcMarkSweepGC I will observe the behaviour now. One more thing is after changing this the stdout format has changed, earlier the lines user to be
nnnn.nnn [GC ….
but now the format changed like
par…
not sure if this indicates any concern

Reply
Add Your Reply