Coherence Cluster : tangosol.net. RequestTimeoutException Timeout during service start
Fusion Middleware (11g R1 PS2 onwards) can be configured to use Coherence between WebLogic Cluster components. I recently faced issue while starting application (ODI), configured to use coherence as cluster cache.
Two weblogic managed servers (running on server1 and server2) were configured as part of weblogic cluster. Managed server were configured to start with below parameter (9088 here is coherence port)
-Doracle.odi.coherence.wka1=server1 -Doracle.odi.coherence.wka1.port=9088 -Doracle.odi.coherence.wka2=server2 -Doracle.odi.coherence.wka2.port=9088 -Dtangosol.coherence.localport=9088
- Machine (server1) was running with two network cards – 10.7.42.21 (eth0) & 192.168.43.21 (eth1) with server1 resolving to 10.7.42.21
- Machine (server2) was running with two network cards – 10.7.42.22 (eth0) & 192.168.43.22 (eth1) with server2 resolving to 10.7.42.22
Error message reported in weblogic managed server log file ($DOMAIN_HOME/servers/<ms1>/logs)
_________________________________________
2011-08-25 12:14:27.949/34.218 Oracle Coherence GE 3.6.0.4 <D4> (thread=[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’, member=n/a): TCMP bound to /192.168.43.21:9088 using SystemSocketProvider
####<Aug 25, 2011 11:45:05 AM BST> <Error> <HTTP> <server1> <odi_server1> <[STANDBY] ExecuteThread: ‘2’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <0000J80XZzX5MeD_z9T4iW1ELYPm000002> <1314269105143> <BEA-101216> <Servlet: “AgentServlet” failed to preload on startup in Web application: “oraclediagent”. com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=ClusterMemberSet=ServiceMemberSet(OldestMember=n/a ActualMemberSet=MemberSet(Size=0, BitSetCount=0) MemberId/ ServiceVersion/ ServiceJoined/ MemberState)) at com.tangosol. coherence. component.util. daemon.queueProcessor. service.Grid. onStartupTimeout(Grid.CDB:6)
2011-08-25 12:14:27.949/34.218 Oracle Coherence GE 3.6.0.4 <D4> (thread=[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’, member=n/a): TCMP bound to /192.168.43.21:9088 using SystemSocketProvider
2011-08-25 12:02:07.266/27.308 Oracle Coherence GE 3.6.0.4 <Info> (thread=[ACTIVE] ExecuteThread: ‘0 ‘ for queue: ‘weblogic.kernel.Default (self-tuning)’, member=n/a): Loaded cache configuration from” zip:/u01/fmw/odi/Oracle_ODI1/ oracledi.sdk/lib/ coherence.jar!/ coherence-cache-config.xml”
2011-08-25 12:02:12.554/32.596 Oracle Coherence GE 3.6.0.4 <D4> (thread=[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’, member=n/a): TCMP bound to /192.168.43.21:9088 using SystemSocketProvider
2011-08-25 12:02:43.143/63.185 Oracle Coherence GE 3.6.0.4 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2011-08-25 12:02:12.559, Address=192.168.43.21:9088, MachineId=17358, Lo cation=process:5375, Role=WeblogicServer) has been attempting to join the cluster using WKA list
[server2.co.uk/10.7.42.22:9088, server1.co.uk/10.7.42.21:9088] for 30 seconds without success; this could indicate a mis-configured WKA, or it may simply be the result of a busy cluster or active failover._______________________________________________
Message “TCMP bound to /192.168.43.21:9088” was indication that it was trying to start coherence on second network card (eth1) . where TCMP – Tangosol Cluster Management Protocol is a clustered IP-based protocol, for server discovery, cluster management, service provisioning and data transmission.
Fix: include parameter *.coherence.localhost=<localserver> as shown below so that coherence
- On first maanged server running on server1
-Doracle.odi.coherence.wka1=server1 -Doracle.odi.coherence.wka1.port=9088 -Doracle.odi.coherence.wka2=server2 -Doracle.odi.coherence.wka2.port=9088 -Dtangosol.coherence.localhost=server1 -Dtangosol.coherence.localport=9088
- On second maanged server running on server2
-Doracle.odi.coherence.wka1=server1 -Doracle.odi.coherence.wka1.port=9088 -Doracle.odi.coherence.wka2=server2 -Doracle.odi.coherence.wka2.port=9088 -Dtangosol.coherence.localhost=server2 -Dtangosol.coherence.localport=9088