addPartnerForMultiDataCentre is not updating MDC partners in oam-config.xml in OAM 11g R2 PS2

In OAM 11g R2 PS2, I was working on Multi Data Center setup by following the documentation. I had to run the WLST command addPartnerForMultiDataCentre by giving partnerInfo.properties file as input.

What does this command do?

In MDC, when the failover happens from DC1 to DC2, all the webgate requests will be routed to DC2 to serve. The user session would have cookies/session pertaining to DC1. When the DC2 OAM servers serve the user request, then DC2 specific cookies/session has to be present. Before that, DC2 OAM servers will talk to DC1 OAM servers through back channel using Access Gate.

partnerInfo.properties contains the below details:

remoteDataCentreClusterId=DC2_CLUSTER
oamMdcAgentId=ACCESS_GATE_NAME
PrimaryHostPort=DC2_OAM_SERVER_NAME:port
SecondaryHostPort=
AccessClientPasswd=ACCESS_GATE_PASSWORD
oamMdcSecurityMode=open
agentVersion=11g
trustStorePath=
keyStorePath=
globalPassPhrase=
keystorePassword=

Let me explain every parameter:

remoteDataCentreClusterId: This is the secondary Data center Cluster Name.

oamMdcAgentId: Access Gate name which is making back channel call to DC1 for validating/requesting user session details. By default in OAM 11g R2 PS2, accessgate-oic is created. I have used this in my case. Also, ensure that Allow Management operations flag is enabled in this AG profile. For quick test, you can verify the regular webgate profiles and see that this flag is disabled by default.

PrimaryHostPort & SecondaryHostPort: Secondary DC OAM server host name and port eg., oam2.oracle.com:5575 and oam2.oracle.com:5576 respectively.

oamMdcSecurityMode: Mode in which AG is running.

agentVersion: AG version defined in profile.

trustStorePath, keyStorePath, globalPassPhrase, keystorePassword: If AG is set in simple/cert mode, provide the keystore and relevant details.

Steps to run the command:

Run ./wlst.sh from $ORACLE_HOME/common/bin
connect to weblogic admin server.
Run WLST command addPartnerForMultiDataCentre(propfile=”/opt/oam/MDC/partnerInfo.properties”)
You should see successful message as shown belowls:/oam_domain/serverConfig>addPartnerForMultiDataCentre(propfile=”/opt/oam/MDC/partnerInfo.properties”)
Partner added successfully.
success:

I had executed this command in both data centers. After execution, we can verify this in oam-config.xml under the section MultiDataCenterPartners as shown below

<Setting Name=”MultiDataCenterPartners” Type=”htf:map”>
<Setting Name=”CLUSTER_NAME” Type=”htf:map”>
<Setting Name=”oamMdcSecurityMode” Type=”xsd:string”>open</Setting>
<Setting Name=”periodForWatcher” Type=”xsd:string”>2000</Setting>
<Setting Name=”maxConnPool” Type=”xsd:string”>10</Setting>
<Setting Name=”minConnPool” Type=”xsd:string”>1</Setting>
<Setting Name=”delayForWatcher” Type=”xsd:string”>1000</Setting>
<Setting Name=”oamMdcAgentId” Type=”xsd:string”>accessgate-oic</Setting>
<Setting Name=”accessClientPasswd” Type=”xsd:string”>qqwer3235123asdf</Setting>
<Setting Name=”PrimaryHostPort” Type=”xsd:string”>HOST:PORT</Setting>
<Setting Name=”agentVersion” Type=”xsd:string”>11g</Setting>
<Setting Name=”serverConnTimeout” Type=”xsd:string”>3600</Setting>
<Setting Name=”SecondaryHostPort” Type=”xsd:string”></Setting>
</Setting>

It worked as expected in DC1. When I executed in DC2, it displayed successful message but it is not updated in oam-config.xml.

Fix:

DC2 MDC cluster is write protected. To verify, open the oam-config.xml and look for the element WriteEnabledFlag as shown below.

<Setting Name=”WriteEnabledFlag” Type=”xsd:boolean”>false</Setting>
Since it is set to false, any changes made through WLST will not take effect. So run below WLST command to fix this or you can manually edit the oam-config.xml carefully.

setMultiDataCenterWrite(WriteEnabledFlag = "true")

About the Author Mahendra

I am engulfed in Oracle Identity & Access Management domain. I have expertise on providing the optimized solutions for user provisioning, web access management, Single Sign-On and federation capabilities etc., I am also well versed with complex integrations within Identity Management and other product domains. I have expertise on building demos and implementation experience on products Oracle Access Manager, Oracle Adaptive Access Manager, Oracle Entitlement Server, Oracle Virtual Directory, Oracle Internet Directory etc., Look @ my blog: http://talkidentity.blogspot.com

Leave a Comment:

19 comments

aengineer says May 15, 2014

Hi Mahendra,

Excellent post.

I have a MDC related question which I can use some help with.

Once you configure MDC, do we expect all agent definitions to be automatically updated to reference the DC2 OAM server instances either it its primary or secondary server list? If this is not automatic, do I need to add the DC2 OAM instances manually to the agent? Without doing so I fail to understand how any agent will even know about the presence of DC2.

Thanks
Aspi

Mahendra says May 15, 2014

Thank you. When you export and import partners, all your partners will be updated with Primary Server list with DC2 OAM Servers. However the max no. of connections for every partner will be updated with 8 by default after import into DC2.

Vishwajeet says October 3, 2014

OAM Multi-Data Center Failback Issues.

I have been successful in getting the initial failover from DC1 to DC2 to happen without reauthentication. This is the test I ran to do this:

1. Both DCs are up and running with one OAM server on each side.
2. Access a protected resource that has a 10g WebGate.
a. The WebGate has a primary access server of OAM DC1
b. The WebGate has a secondary access server of OAM DC2
3. Cert is chosen and I am successfully authenticated to the resource via OAM DC1.
4. I see a new session generated only on OAM DC1.
5. Shutdown OAM DC1.
6. Access same protected resource again.
7. No cert is prompted and I am able to see a new session generated on OAM DC2.
8. Start OAM DC1 back up.
9. Access same protected resource again.
10. A cert is prompted and I am forced to re-authenticate to OAM DC1.

During steps 9 and 10 when the WebGate fails back to OAM DC1 I should not have to re-authenticate. OAM DC1 should read the ObSSOCookie and OAM_ID to validate that these sites are MDC together. Why am I forced to re-authenticate in this situation? Are there any settings I may be missing that would cause this?

Sekhar says January 29, 2015

Hi Mahendra,

I am working on setting up MDC but in our case we will be using the same database by both data centers, In this case when I tried to run the steps to set up MDC it failed on clone server with database constraint violation error. How to address this situation.

Sekhar says January 29, 2015

Mahendra says January 29, 2015

Sekhar,

It would fail if you have same database. Besides that it is not recommended to have single DB for both data centers. You may try creating a separate schema for DC2 and specify it during cloning process.. However it may still have some other schemas etc., that may be common to both DC1 and DC2 schemas and cloning may fail…

Thanks
Mahendra.

Sekhar says February 24, 2015

Thank you Mahendra. We are getting a separate database server. Another question I have is regarding Load balancer set up for MDC. Do you have any guidelines on how to configure the load balancers at the cluster level and also at the global level.

cchiappe says March 4, 2015

Hi Mahendra, does EVERY property synchronize fromo DC 1 to DC 2?. I’m asking this because in my scenario the Authn Schemes and Webgates get updated but the Access Manager Settings and User Identity Stores doesn’t. Thank you!.

Mahendra says March 4, 2015

Hi ,

Settings and User stores has to be manually created in DC2. Otherwise create everything in DC1 and do the T2P cloning process so that way everything comes to DC2.

Hope this helps.

Thanks
Mahendra.

cchiappe says March 4, 2015

Ok, we’re using APS (Automated Policy Synchronization) to automatically replicate the configurations from DC1 to DC2. Have you used APS?. Thanks!

Carlos.

Mahendra says March 4, 2015

Yes, I’d used APS.

Sekhar says March 4, 2015

Hi Mahendra,

I am getting error oracle.security.jps.JpsException: JPS-09008: Application policy for application OAM11gApplication does not exist. Failed to rollback data. Reason: JPS-10000: There was an internal error in the policy store.

when running pasteconfig.sh, any idea how to address this.

Also another question i have is can i set up OAM on DC2 as if its an independent domain and then use the DC1 and DC2 to set up MDC, What i mean is do i have to use T2P to set up MDC

Mahendra says March 5, 2015

Yes Sekhar. You will have to T2P for MDC setup. Setting up MDC as independent domain is not supported.

Sekhar says March 12, 2015

Hi Mahendra,

I was successfully able to clone using T2P. After cloning I was able to start the admin server and the oam server, but I am not able to login in to OAM Console. It throws a login form but after I enter the credentials and click on the sign in button the form comes back with out any error message. IS there anything missing as part of MDC set up for me to able to login to oamconsole. It works fine in the master cluster.

Mahendra says March 12, 2015

Sekhar, do you see any errors in OAM Admin Server logs?

Are you still using Embedded LDAP as system store? If yes, are you using same password as Master to login to console?

Is weblogic console login working fine?

Sekhar says March 13, 2015

Hi Mahendra,

this is the only error message i see in the oam adminserver.log file

AdminServer MESSAGE = [ServletContext@852389476[app:oam_admin module:oamconsole path:/oamconsole spec-version:2.5 version:11.1.2.0.0]] Servlet failed with Exception
java.lang.AssertionError: Assertion violated

yes i am using Embedded LDAP as system store (which will change later). I tried with same password as master and still cannot login.

The weblogic console is working absolutely fine.

Sekhar says March 19, 2015

Hi Mahendra,

After setting up APS, i don’t see the sync not happening from master to cone. Am i missing anything here. Where does the change logs exist on master for APS settings.

Mahendra says March 20, 2015

You should see the APS calls in OAM Server logs in Clone DC.

Sekhar says March 21, 2015

Yes i do see the Replication transactions in the server logs but do not see any errors. Also if there aren’t any frequent changes Happening on master which is preferred T2P or APS. Also in APS is there a way to manually force the Sync?

Add Your Reply