OBIEE 11g by default uses its own authentication engine against users stored in WebLogic’s Embedded LDAP Server.
Oracle Access Manager (OAM) is Single Sign-On (SSO) solution from Oracle and there are two versions of OAM i.e. 10g and 11g. More here on differences between OAM 10g and 11g .

OBIEE can be configured for Single Sign-On (OAM) with user repository is LDAP Server (OID or AD). This post assumes that User Repository for OBIEE is OID. If you wish to use AD as user repository then replace OID with AD while using this post. To know more about Oracle Access Manager 11g, check my book at amazon

High Level Steps to integrate OBIEE with OAM (for Single Sign-On)

1. Install OBIEE 11g (11.1.1.5)

2. Install OHS 11g (install 11.1.1.2 and then apply patch 11.1.1.5)

3. Configure access to OBIEE via OHS (mod_wl_ohs)

4. Install OID 11g (install 11.1.1.2 and then apply patch 11.1.1.5)

5. Integrate OBIEE 11g with OID for user repository

6. Install OAM 11g (11.1.1.3 or 11.1.1.5)

7. Integrate OAM 11g with OID for user repository

8. Create Instance of WebGate in OAM 11g

9. Install WebGate with OHS 11g (installed in step 2)

10. Configure OAM Identity asserter as authentication provider in Weblogic Domain hosting OBIEE

11. Configure Response (header variable OAM_REMOTE_USER) in protected authenticated and authorisation policy

12. Enable SSO in OBIEE (including logon URL, Logoff URL) using FMW Enterprise Manager Control

13. Test OBIEE Single Sign-On configuration

Steps

1. Install OBIEE 11g  - Follow OBIEE installation steps here, here and here.
This step will create middleware home (MW_HOME), OBIEE Oracle Home (ORACLE_HOME) and OBIEE Oracle Instance (ORACLE_INSTANCE)

2. Install OHS 11g - You can access OBIEE directly by using managed server port. In SSO environment, request to OBIEE managed servers should come via this HTTP Server. A Policy Enforcement Point (PEP) which is WebGate in this case is configured that will communicate to Oracle Access Manager (SSO server).

This step will create OHS Oracle Home (ORACLE_HOME) and OHS Oracle Instance (ORACLE_INSTANCE)

Note: ORACLE_HOME of OBIEE and OHS must be installed in different directory, similarly ORACLE_INSTANCE of OBIEE and OHS must be installed in different directory.

• Follow OHS Installation guide here   (Do not select WebCache during configuration)

3. Configure access to OBIEE via OHS (mod_wl_ohs)

Configure mod_wl_ohs in OHS to forward request to OBIEE Managed Server (bi_server1) and restart OHS. Test if you can access OBIEE via OHS Server . More on mod_wl_ohs here and here

http://http_server:http_listen_port/analytics

4. Install OID 11g - Enterprise Users that are going to access OBIEE via SSO (OAM in this case) will be stored in OID. Follow OID installation steps here, here, here, and here (DIP, OVD and OIF are optional components and are not required for this integration).

This step will create middleware home for OID, OID Oracle Home (ORACLE_HOME) and OID Oracle Instance (ORACLE_INSTANCE).

Note: You can install OBIEE and OID in same Middleware Home but this is not recommended.

ORACLE_HOME for OID, OBIEE, and OHS must be in different directory.
ORACLE_INSTANCE for OID, OBIEE, and OHS must be in different directory.

5. Integrate OBIEE 11g with OID for user repository. - By default OBIEE 11g authenticates against WebLogic’s embedded LDAP server using Default Authentication Provider. You must add additional Authentication Provider of type OID in WebLogic Security Realm (steps here and here  ) so that OBIEE/WebLogic can authenticate users against OID.

6. Install OAM 11g - If you are installing same version of OID and OAM i.e. 11.1.1.1.5 then these two (OID and OAM) can be installed in same Middleware Home (MW). This step will create OAM Oracle Home (ORACLE_HOME).

Note: You can install OID and OAM in different Middleware Home then this step will also create Middleware Home (MW_HOME) for OAM.

Note: ORACLE_HOME for OAM, OID, OBIEE, and OHS must be in different directory.

Note: OAM comes with additional components like OIM, OAAM, OIN and these components are optional.

7. Integrate OAM 11g with OID for user repository - By default OAM 11g uses WebLogic’s embedded LDAP server as its Identity/User Store. You must add additional identity store in OAM to point to same OID which is configured with OBIEE.  Screen to configure Identity Store in OAM 11.1.1.3 and 1.1.1.5 are slightly different. For 11.1.1.3 click here  and for OAM 11.1.1.5 click here

More on OAM integration with OID in my book here

8. Create Instance of WebGate in OAM 11g - WebGate is web server plug-in which intercepts user request and communicates to OAM Server. Both 10g WebGate and 11g WebGate can be used with OAM 11g . (If you are using Webgate 11g then all requests are protected by default and you should un-protect any public page.). To create WebGate11g  instance you can use RREG

Note: Ensure that /analytics , /analytics/…/* ,  /xmlpserver , and  /xmlpserver/…/* are protected resource in OAM

9. Install WebGate with OHS 11g (installed in step 2)

10. Configure OAM Identity asserter as provider in Weblogic Domain hosting OBIEE - Configure Identity Asserter for OAM as explained here

11. Configure Response (to return OAM_REMOTE_USER as header variable $user.userid ) in protected authenticated and authorisation policy in OAM.

12. Enable SSO in OBIEE (including logon URL, Logoff URL pointing to OAM server) using FMW Enterprise Manager Control as shown here

13. Test OBIEE Single Sign-On (SSO) integration using OAM - Access OBIEE url via HTTP Server, that should redirect user to OAM login page. After username/password it should take user straight to OBIEE URL.

Note: Steps mentioned in 9.2 of OBIEE Enterprise Deployment Guide are for OAM 10g and should not be used with 11g OAM.