On one of our cloned 11i Instance, I faced issues while trying to start Database Control (aka Enterprise Manager Console) using “emctl start dbconsole“.
This post covers issues and resolution (Issues mentioned here are specific to particular Database and Operating System Version)
.

Instance Details

Operating System : Solaris Sparc 64 Bit
Database : Oracle Database Version 10.2.0.2
Apps Version : 11.5.10.2

.
Things good to know about Enterprise Manager

Database Control : is Web-based tool to manage database. Using Database Control (OEM) you can perform administrative task like adding tablespace, table, index, managed space or view performance statistics…

Files/Directory for Database Control (DB Console) :
a) $ORACLE_HOME/sysman
b) $ORACLE_HOME/[hostname_sid]/sysman
c) $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_[hostname]_[sid]

Files useful in troubleshooting : $ORACLE_HOME/[hostname_sid]/sysman/log/

How to use Database Control

a) ORACLE_HOME/bin/emctl start dbconsole  [To start DB Control]
b) ORACLE_HOME/bin/emctl status dbconsole [To check status of DB Control]
c) ORACLE_HOME/bin/emctl stop dbconsole   [To stop DB Control]

Issue 1 : On running “emctl start dbconsole“, I received error message

OC4J Configuration issue. $OH/oc4j/j2ee/OC4J_DBConsole_$hostname_$sid not found

This means enterprise manager was not configured and Fix was to configure enterprise manager using emca (Enterprise Manger Configuration Assistant).
Steps to create enterprise manager configuration files & repository are explained in “Oracle Enterprise Manager Advanced Configuration Guide“  here

.

How to create Database Control(DB Console)

emca (Enterprise Manager Configuration Assistant) is utility to create Database Control Configuration files and Repository Objects

For 10.2.X Database I used
emca -config dbcontrol db -repos create

.

Issue 2 : While running emca, I got below error message

UnsatisfiedLinkError exception loading native library: njni10
Exception in thread “main” java.lang.UnsatisfiedLinkError: get
at oracle.net.common.NetGetEnv.get(Native Method)
at oracle.sysman. assistants. util.sqlEngine. SQLEngine. getEnvParams (SQLEngine.java:409)

as per Metalink note 553154.1, Fix was to set environment variable and relink executable

Fix 2 :
export LD_LIBRARY_PATH=$ORACLE_HOME/lib32
export LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
cd $ORACLE_HOME/network/lib
make -f ins_net_client.mk client_sharedlib
relink all

Issue 3 : While running “relink all“, there was another error

ld: fatal: library -ljava: not found
ld: fatal: library -ljvm: not found
ld: fatal: File processing errors. No output written to $OH/10.2.0/sysman/lib/libnmemso.so
*** Error code 1
make: Fatal error: Command failed for target `$OH/10.2.0/sysman/lib/libnmemso.so’
Current working directory $OH/10.2.0/sysman/lib
*** Error code 1
make: Fatal error: Command failed for target `libnmemso’
Fix 3 : As per Metalink note 743833.1, issue was because there are references of source system in make file and fix was to change references from source to target instance. (DB was cloned as part of 11i Cloning steps)

cp $ORACLE_HOME/sysman/lib/env_sysman.mk $ORACLE_HOME/sysman/lib/env_sysman.mk.bak

change
JRE_LIB_DIR=$SOURCE_ORACLE_HOME/10.2.0/jdk/jre/lib/sparc  (It was pointing to source Directory)
to
JRE_LIB_DIR=$TARGET_ORACLE_HOME/10.2.0/jdk/jre/lib/sparc

and relink again using “relink all”
.

Issue 4 : Just when I was expecting all issues to be resolved by now I got another (different) error message while  running “relink all”

ld: warning: file $OH/10.2.0/lib/libclntsh.so.10.1: wrong ELF class: ELFCLASS64
Undefined                       first referenced
symbol                             in file
ztch                              $OH/10.2.0/lib32//libnmemso.so

ld: fatal: Symbol referencing errors. No output written to $OH/10.2.0/sysman/lib/emagent
*** Error code 1
make: Fatal error: Command failed for target `$OH/10.2.0/sysman/lib/emagent’
Current working directory $OH/10.2.0/sysman/lib
*** Error code 1
make: Fatal error: Command failed for target `emagent’
ld libclntsh.so.10.1 wrong ELF class ELFCLASS64

Fix 4 : another note from Metalink 395731.1, and fix was to create soft link for file “ldflags” like

ln -s $ORACLE_HOME/lib/ldflags $ORACLE_HOME/lib32/ldflags

After creating softlink, I did manage to relink all executable in ORACLE_HOME successfully and it was time to run emca for enterprise manager configuration

“emca -config dbcontrol db -repos create”

.

Issue 5 : While running above emca command, scripting was prompting for dbsnmp password again and again stating

“Invalid username/password” (Even though I could connect to dbsnmp/ dbsnmp_passsword from sqlplus)

Error message 5 : On checking emca logs at  $OH/cfgtoollogs/emca/ emca_$timestamp.log

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available

.
Fix 6 : It was time to search in metalink again and this time I got note # 337260.1 according to this note issue was because of user profile (password limit and resource limit)

SQL> select u.username, u.profile, p.resource_name, p.limit
from dba_profiles p, dba_users u
where p.profile=u.profile
and u.username in (‘DBSNMP’)
and p.resource_type = ‘PASSWORD’
order by u.username, p.resource_name;

Profile for user dbsnmp was set according to metalink 337260.1 which means issue was something else, on doing some hit & trial I realized I was using SID in uppercase where as in database listener.ora SID was in lower case.

When I enetered SID name in lowercase, emca script accepted dbsnmp password and finally after 3 hours I managed to install/configure enterprise manager console on cloned target 11i Database Instance.

30-Oct-2008 16:51:05 oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://machinename:<port_number>/em <<<<<<<<<<<
Enterprise Manager configuration completed successfully
FINISHED EMCA at 30-Oct-2008 16:51:05

—
More on how to monitor database using database control (dbconsole) coming soon …

References

• 278100.1  How To Drop, Create And Recreate DB Control In A 10g Database
• 395162.1  How to manage DB Control 10.2 for RAC Database with emca
• 743833.1  Rapid Clone & Em Dbconsole
• 549079.1  Troubleshooting Database Control Startup Issues
• 553154.1  “UnsatisfiedLinkError: njni10″ When Tring to Create the dbconsole With EMCA
• 337260.1  Dbsnmp Password Not Accepted
• Enterprise Manager Advanced Configuration Guide