Document Display

How to Troubleshoot Failed Login Attempts to DB Control [ID 404820.1]

Modified: May 22, 2012 Type: HOWTO Status: PUBLISHED Priority: 3

Page 1 of 10

In this Document Goal Fix References

Applies to:
Enterprise Manager for Oracle Database - Version 10.2.0.1 to 11.2.0.3 [Release 10.2 to 11.2] Information in this document applies to any platform.

Goal
The purpose of this Note is to fast-track the debugging process with regard to the inability to connect or login to DB web-browser.

Fix
The following checks can be undertaken:

1. If failing to connect to DB Console via remote connection to the DB Console server, are you able to login to DB Console using a web-browser that is running on the actual DB Console Server? If so then the issue is at the remote workstation (n connectivity to the DB Console Server, firewall etc). Preferably you should reference the DB Console server using a Fully Qualified Hostname in the web-browser URL.

1.0 If a firewall is being used, ensure that the EM Port (default non-secure port 5500 or default secure port 1158) is open communication in the firewall e.g.

[oracle@abc ~]$ telnet xyz.oracle.com 5500 Trying <IP Address for xyz.oracle.com>... Connected to xyz.oracle.com (<IP Address for xyz.oracle.com>). Escape character is '^]'.
Note: The above example shows the output from a successful telnet. Should telnet be disabled & ssh communication enabled then the following command can be used instead:

ssh -X xyz.oracle.com -p 5500

Note: A successful connect to the port using SSH will result in no output otherwise if not successful the following error wi

ssh: connect to host xyz port 1158: Connection refused

To eliminate the potential for a port conflict then the following command can also be used to determine if the taken by another process:

netstat -an|grep LISTEN

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 2 of 10

Note: In linux the netstat command can report the pid of the owning process. On Solaris the command "pfiles" can be u track down a running process that is hanging onto a port.

1.1 Are you able to ping the DB Console server from your machine? If not then get in contact with your network admin to this networking issue.

1.2 If you are able to successfully ping the DB Console server from your machine then you need to determine the order o between the "hosts" file & DNS. To do this in a UNIX environment then issue the following command on your machine:

grep -i hosts /etc/nsswitch.conf

If the result has "files" before "dns" as follows: "hosts: files dns"

then your "/etc/hosts" file is taking precedence over DNS. Sometimes machines are configured in this way to minimized t overhead on the network. You need to ensure that your "/etc/hosts" file is setup correctly with a reference to the DB Con server. Refer to Note.203280.1 Ext/Pub How to Configure the hosts File for 9iAS 9.0.2 Installations. The contents of this note do not only apply to 9iAS 9.0.2 and are as follows: Verify that /etc/hosts.* has the following format:

IP_ADDRESS FULLY_QUALIFIED_HOSTNAME SHORT_HOSTNAME ALIASES The following example shows a properly configured /etc/hosts.* file: <ip address> <hostname.domain> <hostname> <alias>

1.3 If on the other hand, you can ping the DB Console server from your machine but the returned value from issuing the hosts /etc/nsswitch.conf" command has the "dns" ahead of "hosts" then DNS is taking precedence. You then need to ens an nslookup on the DB Console Hostname returns the DB Console host's correct IP address & (b) an nslookup on IP address (Reverse NSLOOKUP) returns the fully qualified name of the DB Console host. If this is not the case then plea with you network admin.

A good area to check for issues with connectivity & login to DB Console are the following files found on the DB Console se

-> $ORACLE_HOME/<hostname>_<sid>/sysman/log/emoms.*

Login failures in DB Console will be logged in the emoms.* files. 2. Are you using the correct port in your URL reference to the DB Console? There are 2 ways to check that you are using the correct port. They are as follows:

2.1 Go to the $ORACLE_HOME/install directory & take a look at the portlist.ini file. The portlist.ini will contain the port assignments. Look for the "Enterprise Manager Console HTTP Port (ORACLE_SID) = < Number>" entry. Default DB Console port being 5500 (unsecured) whilst 1158 is the default for DB Console if it has been So the URL to access DB Console should look like the following: http://<DB_Console_Host>:5500/em or https://<DB_Console_Host>:1158/em i.e. http(s)://<DB_Console_Host>:<Port Number>/em

2.2 Go to the $ORACLE_HOME/hostname_sid/sysman/config directory & look for the following entries in the emoms.prop

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 3 of 10

oracle.sysman.emSDK.svlt.ConsoleServerPort=<DBConsole_Port_Number>

oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort=<DBConsole_Port_Number>

2.3 Run "emctl status dbconsole" - this returns the url that the console is using: eg. D:\10204\BIN>emctl status dbconsole Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0 Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved. https://myhost.oracle.com:1158/em/console/aboutApplication Oracle Enterprise Manager 10g is not running. 3. Is DB Console up & running? From the $ORACLE_HOME/bin directory run
emctl status dbconsole

to ensure that the DB Console is up & running. You should receive the following message if it is: "Oracle Enterprise Manager 10g is running." Additionally the following check can be made to see if the DB Console is up & running:

On Linux & other Unix platforms use: ps -ef|grep -i dbconsole On the Solaris Platform use: /usr/ucb/ps -ex -awww |grep -i dbconsole |cut -c1-130

If the DB Console is not running then issue the

emctl start dbconsole

command from the same directory. If any errors are reported in the starting up of the DB Console then please consult the log & trace file found in the $ORACLE_HOME/hostname_sid/sysman/log/ directory. In particular the emoms.* & emagent.* files. 4. Ensure that the Database is up & running as follows: From the Database host issue:

sqlplus "/ as sysdba" SQL> startup immediate;

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 4 of 10

OR issue from the command prompt:

ps -ef |grep $ORACLE_SID

4.1 Ensure that the Database is not in the MOUNT state which means connections for normal users will fail with the error ERROR: ORA-01033: ORACLE initialization or shutdown in progress

To Correct this: Connect as 'sys as sysdba' user and open the database and verify that the database is in 'READ WRITE' mode for all user

SQL> alter database open;

SQL> select name, OPEN_MODE from v$database;

NAME OPEN_MODE --------- --------<DB_NAME> READ WRITE

5. Ensure that the Listener is running & listening for the Database by issuing the following command:

lsnrctl status

You will see a result similar to the following output if it is up & running:

Service "ORACLE_SID" has 2 instance(s). Instance "ORACLE_SID", status UNKNOWN, has 1 handler(s) for this service... Instance "ORACLE_SID", status READY, has 1 handler(s) for this service...

5.1 Ensure that the listener does not reference any symbolic links (UNIX environment). If it does then copy across the ac i.e. tnsnames.ora & listener.ora to their expected locations, removing the symbolic links. As the DB Console agent is unab resolve the references from symbolic links.

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 5 of 10

6. Ensure that the SYSMAN', 'DBSNMP','MGMT_VIEW' accounts are not locked as follows: 6.1 From sqlplus, login as SYS & run the following command:

select username, account_status from dba_users where username in ('SYSMAN', 'DBSNMP','MGMT_VIEW');

Take note of the accounts that are locked. 6.2 Stop the DB Console by running the following command:

$ORACLE_HOME/bin/emctl stop dbconsole

6.3 Unlock the Accounts that are locked by running the following command:

alter user <username> account unlock;

6.4 Start the DB Console once more by running.

$ORACLE_HOME/bin/emctl start dbconsole

7. Check to see whether the SYSMAN or DBSNMP user passwords have recently changed. If so, find out what the new pa use it to login to DB Console as the SYSMAN user. Then remap the new DBSNMP password into the "Monitoring Credentia Agent that is monitoring the Database if the DBSNMP password has also changed. Refer to Note 259387.1

7.1 Ensure that the OEM_MONITOR privilege exists & has been granted to the SYSMAN user account or any other Admin level account that has been created for the purpose of monitoring & administering the Database. Refer to Note.152593.1 Grant OEM_MONITOR Role to Database Preferred Credentials:

8. Other connection Related Problems with DB Console --------------------------------------------------[Problem:] ORA-1017 ERROR when logging into DB Console. Problems with the credentials, when trying to connect with sys as sysdba locally there is no problem, but when trying to c remotely an ORA-1017 ERROR is experienced. [Answer:] This normally has to do with the remote_login_passwordfile parameter, needing to be set to EXCLUSIVE. Refer to Note.204166.1 Connecting To Database From EM to db target as SYSDBA Fails With ORA-1031

[Problem:] Agent sends message "Failed to connect to database instance: ORA-12537: TNS:connection closed (DBD ERR OCIServerAttach)." though everything seems to be fine. [Answer:] The ONS may be interfering with effective listener operation as described in : Note.284602.1 10g Listener: High CPU Utilization - Listener May Hang

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display
Explicitly disable the 10g TNS Listener from subscribing to the ONS. Edit the active 10g listener.ora file and add the parameter below:

Page 6 of 10

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

where <listener_name> would be replaced with the actual listener name configured in the listener.ora file. For example, if the listener name is LISTENER (default), the parameter would be:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

then restart the listener This error message should disappear. [Problem:] when connecting to DB Console as SYSDBA in a Windows Environment. It returns insuffient privleges ORA [Answer:] 1. Include the following to the sqlnet.ora

SQLNET.AUTHENTICATION_SERVICES= (NTS)

2. Ensure the the OS user has the "Logon as a Batch Job" privilege

[Problem:] New basic installation Of Enterprise Edition Database Control in Windows. The installation is completed. Can lo database using SQL plus, but can not access the database control. It display a blank page. [Answer:] Ensure the appropriate services have been started & set to auto-startup as follows: To autostart make the following changes to the registry.

HKEY_LOCAL_MACHINE->SOFTWARE->ORACLE->KEY_<ORACLE_SID>_home1->

ORA_<ORACLE_SID>_AUTOSTART=TRUE

ORA_<ORACLE_SID>_SHUTDOWN=TRUE

ORA_<ORACLE_SID>_SHUTDOWN_TIMEOUT=90

ORA_<ORACLE_SID>_SHUTDOWNTYPE=immediate

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 7 of 10

Also in "Services" ensure that the "OracleService<ORACLE_SID>" is set to autostartup. As well as the TNSLISTENER & D services. [Problem:] Attempt to start the dbconsole with emctl start dbconsole generates the following message:

OC4J Configuration issue.

$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid> not found.

[Answer:] When upgrading the database from a previous version, the emca is not ran thus the dbconsole is not configure Depending on the existing configuration different approaches are done:

If you upgraded the database from a previous version(9i), then you must drop the existing SYSMAN repository and create one using emca. If you manually create a database and no SYSMAN reporistory was created, run the emca

If you used the DBCA and created the scripts with the Database Control option, the repository was created, you will need

emca -r

Refer to the following notes: Note.456111.1 Troubleshooting DbConsole Error - OC4J Configuration issue Note.297363.1 Oc4j Configuration Issue Starting Em Dbconsole Oc4j 'CONFIG ORA-00922 missing or invalid option':

[Problem:] Migrated a single instance DB from host abc123 to RAC hostname def456. The services DB is up fine, into Enterprise Control Manager. It references the wrong connection string. When logging in, it attempts to connect to ab instead of def456. It seems to be acquiring this string from the DB it, how to modify it so that it picks up host def456? [Answer:] Drop and recreate the DB CONTROL with the correct hostname as follows:

A. Login to SQLPLUS as user SYS or SYSTEM, and drop the sysman account & mangement objects:

-> Clean up the Db Console repository a. drop user sysman cascade; b. drop role MGMT_USER; c. drop user MGMT_VIEW cascade;

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 8 of 10

d. drop public synonym MGMT_TARGET_BLACKOUTS; e. drop public synonym SETEMVIEWUSERCONTEXT; f. issue a "commit;"

B. Export the correct values for the ORACLE_HOME and ORACLE_SID environment variab

C. Change directories to the $ORACLE_HOME/bin directory

D. Cleanup the External DB Console Configuration files by issuing: emca -deconfig dbcontrol db -repos drop

E. Recreate the DB Console Repository & external Configuration files by issuing: emca -config dbcontrol db -repos create

9. As a "Catch All", the DB Console can be recreated. Please keep in mind that *RE-CREATING THE DB CONSOLE WILL W ALL SCHEDULED JOBS*. But it will keep all the existing job definitions in-tact. This is a known issue for Database Control version 10.2.0.2. If you are fine with the possibility of having all your job schedules wiped out then you may the steps to re-create the DB Console configuration files. If a re-build of the *DB Console configuration files ONLY* was in resolving the issue, if you can afford to lose the existing Repository information then also rebuild the Repository in DB Console Configuration files. SCENARIO 1: Delete the DB Control using scripts:

A) To delete only the configuration files: In 10g R1 run: <ORACLE_HOME>/bin/emca -x <sid> In 10g R2 run: <ORACLE_HOME>bin/emca -deconfig dbcontrol db

B) To delete only the DB Control repository: In both 10g R1 and R2 run: <ORACLE_HOME>/sysman/emdrep/bin/RepManager <hostname> <listener_port> <sid>

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 9 of 10

C) To delete both the configuration files and the repository: In 10g R1 do:

<ORACLE_HOME>/bin/emca -x <sid>

<ORACLE_HOME>/sysman/emdrep/bin/RepManager <hostname> <listener_port> <sid> In 10g R2 do:

<ORACLE_HOME>/bin/emca -deconfig dbcontrol db -repos drop

SCENARIO 2: Manually delete the DB Control: The procedure is the same for 10g R1 and R2 A) To delete the configuration files: - remove the following directories from your filesystem:

<ORACLE_HOME>/<hostname>_<sid>

<ORACLE_HOME>/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>

SCENARIO 3: Create the DB Control

A) To create only the DB Control configuration files, skipping the repository creation (this would be done for instance if you dropped only the files and left the repository in place): In 10g R1 do: <ORACLE_HOME>/bin/emca -r

In 10g R2 do: <ORACLE_HOME>/bin/emca -config dbcontrol db

B) To create the DB Control configuration files and repository: In 10g R1 do:

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012

Document Display

Page 10 of 10

<ORACLE_HOME>/bin/emca In 10g R2 do <ORACLE_HOME>bin/emca -config dbcontrol db -repos create

You can use the steps above to recreate the DB Control (configuration files and repository). There are 2 typical scenarios - you only recreate the DB Control configuration files (i.e. if you suspect that the repository is intact) - you recreate the configuration files as well as the repository (i.e. if you suspect that the repository may be NOTE: In 10g R2 you can use the following comand to recreate (drop and create) the DB Control configuration files and <ORACLE_HOME>/bin/emca -config dbcontrol db -repos recreate

For additional information on emca commandline options, please see the Oracle Enterprise Manager 10g Advanced Config Guide. 10g R1: http://download.oracle.com/docs/pdf/B12013_03.pdf 10g R2: http://download.oracle.com/docs/cd/B16240_01/doc/em.102/e10954.pdf

10. A piece of advice: Please include the backup of the $ORACLE_HOME/hostname_sid/sysman/config/*.* files in your ba strategy for DB Console esp. the emoms.properties file. AIX Based Systems

NOTE:332805.1 - Database Control failing on AIX5L system with DISPLAY set incorrectly - unable to access Enterprise Ma website. NOTE:399403.1 - Db Control Hangs When Shutting Down

References

NOTE:259387.1 - How to Change DBSNMP Password in Database 10g and 11g Monitored by DB Control @NOTE:272971.1 - How to: Manually Kill the DBconsole if emctl stop Fails NOTE:273758.1 - How to Configure a 10g Database When DB Control Shows the Status Of Unavailable ? NOTE:274661.1 - Cannot Start DBCONSOLE on DHCP Machine after Reboot NOTE:276950.1 - How to setup Enterprise Manager Database Control for SSL (HTTPS) secure connection NOTE:278100.1 - How To Drop, Create And Recreate the Database Control (DB Control) Release 10g and 11g NOTE:309595.1 - Attempting to use EM 10g Java Console gets error: Procedure Entry Point Kpuhhalo Could Not Be Locat Dynamic Link Lib Oci.Dll NOTE:330130.1 - Overview Of The EMCA Commands Available for DB Control 10.2 Installations NOTE:335571.1 - Enterprise Manager Database Control only starts on one node of my RAC cluster. How do I start it on a node? NOTE:383085.1 - Problem: Running EMCA Fails With "Could not complete the configuration" When DBSNMP Password is Specified at During DB Control Creation NOTE:393727.1 - Failed To Connect To The Database: Io Exception NOTE:395162.1 - How to manage DB Control 10.2 for RAC Database with emca NOTE:398499.1 - EM DB Console Failing To Start NOTE:399751.1 - Cannot Logon To Oem Console After Changing IP Address NOTE:400786.1 - Unable To Access Dbconsole After Server Reboot. Manually Starting Fails with "OC4J Configuration issu NOTE:456111.1 - Troubleshooting DbConsole Error - OC4J Configuration issue

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_adf.ctrl-state=6bw2hunet_61 11/13/2012