One of the QA machine OEM13c was not in healthy status, when i tried to start the OMS and found the below errors.
2022-10-09 16:17:11,965 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>This Exception occurred at Sun Oct 09 16:17:11 CDT 2022.
2022-10-09 16:17:11,966 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Access to domain 'GCDomain' for user 'nodemanager' denied]. Please check Node Manager log for details.
2022-10-09 16:17:11,967 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Node Manager is not up, sleeping for five seconds...
2022-10-09 16:17:17,008 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Connecting to Node Manager ...
2022-10-09 16:17:17,264 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>This Exception occurred at Sun Oct 09 16:17:17 CDT 2022.
2022-10-09 16:17:17,265 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Access to domain 'GCDomain' for user 'nodemanager' denied]. Please check Node Manager log for details.
2022-10-09 16:17:17,266 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Node Manager is not up, sleeping for five seconds...
2022-10-09 16:17:22,280 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Connecting to Node Manager ...
2022-10-09 16:17:22,503 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>This Exception occurred at Sun Oct 09 16:17:22 CDT 2022.
2022-10-09 16:17:22,503 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Access to domain 'GCDomain' for user 'nodemanager' denied]. Please check Node Manager log for details.
2022-10-09 16:17:22,504 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Node Manager is not up, sleeping for five seconds...
2022-10-09 16:17:27,517 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Connecting to Node Manager ...
2022-10-09 16:17:27,796 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>This Exception occurred at Sun Oct 09 16:17:27 CDT 2022.
2022-10-09 16:17:27,797 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Access to domain 'GCDomain' for user 'nodemanager' denied]. Please check Node Manager log for details.
2022-10-09 16:17:27,797 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Node Manager is not up, sleeping for five seconds...
2022-10-09 16:17:32,806 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Connecting to Node Manager ...
2022-10-09 16:17:33,255 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>This Exception occurred at Sun Oct 09 16:17:33 CDT 2022.
2022-10-09 16:17:33,256 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Access to domain 'GCDomain' for user 'nodemanager' denied]. Please check Node Manager log for details.
2022-10-09 16:17:33,262 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Node Manager is not up, sleeping for five seconds...
2022-10-09 16:17:38,269 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Not connected to Node Manager
2022-10-09 16:17:38,273 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Failed to connect to node manager after starting it
2022-10-09 16:17:38,274 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Failed to start node manager
2022-10-09 16:17:38,274 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>Exception: exceptions.SystemExit value:
2022-10-09 16:17:38,274 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>
2022-10-09 16:17:38,275 [Thread-6] INFO commands.BaseCommand run.605 - <OUT>_END_
2022-10-09 16:17:38,275 [Thread-7] INFO commands.BaseCommand run.605 - <ERR>_END_
2022-10-09 16:17:38,364 [main] INFO commands.BaseCommand logAndPrint.653 - WebTier Successfully Started
2022-10-09 16:17:38,389 [main] WARN commands.BIPCommands processStatusBIP.340 - Connection exception when trying to hit BIP page. BIP is down
2022-10-09 16:17:38,390 [main] ERROR commands.BaseCommand logAndPrint.651 - Node Manager Could Not Be Started
Check Node Manager log file for details.
2022-10-09 16:17:38,479 [main] WARN commands.BIPCommands processStatusBIP.340 - Connection exception when trying to hit BIP page. BIP is down
2022-10-09 16:17:38,481 [main] INFO commands.BaseCommand logAndPrint.653 - BI Publisher Server is Down
2022-10-09 16:17:55,850 [main] INFO wls.OMSController main.220 - Executing emctl command : start
2022-10-09 16:17:55,861 [main] ERROR wls.OMSController main.253 - OMSController failed for start oms
2022-10-09 16:17:55,863 [main] ERROR wls.OMSController main.254 - OMSController Error: Unknown option: -h. Re-check syntax.
java.lang.Exception: Unknown option: -h. Re-check syntax.
In short webtier was started, BI Publisher server is down and failed to start the OMS. Stopped all the OMS components.
[oracle@oemhost:oemqa] /home/oracle > emctl stop oms -all -force
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Node Manager Not Running
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher Server is Down
[oracle@oemhost:oemqa] /home/oracle > Tried to start the Node Manager
[oracle@oemhost:oemqa] /home/oracle > /opt/oracle/emc13c/gc_inst/user_projects/domains/GCDomain/bin/startWebLogic.sh
.
.
JAVA Memory arguments: -Xms256m -Xmx512m
.
CLASSPATH=/opt/oracle/emc13c/Middleware/bi/bifoundation/jdbc/jdk16/bijdbc.jar:/opt/oracle/emc13c/Middleware/oracle_common/modules/features/com.oracle.db.jdbc7-dms.jar:/opt/oracle/emc13c/Middleware/oracle_common/jdk/lib/tools.jar:/opt/oracle/emc13c/Middleware/wlserver/server/lib/weblogic_sp.jar:/opt/oracle/emc13c/Middleware/wlserver/server/lib/weblogic.jar:/opt/oracle/emc13c/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/opt/oracle/emc13c/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar:/opt/oracle/emc13c/Middleware/wlserver/../oracle_common/modules/glassfish.jaxp_1.4.5.0.jar:/opt/oracle/emc13c/Middleware/oracle_common/modules/internal/features/jrf_wlsFmw_oracle.jrf.wls.classpath_12.1.3.jar:/opt/oracle/emc13c/Middleware/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.1.0.0.jar::/opt/oracle/emc13c/Middleware/wlserver/common/derby/lib/derbynet.jar:/opt/oracle/emc13c/Middleware/wlserver/common/derby/lib/derbyclient.jar:/opt/oracle/emc13c/Middleware/wlserver/common/derby/lib/derby.jar:/opt/oracle/emc13c/Middleware/wlserver/server/lib/xqrl.jar
.
PATH=/opt/oracle/emc13c/Middleware/wlserver/server/bin:/opt/oracle/emc13c/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/opt/oracle/emc13c/Middleware/oracle_common/jdk/jre/bin:/opt/oracle/emc13c/Middleware/oracle_common/jdk/bin:/usr/bin:/usr/local/bin:/etc:/usr/sbin:/usr/ucb:/home/oracle/bin:/usr/bin/X11:/sbin:.:/opt/oracle/emc13c/Middleware/bin:/opt/oracle/product/12.1.0/dbhome_2/OPatch
.
***************************************************
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console *
***************************************************
starting weblogic with Java version:
java version "1.7.0"
Java(TM) SE Runtime Environment (build pap6470sr9fp50-20160720_02(SR9fp50))
IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 20160630_309948 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR9_20160630_1817_B309948
JIT - tr.r11_20160630_120374
GC - R26_Java726_SR9_20160630_1817_B309948_CMPRSS
J9CL - 20160630_309948)
JCL - 20160719_01 based on Oracle jdk7u111-b13
Starting WLS with line:
-- Access to node Manager
For node manager details, check /opt/oracle/emc13c/Middleware/install/setupinfo.txt
Use the following URL to access:
1. Enterprise Manager Cloud Control URL: https://oemhost.localdomain:7799/em
2. Admin Server URL: https://oemhost.localdomain:7101/console
3. BI Publisher URL: https://oemhost.localdomain:9803/xmlpserver
https://oemhost.localdomain:7101/console
Secure the Admin and Node Manager password
oracle@oemhost:oemqa] /domains/GCDomain > emctl secure create_admin_creds_wallet -admin_pwd Oracle123** -nodemgr_pwd Oracle123**
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Creating Admin Credentials Wallet...
Created Admin Credentials Wallet successfully
Run this command on all OMS's in your environment
[oracle@oemhost:oemqa] /domains/GCDomain >Now start the OMS
[oracle@oemhost:oemqa] /domains/GCDomain > emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
[oracle@oemhost:oemqa] /domains/GCDomain >Start the Agent and check the status of the Agent
[oracle@oemhost:oemqa] /home/oracle > emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Starting agent ................................................ started.
[oracle@oemhost:oemqa] /home/oracle > emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.2.0.0.0
OMS Version : 13.2.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /opt/oracle/emagent//agent_inst
Agent Log Directory : /opt/oracle/emagent//agent_inst/sysman/log
Agent Binaries : /opt/oracle/emagent/agent_13.2.0.0.0
Core JAR Location : /opt/oracle/emagent/agent_13.2.0.0.0/jlib
Agent Process ID : 37487022
Parent Process ID : 17957098
Agent URL : https://oemhost.localdomain:3872/emd/main/
Local Agent URL in NAT : https://oemhost.localdomain:3872/emd/main/
Repository URL : https://oemhost.localdomain:1159/empbs/upload
Started at : 2022-10-09 17:16:46
Started by user : oracle
Operating System : AIX version 7.1 (ppc64)
Number of Targets : 41
Last Reload : (none)
Last successful upload : 2022-10-09 17:17:31
Last attempted upload : 2022-10-09 17:17:31
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 32
Size of XML files pending upload(MB) : 0.04
Available disk space on upload filesystem : 15.14%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2022-10-09 17:17:26
Last successful heartbeat to OMS : 2022-10-09 17:17:26
Next scheduled heartbeat to OMS : 2022-10-09 17:18:29
---------------------------------------------------------------
Agent is Running and Ready
[oracle@oemhost:oemqa] /home/oracle >Now start the OMS, after that all the OEM components were started without any issues.
[oracle@oemhost:oemqa] /domains/GCDomain > emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
[oracle@oemhost:oemqa] /domains/GCDomain >View the OMS status Details
[oracle@oemhost:oemqa] /domains/GCDomain > emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : oemhost.localdomain
HTTP Console Port : 7788
HTTPS Console Port : 7799
HTTP Upload Port : 4889
HTTPS Upload Port : 1159
EM Instance Home : /opt/oracle/emc13c/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /opt/oracle/emc13c/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oemhost.localdomain:7799/em
Upload URL: https://oemhost.localdomain:1159/empbs/upload
WLS Domain Information
Domain Name : GCDomain
Admin Server Host : oemhost.localdomain
Admin Server HTTPS Port: 7101
Admin Server is RUNNING
Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oemhost.localdomain
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up
BI Publisher HTTP Managed Server Port : 9701
BI Publisher HTTPS Managed Server Port : 9803
BI Publisher HTTP OHS Port : 9788
BI Publisher HTTPS OHS Port : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://oemhost.localdomain:9851/xmlpserver
BI Publisher Server Logs: /opt/oracle/emc13c/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log : /opt/oracle/emc13c/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log
[oracle@oemhost:oemqa] /domains/GCDomain >
