Monday, February 22, 2010 time out while waiting for a managed process to start Issue We deployed third

party service in oracle 10giAS 10.1.3.0. While starting opmn s ervices we noticed given below error on $ prompt. opmnctl: starting opmn and all managed processes... ================================================================================ opmn id= eos123:4201 1 of 2 processes started.

ias-instance id=ias_1013.eos123.orbit.org ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ias-component/process-type/process-set: default_group/ORBIT/default_group/

Error --> Process (index=1,uid=1201922234,pid=63455) time out while waiting for a managed process to start Log: /usr01/oracle/10giAS1013/opmn/logs/default_group~ORBIT ~default_group~1.log

Solution 1. Make timeout change in $IAS_ORACLE_HOME/opmn/conf/opmn.xml file <start timeout="1200"/> 2. Reload opmn services $IAS_ORACLE_HOME/opmn/bin/opmnctl reload 3. Validate issue Posted by Jay at 2/22/2010 09:09:00 AM 0 comments Friday, February 5, 2010 ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose] I had to restart oracle applications R12 database (10.2.0.3) and application ser vices. While stopping database services I noticed given below error

Error SQL> shutdown immediate Database closed. Database dismounted. ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [] , [], [], [], [], []

Resolution This error can be ignored. This problem is fixed in 11.1.0.6 release.

Reference Metalink Id : 365103.1 Posted by Jay at 2/05/2010 08:04:00 PM 0 comments Unable to Start Output Post Processor Concurrent Manager

After starting concurrent manager I noticed "Output Post Processor" did not star t. I tried to start "Output Post Processor" using "Administrator Concurrent Mana ger" screen but it failed with given error in manager log file

Error Unable to initialize state monitor. oracle.apps.fnd.cp.gsm.GenCartCommException: ORA-01403: no data found ORA-06512: at "APPS.FND_CP_GSM_IPC", line 499 ORA-06512: at line 1

at oracle.apps.fnd.cp.gsm.GenCartComm.initService(GenCartComm.java:233) at oracle.apps.fnd.cp.gsm.GenCartComm.<init>(GenCartComm.java:80) at oracle.apps.fnd.cp.gsf.GSMStateMonitor.init(GSMStateMonitor.java:74) at oracle.apps.fnd.cp.gsf.GSMStateMonitor.<init>(GSMStateMonitor.java:62) at oracle.apps.fnd.cp.gsf.GSMServiceController.init(GSMServiceController.jav a:111) at oracle.apps.fnd.cp.gsf.GSMServiceController.<init>(GSMServiceController.j ava:66) at oracle.apps.fnd.cp.gsf.GSMServiceController.main(GSMServiceController.jav

a:428)

Resolution I notice "Service Manager" was down. So I restarted "Service Manager" and then r estarted "Output Post Processor". Concurrent manager started successfully.

Posted by Jay at 2/05/2010 07:57:00 PM 0 comments Unable to Start Conflict Resolution Manager I had to restart application services to resolve Self Service connection issue. Concurrent manager was not starting. Strange thing was $APPLCSF/$APPLLOG/$TWO_TA SK_<mm><dd>.mgr (Internal Manager) log was not showing getting updated. After co uple of hour of struggle we noticed file system was full by 100%. Some time we m iss very small thing to solve such issues. After starting concurrent manager I n oticed Concurrent Resolution Manager (FNDCRM) was not coming up. I performed following check. 1. Lots of requests where getting assigned to "Concurrent Resolution Manager" so I had to clear queue 2. There was on incompatibility associated to either Standard concurrent mana ger or Conflict Resolution Manager' 3. I also executed cmclean.sql script (from oracle) and restarted concurrent but it did not helped. 4. Finally I relinked FNDCRM concurrent manager executable and restarted conc urrent manger. It resolved issue. Source application environment adrelink.sh force=y "fnd FNDCRM"

Posted by Jay at 2/05/2010 07:49:00 PM 0 comments The Distributed Caching System failed to initialize on port Today was terrible day in office I had to face lots of issue for critical instan ces. All of sudden it was noticed that one of R12 instance was irresponsive. I w as asked to jump to issue. I was asked not to restart concurrent manager. I was only suppose to bound applications (web server). In order to resolve issue I per formed following. 1. Stopped Web services in following order Sourced application environment $ADMIN_SCRIPTS_HOME/adformsrvctl.sh stop

$ADMIN_SCRIPTS_HOME /jtffmctl.sh stop $ADMIN_SCRIPTS_HOME /adoafmctl.sh stop $ADMIN_SCRIPTS_HOME /adoacorectl.sh stop $ADMIN_SCRIPTS_HOME /adapcctl.sh stop $ADMIN_SCRIPTS_HOME/adopmnctl.sh stop $ADMIN_SCRIPTS_HOME/mwactl.sh -login <sysadmin password> stop_force <MWAPo rt> 2. Started Services Sourced application enviroment $ADMIN_SCRIPTS_HOME/adopmnctl.sh start $ADMIN_SCRIPTS_HOME/adapcctl.sh start $ADMIN_SCRIPTS_HOME/adoacorectl.sh start $ADMIN_SCRIPTS_HOME/adoafmctl.sh start $ADMIN_SCRIPTS_HOME/jtffmctl.sh start ${SCRIPT_DIR}/mwactl.sh start $ADMIN_SCRIPTS_HOME/adopmnctl.sh status $ADMIN_SCRIPTS_HOME/adformsrvctl.sh start Even after restarting services did not came up. I noticed given below errors for "s_java_object_cache_port" context variaable (from $CONTEXT_FILE) port. $LOG_HOME/logs/ora/10.1.3/opmn/oafm_default_group_1/oafmstd.err 10/02/05 11:47:38 Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found i n application.xml. 10/02/05 11:47:38 Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found in application.xml. 10/02/05 11:48:06 Exception in static block of jtf.cache.CacheManager. Stack tra ce is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa tion failed. The Distributed Caching System failed to initialize on port: s_java _object_cache_port. The list of hosts in the distributed caching system is: <ipa ddress>. The port <port> should be free on each host running the JVMs. at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335) at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)

at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:768) $LOG_HOME /logs/ora/10.1.3/opmn/oacore_default_group_2/oacorestd.err Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be boots trapped since corresponding module declaration was not found in application.xml. 10/02/05 11:47:32 Exception in static block of jtf.cache.CacheManager. Stack tra ce is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa tion failed. The Distributed Caching System failed to initialize on port: s_java _object_cache_port. The list of hosts in the distributed caching system is: <IPA DDRESS>. The po rt s_java_object_cache_port should be free on each host running the JVMs. at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335) at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451) $LOG_HOME /logs/ora/10.1.3/opmn/oacore_default_group_3/oacorestd.err Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be boots trapped since corresponding module declaration was not found in application.xml. 10/02/05 11:46:49 Exception in static block of jtf.cache.CacheManager. Stack tra ce is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa tion failed. The Distributed Caching System failed to initialize on port: s_java _object_cache_port. The list of hosts in the distributed caching system is: <IPA ddress> The po rt <port> should be free on each host running the JVMs. at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335) at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451) at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:768) at oracle.apps.jtf.cache.CacheManager.<clinit>(CacheManager.java:390)

Solution I had to restart oracle applications Rel 12 web services along with concurrent m anager. I performed following. 1. 2. 3. 4. Source environmennt Stoped all web services running on middle tier Stopped concurrent manager Started web services along with concurrent manager

5. I also checked $LOG_HOME /logs/ora/10.1.3/opmn/*-default_group_?/oa*std.er r file and did not noticed above errors

Posted by Jay at 2/05/2010 07:22:00 PM 1 comments Friday, January 29, 2010 An error occurred while attempting to establish an Application File Server I had to restart oracle applications 12i services. I started database services, listener, middle tier and concurrent tier services. Concurrent manager started f ine. While accessing completed requests log or output file I noticed given below error. Issue An error occurred while attempting to establish an Application File Server conne ction with the node <ConcurrentTierServer> . There may be a network configuratio n problem, or the TNS listener on node <ConcurrentTierServer> may not be running . Please contact your system administration. Troubleshooting 1. I insured there was no sqlplus connectivity issue from Middle Tier, Concur rent Manager Tier to Database. 2. Validated tns lister running on concurrent tier $ps ef grep tnslsnr grep APPS_<TWO_TASK>

APPS_$TWO_TASK process must be running 3. Relinked all fnd executable ($FND_TOP/bin ) using adadmin Solution Above did not helped resolving issue. I had to perform following to resolve issu e. 1. Relinked FNDFS, FNDSM executable using adrelink Source Application Enviroment

$. INST_TOP/ora/10.1.3/$ APPLFENV adrelink.sh force=y "ad FNDSM" adrelink.sh force=y "ad FNDFM" 2. Restarted Concurrent Manage and apps listener serices 3. Login to OES as sysadmin Concurrent Request Search for some request and try to see log or output fi le. You should be able to view log file