gcp4al19:gr417a$ ps -ef | grep introscope gr417a 1944 32527 0 06:26 pts/9 00:00:00 grep introscope beaadmin 4314 19818

12 04:34 ? 00:14:14 /opt/jdk1.6.0_22/jre/bin/java -Dweblogic.Name=edbms1905 Djava.security.policy=null -Dweblogic.management.server=http://gcp4al27a00.cif.att.com:8004 Djava.library.path=/opt/jdk1.6.0_22/jre/lib/i386/server:/opt/jdk1.6.0_22/jre/lib/i386:/opt/jdk1.6.0_22/jre/../lib/i386::/op t/bea103/wlserver/server/native/linux/i686:/opt/bea103/wlserver/server/native/linux/i686/oci920_8:/usr/java/packages/li b/i386:/lib:/usr/lib Djava.class.path=/opt/edb/lib/ojdbc14.jar:/opt/bea103/patch_wlw1030/profiles/default/sys_manifest_classpath/weblogic_ patch.jar:/opt/bea103/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/bea103/patch_cie6 60/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/jdk6/lib/tools.jar:/opt/bea103/wlserver/server/lib/web logic_sp.jar:/opt/bea103/wlserver/server/lib/weblogic.jar:/opt/bea103/modules/features/weblogic.server.modules_10.3.0. 0.jar:/opt/bea103/wlserver/server/lib/webservices.jar:/opt/bea103/modules/org.apache.ant_1.6.5/lib/antall.jar:/opt/bea103/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/antcontrib.jar:/opt/bea103/wlserver/server/lib/xqrl.jar:/opt/edb/mserver/edbms1905/conf:/opt/edb/app/wilyAgent/wily/Web AppSupport.jar Dweblogic.system.BootIdentityFile=/opt/edb/mserver/edbms1905/servers/edbms1905/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true Dweblogic.ReverseDNSAllowed=false -Djava.compiler=NONE -Xms2048m -Xmx2048m -XX:CompileThreshold=8000 XX:PermSize=256m -XX:MaxPermSize=256m -verbose:gc -Xloggc:logs/gc.log -XX:+PrintGCDetails XX:+PrintGCTimeStamps -Dplatform.home=/opt/bea103/wlserver -Dwls.home=/opt/bea103/wlserver/server Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dweblogic.Name=edbms1905 Dweblogic.Stdout=/opt/edb/mserver/edbms1905/logs/weblogic.server.log Dweblogic.Stderr=/opt/edb/mserver/edbms1905/logs/weblogic.server.log -Dconf.dir=/opt/edb/mserver/edbms1905 Dapp_root_dir=/opt/edb/app DSERVICELOGGER_CONF_FILE_PATH=/opt/edb/mserver/edbms1905/conf/serviceloggerxml.xml Dweblogic.deployment.HeartbeatPeriodSeconds=600 -Dweblogic.console.systemstatus.defaultMinimized=true Dweblogic.classloader.noJarSigners=true -Dweblogic.http.client.defaultReadTimeout=30000 -DUseSunHttpHandler=true javaagent:/opt/edb/app/wilyAgent/wily/Agent.jar Dcom.wily.introscope.agentProfile=/opt/edb/app/wilyAgent/wily/IntroscopeAgent.profile weblogic.Server beaadmin 8236 19818 5 04:42 ? 00:05:27 /opt/jdk1.6.0_22/jre/bin/java -Dweblogic.Name=edbms1904 Djava.security.policy=null -Dweblogic.management.server=http://gcp4al27a00.cif.att.com:8004 Djava.library.path=/opt/jdk1.6.0_22/jre/lib/i386/server:/opt/jdk1.6.0_22/jre/lib/i386:/opt/jdk1.6.0_22/jre/../lib/i386::/op t/bea103/wlserver/server/native/linux/i686:/opt/bea103/wlserver/server/native/linux/i686/oci920_8:/usr/java/packages/li b/i386:/lib:/usr/lib Djava.class.path=/opt/bea103/patch_wlw1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/bea103/p atch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/bea103/patch_cie660/profiles/default/sys_ manifest_classpath/weblogic_patch.jar:/opt/jdk6/lib/tools.jar:/opt/bea103/wlserver/server/lib/weblogic_sp.jar:/opt/bea10 3/wlserver/server/lib/weblogic.jar:/opt/bea103/modules/features/weblogic.server.modules_10.3.0.0.jar:/opt/bea103/wlse rver/server/lib/webservices.jar:/opt/bea103/modules/org.apache.ant_1.6.5/lib/antall.jar:/opt/bea103/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/antcontrib.jar:/opt/bea103/wlserver/server/lib/xqrl.jar:/opt/edb/mserver/edbms1904/conf:/opt/edb/app/wilyAgent/wily/Web AppSupport.jar Dweblogic.system.BootIdentityFile=/opt/edb/mserver/edbms1904/servers/edbms1904/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true Dweblogic.ReverseDNSAllowed=false -Djava.compiler=NONE -Xms1024m -Xmx1024m -XX:CompileThreshold=8000 XX:PermSize=256m -XX:MaxPermSize=256m -verbose:gc -Xloggc:logs/gc.log -XX:+PrintGCDetails XX:+PrintGCTimeStamps -Dplatform.home=/opt/bea103/wlserver -Dwls.home=/opt/bea103/wlserver/server Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dweblogic.Name=edbms1904 Dweblogic.Stdout=/opt/edb/mserver/edbms1904/logs/weblogic.server.log Dweblogic.Stderr=/opt/edb/mserver/edbms1904/logs/weblogic.server.log -Dconf.dir=/opt/edb/mserver/edbms1904 Dapp_root_dir=/opt/edb/app DSERVICELOGGER_CONF_FILE_PATH=/opt/edb/mserver/edbms1904/conf/serviceloggerxml.xml Dweblogic.deployment.HeartbeatPeriodSeconds=600 -Dweblogic.console.systemstatus.defaultMinimized=true Dweblogic.classloader.noJarSigners=true -Dweblogic.http.client.defaultReadTimeout=30000 javaagent:/opt/edb/app/wilyAgent/wily/Agent.jar Dcom.wily.introscope.agentProfile=/opt/edb/app/wilyAgent/wily/IntroscopeAgent.profile weblogic.Server gcp4al19:gr417a$

use the Introscope Workstation to view and manipulate data that is stored by the Enterprise Manager. As a managed application runs, probes relay collected data to the agent. The agent then collects and summarizes the data and sends it to the Enterprise Manager.3 The time displayed on data viewers now includes the time zone (for example, PST for Pacific Standard Time) on the machine hosting the Enterprise Manager. Workstation locates where the computer locates for user, Epagent, Agent monitors the server. Introscope agent installs on our server and then for weblogic,websphere we use introscope in start up class. We have to give t the introscope enterprise manager as host at starting and port to get the workstation.

click Start > Introscope > Introscope Workstation, or run Introscope Workstation.exe. For Java Web Start deployments of the Workstation, open a browser to: http://EMhost:8081/workstation

In a URL, the same combination would be referred to like this: http://<localhost>:8081/workstation?page=investigator&agent=SuperDomain|localhost|WebLogic|WebLogic%20 Agent
In the login dialog, enter: the Host name or IP address Note You can use the IP address instead of the host name only if both your client machine and the host machine support the same IP protocol. the Port number your User Name your Password Tip The Workstation will remember the last five login attempts, so if you have entered host and user information previously, the Host, Port and User ID fields will be drop-downs from which you can select the credentials you want to use.

About the Workstation Console

The Console is the default view when you start the Workstation, and contains dashboards that show performance data in graphical views. Dashboards are basic tools for viewing management data in Introscope.

The Introscope Sample Management Module provides a set of sample dashboards. Authorized users can create custom dashboards using the Dashboard Editorsee the Introscope Configuration and Administration Guide for more information. You can have more than one Console window open at the same time. To open a new Console window, select Workstation > New Console.

About the Workstation Investigator

You use the Investigator to view application and system status, to search, and to browse metric data using a tree structure. Select Workstation > New Investigator.

This is java

This is .net app The Investigator contains these sections: The Investigator tree provides information about each host and application managed by the Enterprise Manager. The metrics that appear in the Investigator tree are a function of the resources your applications use and the data that your Introscope agents are configured to report. The Viewer pane on the right side of the Investigator presents details, often graphical, for the resource or metric in the tree. You can select View tabs to open different views of data. The tabs that are

available vary, depending on the item selected in the tree. For some views, options might be available in the bottom section of the Viewer pane to control the data displayed in the Viewer.

About the Management Module Editor

You use the Management Module Editor to create or edit a Management Module, which contains a set of Introscope monitoring configuration information. Management Modules are listed for each domain, and contain objects, known as elements, that contain and organize data with monitoring logic alerts, actions, and dashboards.

The Dashboard Editor provides tools for creating and laying out Data Viewers, shapes, lines, text boxes, and connectors. Users with appropriate permissions can create and edit dashboards and dashboard objects such as imported images, shapes, lines, and textsee the Introscope Configuration and Administration Guide for more information.

In management module we can keep the barchart, GraphicEqualizer,String viewer etc we can change this by right clicking in management module.

In Introscope, Workstation users are assigned user permissions. Each Workstation user is assigned a user name, password, and certain permissions. Permissions are granted at the Domain and Enterprise level.

User preferences
You use Introscope user preferences to specify: a home dashboard whether to display Management Module names alongside dashboard names in

the Console, low-threshold execution-time warnings for Transaction Tracer. To change your home dashboard: 1 Select Workstation > User Preferences.

Click Choose, enter a search string to narrow the selection, and select from the remaining list.

You can use the same name for dashboards that are in different Management Modules, and use the same name for Management Modules that are in different Domains. 1 Select Workstation > User Preferences. 2 Check Show Module and Domain name with Dashboard name. 3 Click Apply. The Management Module and domain that contain the dashboard appear after the dashboard name. To turn off the warnings about low-threshold execution time: 1 Select Workstation > User Preferences. 2 Click the Transaction Tracer tab. 3 Check the Dont warn when threshold is less than 1 second checkbox.

Navigating among dashboards in the Console

Dashboard drop-down list
You can select dashboards from the drop-down list at the top of the Console page. You can type all or part of the dashboard name, to narrow the selections in the list.

Automatic hyperlinksIntroscope automatically links a Data Viewer to the metric grouping it is based uponthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor. Similarly, dashboards that contain Data Viewers based on the same metric grouping are automatically linked, and you can navigate between them using the Links menu. We can done its in management module- dashboard-right click and link to. Custom hyperlinksYou can define custom links for dashboard items, to link to other dashboards or to web pages. You can define custom links if you have dashboard editing permission.

See a list of available links Select a dashboard object and select Properties > Links. Right-click the dashboard object and select Links from the context menu.

1 Click the graph in the Console to select it. 2 Show the minimum and maximum values in one of two ways: Right-click the Data Viewer and select Show Minimum and Maximum. Select Properties menu, and select Show Minimum and Maximum. To open a tool tip: Mouse over any element in the Workstation metrics tree or in a Data Viewer, such as a point on a graph. The illustration below displays information about a particular data point in the graph, showing: Metric name Exact value of the metric Min/max values for the metric across the period represented by the data point. (See How time range affects data points, below.) The count of 15-second intervals represented by the data point. (See How time range affects data points, below.) The date and time for the data point in the graph. Moving metrics to front/back in graph

When a graph contains multiple metrics, it is possible for data points to overlay each other. You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics.

To view the scale of a graph chart: Click on a chart to select it, and then: Select Viewer > Scale Options, or Right-click the chart and select Scale Options from the context menu. The Data Options dialog box opens.

To force minimum and maximum values: 1 Click on a chart to select it. 2 Select Viewer > Scale Options. 3 Select Pin at on both the Minimum and Maximum sides of the dialog, and enter a value for the minimum and maximum points of the data access. 4 Click OK. For example, if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000, it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data:

1 Click on a chart to select it. 2 Select Viewer > Scale Options. 3 Choose Auto Expand on both the Minimum and Maximum side of the dialog. 4 Click OK. 2 Select Viewer > Scale Options. 3 Select AutoScale on both the Minimum and Maximum sides of the dialog. The resulting charts data axis is reset based on the data in the chart, as shown in the illustration below. This often results in sharper valleys and peaks in the graph display:

To copy a Data Viewer to the clipboard: 1 Open a Console and select a Data Viewer 2 Select Viewer > Copy to Clipboard as Image. To export data from a Data Viewer: 1 In the Console, select a Data Viewer. 2 Select Viewer > Export Data. You can view live data in the Console, or select a range of time to view historical data. The default view of data is Live:

To view historical data: 1 Select the metric or dashboard for which you want to see historical data. 2 Select a time range for the historical view from the Time Range drop-down menu.

Now you can select a resolution to adjust the granularity of the view, by increasing or decreasing the number of data points that appear. Each pre-defined time range is associated with a default resolution. You normally do not need to change this. Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default. For 8 mins its 15 seconds, 1 day its 12 mins seems the data its showing spike in that time only.

Drag the slider on the time bar to change the time range: Click the arrows to move backward and forward in time: The single arrows move backward or forward in small increments; the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range. Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu.

Use the calendar controls to select the start and end dates and times.

When you view historical data in a graph, you can zoom in on data by clicking the mouse pointer on a graph position and dragging, to specify the time range:

Set the global time range and the Time Range control to match the zoomed view: select Viewer > Set Time Range From Zoomed Range, or click the Set Time Range from Zoomed Range icon . Lock your selected resolution by clicking the Lock icon . This maintains your selected resolution as you select different time ranges by zooming in on data. Hold down the shift key while you zoom, to constrain zooming to the time axis. You use the Console Lens to filter metric data for the agents that are reporting data. In a dashboard that shows data for more than one agent, you can use the Console Lens to view data only for selected agents. Click the Lens button or select Dashboard > Lens.

If you select and click apply it would displays only that data in the console and dont show for other agents or process. Again make it to all by clicking remove.
In the Select Agent dialog box, select a single agent, or select multiple agents (click and drag, or CTRL/click) on which to filter. Click Apply or press Enter. The dashboard refreshes to show only data for the selected agent(s). The Lens button shows a black arrow: To clear the Console Lens: 1 Click Lens. 2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box.

Workstation Investigator
The Investigator contains two main panes: Investigator TreeThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into. Metrics are organized hierarchically by host, process, agent, and resource type. Selecting items in the tree causes predefined views to be presented in the Viewer pane. Inactive metrics appear grayed out in the tree. ViewerThe Viewer pane on the right side of the Investigator presents details, often graphical, for the resource or metric currently selected in the tree. Depending on the item selected in the tree, tabs at the top of the Viewer pane allow you to select one or more views, including the General, Overview, Errors, Traces, and Search views. For some views, options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer.

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected. Metrics are organized in a Host|Process|Agent hierarchy. The nodes immediately under the SuperDomain node are virtual and physical hosts. Custom Metric Host (Virtual)This node does not correspond to a physical host machine. It is a virtual host that contains metrics that are not reported by a specific, individual agent. For example, if you have configured calculators that create custom metrics, or have configured aggregated agents, they typically appear under the Custom Metric Host. HostsOne node for each machine that hosts an agent. Each host node contains a process node for the instance of the application being monitored, which in turn contains an agent node. The agent node contains nodes that correspond to application and system resources, which contain metrics. The SuperDomain is that which includes all user-defined domains and agents. The Enterprise Manager administrator can set up the EM to display child domains with separate permissions. This illustration shows two child domains, myDomain1 and myDomain2, listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process.

Supportability metrics
Supportability metrics give information about the state of the Enterprise Manager and the machine it runs on. You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager. The Introscope Sizing Guide contains extensive information about the supportability metrics. To change Data Viewer type: 1 Select Properties > View As. 2 Select the viewer display type to change to.

Top ten views

When you select certain resources in the Investigator, the General tab of the Viewer pane shows the top ten matching metrics for the selected resource. Java resources include servlets, JSP, EJBs, and JDBC; for .NET, resources include ASP.NET, ADO.NET, and serviced components. These metrics appear in a bar chart in the Investigator Preview pane.

Select Workstation > New Investigator. General tab When you select a metric, the General tab shows a graphic view of the metric either for live data, or for a selected historical period. See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view. For some nodes in the tree, the General tab shows the path to that node object in the Investigator hierarchy. For example, when the Frontends node is selected, the General tab shows this path:

Application Overview
The Application Overview is available when you select an agent in the Investigator tree, and enables application monitoring and triage. It shows highlevel health indicators, and a log of related events and historical metric information. The Overview shows a row of lights for each application managed by the currently selected agent. Introscope presents this data for each application it discovers when a servlet executes, Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application. After the application starts, the Overview tab automatically updates to display a row of lights for it.

This illustration shows one application -- its name is truncated to .CatalogServletCont... -- managed by this agent. For this application, you can view alerts showing the state of:
Double-clicking one of the alert indicators under the User column links to the URL metrics for the selected application.

User Indicates how satisfactory the end-users interactions with the application are likely to be. Satisfaction is a function of response time, waits, stalls, and errors. Greennormal, satisfactory user interactions with the application. Yellowan attempt to use the application is likely to yield

unsatisfactory results, for instance poor response time or errors. Redindicates a serious availability issue and that an attempt to use the application will probably fail. VM Indicates the health and availability of server resources, such as resource pools and CPU. Greennormal health of server resources. Yellowresource limitations or outages Redserious resource limitations or outages. Backend Summary Indicates the worst health and availability across all backends accessed by the application. For example, if one of three backends has a serious resource limitation or outage, the All Backends light is red. The purpose of the All Backends light is to allow th e user, with minimal scrolling, to quickly assess whether any of the backends have problems that require investigation. Greennormal backend health and availability across all backends accessed by the application. Yellowat least one backend accessed by the application is experiencing errors or stalls, or poorer than expected response times. Redat least one backend accessed by the

Using alerts to drill down for more data You can double-click an alert from the overview tab to display the underlying data for that application tier. For example, if you double-click the User alert, the Workstation will display the URLs node for that agent.

If you click VM in the topmost screen it shows where the error is , click the red button so it goes there.

Application Overview metrics for historical mode

Over a historical range, an alert color reflects the worst-case value of the heuristic at any point in the historical range. For example, if at any time during a historical range the User heuristic for an agent was yellow, but never red, the Overview tab for that historical range is yellow.

Whats Interesting events

The lower half of the Overview lists Whats Interesting events, which Introscope generates automatically when the color of an alert changes to yellow or red. In Live mode, the previous 20 minutes of events appear.

If you double click the error in overview then it shows the error message in intresting events.

Shows the time at wt time it changes. And also the message.

EM overview

You can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent:

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans, Session beans, and Message Driven beans:

Frontend overviews
Overviews for Frontend nodes show graphed application metrics, and statistics related to transactions in the application:

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node:

The garbage collection (GC) heap overview shows heap use:

The Instance Counts overview tab shows the classes instantiated on the JVM.

The JTA overview tab data about JTA components:

The LeakHunter overview shows statistics graphically and in a table. Leak tabs appear for nodes under LeakHunter, and show details of the leak and a graph of the number of collections over time:

The Servlet overview shows a table of servlets in the node. When you select a servlet, the Investigator shows its statistics in a graph:

The socket overview shows tables for client and server sockets, and socket information for each port:

The Struts Overview tab shows an overview of Struts components, with a display of the average response time for all components.

The Threads overview shows all active threads being processed through an agent:

The Overview tab for the XML node displays metrics for XML components.

Search tab
The Search tab is available when you select a node in the Investigator tree that contains metrics. It enables you to quickly find metrics. Traces tab The Traces tab, available when a resource or component is selected in the Investigator tree,

Errors tab The Errors tab, available when a resource or component is selected in the Investigator tree, lists errors and error details for the selected item.

Metric Count tab

Many of the nodes in the Investigator tree now have a new Metric Count tab, which displays a pie chart of the metric distribution for the node.

Frontend metrics
These metrics are listed for each frontend: Average Response Time (ms) Concurrent Invocations Errors Per Interval Responses Per Interval The Heuristics node shows the metric values related to the alerts displayed in the Overview tab, when an agent is selected. The URLs node under a front end node shows these metrics for each URL group that is configured for the front end:

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group: Viewing metrics for Backends in the Investigator For each database backend, these metrics reflect the activity and performance of the backend across all applications it serves:

To change an alert view: Display an alert in the Investigator Preview pane and select Properties > AlertView . Alert messages are triggered by an action associated with an alert status. These alerts appear automatically. You can also view alert messages by selecting Workstation > Show Alert Messages.

Dashboard alerts that show overall status

Each sample dashboard includes alerts that show the overall state of the environment, and how key performance indicators are affecting the environment:

The Overview dashboard

The Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment.

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem.

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction, enabling authorized users to trace the transaction activity at the component level. Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability:
WebLogic Server 8.0 and later WebSphere 6.x.

To start a Transaction Trace session: 1 Select Workstation > New Transaction Trace Session. The New Transaction Trace Session window opens.

In the Trace transactions section of the window, specify the threshold execution time. Select milliseconds or seconds from the drop-down list. To stop a Transaction Trace session: Click Stop, or Select Trace > Stop Tracing Session. To restart a Transaction Trace session: Click Restart, or Select Trace > Restart Tracing Session. To turn off the warnings about low-threshold execution time: 1 Select Workstation > User Preferences. 2 Click the Transaction Tracer tab. 3 Check the Dont warn when threshold is less than 1 second checkbox. 4 Click Apply. 1 Select Trace > Show Traced Agents. The Tracing Agent(s) dialog box appears.

Using the Transaction Trace Viewer

The Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session.

Summary View shows metrics for the components in the selected transaction. Metrics include the path, number of calls, the length of the call in milliseconds, and the minimum, average, and maximum call times.

Trace View shows the selected transaction in graphical format:

1 Select Workstation > Query Historical Events. The Historical Query Viewer opens. 2 In the Query field, enter a combination of: the query option type: to include all transaction events that match the specified type. a query stringto search for errors that contain or match a string. If you dont enter a query string, all errors events are returned. query optionsto limit your search based on event parameters,

type:errorsnapshot

type:whatsinteresting

type:normal

type:sampled

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24.
We would like to extract the error details for particular application(agent).I tried with C:\Program Files\CA Wily\Introscope8.0.2.0\lib>java -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstation.jar get historical events matching (type:errorsnapshot) and (agent:wb1)and time:[2010062712 TO 2010062718]. For these things we create xml file and that stores through trace(open through xml) that works fine.

Querying for similar events

In Introscope you can query for events that are similar to a selected event. For example, similar events might be events that all contain the same components (Servlet > EJB > SQL) with varying response times. Introscope considers events similar if 60% of the strings within them (component names, SQL tables names, and so forth) overlap. To save Transaction Trace data to an XML file: 1 In the Transaction Trace Viewer, select the Transaction Traces to save: CTRL + click to select multiple Transaction Traces. Edit > Select All to select all Transaction Traces in the window. 2 Click Save As... 3 You can open the file now, or select a location to save the file into, enter a filename, and click Save. To open saved Transaction Trace data in an XML file: 1 Select Workstation > Query Historical Events 2 Select Trace > Open Saved Events (XML). 3 Select the XML file from the browser window, and click Open. 1 In the Transaction Trace Viewer, select the Transaction Traces to export: CTRL + click to select multiple Transaction Traces Edit > Select All to select all Transaction Traces in the window. 2 Select Trace > Export.

Manage Module editor: A Management Module contains a set of Introscope monitoring configuration information. You configure Introscope's monitoring logic by using Management Modules and elements, which organize metric data in the Workstation. Management Modules for each domain contain elements. Elements are objects that contain and organize metric data with monitoring logic, for presentation in the Workstation. Introscope elements are:

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command, E-mail, and Workstation actions) Dashboards

Calculators Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope. This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic. You must create other Management Modules for other created domain. Triage Map Configuration Management Module:Another useful Management Module is the Triage Map Configuration Management Module. This Management Module saves definitions of alerts and other objects you create directly from the application triage map. You can also use this management module to do the following:Create alert downtime schedules for triage map alerts.

Define alert actions for use in triage map alerts.

To create a new metric grouping from an existing metric: 1. Right-click the metric and select New Metric Grouping from Metric <Name> from the menu. 2. Accept the default name for the metric grouping. 3. Choose a Management Module to contain the metric grouping in one of these ways: o Select a Management Module from the drop-down list box. o Click Choose, select a Management Module from the list, then click Choose again. 4. Click OK. The new metric grouping you created is highlighted in the Management Module Editor tree, under the Management Module in which you saved it.Note: The metric grouping is active when it is created, and cannot be de-activated. In the Settings pane for the metric grouping, metric grouping Agent Expressions and Metric Expressions fields already contain the metric information. Note: Filtering based on agent name uses the property introscope.agent.perfmon.agentExpression. A valid agent expression would be: ProcessName|AgentName or *|MyAgent

5. Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field. This field should contain no more than 64 KB of data. After it is applied, it will be persisted in the management module jar. 6. Select which Agent Expressions to use: o Select Use Management Module Agent Expressions to use the Agent Expressions defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for this metric grouping 7. Click Apply. To create a new metric grouping from the Elements menu: 1. 2. 3. 4. From any Workstation window, select Workstation > Management Module Editor. In the Management Module Editor window, select Elements > New Metric Grouping. In the Name field, enter a name for the metric grouping. Choose a Management Module to contain the metric grouping in one of these ways: o Select a Management Module from the drop-down list box. o Click Choose, select a Management Module from the list, then click Choose again. 5. Click OK.

To create a new Management Module: 1. From any Workstation window, select Workstation > New Management Module Editor. 2. In the Management Module Editor window, select Elements > New Management Module. 3. Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree.) 4. Enter a .jar file name for the Management Module, using alphanumeric characters without spaces (to comply with all operating systems). 5. In the Domain Name field, use the pull-down menu to select which domain contains the Management Module. 6. Click OK. The Management Module appears in the Management Module Editor tree. Modules are active and editable when they are created.

In the Investigator, click and hold a metric in the tree. Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard. The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard.

Creating an empty data viewer

To create an empty data viewer: 1. In the Console, open the dashboard to edit by selecting Dashboard > Edit dashboard. 2. Select a dashboard object creation tool from the Tools Palette. 3. Click and drag the location and size of the empty data viewer on the dashboard area. 4. Release the mouse button. The data viewer appears on the dashboard as drawn. You can now manipulate the data viewer. 5. Select File > Save to save dashboard changes.

Adding data to a data viewer by dragging and dropping

You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor. You can use this function to: Add data to an empty data viewer.
Replace the data displayed in existing data viewer.

To add data to a data viewer by dragging and dropping:

1. In the Console, open the dashboard to edit by selecting Dashboard > Edit dashboard.
2. Select the data to add to the dashboard:
To add a metric to the dashboard, open an Investigator window and position it so both the Investigator window

and dashboard are visible, then click and hold a metric in the tree.
To add an element to the dashboard, open the Management Module Editor, and click and hold an element in the

tree.

Creating report templates

In the Management Module Editor, select Elements > New Report Template. Note The New Report Template menu item is disabled if you do not have writepermission. The New Report Template dialog box opens.

a Type the Name for the new report template. b Select Force Uniqueness to ensure that the report name is unique. If you select this option and you then enter a name that is not unique, Introscope adds a number to the name to make it unique. Note The appended number appears after the report template is created,when you view it in the Management Module Editor. If you dont select Force Uniqueness and an identical report template name exists,Introscope displays an error message and does not create the report. c Select a Management Module from the drop-down list box to choose the Management Module that will contain the report. d Optional: Instead of selecting an existing Management Module to contain the report, click Choose, then click New Management Module and assign a name to the new Management Module. The illustration below shows these options.

For more information about creating Management Modules, see the Introscope Administration and Configuration Guide. The new report template is added to the Management Module Editor, and the settings pane opens.

In the settings pane, select the Active check box if you are ready to activate the report template. Click Open Template Editor to define report data.

Adding report elements to reports 1 If the report template editor is not already open, open it: a With the Management Module Editor open, select the report in the pane on the left. b Click Open Template Editor. 2 Right-click the Report listed in the upper left pane, and choose Add. A list of available elements appears.

A calendar dialog appears, with the current date (Today) circled. Use the calendar dialog to set the date, and edit the clock time in the text field after the dialog is closed.

Management modules application Metric groups . here we can check all the things.

So here we are putting the thresholds and if it cross its sending tickets and mails.

We can change the mail names.

To run the Introscope Enterprise Manager control scripts on UNIX: Use the following commands at the bin directory: ./EMCtrl.sh startThis starts the Enterprise Manager.
./EMCtrl.sh stopThis stops the Enterprise Manager.

 ./EMCtrl.sh statusThis gives the status of the Enterprise Manager. ./EMCtrl.sh helpThis shows the supported options for the script.

Gurusekhar(2:40:21 AM): Do you know the introscope process name in systems ? Som(2:42:27 AM): it runs as a startup class on managed server process Som(2:42:32 AM): no seperate process

Introscope process starts along with the weblogic servers, its add as one argument to that
241 ##################### INTROSCOPE CONFIGURATION #################### 242 WILY_HOME=/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily 243 WILY_BOOTPATH=-javaagent:/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily/Agent.jar 244 WILY_OPTIONS=Dcom.wily.introscope.agentProfile=/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily/In troscopeAgent.profile 245 WILY_CLASSPATH=/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily/WebAppSupport.jar:/opt/BEA/weblogic 10.3/wlserver_10.3/wilyAgent/wily/Agent.jar 246 CLASSPATH="${CLASSPATH}:${WILY_CLASSPATH}" 247 arguments="$arguments Dcom.wily.introscope.agentProfile=/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgen t/wily/IntroscopeAgent.profile" 248 arguments="$arguments javaagent:/opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily/Agent.jar" 249 ##################### INTROSCOPE CONFIGURATION End #################### su $PRE_USER -c "umask $UMASK;export LD_LIBRARY_PATH=$LD_LIBRARY_PATH;${JAVA} $arguments ${JAVA_OPTIONS} weblogic.Server 1>>$stdout_file 2>>$stderr_file &" $ cd /opt/BEA/weblogic10.3/wlserver_10.3 $ ls -lrt total 88190 drwxr-xr-x 2 root other 96 Sep 1 2010 L10N drwxr-xr-x 9 root other 1024 Sep 1 2010 common drwxr-xr-x 11 root other 1024 Sep 1 2010 server drwxr-xr-x 3 root other 96 Sep 1 2010 inventory drwxr-xr-x 2 root other 96 Sep 1 2010 uninstall drwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent -rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgent.tar drwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope -rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176.sh

$ pwd /opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily/logs
$ view IntroscopeAgent.20131104-230606.log
-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 20:35 IntroscopeAgent.20131104-213326.log -rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 20:35 IntroscopeAgent.20131104-213320.log -rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 20:41 IntroscopeAgent.wlsesmgt__wlsesmgt1.log -rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 20:41 IntroscopeAgent.wlmsgctr__wlmsgctr1.log -rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 20:41 IntroscopeAgent.wllogin__wllogin1.log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 20:41 IntroscopeAgent.wlprfadm__wlprfadm1.log -rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 20:41 IntroscopeAgent.wllogin__wllogin2.log -rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 20:41 IntroscopeAgent.wlprfadm__wlprfadm0.log -rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 20:41 IntroscopeAgent.wllogin__wllogin0.log -rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 20:41 IntroscopeAgent.wlmsgctr__wlmsgctr0.log -rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 20:41 IntroscopeAgent.wlslfreg__wlslfreg0.log -rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 20:41 IntroscopeAgent.wlsesmgt__wlsesmgt0.log -rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 20:41 IntroscopeAgent.wlslfreg__wlslfreg1.log -rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp18.log -rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp17.log -rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp14.log -rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp15.log -rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 20:45 IntroscopeAgent.common_Common_cluster_dynco11.log -rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp19.log -rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp12.log -rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 20:45 IntroscopeAgent.common_Common_cluster_dynco13.log -rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp13.log -rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 20:45 IntroscopeAgent.common_Common_cluster_dynco12.log -rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp16.log -rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 20:45 IntroscopeAgent.cp_cp_cluster_dyncp11.log -rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 20:48 IntroscopeAgent.WebLogic_Agent.log -rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 21:05 IntroscopeAgent.20131104-220350.log -rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 21:36 IntroscopeAgent.20131104-223504.log -rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 22:07 IntroscopeAgent.20131104-230606.log

$ pwd ; ls -lrt /opt/BEA/weblogic10.3/wlserver_10.3/wilyAgent/wily total 13490 drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy -rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupport.jar -rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agent.jar -rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupport.jar -rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-config.xml -rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMap.jar -rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogic.xml drwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples -rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefile.so -rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvm.pbd -rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsf.pbd -rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typical.pbd -rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-full.pbd -rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2.pbd -rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2ee.pbd -rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errors.pbd -rwxrwxr-x 1 eia eia 713 Mar 25 2011 di.pbd -rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecording.pbd -rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-http.pbd -rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmap.pbd -rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soa.pbd -rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejb.pbd -rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEW.pbd -rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecorator.pbd -rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterString.txt -rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSB.pbd -rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typical.pbl -rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typical.pbd -rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-full.pbd -rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-full.pbl -rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgent.properties -rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEW.pbd -rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90x.pbd -rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100x.pbd -rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecounts.pbd -rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errors.pbd -rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremq.pbl -rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremq.pbd -rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extra.pbd -rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservices.pbd drwxrwxr-x 2 eia eia 1024 Mar 25 2011 version -rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibs.pbd -rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-6.1.pbd -rwxrwxr-x 1 eia eia 2107 Mar 25 2011 struts.pbd -rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-6.1.pbd -rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-6.1.pbd -rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagent.pbd -rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasa.pbd -rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentext.pbd -rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogic.pbd -rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typical.pbd -rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_full.pbd -rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errors.pbd -rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportal.pbl -rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportal.pbd

-rwxrwxr-x -rwxrwxr-x drwxrwxr-x drwxrwxr-x drwxrwxr-x drwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x -rwxrwxr-x drwxrwxr-x

1 eia 1 eia 2 eia 2 eia 2 eia 2 eia 1 eia 1 eia 1 eia 1 eia 1 eia 1 eia 1 eia 1 eia 1 eia 2 eia

eia eia eia eia eia eia eia eia eia eia eia eia eia eia eia eia

1867 Mar 25 2011 oraclejdbc.pbd 3494 Mar 25 2011 leakhunter.pbd 96 Apr 26 2011 connectors 96 Apr 26 2011 tools 96 Apr 26 2011 dynamic 2048 Jun 20 2011 ext 30276 Aug 9 2011 sqlagent-full.pbd 30186 Aug 9 2011 sqlagent.pbd 6072 Sep 20 2011 toggles-typical.pbd 2107 May 29 2012 weblogic-full.pbl 6034 Jul 11 2012 toggles-full.pbd 2119 Aug 31 2012 weblogic-typical.pbl 60694 Dec 5 2012 IntroscopeAgent.profile_168 60694 Dec 6 2012 IntroscopeAgent.profile 405 Jan 10 2013 purgeIntroscopeLogs.sh 40960 Nov 4 22:08 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below:
Login to the Weblogic Console In the left pane, expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter Introscope Startup Class In the class name field enter com.wily.introscope.api.weblogic.IntroscopeStartupClass Click next Place a check beside the managed servers and then click finish. Click on Activate Changes Button

245 <startup-class> 246 <name>Introscope Startup Class</name> 247 <target>Common,dynco3,dynco1,dynco2,dynco13,dynco12,dynco11</target> 248 <classname>com.wily.introscope.api.weblogic.IntroscopeStartupClass</class-name> 249 </startup-class>

The application triage map

The application triage map presents a graphical visualization of the components that make up your application, showing application health and errors. This map is automatically generated from performance and analysis of Introscope metrics, errors, and events. It presents applications in the business-centric terms that youve defined. The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems.

Management module is container that holds introscope elements,Every thing in management module stored in a jar file.the jar file contains XML file where the definition of management module and its elements. We can reuse the management modules in another EM. Action: specifies what occurs when alert crosses user defined threshold. AlertDowntimeschedule: Alerts: notification generates based on threshold. Dashboard:where we can see the console workstation. Metric grouping:logical group of metrics treated as single element. ReportTemplate:definition of structure and data that displays when a report is run. It gets their data for metric grouping. SNMP:simplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB). An agent expression is filter that uses the default agents to be used in the management module. (.*)\|(.*)\|(.*) --- any agent on any host. EJB\\|Session\\|MyBean\\|service:stailed Method Count --- Match only EJB |session|Mybean|service:stalled method count CPU.*.Utlization.* - match any metric beginning with utilization in the cpu category. RBC_Back end stall count: backends\|ctgclient\jdbc.*wily):stallcount Rbc_sql_query: : backends\|(*.wily(\(|SQL*Qeury\|(.*) Averageresponse(.*) Elements New Managemenet module - give the name and jar file, in the domain field use the drop down to select the domain(super domain). Management module with in the same domain must have unique name, In new management module when force uniqueness selected it changes non unique names to unique. Management module and edits are saved as XML and stored in jar files. These are maintained by EM and should not change directly. After you create the management module we have to define the parameters for the management.

In the settings under configuration check the active box this global switch all functionality on. You configure the agent expression which is default set of agent to use with this management module.click add to define agent expression. This jar file locates in EM/config directory.

Management module you can delete or copy between domains and between EMS also. Metric grouping is logical set of one or more metrics.uses a filter to group metrics from multiple sources at the same time. Individual metrics are uniquely identified with fully qualified metric names. Host|weblogic|default|servlets|executorservlet:average response time.

A Metric name average response,stall coutn indicated by :. A metric grouping expression filter in a regex that defines the metric included in metric grouping. AgentExpression: Define the agent from which the metric grouping will retrieve data. .*\|.*\|Agent1 First the part of the host that runs processes to be monitored, second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data, Third the agent responsible for collecting the data.

AB Server\|WAS_NODE\|AB_RealTime.* : on host Abserver, was node and the AB_Real time jvm.
ABServer\|.* --- All nodes in ABSERVER host.
MetricExpression:Defines the metrics to include in the metric grouping.

Servlets\|.*:Average response

time.

Here under servlets many servlets are there and for all it gets the average response time. We can create metric grouping in investgate only.

Give the name and management module, now it appears in management module-Metric grouping. In metric grouping for management module change the settings,

the metric grouping agent expression looks like, here it gets the GC Heap for the s1_66 node in all the JVMS.

For the management module expression we use the Enterprisemanager\|CPU:EM CPU Used\(%\) Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined threshold. Multiple alerts will enable you to display the worst conditions among all the simple alerts. You can create actions to notify you when an alert occur so that we can react. Creating alert blackout enables you to temporarly disable alerts during routine service,maintenance or load testing. Alerts generate when metric value crosses user defined threshold. It contains two thresholds one for danger-red and antoher is caution-yellow.

Metricgrouping:defines the metric groping you want to watch the alert, DB connection is the metric grouping for the alert,Resolution

time period between points when the comparison made between the metric value and the threshold.

Comparison operator greater than or less than the threshold. Combination its for any or all, if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only.

Action delay when ever severity changes etc Alerts: white: no input data, Green- acceptable values,yellow- minor problem, Red- errors. Right click on the metric grouping and simple alerts,name and management module to select then it shows in the alerts side. Notify the alert when to trigger,like the above settings. While selecting you can find one option notify by individual metric, dont select this it created notification script that parses alert message text and then makes decision to route based on some text. Creating summary alerts: status of two or more simple alerts, state is defined as the worst among the simple alerts monitor. Includes time stamp, name of the summary alert, current state of summary alert. Summary alerts dont have the user define time period for checking the current state of underlying simple alert. Summary alerts are viewable with alert state data viewer only. Element Alert- Summary alert. Summary alert displays in the dashboard at starting to show the system health status. All Alerts locates in em/logs/alerts.txt.

Actions: multiple alerts can share actions, Element-new snmp notification action/new send smtp mail action/shell command action/console notification action. Use the console notification action ---- It displays the alert notification to all workstation it connected, it comes infrontof the console saying errors are there.

Shell command action: give the shell as echo %1 >> /alertlog.txt

Send smtp mail action:

Send mails to the team when ever the alert generates, in Alerts only we can assign alerts.

Add , you can mention any type of actions . Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases. Alert blackout to schedule the deactivaton and reactivation of alerts, elment Alert blackout give the name. Here select simple schedule or cron schedule. For simple schedule mention the time and cron schedule check the cron . This can be done in element- Alert blackout These all stores in EM/config/directory. <blackout alerts> <blackout enabled=true crong schedule 0 30 3 ? *SUN* duration in mins =120> <alert-regx mgmt ---------->

Dashboard on the dashboard of console . Data viewers display elements mapped to data as metrics,metric grouping,alerts and calculators. Link provides navigation between dashboards.

click Dashboard - new dashboard

You can get the dash board editor, you can see the graph, string viewer, txt viewer, bar chate , rectangle etc Configure pixel size also. From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor. or Right click in the dataviewer- select data option, there you can see all the metric groupings,alerts, calculator, and metrics select any. Right click dataviewer- scale pin at 0 to maximum auto expand, and in sort/filter you can select the show the top 10 metrics by excluding the metrics over there. Resolution configure the update period of the viewer in data like 15 seconds it refreshes. In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module.
Summary reports like response per interval in the back end, errors per interval in backend, worst performance sql querys etc want to generate on request. It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc for sql query you many slect metric data table also. Reports are generated through CLW or the work station, it provides critical information about enterprise web application.Elements

New Report template name and open template editor. There you can see the toolbar,graphs and button. Other side cover page,default data, report and default display properties. Drag and drop the template. Preview selected elements. On cover page give any name , for the default data properties, start time, end time period etc Report page report signature, locale settings,language used.Default display parameters. In the report editor right click data , text, display properties create your own and do the link. Generate report- workstation generate report. Select the report and click generate preview and ok save report to pdf.

If we want SQL Query graph in editor select sql query time graph, click the display properties use the option like show volume on, yellow line value , label, redline value and label.

Detect and verify the problem: if the problem is reoccurring verify the reports. Prioritize the problem, isolate the problem, document the problem and forward it to specialist. Four kind of problems, consistent- progressive periodic- sudden. Load the problem due to more user load, code issues, configuration issues and backend issues. Metrics with increased value Average response time , Metrics with decrease value: response per interval, GC Bytes in use: stalle method , concurrent invocation. Consistent problem not belongs to time/load dependent, not helps after adding capcity, not solved to restart the system also. Affects for some function usage. It may be the too many layers between components, over use of external system, backend and code issues.

Many Layers like object translation performance between two other objects needing to communicate
with each other, having too many layers between components can cuase slow behavior. A SQL query result is put into xml, enhance that using XSLT, read by data management layer DTO and retrun to EJB these all are many layers. To overcome above one, sql query returns set of database record packages to specialized designed object of arrays, other structure data might be returned to comma delimited text or electronic data exchange. If we have so many standards also program needs to translate like one standard to another and come back. Too many layers impacts: Metrics with increase value as AVg resp time, concurrent invocations and stall count. Decreased value as Response per interval. This is code problem so report to the developer

Over useExternal system: using LDAP to frequently lookup credentials,sending emails to many
users instead of one message n address list. This happens for using back end system frequently. Metrics with incrased value: Avg resp time, concurrent invocation. Metrics with decreased value: Available threads. This can be to the developer .

Slow back end response: changing any server settings might cause this issue, slow query
transactions. Metrics increased : Avg resp, concurrent invocation and stall count . Decreased Available threads. Developer or database

Code issues: too many loops, calling same methods, deadlock redundant objects. Metric increased:Avg
resp time, concurrent invocation, cpu utilization and decreased values are available threads and response per interval. Developer report this.

Progressive issues: severity range is abroad, cpu and memory use increased,problem usually effects all functions and server can eventually look up. This can be due to garbage collector, resource leaks and memory leaks as relationship with GC and CPU, instance count in GC and non expirting session objects. GC reclaims which objects are using the app, make heap size available to other. If any JVM locked it can assign to other. For all progressive problem check the GC , you can see this investigator for all app, CPU , JVM etc

Resource Collector: File I/O , sockets and Threads opened and never closed. When two threads occurs
each other then dead lock occurs this is code issues. Metrics incrased:concurrent invocations, stalled methods threads under servlets-threadpool etc Metrics decreased value: responsible per interval, available threads, cpu utilization etc Take threaddump to know the error.

Memory leaks: objects are added to the data structure and never released, session expired but that
session not removed from collection objects. If this not used then only it can be removed by garbage collector. Here cpu will reach 100 percent. Metrics increased:Avg Resp time,bytes use, cpu utilization and instance counts. Metrics decreased: Resp per interval. Relationship between GC and CPU:Garbage collect more often with leak, cpu utilization spike ever time garbage is collected. The GC hep byte use metric grows progressively. The cpu spike with each GC dependent. Instance Count increases during the leak , decreases during the GC. Non expiring session objects:session not expire after certain time. Contact dev.

Sudden problems: is instantaneous(not linear or progressive), happens with out warning(worked fine
one minute, but not in the text. Is recurring(but irregular time). The application server can freeze up , start producing errors, act chaotically.

Live locks are like Runaway process that consumes resources,loop statement that doesnot specify when
repletion stops. Metrics with increased value:Avg resptime, stall coutn, cpu utilization count. Decreased:response interval. Unending retries:when component makes constant call to unavailable systems,is the most common source of hang-ups and freezes.requires backd end lookup. Here the threads drops to zero, stall count metrics high , cpul utliztion goes down.

Pig and python: The available memory is consumed with very large result set, the jvm crashes as the bytes in use metric equals the total bytes metric. Out of memor errors are received. Increased: resp time, gc bytes use,stall count, concurrent invocation decreased resp per interval.
Large result set should broken up to small one.

Sudden slow backend:Network issues, connection with another user, less database
resources, content with antoher process and poor database design. Metrics increases are avg resp
time, stall count and concurrent invocations. Decreased: available threads. Dead lock also,

Periodic problems: time of day or load time, it recovers after some time.
Over load metrics:available threads are consumed as response takes longer, concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases. Under allocation: application server configured issues. Increased metric:Avg resp time,concurrent invocaton. Decreased: cpu utilztion, available threads, available connections. In sufficient code logic is under load like process only one request at a time etc ..

Load backend: poorly written SQL, Stored procedure. Database server settings and Maiframe settings.

, in network issues every thing is normal at our end.

Network issue

Calculators are the first example generating the custom metrics which exist on EM. ProbeBuilder file
extension(pbd) files are associated with java and .net agents and defined what to monitor inside enterprise application. EPAgents exist on managed server, these are monitor from
outside of the enterprise applications,retrieving information from external processes and logs such as the JVM or CLR. Calculators are components that generate new metric based on calculations performed on other metrics. Its deployed on EM where metrics can be intercepted and used. Management and java script calculator performs customizable calculations on specified input metrics to produce new metric, there is tradeoff between the simplicity of management calcualtors and the java script calculators. Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents.

1 . Management module calculators: develpement time:short, flexibility:limited, scope:narrow. 2 . Java script calculators: Development time:long , flexibility:great, scope:narrow. 3 . Virtual agents: Development time:short,flexibility:limited,scope:wide. Mangement module calculator performs the sum(counter:stal count and concurrent invocations, interval count:response per interval and errors per interval) and average operations. Metric grouping New calendar from give the name you can see this in calenders. Java script calculators provide the full functionality of java script language, create new metrics on any agent resides the EM/scripts/*js files samples can be EM/examples/scripts. Java script calculator functions: execute(): works with input metric data and produce output javascript resultset helper. getAgentRegex() specifies which agents are inputs, is called once at deployment.getMetricRegex() specifies which metric are inputs, is called once at deployment. getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create, is called once at deployment.

Virtual agents: across multiple agents into new virtual agent. Suppose in cluster(all of nodes) we have 16
agents each response per interval is 2,5,12 etc then crate virtual agent for this its avg response is like total by 16. It applicable application deployed to clusterd application servers,multiple instance of same application deployed on different jvms.The metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm. Its like in all jvms one jsp is there it averages the response from all.

These virtual agents reside on EM or Manager of managers in EM Cluster. Virtual agent calculators can include agents assigned to other virtual agent calculators,cannt include other virtual agent calculators, cant include other virtual agent calculators, must be uniquely named. Virtualagent config file: EM/config/agentcluster.xml. <agentclustername=Myvirtualagenet domain=superdomain> host|process|agentname

Java agents can change metric data by altering, configuring or customizing by using pdb and URL grouping. The agent obtain list of applicable pbd files from introscopeagent.profile file and parses their directives. Through probebuilder,the agent inserts probes report performance and execution data to the agent, the agent collects the data and reports metrics over the network to the EM. During the running of application, the probe reports performance and execution data to the agent, the agent collect this data and reports metrics over the network to EM. JVM(Introscope agent)-introscope agent.profile- we mentioned either pbl or pbds in pbl we mentions pbds.
41 # Directives Files 42 # 43 # ================ 44 # This property specifies all the directives files that determine 45 # how Introscope AutoProbe performs the instrumentation. Specify 46 # a single entry, or a comma-delimited list of entries. The list 47 # may include any combination of: 48 # - directives (.pbd) files 51 # 53 # 49 # - directives list (.pbl) files 50 # - directories that will be scanned about once per minute for will be loaded automatically, without any need to edit this 54 # directives will take effect immediately without an app reboot.

.pbd files. Directives files placed in a listed directory

52 #

Agent profile. If dynamic instrumentation is enabled, the

55 # Non-absolute names will be resolved relative to the location of 56 # this properties file. 57 # IMPORTANT NOTE: This is a required parameter and it MUST be set 58 # to a valid value. 59 # - If the property is not specified or the values are invalid, 60 # the Introscope Agent will not run! 61 # - If the property is set to include a directory, and invalid 62 # directives files are placed in the directory, AutoProbe 63 # metrics will no longer be reported! 64 # - If the property is set to include a directory, and loaded 65 # directives files are removed from the directory, AutoProbe 66 # metrics will no longer be reported! 67 # You must restart the managed application before changes to this property 68 # take effect. However, if the property includes one or more directories, 69 # and dynamic instrumentation is enabled, the Introscope Agent will load 70 # directives files from the specified directories without an app restart, 71 # as noted above.

73 introscope.autoprobe.directivesFile=weblogic-typical.pbl,hotdeploy gcp4al25:gr417a$ pwd /opt/edb/app/wilyAgent/wily

gcp4al25:gr417a$ ls -lrt *pbl -rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-full.pbl -rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typical.pbl -rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typical.pbl -rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportal.pbl -rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-full.pbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremq.pbl

gcp4al25:gr417a$ view weblogic-typical.pbl 1 ######################################################################## 2 # Introscope ProbeBuilder List File 3# 4 # CA Wily Introscope(R) Version 9.0 Release 9.0.6.2 5 # Copyright (c) 2011 CA. All Rights Reserved. 6 # Introscope(R) is a registered trademark of CA. 7 ######################################################################## 8 9 ################################################################## 10 # This file contains a list of ProbeBuilder directives files. # 11 # These files contain the ProbeBuilder directives used when 12 # adding probes into BEA WebLogic Server 9.0.x, 9.2, 10.x 13 # or 10.3.x # # #

14 ################################################################## 15 16 ####################### 17 # List Title 18 # ================ 19 20 Title: BEA WebLogic Server [Typical] (9.0.x, 9.2, 10.x or 10.3.x) 21 22 23 ####################### 24 # List Description 25 # ================

26 27 Description: Default Typical Introscope settings plus support for BEA WebLogic Server versions 9.0.x, 9.2, 10.x or 10 .3.x 28 29 ####################### 30 # Directives Files 31 # ================ 32 # One directives file name per line. Relative names 33 # are resolved against the location of this file. 34 35 toggles-typical.pbd 36 jvm.pbd 37 j2ee.pbd 38 java2.pbd 39 struts.pbd 40 taglibs.pbd 41 oraclejdbc.pbd 42 sqlagent-typical.pbd 43 errors.pbd 44 summary-metrics-6.1.pbd 45 sql-agent-summary-metrics-6.1.pbd 46 wl90x.pbd 47 wl100x.pbd 48 jsf-toggles-typical.pbd 49 jsf.pbd 50 biz-trx-http.pbd 51 #appmap.pbd

52 #appmap-ejb.pbd 53 #bizrecording.pbd 54 di.pbd 55 56 ####################### 57 # Servlet Header Decorator 58 # 59 # Enable this PBD for CEM integration. It appends additional 60 # information to the HTTP response that is then captured by CEM. 61 # ================ 62 63 #ServletHeaderDecorator.pbd 64 65 66 ####################### 67 # LeakHunter 68 # 69 # Enable this PBD to help locate the source of potential memory leaks. 70 # This will incur overhead, so enable LeakHunter only when necessary. 71 # See LeakHunter documentation for usage and configuration details. 72 # ================ 73 74 #leakhunter.pbd

gcp4al25:gr417a$ view CTG_ECI_Tracer_For_SYSVIEW.pbd 1 ######################################################################## 2 # IBM CTG Java Gateway Tracer

3# 4 # Version: 9.0.6.2 5 # Build: 115259 6 # Copyright (c) 2011 CA. All Rights Reserved. 7 # Introscope(R) is a registered trademark of CA. 8# 9 ######################################################################## 10 SetTracerClassMapping: JavaGatewayTracing com.wily.powerpack.sysview.ctgtracer.eci.JavaGatewayFlowTracer com.wily.int roscope.probebuilder.validate.ResourceNameValidator 11 12 SetFlag: JavaGatewayTracer 13 TurnOn: JavaGatewayTracer 14 15 IdentifyClassAs: com.ibm.ctg.client.JavaGateway JavaGatewayTracer

JavaGatewayTracer flow JavaGatewayTracing "CTGTracer|JavaGateway|Flow|{server} |{program}"

Here we can find lot of pbd files, many for the normal like jvm, jsp there you can see the normal classes to exclude or monitor. And some for application specific like above.

16 TraceOneMethodWithParametersIfFlagged:

Two types of pbl , application-full.pbl and applicaton-typical pbl(more visibility, more overhead better for QA and development environments). Default metric set is modified as 73 introscope.autoprobe.directivesFile=weblogic-typical.pbl,hotdeploy.pbd and pbl updates require to restart the jvm. Turnon key activate a tracer group in the toggles-full.pbd or togglestypical.pbd files. gcp4al25:gr417a$ ls -lrt togg*

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typical.pbd

Example: Web application using number of sockets to write large files in a remote system,but we are not able to see this due to networksocket I/O is disable in toggles-typical.pbd file. So go and uncheck that like turn on socket tracing.

toggles-typical.pbd
TurnOn: EJB2StubTracing , TurnOn: SessionBeanTracing , TurnOn: EntityBeanTracing, TurnOn: MessageDrivenBeanTracing, TurnOn: EJB3StubTracing, TurnOn: SessionBean3Tracing , TurnOn: EntityBean3Tracing , TurnOn: MessageDrivenBean3Tracing , # TurnOn: EJB3MethodLevelTracing, TurnOn: HTTPServletTracing, # TurnOn: JTACommitTracing etc Suppose if we want reduce overhead on jma transport then do #turnon:javamailtransportflag.

Over view of URL groups:how metrics appear in front end metrics gateway configured in the introscopeagent.profile, url group defines how metrics will be segmented according to http request parameters.

####################### # URL Grouping Configuration # # ================ # Configuration settings for Frontend naming. By default, all frontends # go into the "Default" group. This is done so that invalid URLs (i.e. # those that would generate a 404 error) do not create unique, one-time # metrics -- this can bloat the EM's memory. To get more meaningful # metrics out of the Frontends|Apps|URLs tree, set up URL groups that # are relevant to the deployment # Changes to this property take effect immediately and do not require the managed application to be restarted.

introscope.agent.urlgroup.keys=default ( so here in url we can see all the jsps) introscope.agent.urlgroup.group.default.pathprefix=* introscope.agent.urlgroup.group.default.format=Default
if we give the above as introscope.agent.urlgroup.keys=account,car,hotel then you can see these 3 url groups.

We can use dynamic url also by introscope.agent.urlgroup.group.pages.fromat={path_delimited:/:2;4}.

Custom pbds are user defined pbds. Select the suitable components from the system, install the new pbd file in introscope system and update the introscopeagent.profile and pbl files. Example, the three custom components that participate in application are customer, email daemon and XML processer. Customer is simple data holding object associated with an order whether object should go or emaildaemon provide confirmation of ordered email for shipment.B2B processer is strong it implements a use case, the full fillment of orders from business partners such as car rental, hotel reservation through xml or webservice. Pbd files identify the classes and methods to trace, the type of metrics to capture, how the metrics will be displayed in the workstation investigator. Files are divided into tracer groups, set of probes associated with group of classes, tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions. Pdb servlet tracer group(1), jms tracergroup(2), B2B order tracergroup(3). There are four steps to create 1)declares tracer group, associate classes with your tracer group, identify the methods to trace and how to trace them, Activate the tracer group. To declare trace group, use the set flag directive in pbd file to declare tracer group name, example setflag:b2b processorTracing. Use the identifyclass as:com.rbc.B2BOrderProcessor B2BProcessorTracing. The tracer directive often have four parameters: the tracer groupname, the method name to trace, tracer type to use, destination where the resulting metric . Traceonemethodifflagged:B2B processerTracing, process orders(name of the method),Blame point tracer(tracer to use , contains avg resp tme, resp per interval, errors per interval etc . Custom|B2bOrder processor|processorders. In pbd turn on:B2B processor tracing. Deploy using introscope agent.profile or pbl(pbd file updates).check the Auto probe log.

For subclass:identify inherited as:associate all child classes Identify matchclass:

This examples show the updated version of the custom PBD file created earlier, monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methods.once you made changes to the pbd file has to restart the EM. Setflag:B2B processor Tracing

Identify class as: com.rbc.B2BOrderProcessor B2Bprocessortracing Identify inherited as:com.rbc.B2Borderprocessor B2BProcesstracing TraceComplexMethodFlagged:B2B ProcessTracing BlamepointTracerDifferentInstancesCustom|{classname}|{method} Turnon B2BProcessTracing. We can skip classes and method also.

There are two ways of setting the stall count threshold in introscope. Per agent and per method. Permethod threshold can be define by using the stalled method tracer in the tracer group. The peragent is defined in theintroscopeagent.profile introscope.agent.stalls.thresholdseconds=30.

EPAgent is standalone java program running on JVM,retrieves metric data from plugin like normal agent from probes(pbd). EPAgent input metric into CA introscope that extend beyond java and .net monitoring. Agent looks java classes etc but EPAgent looks like WLS logreader plugin(binary format) etc EPAgent starts plugins starts. You can install the EPAgent by bundle or standalone. EM/config/EPAgent.properties