8 Using Oracle Enterprise Manager for Monitoring and Detection

8.2.1 Set Up Default Notification Rules for Each System

Notification Rules are defined sets of alerts on metrics that are automatically applied to a target when it is discovered by Enterprise Manager. For example, an administrator can create a rule that monitors the availability of database targets and generates an e-mail message if a database fails. After that rule is generated, it is applied to all existing databases and any database created in the future. Access these rules by navigating to Preferences and then choosing Rules.

The rules monitor problems that require immediate attention, such as those that can affect service availability and Oracle or application errors. Service availability can be affected by an outage in any layer of the application stack: node, database, listener, and critical application data. A service availability failure, such as the inability to connect to the database, or the inability to access data critical to the functionality of the application, must be identified, reported, and reacted to quickly. Potential service outages such as a full archive log directory also need to be addressed correctly to avoid a system outage.

Enterprise Manager provides a series of default rules that provide a strong framework for monitoring availability. A default rule is provided for each of the preinstalled target types that come with Enterprise Manager. These rules can be modified to conform to the policies of each individual site, and new rules can be created for site-specific targets or applications. The rules can also be set to notify users during specific time periods to create an automated coverage policy.

Consider the following recommendations:

• Modify each rule for high-value components in the target architecture to suit the required availability requirements by using the rules modification wizard. For the database rule, set the events in Table 8-1, Table 8-2, and Table 8-3 for each target. The frequency of the monitoring is determined by the service level agreement (SLA) for each component.

• Add Notification Methods and use them in each Notification Rule. By default, the easiest method for alerting an administrator to a potential problem is to send e-mail. Supplement this notification method by adding a callout to an SNMP trap or operating system script that sends an alert by some method other than e-mail. This avoids the problem that might occur if a component of the e-mail system has failed. Set additional Notification Methods by using the Set-up link at the top of any Enterprise Manager page.

• Modify Notification Rules to notify the administrator when there are errors in computing target availability. This may generate a false positive reading on the availability of the component, but it ensures the highest level of notification to system administrators.

Figure 8-2 shows the Notification Rule property page for choosing availability states. Down, Agent Unreachable, Agent Unreachable Resolved, and Metric Error Detected are chosen.

Figure 8-2 Setting Notification Rules for Availability

Description of the illustration avail.gif

In addition, modify the metrics monitored by the database rule to report the metrics shown in Table 8-1, Table 8-2, and Table 8-3. This ensures that these metrics are captured for all database targets and that trend data will be available for future analysis. All of the events described in Table 8-1, Table 8-2, and Table 8-3 can be accessed from the Database Homepage. Choose All Metrics > Expand All.

Space management conditions that have the potential to cause a service outage should be monitored using the events shown in Table 8-1.

From the Alert Log Metric group, set Enterprise Manager to monitor the alert log for errors as shown in Table 8-2.

Monitor the system to ensure that the processing capacity is not exceeded. The warning and critical levels for these events should be modified based on the usage pattern of the system. Set the events from the Database Limits metric group. Table 8-3 contains the recommendations.

Figure 8-3 shows the Notification Rule property page for setting choosing metrics. The user has chosen Critical and Warning as the severity states for notification. The list of Available Metrics is shown in the left list box. The metrics that have been selected for notification are shown in the right list box.

Figure 8-3 Setting Notification Rules for Metrics

Description of the illustration metrics.gif

8.2.2 Use Database Target Views to Monitor Health, Availability, and Performance

The Database Targets page in Figure 8-4 shows an overview of system performance, space utilization, and the configuration of important availability components like archived redo log status, flashback log status, and estimated instance recovery time. Alerts are displayed immediately. Each of the alert values can be configured from links on this page

Figure 8-4 Overview of System Performance

Description of the illustration overv.gif

Many of the metrics from the Enterprise Manager pertain to performance. A system without adequate performance is not an HA system, regardless of the status of any of the individual components. While performance problems seldom cause a major system outage, they can still cause an outage to a subset of customers. Outages of this type are commonly referred to as application service brownouts. The primary cause of brownouts is the intermittent or partial failure of one or more infrastructure components. IT managers must be aware of how the infrastructure components are performing (their response time, latency, and availability) and how they are affecting the quality of application service delivered to the end user.

A performance baseline, derived from normal operations that meet the SLA, should determine what constitutes a performance metric alert. Baseline data should be collected from the first day that an application is in production and should include the following:

• Application statistics (transaction volumes, response time, web service times)

• Database statistics (transaction rate, redo rate, hit ratios, top 5 wait events, top 5 SQL transactions)

• Operating system statistics (CPU, memory, I/O, network)

You can use Enterprise Manager to capture a snapshot of database performance as a baseline. Enterprise Manager compares these values against system performance and displays the result on the database Target page. It can also send alerts if the values deviate too far from the established baseline.

Set the database notification rule to capture the metrics listed in Table 8-4 for all database targets. Analysis of these parameters can then be done using one tool and historical data will be available.

8.2.3 Use Event Notifications to React to Metric Changes

There are many operating system events that can be used to supplement a suggested metric. Such operating system events are not required for each host and instance. All metrics defined here can be set individually by instance or database using the Manage Metrics link at the bottom of the navigation bar of the object target page. The values that trigger a warning or critical alert can be changed here, and an operating system script can be activated to respond to an metric threshold, in addition to the standard alert being generated to the Oracle Enterprise Manager 10g Grid Control.

8.2.4 Use Events to Monitor Data Guard system Availability

Set Enterprise Manager metrics to monitor the availability of logical and physical Data Guard configurations. If a Data Guard environment is registered with the Data Guard Manager extension of Enterprise Manager, then set the events shown in

8.3.1 Check Enterprise Manager Policy Violations

Enterprise Manager comes with a pre-installed set of policies and recommendations of best practices for all databases. These policies are checked by default, and the number of violations is displayed on the Targets page in Figure 8-4. Select Policy Violations from the Targets page to see a list of all violations.

8.3.2 Use Enterprise Manager to Manage Oracle Patches and Maintain System Baselines

You can use Enterprise Manager to download and manage patches from http://metalink.oracle.com for any monitored system in the application environment. A job can be set up to routinely check for patches that are relevant to the user environment. Those patches can be downloaded and stored directly in the Management Repository. Patches can be staged from the Management Repository to multiple systems and applied during maintenance windows.

You can examine patch levels for one machine and compare them between machines in either a one-to-one or one-to-many relationship. In this case, a machine can be identified as a baseline and used to demonstrate maintenance requirements in other machines. This can be done for operating system patches as well as database patches.

8.3.3 Use Enterprise Manager to Manage Data Guard Targets

Enterprise Manager can be used to set up logical and physical standby databases for any database target. It also provides the ability to manage switchover and failover of database targets other than the database that contains the Management Repository.

Enterprise Manager can also be used to monitor the health of a Data Guard configuration at a glance. From any database target page, navigate to the Data Guard status section by using the link in the High Availability section. The page shows the active standby databases for the primary target, the amount of log data waiting for shipment and receipt by the standby database and the data protection mode. You can also modify the data protection mode from this page.

This page contains a link to the Verify function, which checks the Data Guard environment and log transport services and displays warnings and errors. The Verify function must be run manually; it is not automatic.

8.4.1 Recommendations for an HA Architecture for Enterprise Manager

The following recommendations are described in this section:

• Protect the Repository and Processes As Well as the Configuration They Monitor

• Place the Management Repository in a RAC Instance and Use Data Guard

• Configure At Least Two Management Service Processes and Load Balance Them

• Consider Hosting Enterprise Manager on the Same Hardware as an HA System

• Monitor the Network Bandwidth Between Processes and Agents

8.4.2 Unscheduled Outages for Enterprise Manager

Enterprise Manager is the primary control interface for managing your data center. An outage of Enterprise Manager causes a critical lack of visibility into the performance and availability metrics that allow the DBA to manage overall system performance. Table 8-6 describes the outages that can occur to any of the tiers involved in Enterprise Manager and how to recover from each outage.

8.5.1 Configure a Separate Listener for Enterprise Manager

Traffic from the Management Agents is routed to the Management Service processes and then to the Management Repository by Oracle Net. To isolate this traffic from other application traffic and to support Data Guard if required for site failover, configure the Enterprise Manager traffic through a separate listener. The listener is active only on the node where the active Enterprise Manager instance is running. Do not set the GLOBAL_DBNAME parameter in the listener.ora file because setting it disables Transparent Application Failover (TAF) and connect-time failover. Configure the LOCAL_LISTENER and REMOTE_LISTENER initialization parameters to enable dynamic service registration and cross-registration. The following is an example of a listener configuration:

LISTENER_N1=
        (description_list=
             (description=
               (address_list=
                (address=(protocol=tcp)(port=1521)(host=EMPRIM1.us.oracle.com))
               )
             )
        )
SID_LIST_LISTENER_N1 =
  (SID_LIST =
    (SID_DESC =
      (ORACLE_HOME = /mnt/app/oracle/product/10g)
      (SID_NAME = EM1)
    )
  )

LISTENER_DG=
        (description_list=
             (description=
               (address_list=
                (address=(protocol=tcp)(port=1529)(host=EMPRIM1.us.oracle.com))
               )
             )
        )

8.5.2 Install the Management Repository Into an Existing Database

To avoid installation problems when building a RAC-based repository, it is easier to install the Enterprise Manager into an existing database and build any tablespaces in advance. Certain versions of the Oracle Universal Installer do not handle installing the repository into a RAC database. Build the database first; then use the Install option to install into an existing database.