9 Tuning and Performance

Web Crawling Strategy

Your Web crawling strategy can be as simple as identifying a few well-known sites that are likely to contain links to most of the other intranet sites in your organization. You could test this by crawling these sites without indexing them. After the initial crawl, you have a good idea of the hosts that exist in your intranet. You could then define separate Web sources to facilitate crawling and indexing on individual sites.

However, in reality, the process of discovering and crawling your organization's intranet is an interactive one characterized by periodic analysis of crawling results and modification to crawling parameters to direct the crawling process somewhat. For example, if you observe that the crawler is spending days crawling one Web host, then you might want to exclude crawling at that host or limit the crawling depth.

Monitoring the Crawling Process

Monitor the crawling process by using a combination of the following methods:

• Monitoring the schedule status with the administration tool

• Monitoring the real time schedule progress with the administration tool

• Monitoring the crawler statistics with the administration tool

• Monitoring the log file for the current schedule

URL Looping

URL looping refers to the scenario where, for some reason, a large number of unique URLs all point to the same document. One particularly difficult situation is where a site contains a large number of pages, and each page contains links to every other page in the site. Ordinarily, this would not be a problem, because the crawler eventually analyzes all documents in the site.

However, some Web servers attach parameters to generated URLs to track information across requests. Such Web servers might generate a large number of unique URLs that all point to the same document.

For example, http://mycompany.com/somedocument.html?p_origin_page=10 might refer to the same document as http://mycompany.com/somedocument.html?p_origin_page=13 but the p_origin_page parameter is different for each link, because the referring pages are different. If a large number of parameters are specified and if the number of referring links is large, then a single unique document could have thousands or tens of thousands of links referring to it. This is an example of how URL looping can occur.

Monitor the crawler statistics in the Ultra Search administration tool to determine which URLs and Web servers are being crawled the most. If you observe an inordinately large number of URL accesses to a particular site or URL, then you might want to do one of the following:

• Exclude the Web Server: This prevents the crawler from crawling any URLs at that host. (You cannot limit the exclusion to a specific port on a host.)

• Reduce the Crawling Depth: This limits the number of levels of referred links the crawler will follow. If you are observing URL looping effects on a particular host, then you should take a visual survey of the site to find out an estimate of the depth of the leaf pages at that site. Leaf pages are pages that do not have any links to other pages. As a general guideline, add three to the leaf page depth, and set the crawling depth to this value.

Be sure to restart the crawler after altering any parameters in the Crawler Page. Your changes take effect only after restarting the crawler.

Understanding the Launcher

The launcher is the sub-system that listens for launch requests and launches remote crawler processes. When you register a remote crawler (either RMI-based or JDBC-based), you are actually registering the launcher with the Ultra Search backend. By registering a launcher, you make it available to be used by all Ultra Search instances within an Ultra Search installation. Thus, after registration, an administrator or an Ultra Search instance may subsequently choose to associate the launcher and assign schedules to be launched with that launcher. There is no way to restrict a launcher to specific Ultra Search instances. Once registered, all Ultra Search instances may use it. However, the launcher is only a sub-system for launching remote crawler processes. Having multiple instances use the same launcher for launching purposes poses no security problems for most customers. Both RMI and JDBC launchers are simply Java processes themselves. They are started from the command line. Oracle provides scripts for starting these launchers, described in the following section. Also, the JDBC launcher must establish JDBC connections to the Ultra Search backend database to listen for launch events. You must specify the launch user (or role) at registration time. Oracle strongly recommends that you create a new database user (or role) specifically for the purposes of launching remote crawlers. You should not use this user (or role) for any other purposes.

RMI-Based Remote Crawling

RMI-based remote crawling depends on the standard RMI infrastructure. Therefore, each remote crawler host must have an RMI registry and an RMI daemon running. These are started when you run the scripts to start the RMI-based launcher.

• When a crawling schedule is activated, the Ultra Search scheduler launches a Java program as a separate process on the database host. This Java program is known as the ActivationClient.

• This program attempts to connect to the remote crawler host through the RMI registry port specified at installation time. If successful, then the ActivationClient receives a remote reference to a Java object running on the remote host. This remote Java object is known as the ActivatableCrawlerLauncher.

• The ActivationClient then instructs the ActivatableCrawlerLauncher to launch the Ultra Search crawler on the remote host. The ActivatableCrawlerLauncher launches the Ultra Search crawler as a separate Java process on the remote host.

The RMI registry and daemon ports are inflexible. Therefore, if you have other RMI services running on the same host, you will not be able to use RMI-based remote crawling. Also, you cannot run two RMI-based launchers, because they will both conflict on the RMI ports.

JDBC-Based Remote Crawling

JDBC-based remote crawling requires that the launcher be up and running.

• When a crawling schedule is activated, the Ultra Search scheduler sends a message to the launcher.

• If the launcher is running and properly connected to the database as the appropriate launch user (or role), then it can receive the launch messages. Otherwise, the message times out after 30 seconds and launch failure is reported.

• The launcher then deciphers the launch message and spawns an Ultra Search crawler as a separate Java process on the remote host.

The launcher maintains two permanent JDBC connections to the backend database. If either connection goes down at any time, then the JDBC launcher attempts to reestablish it. The number of attempts to reestablish connections is configurable as a command line parameter. The wait time between attempts is also configurable.

Security With Remote Crawlers

When launching a remote crawler, the Ultra Search backend database communicates with the remote computer through Java remote method invocation (RMI) or JDBC.

Ultra Search encrypts all RMI communication. However, the JDBC launcher uses the Oracle Thin JDBC driver. If security is a concern, then encrypt all JDBC traffic by securing the Oracle Thin JDBC driver.

Scalability and Load Balancing

Each Ultra Search schedule can be associated with exactly one crawler. The crawler can run locally on the Oracle database host or on a remote host. There is no limit to the number of schedules that can be run. Similarly, there is no limit to the number of remote crawler hosts that can be run. However, each remote crawler host requires that the Ultra Search middle tier be installed on its host.

By using several remote crawler hosts and carefully allocating schedules to specific hosts, you can achieve scalability and load balancing of the entire crawling process.

Installation and Configuration Sequence

1. Make sure that you have installed the Ultra Search Backend server component as well as a Server component on each host that is to be used to run remote crawlers.

   
   

   See Also: Chapter 3, "Installing and Configuring Ultra Search"

2. Understand the cache and mail archive directories.

   All remote crawlers must cache crawled data into a common file system location, that is, accessible by the backend Ultra Search database. Likewise, when crawling Email sources, all emails must be saved in a common, central location. The simplest way to achieve this is by ensuring that the cache and mail archive directories seen by the remote crawler uses are mounted through NFS to point to the cache and mail directories used by the Ultra Search backend database.

   For example, your Ultra Search installation might consist of four hosts: one database server (host X) running Solaris on which the Ultra Search backend is installed; one remote crawler host (host Y1) running on Windows; one remote crawler host (host Y2) running on Solaris; and one remote crawler host (host Y3) running on Linux.

   In this scenario, export the shared directories on host X using the UNIX export command. Then use the UNIX mount command on hosts Y2 and Y3 to mount the exported directories. For host Y1, you must purchase a third-party NFS client for Windows and use that to mount the shared directories. If host X is a Linux server, you can create Samba shares and thereby mount those shares on Windows without needing any third party software.

   If for some reason there is no shared file system between the database and remote crawler hosts, you can instruct the remote crawler to transfer all cache and mail archive data across JDBC to the database host. The files are then saved locally on the database host. You can choose this option by selecting through JDBC connection for the Cache file access mode setting in the next step.

3. Configure the remote crawler with the administration tool.

   To edit the remote crawler profile, navigate to the Crawler: Remote Crawler Profiles page and click Edit for the remote crawler profile you want to edit. Edit that profile by manually entering all mount points for the shared crawler resources that you defined.

   Cache and mail archive directories. If the backend database host and remote crawler host are on a shared file system (such as NFS), select "through mounted file system" for the Cache file access mode setting. Then specify values for the following parameters:

   • Mount point for cache directory path as seen by the remote crawler

   • Mount point for mail archive path as seen by the remote crawler (if you are using the Ultra Search mailing list feature)

   Otherwise, if there is no shared file system between the remote crawler host and the backend database host, you must select through JDBC connection for the Cache file access mode setting. Then, specify values for the following parameters:

   • Local cache directory as seen by local crawlers on the backend database host

   • Local mail archive directory as seen by local crawlers on the backend database host

   Crawler log directory. It is not necessary that the remote crawler log directory be an NFS mount a central location accessible by the backend Ultra Search database. However, it is beneficial to do so if you want to be able to monitor all crawler logs (local as well as all remote crawlers) in one central location.

   Additionally, you must specify the following crawler parameters before you can begin crawling:

   • number of crawler threads that the remote crawler uses for gathering documents

   • number of processors on the remote crawler host

   • initial Java heap size

   • maximum Java heap size

   Java classpath. It is not usually necessary to specify this classpath. The classpath that remote crawler processes use is inherited from the RMI subsystem. The RMI subsystem classpath is configured by the scripts used to launch it. You will need to modify the classpath only in special circumstances where you require the classpath to be different from the RMI subsystem classpath.

4. Complete the crawler configuration with the administration tool.

   Create schedules and data sources. Assign one or more data sources to each schedule.

   Each schedule must then be assigned to a remote crawler or the local crawler. (The local crawler is the crawler that runs on the local Oracle database host itself). To assign the a schedule to a remote crawler host or the local database host, click the host name of a schedule in the Schedules page.

   You can also turn off the remote crawler feature for each schedule, thereby forcing the schedule to launch a crawler on the local database host, instead of the specified remote crawler host. To turn off the remote crawler feature, click the host name of a schedule in the Synchronization Schedules page. If a remote crawler host is selected, the RMI-based remote crawler hostname (or JDBC-Based launcher name) will be displayed. Change this to the local database host in order to disable remote crawling.

   
   

   See Also: Chapter 7, "Understanding the Ultra Search Administration Tool"

5. Start the remote crawler launching sub-system on each remote crawler host.

   Use the helper scripts in $ORACLE_HOME/tools/remotecrawler/scripts/operating_system to do this.

   • If the remote crawler is running on a UNIX platform, then source the $ORACLE_HOME/tools/remotecrawler/scripts/unix/runall.sh Bourne shell script for RMI-based remote crawling. Source runall_jdbc.sh for JDBC-based remote crawling.

   • If the remote crawler is running on a Windows host, then run the %ORACLE_HOME%\tools\remotecrawler\scripts\winnt\runall.bat file for RMI-based remote crawling. Runrunall_jdbc.bat for JDBC-based remote crawling.

   For RMI-based remote crawling, the runall scripts perform the following tasks in sequence:

   1. define_env is invoked to define necessary environment variables.

   2. runregistry is invoked to start up the RMI registry.

   3. runrmid is invoked to start up the RMI daemon.

   4. register_stub is invoked to register the necessary Java classes with the RMI subsystem.

   
   

   Note: You can invoke runregistry, runrmid, and register_stub individually. However, you must first invoke define_env to define the necessary environment variables.

   For JDBC-based remote crawling, the runall_jdbc scripts perform the following tasks in sequence:

   1. define_env is invoked to define necessary environment variables

   2. The JDBC launcher is started with a command line Java process invocation. There are the following command line arguments for the JDBC launcher:

   • -name: name of launcher (that you used to register this launcher)

   • -url: JDBC connection URL to the backend Ultra Search database

   • -user: database user to connect

   • -password: database user password

   • -rw: wait time (in seconds) between attempts to reestablish JDBC connections

   • -ra: maximum number of attempts to reestablish JDBC connections

   • -kw: wait time (in milliseconds) between keep-alive signals

   You must edit the contents of the runall_jdbc script and specify the values for each parameter before running it.

6. Launch the remote crawler from the administration tool, and verify that it is running.

   The state of the schedule is listed in the Schedules page. The remote crawler launching process takes up to 90 seconds to change state from LAUNCHING to FAILED if failure occurs.

   To view the schedule status, click the crawler status in the schedules list. To view more details, especially in the event of failure, click the schedule status itself. This brings up a detailed schedule status.

   The RMI-based remote crawler fails to launch if any one of the following requirements are not met:

   • The RMI registry is not running and listening on the port specified at installation.

   • The RMI daemon is not running and listening on the port specified at installation.

   • The necessary Java objects have not been successfully registered with each RMI registry.

   The JDBC-based remote crawler fails to launch if any one of the following requirements are not met:

   • The JDBC launcher is not running.

   • The JDBC launcher is running, but the connect user (or role) specified is incorrect.

   After a remote crawler is launched, verify that it is running with one or more of the following methods:

   • For RMI-based crawling, check for active Java processes on the remote crawler host. A simple way to confirm that remote crawler is running on the remote crawler host is to use an operating system command, such as ps on UNIX systems. Look for active Java processes.

   • For JDBC-based crawling, check that the launcher is up and running and that there are no errors. When you start the JDBC-based launcher, it will output text to standard output. You may optionally redirect output to a file. Monitor this output for any errors.

   • Monitor the contents of the schedule log file. If the remote crawler is running successfully, then you should see the contents of the schedule log file changing periodically. The schedule log file is located in the shared log directory.

Configuring Storage Access

The disk of any node in a Real Application Clusters system can be shared (cluster file system) or not shared (raw disk). For Real Application Clusters on a cluster file system (CFS), the cache files generated by the crawler on any node are visible to any Oracle instance and can be indexed by any Oracle instance that performs index synchronization. If the disk is not shared, then the crawler must run on one particular Oracle instance to ensure that all cache files can be indexed.

This is due to the nature of Oracle Text indexing, where rows inserted into one table by different sessions go to the same pending queue, and whoever initiates index synchronization attempts to index all of the inserted rows. Because of this limitation, on a CFS, Ultra Search is configured to launch the crawler on any database instance. If it is not on a CFS, then Ultra Search launches the crawler on the database instance where INSTANCE_NUMBER = 1.

The Ultra Search administrator can configure which instance runs the crawler with the following PL/SQL API:

WK_ADM.SET_LAUNCH_INSTANCE(instance_name, connect_url);

where instance_name is the name of the launching instance (or the database name if it is to be launched on any node) and connect_url is the connect descriptor.

For connection to a single database instance, the descriptor can be in the short form "host:port:SID" or the connect descriptor (Oracle Net keyword-value pair). For example:

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=cls02a)(PORT=3999)))(CONNECT_DATA=(   
  SERVICE_NAME=acme.us.com))) 

To connect to any database instance, the full database connect descriptor must be used. For example:

(DESCRIPTION=(LOAD_BALANCE=yes)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=cls02a)(PORT=3999 
  ))(ADDRESS=(PROTOCOL=TCP)(HOST=cls02b)(PORT=3999)))(CONNECT_DATA=(SERVICE_NAME=acme.us.com))) 

You cannot configure Ultra Search to launch the crawler on any node on a non-cluster file system.

To query on the existing launching instance configuration, use the following PL/SQL API:

WK_ADM.GET_LAUNCH_INSTANCE RETURN VARCHAR2;

This returns the name of the launching instance or the database name if any node can launch the crawler.

Logging on to the Oracle Instance

All components of Ultra Search use the JDBC Thin Driver with the connect string consisting of "hostname:port:SID" or the full connect descriptor as seen in tnsnames.ora.

The administration middle tier connects to the Oracle database with a JDBC connection specified in the ultrasearch.properties file. If the client serving node is down, then you must manually edit the ultrasearch.properties file to connect to a different Oracle instance.

"jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=yes)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=cls02a)(PORT=3999
  ))(ADDRESS=(PROTOCOL=TCP)(HOST=cls02b)(PORT=3999)))(CONNECT_DATA=(SERVICE_NAME=acme.us.com)))" 

WK_ADM.SET_JDBC_DRIVER(driver_type)

WK_ADM.GET_JDBC_DRIVER RETURN NUMBER;

Ultra Search Failover in a RAC Environment

When RAC uses the Cluster File System (CFS), the Ultra Search crawler can be launched from any of the RAC nodes, as long as at least one RAC node is up and running.

When RAC is not using CFS, the Ultra Search crawler always runs on a specified node. If this node stops operating, then you must run the wk0reconfig.sql script to move Ultra Search to another RAC node.

where:

instance_name is the name of the RAC instance that Ultra Search uses for crawling. After connecting to the database, simply SELECT instance_name FROM v$instance to get the name of the current instance.

connect_url is the JDBC connection string that guarantees a connection only to the specified instance. For example:

"(DESCRIPTION= 
        (ADDRESS_LIST= 
          (ADDRESS=(PROTOCOL=TCP) 
                   (HOST=<nodename>) 
                   (PORT=<listener_port>))) 
        (CONNECT_DATA=(SERVICE_NAME=<service_name>)))"

When preserving the crawler cache, if Ultra Search is switched from one RAC node to another, you loose the contents of the cache. Force a re-crawl of the documents after switching instances.

Synchronizing Crawling of Oracle Databases

Before creating log tables and log triggers, make sure that the Ultra Search instance schema has the CREATE ANY TABLE and CREATE ANY TRIGGER system privileges. For tables in Oracle databases, data definition language (DDL) statements are provided to create the following:

ID

NUMBER

NAME

VARCHAR2(200)

ADDRESS

VARCHAR2(400)

TELEPHONE

VARCHAR2(10)

USERNAME

VARCHAR2(24)

CREATE TABLE WK$LOG(
K1 VARCHAR2(1000),
K2 VARCHAR2(1000),
MARK CHAR(1) default 'F');

CREATE TABLE employees (id NUMBER, name VARCHAR2(10));

CREATE OR REPLACE TRIGGER wk$ins
AFTER INSERT ON employees
FOR EACH ROW

BEGIN
  INSERT INTO WK$LOG(k1,k2,mark)
    VALUES(:new.id,:new.name,'F');
END;
/

CREATE OR REPLACE TRIGGER wk$upd
AFTER UPDATE ON employees
FOR EACH ROW
 
BEGIN
  INSERT INTO WK$LOG(k1,k2,mark)
    VALUES(:old.id,:old.name,'F');
  INSERT INTO WK$LOG(k1,k2,mark)
    VALUES(:new.id,:new.name,'F');
END;/

CREATE OR REPLACE TRIGGER wk$del
AFTER DELETE ON employees
FOR EACH ROW
 
BEGIN
  INSERT INTO WK$LOG(k1,k2,mark)
    VALUES(:old.id,:old.name,'F');
END;/

Synchronizing Crawling of Non-Oracle Databases

For tables in non-Oracle remote databases, you must perform the following steps:

1. Manually create the log table. The log table must conform to the rules for log tables described earlier. Also, it must reside in the same schema and database instance as the base table.

2. Create three triggers that record inserts, updates, and deletes on the base table. These triggers must exhibit the same behavior as the triggers described earlier for Oracle tables.

3. Associate the log table. When you have completed these tasks, choose the "Enable logging mechanism (non-Oracle tables)" option during the creation of an Ultra Search table data source. By choosing that option, the Ultra Search administration tool prompts you for the name of the log table in the remote database. Ultra Search associates this log table with the base table. Ultra Search assumes that you have correctly performed steps 1 and 2.