10 Oracle Ultra Search Developer's Guide and API Reference

End User Query Syntax

The end user query syntax defined by the default query syntax expansion implementation is similar to the standard text query syntax employed by most search engines on the Web.

• Token: A token is a string enclosed in double-quotes ("). It can be a single word or a phrase.

• Operators: The default implementation defines three operators. They are the [+], [-] and [*] operators. These operators are defined by the default implementation. Change these operators to whatever you prefer in your own custom implementation.

  The plus operator [+] specifies that the token immediately following it must appear in all documents included in the search result.

  The minus operator [-] specifies that the token immediately following it cannot appear in any document included in the search result.

  The asterisk [*] specifies a wildcard search. It matches zero or more characters. A token starting with the asterisk is ignored. The asterisk can only be specified at the end (right side) or middle of a token. For example, "hel*o" and "hell*" use the asterisk correctly, but "*ello" is unacceptable.

The following table summarizes the rules for the Oracle Ultra Search end user query syntax:

Scoring Classes

There are three ways documents are matched against an end user query string. These three ways are known as scoring "classes." Documents are scored and ranked higher if they satisfy the requirements for a higher class. Within each class, documents are also ranked differently depending on how well they match the conditions of the scoring class.

Class 1 is the highest class. The score is derived from the number of occurrences of a precise phrase in a document. A document that has more instances of the precise phrase have a higher score than another document that has fewer occurrences of the precise phrase.

Class 2 is the next highest class. In this class, the closer the tokens appear in a document, the higher the score becomes. For example, an end user query string [Oracle Applications Financials] can result in three documents found. None of the three documents contain the precise phrase "Oracle Applications Financials." However, document X contains the all three tokens "Oracle", "Applications", and "Financials" in the same sentence separated by other words. Document Y contains the individual tokens in the same paragraph but in different sentences. Document Z contains the same three tokens, but each token resides in different paragraphs. In this scenario, document X has the highest score, because the tokens are closest together. Likewise, Y has a higher score than Z.

Class 3 is the last class. A document that has more tokens gets a higher score. For example, an end user query string [Oracle Applications Financials] can result in three documents found. Document X might contain all three tokens. Document Y might contain the tokens "Oracle" and "Applications" only. Document Z might contain only the token "Oracle." In this scenario, document X has a higher score than Y. Likewise, Y has a higher score than Z.

Expansion Rules

As mentioned, the end user query is expanded to an Oracle Text query. The expanded query string rules are captured in BNF (Backus Naur Form) notation. Again, these rules are the rules that Oracle Ultra Search uses as a default query syntax expansion implementation.

The rules that define an expanded query:

<expanded query> ::= (<expression> within <title section>)*2, <expression>

<expression> ::= <generic query expression> | <simple query expression>

<generic query expression> ::= (([ <plus expression>*100 & ]) (<main expression>)) [ <minus expression> ]

<simple query expression> ::= (<phrase expression>)*2, (<main expression>)

<main expression> ::= (<near expression>)*2, (<accum expression>)

The following list contains some terms and their meanings, which explain some of the terms used in the preceding rules:

A <plus expression> is an AND expression of all plus tokens.

A <minus expression> is a NOT expression of all minus tokens.

A <phrase expression> is a PHRASE formed by all tokens in the <main expression>

A <near expression> is a NEAR expression of all tokens but minus tokens.

An <accum expression> is an ACCUMULATE expression of all tokens but minus tokens.

A <simple query expression> is used only when the end user query has multiple tokens and does not have any operator or a double quote. Otherwise, a <generic query expression> is used.

If there is no token that is neither plus token nor minus token, then the <plus expression> and the <accum expression> are eliminated.

Examples of Applying the Rules

The following table illustrates how the default query syntax expansion implementation converts end user query strings into Oracle Text compatible query strings.

((({Oracle}) within TITLE__31)*2,({Oracle}))

((((({Applications})*10)*10&(({Oracle};{Applications})*2,({Oracle},{Applications
}))) within TITLE__31)*2,((({Applications})*10)*10&(({Oracle};{Applications})*2,
({Oracle},{Applications}))))  

(((({Oracle})~{Applications}) within TITLE__31)*2,(({Oracle})~{Applications}))

((({Oracle Applications}) within TITLE__31)*2,({Oracle Applications}))

((((Ora%)) within TITLE__31)*2,((Ora%)))

(((({Oracle Applications})*2,(({Oracle};{Applications})*2,({Oracle},{Application
s}))) within TITLE__31)*2,(({Oracle Applications})*2,(({Oracle};{Applications})*
2,({Oracle},{Applications}))))   

<instance> Tag: Connecting to the Oracle Ultra Search Instance

This tag establishes a connection to an Oracle Ultra Search instance. Some basic parameters must be established for this tag to work, such as JDBC connection string, schema user name/password, Oracle Ultra Search instance name, and so on.

This tag defines a scripting variable of the name set by the instance Id property. All the other tag properties correspond to a property in the oracle.ultrasearch.query.QueryInstance class. Either the URL or the dataSourceName attribute should be set, they are unique.

The following example uses the URL property to connect to the database.

<US:instance 
 instanceId="mybookstore"
 url="oracle:jdbc:thin:@dbhost:1521:inst1"
 username="scott"
 password="tiger"
 tablePage="../display.jsp"
 emailPage="../mail.jsp"
 filePage="../display.jsp"
/>

<iterAttributes> Tag: Show All Search Attributes

When a user wants to perform an advanced query, the application needs to show the list of attributes that are available, the list of groups, and the list of languages defined in the instance. This can be done using some iteration tags that define script variables for page rendering.

Each attribute in Oracle Ultra Search has a name, a type, and a display name that is translated depending on the locale that is set for the QueryInstance tag. The attribute type should be used to determine which operators can be used on this attribute and how to parse the user's input.

This tag is an iteration tag. It loops through all the search attributes in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "attribute", which is an oracle.ultrasearch.query.Attribute object. It also defines a string variable named "displayname", which is the localized name of the attribute.

The following example shows all the attributes in "mybookstore" instance, using their English display names.

<US:iterAttributes instance="mybookstore" locale="<%=Locale.ENGLISH%>" >
<%= attribute %>
<%= displayname %>
</US:iterAttributes>

<iterGroups> Tag: Show All Search Groups

Similar to the ShowAttributes tag, the Show Groups tag iterates through all the groups defined in an instance.

This tag loops through all the search groups in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "group", which is an oracle.ultrasearch.query.Group object. It also defines a string variable named "displayname", which is the localized name of the group.

The following example shows all the groups in "mybookstore" instance, using their English display names.

<US:iterGroups instance="mybookstore" locale="<%=Locale.ENGLISH%>" >
<%= group %>
<%= displayname %>
</US:iterGroups >

<iterLanguages> Tag: Show All Search Languages

Similar to the showAttributes tag, the showLanguages tag iterates through all the languages defined in an instance. Because each language is defined by a java.util.Locale object, their display names are not handled by Oracle Ultra Search. Therefore, this tag does not define the displayname scripting variable.

This tag is an iteration tag. It loops through all the search languages in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "language", which is a java.util.Locale object. The display name for the language is provided by Java as a property of the object itself (through the getDisplayName method).

The following example shows all the languages in "mybookstore" instance, using their English display names.

<US:iterLanguages instance="mybookstore">
<%= language %>
<%= language.getDisplayName (Locale.ENGLISH) %>
</US:iterLanguages >

<iterLOV> Tag: Show All Values Defined for a Search Attribute

This tag is an iteration tag. It loops through all the values in a search attribute's LOV. In each loop, it defines a scripting variable named "value", which is either a java.lang.String, java.util.Date, or java.math.BigDecimal object, depending on the attribute type. It also defines a string variable named "displayname", which is the localized display name of the value.

The following example shows all the values for a string attribute named "Dept" in "mybookstore" instance, using their English display names.

<US:iterLOV instance="mybookstore" attribute_name="Dept" attribute_type="String" >
<%= value %>
<%= displayname %>
</US:iterLOV >

Formulating the Query

Oracle Ultra Search supports a set of classes for building queries. Currently these classes do not have any tag equivalents.

<getResult> Tag: Perform Search

This tag performs the search and returns the result by defining a scripting variable of the type oracle.ultrasearch.query.Result.

The <getResult> tag corresponds to the getResult method on the oracle.ultrasearch.query.Instance class. The attributes of tag map to the parameters of the method, with the exception that getResult method can specify the attributes to fetch. The <getResult> tag requires the use of the nested <fetchAttribute> tag to accomplish metadata selection.

The following example shows a search for the first 20 documents of a query in English that appears in French documents.

<US:getResult 
 resultId="searchresult"
 instance="mybookstore"
 query=""
 queryLocale=""
 documentLanguage=""
 from="1" to="20">
</US:getResult>

<fetchAttribute> Tag: Metadata Selection

This tag is used as nested tag inside <getResult>. It specifies which attributes of each document should be fetched along with the query result. Each <getResult> can have any number of nested <fetchAttribute> tags.

Each occurrence of the <fetchAttribute> adds to the list of attributes passed to the getResult invoked by the <getResult> tag.

The following example shows the search in <getResult> tag, fetching title and publication-date attributes of each book.

<US:getResult 
 resultId="searchresult"
 instance="mybookstore"
 query=""
 queryLocale=""
 documentLanguage=""
 from="1" to="20">
<US:fetchAttribute 
 attributeName="title"
 attributeType="string" />
<US:fetchAttribute 
 attributeName="publication-date"
 attributeType="date" />
</US:getResult>

<showHitCount> Tag: Show Estimated Hit Count

After the search is performed, the result must be rendered. If withCount=true is in the <US:getResult> tag, then the result contains a count of total hits, and <showHitCount> tag can be used to display it.

The following shows the result count of the a search result.

<US:showHitCount result="searchresult" />

<iterResult> Tag: Render the Results

This tag is an iteration tag. It loops through all the documents in a search result.

The tag loops through all the documents in a search result and defines a scripting variable "doc" that is a oracle.ultrasearch.query.Document object. In addition, it can have nested tags of <showAttributeValue>, which helps to render the document's attributes. If the result specified is not one obtained from search on the instance specified, then it is an error. In other words, the result must come from the instance.

The following example shows the URL of all documents in a search result.

<US:iterResult
result="searchresult" 
instance="mybookstore">
</US:iterResult>

<showAttributeValue> Tag: Render a Document Attribute

This tag shows an attribute of a document within the <US:iterResult> tag.

This tag searches the document attribute value and displays it on the page. If the attribute was not fetched as a part of the search result, then no output is displayed.

The following example shows the title and publication dates of all documents in a search result.

<US:iterResult
result="searchresult" 
instance="mybookstore">
<US:showAttributeValue attributeName="title" attributeType="string" default="No Title" />
<US:showAttributeValue attributeName="publication-date" attributeType="date" />
</US:iterResult>

Standard Agent

The standard agent returns the list of URLs currently existing in the data source. It does not know whether any of the URLs had been crawled before, and it relies on the crawler to find any updates to the target data source. The standard agent's interaction with the crawler is the following:

• Crawler marks all existing URLs of this data source for garbage collection, assuming they no longer exist in the target data source.

• Crawler calls the agent to get an updated list of URLs. It marks for crawling every URL that already exists. If it is new, it inserts it into the URL table and queue.

• Crawler deletes the URLs that are still marked for garbage collection.

• Crawler goes through every URL marked for crawling and checks for updates.

Smart Agent

The smart agent uses a modified-since time stamp (provided by the crawler) to return the list of URLs that have been updated, inserted, and deleted. The crawler only crawls URLs returned by the agent and does not recrawl existing ones. For URLs that were deleted, the crawler removes them from the URL table. If the smart agent can only return updated or inserted URLs but not deleted URLs, then deleted URLs are not detected by the crawler. In this case, you must change the schedule crawler recrawl policy to periodically run the schedule in force recrawl mode. Force recrawl mode signals to the agent to return every URL in the data source.

The agent API isDeltaCrawlingCapable, it tells the crawler whether the agent it invokes is a standard agent or a smart agent. The agent API startCrawling(boolean forceRecrawl, Date lastCrawlTime) allows the crawler to tell the agent that the last crawl time and whether the crawler is running in force recrawl mode.

Document Attributes and Properties

Document attributes, or metadata, describe document properties. Some attributes can be irrelevant to your application. The crawler agent creator must decide which document attributes should be extracted and saved. The agent also can be created such that the list of collected attributes are configurable. Oracle Ultra Search automatically registers attributes returned by the agent. The agent can decide which attributes to return for a document.

Library Path and Java Class Path

Any other Java class needed by the agent should be included in the agent jar file. This is because Oracle Ultra Search automatically adds the agent jar file to the crawler Java class path, and Oracle Ultra Search does not allow you to add other class paths from the administration interface. To add a new class path, see Appendix B, "Altering the Crawler Java Classpath"

If the agent code also relies on a particular library file (for example, a .ddl file on Windows or a .so file on UNIX), then the library path environment variable (PATH on Windows, LD_LIBRARY_PATH on UNIX) must contain the path to it. Make sure that Oracle is started from this environment. As the crawler is spawned by the Oracle process, it automatically inherits all environment variables from Oracle, including the library path.

Data Source Type Registration

A data source type is an abstraction of a data source. You can define new data source types with the following attributes:

• Name of data source type: For example, Lotus Notes. The name cannot be more than 100 bytes.

• ID of data source type: This is automatically assigned.

• Description of the data source type: This limit is 4000 bytes.

• Agent Java class name: For example, WebDbAgent. The location of this class is predefined by Oracle Ultra Search in $ORACLE_HOME/ultrasearch/lib/agent/ and cannot be changed.

• Agent Java jar file name: The agent class can be stored in a Java jar file. This jar file must be in $ORACLE_HOME/ultrasearch/lib/agent/, where $ORACLE_HOME is the Oracle home directory where the Oracle Ultra Search backend, not the middle tier, is installed.

• Parameters: Parameters are the properties of a data source; for example, seed URL, inclusion pattern, and robots exclusion for a Web data source. Define a parameter by specifying a parameter name (100 bytes maximum) and a description (4000 bytes maximum). By default, a parameter is not encrypted.

• Encryption: Should the value of this parameter be encrypted when stored.

Oracle Ultra Search does not enforce the occurrence of parameters. You cannot specify a particular parameter to have 0 or more, at least 1, or only 1 occurrence.

META-INF/MANIFEST.MForacle/marketing/search/crawleragent/SearchCrawlerAgent.java + other class fileslog4j-1.2.8.jar

Manifest-Version: 1.0Created-By: Praveen DevulapalliMain-Class: oracle.marketing.search.crawleragent.SearchCrawlerAgentClass-Path: log4j-1.2.8.jar

Data Source Registration

After a data source type is defined, any instance of that data source type can be defined:

• Data source name

• Description of the data source, limit to 4000 bytes

• Data source type ID

• Default language; default is 'en' (English)

• Parameter values, for example, seed - http://www.oracle.com depth - 8

Data Source Attribute Registration

You can add new attributes to Oracle Ultra Search by providing the attribute name and the attribute data type. The data type can be string, number, or date. Attributes with the same name but different data type can be added. Attributes returned by an agent are automatically registered if they have not been defined.

User-Implemented Crawler Agent

The crawler agent has the following requirements:

• The agent must be implemented in Java.

• The agent must support the Java agent APIs defined by Oracle Ultra Search.

• The agent must return the URL attributes and properties.

• The agent optionally can authenticate the crawler's access to the data source.

• The agent must "flatten" the data source such that each document is retrieved one by one in a streaming fashion. This is to encapsulate the crawling logic of a specific data source into the agent.

• The agent must decide which document attributes Oracle Ultra Search should keep. Any attribute not defined in Oracle Ultra Search is registered automatically.

• The agent can map attributes to data source properties. For example, if an attribute "ID" is the unique ID of a document, then the agent should return (document_key, 4) where "ID" has been mapped to the property "document_key" and its value is 4 for this particular document.

• If the attribute LOV is available, then the agent returns them upon request.

Interaction Between the Crawler and the Crawler Agent

The crawler crawls data sources defined by the user through the invocation of the user-supplied crawler agent. The crawler can do the following:

• Invoke the crawler agent of the defined data source

• Supply data source parameter information to the agent

• Authenticate itself with the agent if needed

• Retrieve a list of URLs and associate attributes/properties that must be crawled

• Use the URL provided by the agent to retrieve the document

• Detect insert, update, and delete to the data source

• Retrieve attribute LOV data if available

Crawler Agent APIs and Classes

The crawler agent API is a collection of methods used to implement a crawler agent. A sample implementation of a crawler agent SampleAgent.java is provided under $ORACLE_HOME/ultrasearch/extension/.

UrlData: The crawler agent uses this interface to populate document properties and attribute values. Oracle Ultra Search provides a basic implementation of this interface that the agent can use directly or extend if necessary. The class is DocAttributes with a constructor that has no argument. The agent might decide to create a pool of UrlData objects and cycle through them during crawling. In the most simple implementation, the agent creates one DocAttributes object, repeatedly resets and populates the data, and returns this object.

LovInfo: The crawler agent uses this interface to submit attribute LOV definitions.

DataSourceParams: The crawler agent uses this interface to read and write data source parameters.

AgentException: The crawler agent uses this exception class when an error occurs.

CrawlerAgent: This interface lets the crawler communicate with the user-defined data source. The crawler agent must implement this interface.

Compiling and Building the Agent Jar File

The Java source code for the sample agent first must be compiled into class files and put into a jar file in the $ORACLE_HOME/ultrasearch/lib/agent/ directory, where $ORACLE_HOME is the Oracle home directory where the Oracle Ultra Search backend, not the middle tier, is installed.

The classes needed for compilation are the JDK class (classes.zip), Oracle JDBC Thin Driver (classes12.zip), and ultrasearch.jar. For example:

$ORACLE_HOME/jdk/bin/javac -J-ms16m -J-mx96m -O -classpath $ORACLE_HOME/dbjava/lib/classes12.zip: 
$ORACLE_HOME/ultrasearch/lib/ultrasearch.jar SampleAgent.java 

To build the SampleAgent.jar file, enter the following:

$ORACLE_HOME/jdk/bin/jar cv0f  $ORACLE_HOME/ultrasearch/lib/agent/SampleAgent.jar 
SampleAgent.class 'SampleAgent$DocNode.class' 

Creating a Data Source Type

A data source type that uses the sample agent must be created first.

• Name: URL table type

• Description: Table with rows of URLs

• Agent Name: SampleAgent

• Agent Jar File: sampleagent

Defining Data Source Parameters

Define parameters for a data source type:

• Database Connect String (DB connection)

• User Name (schema owner of the URL table)

• Password (schema owner password, encrypted)

• Table Name (URL table name)

• URL Column (Column holding doc URLs)

• Ignore Flag Column (1 for ignoring, 0 otherwise)

• Language Column (Document Language)

• Attribute List (List of column for attributes)

• It is in the following format: [column name/attribute name] <data type> [column name/attribute name] <data type> ... where <data type> 0 is number, 1 is string, and 2 is date. For example, if the document has 4 attributes: Company Name, Category, Revenue, S&P Rating, then it is specified as: [Company Name/Company/1][Category/Classification/1][Revenue/Revenue/0][Rating/Analyst Rating/1]

• Log File Name (log file)

• Log Directory (Location of log file)

Defining a Data Source of this Type

A data source is defined, which initializes the data source parameters. For example, the value specified accesses a table whose schema is the following:

TABLE NEWS (
    ARTICLE_NO    NUMBER,
    NEWS_URL      VARCHAR2(740),
    TITLE         VARCHAR2(200),
    AUTHOR        VARCHAR2(100),
    PUB_DATE      DATE default SYSDATE,
    PUBLISHER     VARCHAR2(100),
    PRICE         NUMBER,
    LANG          VARCHAR2(10),
    IGNORE        NUMBER DEFAULT 0,
    PRIMARY KEY (NEWS_URL)
    );

• Database Connect String: dlsun1710:5521:search

• User Name: SCOTT

• Password: TIGER

• Table Name: NEWS

• URL Column: NEWS_URL

• Ignore Flag Column: IGNORE

• Language Column: LANG

• Attribute List: [ARTICLE_NO/Article Number/0][TITLE/Article Title/1][AUTHOR/Author/1][PUB_DATE/Report Date/2][PUBLISHER/Newspaper/1][PRICE/Download Cost/0]

• Log File Name: testagent.log

• Log Directory: /tmp/ultrasearch/