![[framework]](nodeframe.png)
This guide describes Hyper Estraier's client/server (C/S) and peer to peer (P2P) architecture. If you haven't read user's guide yet, now is a good moment to do so.
There was several problems which where motivation for C/S architecture. estseek.cgi is inefficient because it reconnects to database for each search query. Database updates using estcmd prevents searches on same database because estcmd uses locks when doing update. To solve those problems, estmaster server process is provided. This is resident (daemon) process which has control over database and provides services via network.
This approach also leads to following advantages:
Protocol between clients and servers is based on HTTP, so normal web browsers can be used as simple clients. Clients can be implemented using any languages which supports HTTP protocol like JavaScript or Flash.
Distributed processing is based on peer to peer (P2P) architecture. This allows horizontal scalability. For example, if you use 10 servers, each with million documents, you can search 10 million documents without much additional effort. Since all servers are equal, search service is provided even if some of servers are unavailable. There is notion of relevance of each index which can improve search results (if some parts of index are more important that others).
This guide describes the node API which can be used by client applications to implement search and update capabilities without using network protocol between client and server. The node API is implemented by not only C language but also Java and Ruby.
This section describes the P2P architecture of Hyper Estraier.
When using multiple indexes, it is highly inefficient to run one server for each index. estmaster is server process (daemon) implemented as single process available on network through HTTP protocol on some hight port that provides services for multiple indexes. This component is called node master. Each index has it's own unique URL which is served by node server. You can think about node server as a virtual servers for each index within one node master.
Client application just need to know URL of node server and issue queries to it on any available node masters. Term node corresponds to peer in P2P architecture. Clients can also connect to node masters directly to manage configuration of that master.
Each node server can have unidirectional links to other node servers. This allows distributed searching called meta search. When client sends query to node server, it will forward that query to all other node servers and merge responses before sending result back to client.
Meta search is performed hierarchically. Loops in links are detected and restrained automatically, so searching is always performed on tree structured network of nodes. This allows infinite scalability just by adding additional nodes.
![[tree of meta search]](metatree.png)
There is a notion of relevance for each link between nodes called credit. It's used for weighting of scores when merging results from different nodes. Document coming from node with larger credit will have higher rank in search results. Application can set links and credits for them, and this allows improving of search precision by modifying credits of frequently used nodes.
Client connections to node master or node server are authenticated using login names and passwords. Users and permissions are defined on node masters and all node servers on that master inherit those permissions. Users are divided into two groups: super users and normal users. Normal users have permission to search index while super users also have permission to create and update indexes. Moreover, for each node, normal users are divided into administrators and guest users.
mod_estraier is the most interesting application of the node API of Hyper Estraier. It works as a module of a Web server (Apache) and register contents mediated by its proxy mechanism. Using it as a forward proxy, you can enjoy a search engine whose targets are documents which you (or your comrades) have seen. Using it as a reverse proxy, you can add search feature to any web application as well as BBS and Wiki. That is, the greatest characteristic of the node API is helping you to develop advanced search systems combining various applications.
Concept of P2P seems difficult at first, so it's best to start with a tutorial which will show commands used to setup search network.
First step is creation of the server root directory which contains configuration files and indexes. Following command will create casket, the server root directory:
estmaster init casket
Next, we need to start node master using:
estmaster start casket
To stop node master you can press Ctrl-C on terminal in which it's running on issue following command:
estmaster stop casket
After starting node master you can access it's administration interface pointing web browser at http://localhost:1978/master_ui. This will bring HTTP authorization dialog requesting user name and password. Enter "admin" and "admin". You will be presented with administration commands menu.
First option is Manage Master which provides options to shutdown node master and to synchronize databases. However, they are not used for now.
Since you are logged as admin which has super user privileges, we can edit users. We will create new account with super user privileges and switch to it. We will also erase admin user because it has well-known password and it's possible security hole.
Select Manage Users. There are input boxes for user name, password, flags, full name and miscellaneous information at bottom. Enter following data for new user: "clint", "tnilc", "s", "Clint Eastwood" and "Dirty Harry". Flag "s" denotes user with super user privileges. User name is limited to alphanumeric characters only.
You can now delete admin user using DELE link and confirm that by clicking on sure button.
Select Manage Nodes next. Since we just erased admin user, you will be again asked for user name and password. Enter "clint" and "tnilc" to login again. Input boxes at bottom allows you to enter new index name and label. Index name is limited to alphanumeric characters. Enter "test1" as name and "First Node" as description. Create another node with name "test2" and "Second Node" as description.
We'll now register new documents in index. For that, we will again use command-line utilities. Since terminal in which you started estmaster is busy outputting log messages, open another one.
Documents which are about to be registered in index must be in document draft format described in User's Guide. Create file data001.est with following content:
@uri=data001 @title=Material Girl Living in a material world And I am a material girl You know that we are living in a material world And I am a material girl
We are going to register this document to node test1. Since updating of index needs super user permissions, we will use -auth option to specify user name and password of user that we created in previous step.
estcall put -auth clint tnilc http://localhost:1978/node/test1 data001.est
Document registration will finish shortly, and absence of any error message denotes that it has been successful.
To demonstrate meta search mentioned before, we will register another document to node test2. Create file data002.est with following content:
@uri=data002 @title=Liberian Girl Liberian girl You came and you changed My world A love so brand new
Then, register document using:
estcall put -auth clint tnilc http://localhost:1978/node/test2 data002.est
As you can see, it's possible to register documents from any machine connected to network on which estmaster is running. You can register few additional documents if you want now.
Next step is to find some of documents we just registered. You can try search by executing:
estcall search http://localhost:1978/node/test1 "material world"
This will produce search result with draft of the document including phrase material world.
We can also create a link between two nodes and try meta search. We need super user, URL of source and destination node, link name and credit.
estcall setlink -auth clint tnilc http://localhost:1978/node/test1 \ http://localhost:1978/node/test2 TEST02 8000
This command creates link from node test1 to node test2 called TEST02 and assign it credit of 8000.
You can now repeat search and add option -dpt which will activate meta search over both nodes.
estcall search -dpt 1 http://localhost:1978/node/test1 "girl"
Although your search is accessing node test1, we can see merged results from test2. Since nodes can be located on separate machines, meta search enables us to do distributed processing using P2P architecture described before.
We can also influence ranking of results by increasing credit for particular node. If we change it to 12000 and re-run search we will see different ranking of results.
estcall setlink -auth clint tnilc http://localhost:1978/node/test1 \ http://localhost:1978/node/test2 TEST02 12000
Results can also be returned in XML format which is defined in estresult.dtd.
estcall search -dpt 1 -vx http://localhost:1978/node/test1 "girl"
Each node server also has a web search interface. You can go to http://localhost:1978/node/test1/search_ui and access search interface of node test1.
estmaster is provided as a command to manage the node master. This section describes how to use estmaster.
estmaster is an aggregation of sub commands. The name of a sub command is specified by the first argument. Other arguments are parsed according to each sub command. The argument rootdir specifies the server root directory which contains configuration file and so on.
All sub commands return 0 if the operation is success, else return 1. A running node master finishes with closing the database when it catches the signal 1 (SIGHUP), 2 (SIGINT), 3 (SIGQUIT), or 15 (SIGTERM). Moreover, when a node master running as a daemon catches the signal 1 (SIGHUP), the process is re-start and re-read the configuration files.
A running node server should be finished by valid means by command line or via network. Otherwise, the index may be broken.
The server root directory contains the following files and directories.
The prime configuration file can be edit with a text editor. However, the user account file should not be edit during the node master is running.
If you have an index created by estcmd, move it into the node directory and reboot the server. So, the index will work as a node.
The prime configuration file is composed of lines and the name of an variable and the value separated by ":" are in each line. By default, the following configuration is there.
bindaddr: 0.0.0.0
portnum: 1978
publicurl:
runmode: 1
authmode: 2
recvmax: 1024
maxconn: 30
idleflush: 20
idlesync: 300
sessiontimeout: 600
searchtimeout: 15
searchmax: 1000
searchdepth: 5
rateuri: 1
mergemethod: 2
proxyhost:
proxyport:
logfile: _log
loglevel: 2
backupcmd:
scalepred: 2
scoreexpr: 2
attrindex: @mdate{{!}}seq
attrindex: @title{{!}}str
docroot:
indexfile:
trustednode:
denyuntrusted: 0
cachesize: 64
cacheanum: 8192
cachetnum: 1024
cachernum: 256
specialcache:
helpershift: 0.9
wildmax: 256
limittextsize: 128
snipwwidth: 480
sniphwidth: 96
snipawidth: 96
scancheck: 1
smlrvnum: 32
extdelay: 4096
adminemail: magnus@hyperestraier.gov
uireplace: ^file:///home/mikio/public_html/{{!}}http://localhost/
uireplace: /index\.html?${{!}}/
uiextattr: @author|Author
uiextattr: @mdate|Modification Date
uiphraseform: 2
uismlrtune: 16 1024 4096
Meaning of each variable is the following.
{{!}}". This can be more than once.|". This can be more than once.The user account file is composed of lines and each includes the name, the encrypted password, the flags, the full name, and the miscellaneous information separated by tabs. The character encoding is UTF-8. By default, the following account is there.
admin 21232f297a57a5a743894a0e4a801fc3 s Carolus Magnus Administrator
The password is expressed as MD5 hash value. In the flags, "s" is for super users, and "b" is for banned users. Flags, full name, and miscellaneous information can be omitted.
By accessing the absolute URL "/master_ui" of the node master with a web browser, you can use the administration interface. It requires authentication as a super user.
By accessing the URL which is the URL of a node server followed by "/search_ui", you can use the search interface. There are some anchors with such labels as "LINK#1" in the result page. They express nodes linked from the current node. If you select one of them, you move to the node and then search with the current conditions are performed.
If you select the link "Atom" or "RSS" in the result page, you get the search result in Atom or RSS format. By registering the URL to an application supporting Atom 1.0 or RSS 1.0, you can monitor the search result periodically. In order to work together with such site as Google, Wikipedia and so on, Hyper Estraier supports OpenSearch 1.1. To get the description for OpenSearch, access the URL of the node server trailing "/opensearch".
Communication between nodes and communication between clients and nodes are carried out by a protocol based on HTTP. This section describes the protocol.
The node master and node servers implement HTTP/1.0. As for now, such particular features of HTTP/1.1 as keep-alive connection, chunked encoding, and content negotiation are not supported.
While both of GET and POST are allowed for the request method of HTTP, GET is preferred if the command retrieves information, POST is preferred if the command update the node master or a node server. As the character encoding of parameters is UTF-8, meta characters and multi-byte characters should be escaped by URL encoding (application/x-www-form-urlencoded). The maximum length of data sent with the GET method is 8000. Authentication information is passed in the basic authentication mechanism of HTTP.
If an operation is done successfully, the status code 200 or 202 is returned. On error, one of the following status code is returned.
The result of operation of search or retrieve is sent as message body of response. As the format of the data is plain text whose encoding is UTF-8, it can be structured with tabs and line feeds. If a client supports deflate encoding, compressed data is sent.
To operate the node master, connect to the path "/master" of the server. For example, if the host name is "skyhigh.estraier.go.jp" and the port number is 8888, connect to "http://skyhigh.estraier.go.jp:8888/master". Only super users are granted to operate the node master. There are some sub commands for operations of the node master. The name of a sub command is specified by the parameter "action". Other parameters vary according to each sub command.
To operate a node server, connect to a path which begins "/node/" and is followed by the name of the node. For example, if the host name is "skyhigh.estraier.go.jp" and the port number is 8888 and the name of the node is "foo", connect to "http://skyhigh.estraier.go.jp:8888/node/foo". There are some sub commands for operations of node servers. The name of a sub command is specified after the node name. Parameters vary according to each sub command.
attr1 to attr9 works as with it. The format is as with the one of the core API.mask0=on to mask9=on.Note that while super users has permission to administrate all nodes, an administrator of a node may not be a super user. Moreover, setting of guests of each node have meaning only when the authorization mode is 3 (all).
The format of the entity body of response of search command is alike to multipart of MIME. The following is an example.
--------[2387AD2E34554FFF]-------- VERSION 1.0 NODE http://localhost:1978/node/sample1 HIT 2 HINT#1 give 2 DOCNUM 2 WORDNUM 31 TIME 0.006541 TIME#i 0.000058 TIME#0 0.002907 TIME#1 0.001578 LINK#0 http://localhost:1978/node/sample1 Sample1 10000 2 31 2731304 2 LINK#1 http://localhost:1978/node/sample2 Sample2 4000 3 125 8524522 1 VIEW SNIPPET --------[2387AD2E34554FFF]-------- #nodelabel=Sample Node One #nodescore=7823432 #nodeurl=http://localhost:1978/node/sample1 @id=1 @uri=http://localhost/foo.html %VECTOR give 8502 dispose 7343 griefs 5932 king 2343 void 1232 You may my glories and my state dispose, But not my griefs; still am I king of those. ( Give give it u p, Yo! Give give it up, Yo!) --------[2387AD2E34554FFF]-------- #nodelabel=Sample Node One #nodescore=5623772 #nodeurl=http://localhost:1978/node/sample1 @id=2 @uri=http://localhost/bar.html %VECTOR faster 9304 give 7723 griefs 6632 go 5343 you 3289 The faster I go, the behinder I get. ( Give give it up, Yo! Give give it up, Yo!) --------[2387AD2E34554FFF]--------:END
Each line feed is a single LF. The first line is definition of the border string. Each parts are delimited by the border string. The last border string is followed by ":END". The first part is the meta section. The other parts are document sections.
The format of the meta section is TSV. Meaning of each string is picked out by the first field. There are the following kinds.
TIME#i" means elapsed time for the local inverted index only.LINK#0" means the node it self.SNIPPET".Each document part expresses attributes and a snippet of a document. Top lines to the first empty line expresses attributes. Their format is as with the one of document draft. If keywords are attached to the document, they are expressed with the "%VECTOR" control command. If score is specified explicitly, it is expressed with the "%SCORE" control command. The format of the snippet is TSV. There are tab separated values. Each line is a string to be shown. Though most lines have only one field, some lines have two fields. If the second field exists, the first field is to be shown with highlighted, and the second field means its normalized form.
The following pseudo-attributes are added to each result documents of the search command or the get_doc command includes.
The format of the entity body of response of list command is in TSV format. The following is an example though some strings follows "..." actually.
181 http://localhost/data/ihaveadream.xml 31e51df5f33131943dda22bd0fd755a0 ... 1 http://localhost/prog/hyperestraier-1.0.2/doc/index.html 45368fa3c... 2 http://localhost/prog/hyperestraier-1.0.2/doc/index.ja.html 0e9edf4ae... 3 http://localhost/prog/hyperestraier-1.0.2/doc/intro-en.html ec622d19a... 18 http://localhost/prog/hyperestraier-1.0.2/doc/intro-ja.html 96f743fa6... 5 http://localhost/prog/hyperestraier-1.0.2/doc/javanativeapi/allclasses-fr... 25 http://localhost/prog/hyperestraier-1.0.2/doc/javanativeapi/allclasses-no... 26 http://localhost/prog/hyperestraier-1.0.2/doc/javanativeapi/constant-valu... 20 http://localhost/prog/hyperestraier-1.0.2/doc/javanativeapi/estraier/Cmd.... 1022 http://localhost/prog/hyperestraier-1.0.2/doc/javanativeapi/estraier/Docu...
Each line feed is a single LF. Each line expresses each document and has 14 fields of system attributes. They are "@id", "@uri", "@digest", "@cdate", "@mdate", "@adate", "@title", "@author", "@type", "@lang", "@genre", "@size", "@weight", and "@misc". Fields of undefined attributes are empty strings.
Because URL encoding is not efficient as for large data sent for the put_doc command and edit_doc command, the raw mode is supported. If the value of "Content-Type" is "text/x-estraier-draft", the entity body is treated as a document draft itself. The following is an example.
POST /node/foo/put_doc HTTP/1.0 Content-Type: text/x-estraier-draft Content-Length: 138 @uri=http://gogo.estraier.go.jp/sample.html @title=Twinkle Twinkle Little Star Twinkle, twinkle, little star, How I wonder what you are.
As it is a bother to implement HTTP, the node API is useful. This section describes how to use the node API.
Using the node API, you can implement clients communicating node severs without considering such low level processing as TCP/IP and HTTP. Though the node API has overhead comparing to the core API, it is important to be able to execute at remote host and to perform parallel processing without discrimination of readers and writers.
In each source of applications of the node API, include `estraier.h', `estnode.h', `cabin.h', and `stdlib.h'.
#include <estraier.h> #include <estnode.h> #include <cabin.h> #include <stdlib.h>
To build an application, perform the following command. It is same as with the core API.
gcc `estconfig --cflags` -o foobar foobar.c `estconfig --ldflags` `estconfig --libs`
Because the node API uses features of the core API also, if you have never read the programming guide, please read it beforehand.
For preparation to use the node API, initialize the network environment at the beginning of a program. Moreover, the environment should be freed at the end of the program.
The function `est_init_net_env' is used in order to initialize the networking environment.
The function `est_free_net_env' is used in order to free the networking environment.
The type of the structure `ESTNODE' is for abstraction of connection to a node. A node has its own URL. No entity of `ESTNODE' is accessed directly, but it is accessed by the pointer. The term of node connection object means the pointer and its referent. A node connection object is created by the function `est_node_new' and destroyed by `est_node_delete'. Every created node connection object should be destroyed.
The following is a typical use case of node connection object.
ESTNODE *node;
/* create a node connection object */
node = est_node_new("http://estraier.gov:1978/node/foo");
/* set the proxy, the timeout, and the authentication */
est_node_set_proxy(node, "proxy.qdbm.go.jp", 8080);
est_node_set_timeout(node, 5);
est_node_set_auth(node, "mikio", "oikim");
/* register documents or search for documents here */
/* destroy the object */
est_node_delete(node);
The function `est_node_new' is used in order to create a node connection object.
The function `est_node_delete' is used in order to destroy a node connection object.
The function `est_node_set_proxy' is used in order to set the proxy information of a node connection object.
The function `est_node_set_timeout' is used in order to set timeout of a connection.
The function `est_node_set_auth' is used in order to set the authentication information of a node connection object.
The function `est_node_status' is used in order to get the status code of the last request of a node.
The function `est_node_sync' is used in order to synchronize updating contents of the database of a node.
The function `est_node_optimize' is used in order to optimize the database of a node.
The function `est_node_put_doc' is used in order to add a document to a node.
The function `est_node_out_doc' is used in order to remove a document from a node.
The function `est_node_out_doc_by_uri' is used in order to remove a document specified by URI from a node.
The function `est_node_edit_doc' is used in order to edit attributes of a document in a node.
The function `est_node_get_doc' is used in order to retrieve a document in a node.
The function `est_node_get_doc_by_uri' is used in order to retrieve a document specified by URI in a node.
The function `est_node_get_doc_attr' is used in order to retrieve the value of an attribute of a document in a node.
The function `est_node_get_doc_attr_by_uri' is used in order to retrieve the value of an attribute of a document specified by URI in a node.
The function `est_node_etch_doc' is used in order to extract keywords of a document.
The function `est_node_etch_doc_by_uri' is used in order to extract keywords of a document specified by URI in a node.
The function `est_node_uri_to_id' is used in order to get the ID of a document specified by URI.
The function `est_node_name' is used in order to get the name of a node.
The function `est_node_label' is used in order to get the label of a node.
The function `est_node_doc_num' is used in order to get the number of documents in a node.
The function `est_node_word_num' is used in order to get the number of unique words in a node.
The function `est_node_size' is used in order to get the size of the database of a node.
The function `est_node_cache_usage' is used in order to get the usage ratio of the cache of a node.
The function `est_node_admins' is used in order to get a list of names of administrators of a node.
The function `est_node_users' is used in order to get a list of names of users of a node.
The function `est_node_links' is used in order to get a list of expressions of links of a node.
The function `est_node_search' is used in order to search a node for documents corresponding a condition.
The function `est_node_set_snippet_width' is used in order to set width of snippet in the result from a node.
The function `est_node_set_user' is used in order to manage a user account of a node.
The function `est_node_set_link' is used in order to manage a link of a node.
The type of the structure `ESTNODERES' is for abstraction of search result from a node. A result is composed of a list of corresponding documents and information of hints. No entity of `ESTNODERES' is accessed directly, but it is accessed by the pointer. The term of node result object means the pointer and its referent. A node result object is created by the function `est_node_search' and destroyed by `est_noderes_delete'. Every created node connection object should be destroyed.
The type of the structure `ESTRESDOC' is for abstraction of a document in search result. A result document is composed of some attributes and a snippet. No entity of `ESTRESDOC' is accessed directly, but it is accessed by the pointer. The term of result document object means the pointer and its referent. A result document object is gotten by the function `est_noderes_get_doc' but it should not be destroyed because the entity is managed inside the node result object.
The following is a typical use case of node connection object and result document object.
ESTNODERES *nres;
CBMAP *hints;
ESTRESDOC *rdoc;
int i;
/* create a node result object */
nres = est_node_search(node, cond, 1);
/* get hints */
hints = est_noderes_hints(nres);
/* show the hints here */
/* scan documents in the result */
for(i = 0; i < est_noderes_doc_num(nres); i++){
/* get a result document object */
rdoc = est_noderes_get_doc(nres, i);
/* show the result document object here */
}
/* destroy the node result object */
est_noderes_delete(nres);
The function `est_noderes_delete' is used in order to delete a node result object.
The function `est_noderes_hints' is used in order to get a map object for hints of a node result object.
The function `est_noderes_eclipse' is used in order to eclipse similar documents of a node result object.
The function `est_noderes_doc_num' is used in order to get the number of documents in a node result object.
The function `est_noderes_get_doc' is used in order to refer a result document object in a node result object.
The function `est_resdoc_uri' is used in order to get the URI of a result document object.
The function `est_resdoc_attr_names' is used in order to get a list of attribute names of a result document object.
The function `est_resdoc_attr' is used in order to get the value of an attribute of a result document object.
The function `est_resdoc_snippet' is used in order to get the snippet of a result document object.
The function `est_resdoc_keywords' is used in order to get keywords of a result document object.
The function `est_resdoc_shadows' is used in order to get an array of documents eclipsed by a result document object.
The function `est_resdoc_similarity' is used in order to get similarity of an eclipsed result document object.
Each of node connection objects, node result objects, and result document objects can not be shared by threads. If you use multi threads, make each thread have its own objects. If the precondition is kept, functions of the node API can be treated as thread-safe functions.
The following is the simplest implementation of a gatherer.
#include <estraier.h>
#include <estnode.h>
#include <cabin.h>
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv){
ESTNODE *node;
ESTDOC *doc;
/* initialize the network environment */
if(!est_init_net_env()){
fprintf(stderr, "error: network is unavailable\n");
return 1;
}
/* create and configure the node connection object */
node = est_node_new("http://localhost:1978/node/test1");
est_node_set_auth(node, "admin", "admin");
/* create a document object */
doc = est_doc_new();
/* add attributes to the document object */
est_doc_add_attr(doc, "@uri", "http://estraier.gov/example.txt");
est_doc_add_attr(doc, "@title", "Over the Rainbow");
/* add the body text to the document object */
est_doc_add_text(doc, "Somewhere over the rainbow. Way up high.");
est_doc_add_text(doc, "There's a land that I heard of once in a lullaby.");
/* register the document object to the node */
if(!est_node_put_doc(node, doc))
fprintf(stderr, "error: %d\n", est_node_status(node));
/* destroy the document object */
est_doc_delete(doc);
/* destroy the node object */
est_node_delete(node);
/* free the networking environment */
est_free_net_env();
return 0;
}
The following is the simplest implementation of a searcher.
#include <estraier.h>
#include <estnode.h>
#include <cabin.h>
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv){
ESTNODE *node;
ESTCOND *cond;
ESTNODERES *nres;
ESTRESDOC *rdoc;
int i;
const char *value;
/* initialize the network environment */
if(!est_init_net_env()){
fprintf(stderr, "error: network is unavailable\n");
return 1;
}
/* create the node connection object */
node = est_node_new("http://localhost:1978/node/test1");
/* create a search condition object */
cond = est_cond_new();
/* set the search phrase to the search condition object */
est_cond_set_phrase(cond, "rainbow AND lullaby");
/* get the result of search */
nres = est_node_search(node, cond, 0);
if(nres){
/* for each document in the result */
for(i = 0; i < est_noderes_doc_num(nres); i++){
/* get a result document object */
rdoc = est_noderes_get_doc(nres, i);
/* display attributes */
if((value = est_resdoc_attr(rdoc, "@uri")) != NULL)
printf("URI: %s\n", value);
if((value = est_resdoc_attr(rdoc, "@title")) != NULL)
printf("Title: %s\n", value);
/* display the snippet text */
printf("%s", est_resdoc_snippet(rdoc));
}
/* delete the node result object */
est_noderes_delete(nres);
} else {
fprintf(stderr, "error: %d\n", est_node_status(node));
}
/* destroy the search condition object */
est_cond_delete(cond);
/* destroy the node object */
est_node_delete(node);
/* free the networking environment */
est_free_net_env();
return 0;
}
estcall is provided as a client command to manage the node server. This section describes how to use estcall.
estcall is an aggregation of sub commands. The name of a sub command is specified by the first argument. Other arguments are parsed according to each sub command. The argument nurl specifies the URL of a node. The option -proxy specifies the host name and the port number of a proxy server. The option -tout specifies timeout in seconds. The option -auth specifies the user name and the password of authentication information.
All sub commands return 0 if the operation is success, else return 1.
Operations for the node maser itself is not provided as APIs, use the raw sub command for that purpose. For example, the following command is used in order to shutdown the node master.
estcall raw -auth admin admin \ 'http://localhost:1978/master?action=shutdown'
In order to add a user, perform the following command.
estcall raw -auth admin admin \ 'http://localhost:1978/master?action=useradd&name=mikio&passwd=iloveyou'
In order to use POST method, perform the following command.
echo -n 'action=useradd&name=mikio&passwd=iloveyou' |
estcall raw -auth admin admin \
-eh 'Content-Type: application/x-www-form-urlencoded' \
'http://localhost:1978/master' -
estfraud.cgi is a CGI script working as a node master.
Running a node master bothers you to write start-up and shutdown scripts. And, it may be forbidden to run any daemon process on some shared servers. In that case, pseudo node master is useful. You can publish some indexes as node servers, on any environment where a web server runs and CGI scripts are available. Such virtual node server working on a pseudo master is called pseudo node server. Each pseudo node server can be searched with the node API.
Pseudo node master is useful in case that the number of node is enormous. Different from usual node server which regidents with keeping file descriptios, pseudo node server connects the database on demand of each request. So, file descriptors are not exhausted. Actually, the maximum number of node servers running on a usual node master is about 30. However, a pseudo node master can handle one thousand or more pseudo node servers.
To use pseudo node master, deploy a CGI script estfraud.cgi and a configuration file estfraud.conf into a directory where CGI scripts are available.
The configuration file is composed of lines and the name of an variable and the value separated by ":" are in each line. Lines leaded by "#" are ignored as comments. By default, the following configuration is there.
indexdir: . runmode: 2 pidxsuffix: -pidx pidxdocmax: 256 pidxdocmin: 0 lockindex: 0 searchmax: 1000 rateuri: 1 mergemethod: 2 scoreexpr: 2 wildmax: 256 snipwwidth: 480 sniphwidth: 96 snipawidth: 96 scancheck: 1 smlrvnum: 32 extdelay: 4096
Means of each variable is the following.
For example, if the value of indexdir is "/home/mikio/myindex" and the URL of estfraud.cgi is "http://abc.def/ghi/estfraud.cgi", the URL "http://abc.def/ghi/estfraud.cgi/foo" is to access the index "/home/mikio/myindex/foo".
The following conditions must be satisfied for updating indexes by node servers.
runmode: 1").-pidx".If documents are registered via node server, they are stored in a pseudo index. When the number of files in the pseudo index reaches the number specified by pidxdocmax, content of the pseudo index is merged into the index and the pseudo index become empty. Due to the mechanism, the CGI script which does not resident as daemon can update the index. If pidxdocmax is set larger, faster the updating process performs, but slower the search process performs. To merge the pseudo index into the index explicitly, call the sync method (est_node_sync).
As pseudo node server does not support query relay to other nodes, the depth parameter of search query is ignored. Moreover, pseudo node server does not support changing the URI attribute by document editing (est_node_doc_edit).
estscout.cgi is a gateway for intersection meta search and estsupt.cgi is a gateway for union meta search.
In order to retrieve intersection set of search results from plural indexes by using such identifiers as URI attribute, the intersection meta search gateway is useful. For example, you can search a staff register for records whose name and address match some full-text search conditions, by using URI attribute of staff number as identifier. Moreover, the union meta search gateway is provided in order to get union set of search results from plural intersection meta search gateways.
The two gateways are implemented as CGI scripts and used with parameters specified by URL. No such abstraction interface as API is provided. Because meta search is performed by multi thread or multi process, improvement of scalability for distributed processing is expected.
![[gateways]](gateframe.png)
To use the intersection meta search gateway, deploy a CGI script estscout.cgi and a configuration file estscout.conf into a directory where CGI scripts are available.
The configuration file is composed of lines and the name of an variable and the value separated by ":" are in each line. By default, the following configuration is there.
indexname: casket-1
indexname: casket-2
indexname: casket-3
lockindex: 1
condgstep: 2
dotfidf: true
scancheck: 3
phraseform: 2
wildmax: 256
stmode: 0
idattr: @uri
idsuffix:
ordexpr: @uri STRA
dupcheck: 0
union: 0
tmpdir: /tmp
cclife: 300
logfile:
logformat: {time}\t{REMOTE_ADDR}:{REMOTE_PORT}\t{cond}\t{hnum}\n
Means of each variable is the following.
The intersection meta search gateway receives the following parameters. The order expression is not supported.
In the output, the first line specifies the approximation number of corresponding documents. Each of the subsequent lines specifies identifier and score of a corresponding document. For example, the following is output.
256 file:///home/mikio/tako.html 12561 file:///home/mikio/ika.html 11624 file:///home/mikio/uni.html 9232 file:///home/mikio/kani.html 8293 file:///home/mikio/ebi.html 8312
To use the union meta search gateway, deploy a CGI script estsupt.cgi and a configuration file estsupt.conf into a directory where CGI scripts are available.
The configuration file is composed of lines and the name of an variable and the value separated by ":" are in each line. By default, the following configuration is there.
targeturl: http://searcher1/estscout.cgi
targeturl: http://searcher2/estscout.cgi
stmode: 0
tmpdir: /tmp
cclife: 1800
#shareurl: http://merger1/estsupt.cgi
#shareurl: http://merger2/estsupt.cgi
failfile:
logfile:
logformat: {time}\t{REMOTE_ADDR}:{REMOTE_PORT}\t{cond}\t{hnum}\n
Means of each variable is the following.
The format of output and parameters of the union meta search gateway are same with ones of the intersection meta search gateway.