Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet. Though the DBM has high performance, you might bother in case that multiple processes share the same database, or remote processes access the database. Thus, Tokyo Tyrant is provided for concurrent and remote connections to Tokyo Cabinet. It is composed of the server process managing a database and its access library for client applications.
The server features high concurrency due to thread-pool modeled implementation and the epoll/kqueue mechanism of the modern Linux/*BSD kernel. The server and its clients communicate with each other by simple binary protocol on TCP/IP. Protocols compatible with memcached and HTTP are also supported so that almost all principal platforms and programming languages can use Tokyo Tyrant. High availability and high integrity are also featured due to such mechanisms as hot backup, update logging, and replication. The server can embed Lua, a lightweight script language so that you can define arbitrary operations of the database.
Because the server uses the abstract API of Tokyo Cabinet, all of the six APIs: the on-memory hash database API, the on-memory tree database API, the hash API, the B+ tree database API, the fixed-length database API, and the table database API, are available from the client with the common interface. Moreover, the table extension is provided to use specific features of the table database.
As for now, the server works on Linux, FreeBSD, Mac OS X, Solaris only.
Install the latest version of Tokyo Cabinet beforehand and get the package of Tokyo Tyrant.
When an archive file of Tokyo Tyrant is extracted, change the current working directory to the generated directory and perform installation.
Run the configuration script. To enable the Lua extension, add the `--enable-lua' option.
./configure
Build programs.
make
Install programs. This operation must be carried out by the root user.
make install
When a series of work finishes, the following files will be installed.
/usr/local/include/ttutil.h /usr/local/include/tculog.h /usr/local/include/tcrdb.h /usr/local/lib/libtokyotyrant.a /usr/local/lib/libtokyotyrant.so.x.y.z /usr/local/lib/libtokyotyrant.so.x /usr/local/lib/libtokyotyrant.so /usr/local/lib/ttskelmock.so /usr/local/lib/ttskeldir.so /usr/local/lib/ttskelproxy.so /usr/local/lib/ttskelnull.so /usr/local/lib/pkgconfig/tokyotyrant.pc /usr/local/bin/ttserver /usr/local/bin/ttultest /usr/local/bin/ttulmgr /usr/local/bin/tcrtest /usr/local/bin/tcrmttest /usr/local/bin/tcrmgr /usr/local/sbin/ttservctl /usr/local/share/tokyotyrant/... /usr/local/man/man1/... /usr/local/man/man3/...
To test the server, perform the following command. To finish it, press Ctrl-C on the terminal.
ttserver
To test the client connecting to the above running server, perform the following command on another terminal.
make check
The command `ttserver' runs the server managing a database instance. Because the database is treated by the abstract API of Tokyo Cabinet, you can choose the scheme on start-up of the server. Supported schema are on-memory hash database, on-memory tree database, hash database, and B+ tree database. This command is used in the following format. `dbname' specifies the database name. If it is omitted, on-memory hash database is specified.
ttserver [-host name] [-port num] [-thnum num] [-tout num] [-dmn] [-pid path] [-kl] [-log path] [-ld|-le] [-ulog path] [-ulim num] [-uas] [-sid num] [-mhost name] [-mport num] [-rts path] [-rcc] [-skel name] [-mul num] [-ext path] [-extpc name period] [-mask expr] [-unmask expr] [dbname]Options feature the following.
To terminate the server normally, send SIGINT or SIGTERM to the process. It is okay to press Ctrl-C on the controlling terminal. To restart the server, send SIGHUP to the process. If the port number is not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter. This command returns 0 on success, another on failure.
The naming convention of the database is specified by the abstract API of Tokyo Cabinet. If the name is "*", the database will be an on-memory hash database. If it is "+", the database will be an on-memory tree database. If its suffix is ".tch", the database will be a hash database. If its suffix is ".tcb", the database will be a B+ tree database. If its suffix is ".tcf", the database will be a fixed-length database. If its suffix is ".tct", the database will be a table database. Otherwise, this function fails. Tuning parameters can trail the name, separated by "#". Each parameter is composed of the name and the value, separated by "=". On-memory hash database supports "bnum", "capnum", and "capsiz". On-memory tree database supports "capnum" and "capsiz". Hash database supports "mode", "bnum", "apow", "fpow", "opts", "rcnum", "xmsiz", and "dfunit". B+ tree database supports "mode", "lmemb", "nmemb", "bnum", "apow", "fpow", "opts", "lcnum", "ncnum", "xmsiz", and "dfunit". Fixed-length database supports "mode", "width", and "limsiz". Table database supports "mode", "bnum", "apow", "fpow", "opts", "rcnum", "lcnum", "ncnum", "xmsiz", "dfunit", and "idx". The tuning parameter "capnum" specifies the capacity number of records. "capsiz" specifies the capacity size of using memory. Records spilled the capacity are removed by the storing order. "mode" can contain "w" of writer, "r" of reader, "c" of creating, "t" of truncating, "e" of no locking, and "f" of non-blocking lock. The default mode is relevant to "wc". "opts" can contains "l" of large option, "d" of Deflate option, "b" of BZIP2 option, and "t" of TCBS option. "idx" specifies the column name of an index and its type separated by ":". For example, "casket.tch#bnum=1000000#opts=ld" means that the name of the database file is "casket.tch", and the bucket number is 1000000, and the options are large and Deflate.
The command mask expression is a list of command names separated by ",". For example, "out,vanish,copy" means a set of "out", "vanish", and "copy". Commands of the memcached compatible protocol and the HTTP compatible protocol are also forbidden or allowed, related by the mask of each original command. Moreover, there are meta expressions. "all" means all commands. "allorg" means all commands of the original binary protocol. "allmc" means all commands of the memcached compatible protocol. "allhttp" means all commands of the HTTP compatible protocol. "allread" is the abbreviation of `get', `mget', `vsiz', `iterinit', `iternext', `fwmkeys', `rnum', `size', and `stat'. "allwrite" is the abbreviation of `put', `putkeep', `putcat', `putshl', `putnr', `out', `addint', `adddouble', `vanish', and `misc'. "allmanage" is the abbreviation of `sync', `optimize', `copy', `restore', and `setmst'. "repl" means replication as master. "slave" means replication as slave.
The command `ttservctl' is the startup script of the server. It can be called by the RC script of the bootstrap process of the operating system. This command is used in the following format.
ttservctl startttservctl stopttservctl restartttservctl hupThe database is placed as "/var/ttserver/casket.tch". The log and related files are also placed in "/var/ttserver". This command returns 0 on success, another on failure.
The command `ttulmgr' is the utility to export and import the update log. It is useful to filter the update log with such text utilities as `grep' and `sed'. This command is used in the following format. `upath' specifies the update log directory.
ttulmgr export [-ts num] [-sid num] upathttulmgr import upathOptions feature the following.
This command returns 0 on success, another on failure.
The command `tcrtest' is a utility for facility test and performance test. This command is used in the following format. `host' specifies the host name of the server. `rnum' specifies the number of iterations.
tcrtest write [-port num] [-cnum num] [-tout num] [-nr] [-rnd] host rnumtcrtest read [-port num] [-cnum num] [-tout num] [-mul num] [-rnd] hosttcrtest remove [-port num] [-cnum num] [-tout num] [-rnd] hosttcrtest rcat [-port num] [-cnum num] [-tout num] [-shl num] [-dai|-dad] [-ext name] [-xlr|-xlg] host rnumtcrtest misc [-port num] [-cnum num] [-tout num] host rnumtcrtest wicked [-port num] [-cnum num] [-tout num] host rnumtcrtest table [-port num] [-cnum num] [-tout num] [-exp num] host rnumOptions feature the following.
If the port number is not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter. This command returns 0 on success, another on failure.
The command `tcrmttest' is a utility for facility test under multi-thread situation. This command is used in the following format. `host' specifies the host name of the server. `rnum' specifies the number of iterations.
tcrmttest write [-port num] [-tnum num] [-nr] [-rnd] [-ext name] host rnumtcrmttest read [-port num] [-tnum num] [-mul num] hosttcrmttest remove [-port num] [-tnum num] hostOptions feature the following.
If the port number is not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter. This command returns 0 on success, another on failure.
The command `tcrmgr' is a utility for test and debugging of the remote database API and its applications. `host' specifies the host name of the server. `key' specifies the key of a record. `value' specifies the value of a record. `params' specifies the tuning parameters. `dpath' specifies the destination file. `func specifies the name of the function. `arg' specifies the arguments of the function. `file' specifies the input file. `upath' specifies the update log directory. `mhost' specifies the host name of the replication master. `url' specifies the target URL.
tcrmgr inform [-port num] [-st] hosttcrmgr put [-port num] [-sx] [-sep chr] [-dk|-dc|-dai|-dad] host key valuetcrmgr out [-port num] [-sx] [-sep chr] host keytcrmgr get [-port num] [-sx] [-sep chr] [-px] [-pz] host keytcrmgr mget [-port num] [-sx] [-sep chr] [-px] host [key...]tcrmgr list [-port num] [-sep chr] [-m num] [-pv] [-px] [-fm str] hosttcrmgr ext [-port num] [-xlr|-xlg] [-sx] [-sep chr] [-px] host func [key [value]]tcrmgr sync [-port num] hosttcrmgr optimize [-port num] host [params]tcrmgr vanish [-port num] hosttcrmgr copy [-port num] host dpathtcrmgr misc [-port num] [-mnu] [-sx] [-sep chr] [-px] host func [arg...]tcrmgr importtsv [-port num] [-nr] [-sc] host [file]tcrmgr restore [-port num] [-ts num] [-rcc] host upathtcrmgr setmst [-port num] [-mport num] [-ts num] [-rcc] host [mhost]tcrmgr repl [-port num] [-ts num] [-sid num] [-ph] hosttcrmgr http [-ah name value] [-ih] urltcrmgr versionOptions feature the following.
If the port number is not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter. This command returns 0 on success, another on failure.
Remote database is a set of interfaces to use an abstract database of Tokyo Cabinet, mediated by a server of Tokyo Tyrant. See `tcrdb.h' for entire specification.
To use the remote database API, include `tcrdb.h' and related standard header files. Usually, write the following description near the front of a source file.
#include <tcrdb.h>#include <stdlib.h>#include <stdbool.h>#include <stdint.h>Objects whose type is pointer to `TCRDB' are used to handle remote databases. A remote database object is created with the function `tcrdbnew' and is deleted with the function `tcrdbdel'. To avoid memory leak, it is important to delete every object when it is no longer in use.
Before operations to store or retrieve records, it is necessary to connect the remote database object to the server. The function `tcrdbopen' is used to open a database connection and the function `tcrdbclose' is used to close the connection.
The function `tcrdberrmsg' is used in order to get the message string corresponding to an error code.
const char *tcrdberrmsg(int ecode);The function `tcrdbnew' is used in order to create a remote database object.
TCRDB *tcrdbnew(void);The function `tcrdbdel' is used in order to delete a remote database object.
void tcrdbdel(TCRDB *rdb);The function `tcrdbecode' is used in order to get the last happened error code of a remote database object.
int tcrdbecode(TCRDB *rdb);The function `tcrdbtune' is used in order to set the tuning parameters of a hash database object.
bool tcrdbtune(TCRDB *rdb, double timeout, int opts);The function `tcrdbopen' is used in order to open a remote database.
bool tcrdbopen(TCRDB *rdb, const char *host, int port);The function `tcrdbopen2' is used in order to open a remote database with a simple server expression.
bool tcrdbopen2(TCRDB *rdb, const char *expr);The function `tcrdbclose' is used in order to close a remote database object.
bool tcrdbclose(TCRDB *rdb);The function `tcrdbput' is used in order to store a record into a remote database object.
bool tcrdbput(TCRDB *rdb, const void *kbuf, int ksiz, const void *vbuf, int vsiz);The function `tcrdbput2' is used in order to store a string record into a remote object.
bool tcrdbput2(TCRDB *rdb, const char *kstr, const char *vstr);The function `tcrdbputkeep' is used in order to store a new record into a remote database object.
bool tcrdbputkeep(TCRDB *rdb, const void *kbuf, int ksiz, const void *vbuf, int vsiz);The function `tcrdbputkeep2' is used in order to store a new string record into a remote database object.
bool tcrdbputkeep2(TCRDB *rdb, const char *kstr, const char *vstr);The function `tcrdbputcat' is used in order to concatenate a value at the end of the existing record in a remote database object.
bool tcrdbputcat(TCRDB *rdb, const void *kbuf, int ksiz, const void *vbuf, int vsiz);The function `tcrdbputcat2' is used in order to concatenate a string value at the end of the existing record in a remote database object.
bool tcrdbputcat2(TCRDB *rdb, const char *kstr, const char *vstr);The function `tcrdbputshl' is used in order to concatenate a value at the end of the existing record and shift it to the left.
bool tcrdbputshl(TCRDB *rdb, const void *kbuf, int ksiz, const void *vbuf, int vsiz, int width);The function `tcrdbputshl2' is used in order to concatenate a string value at the end of the existing record and shift it to the left.
bool tcrdbputshl2(TCRDB *rdb, const char *kstr, const char *vstr, int width);The function `tcrdbputnr' is used in order to store a record into a remote database object without response from the server.
bool tcrdbputnr(TCRDB *rdb, const void *kbuf, int ksiz, const void *vbuf, int vsiz);The function `tcrdbputnr2' is used in order to store a string record into a remote object without response from the server.
bool tcrdbputnr2(TCRDB *rdb, const char *kstr, const char *vstr);The function `tcrdbout' is used in order to remove a record of a remote database object.
bool tcrdbout(TCRDB *rdb, const void *kbuf, int ksiz);The function `tcrdbout2' is used in order to remove a string record of a remote database object.
bool tcrdbout2(TCRDB *rdb, const char *kstr);The function `tcrdbget' is used in order to retrieve a record in a remote database object.
void *tcrdbget(TCRDB *rdb, const void *kbuf, int ksiz, int *sp);The function `tcrdbget2' is used in order to retrieve a string record in a remote database object.
char *tcrdbget2(TCRDB *rdb, const char *kstr);The function `tcrdbget3' is used in order to retrieve records in a remote database object.
bool tcrdbget3(TCRDB *rdb, TCMAP *recs);The function `tcrdbvsiz' is used in order to get the size of the value of a record in a remote database object.
int tcrdbvsiz(TCRDB *rdb, const void *kbuf, int ksiz);The function `tcrdbvsiz2' is used in order to get the size of the value of a string record in a remote database object.
int tcrdbvsiz2(TCRDB *rdb, const char *kstr);The function `tcrdbiterinit' is used in order to initialize the iterator of a remote database object.
bool tcrdbiterinit(TCRDB *rdb);The function `tcrdbiternext' is used in order to get the next key of the iterator of a remote database object.
void *tcrdbiternext(TCRDB *rdb, int *sp);The function `tcrdbiternext2' is used in order to get the next key string of the iterator of a remote database object.
char *tcrdbiternext2(TCRDB *rdb);The function `tcrdbfwmkeys' is used in order to get forward matching keys in a remote database object.
TCLIST *tcrdbfwmkeys(TCRDB *rdb, const void *pbuf, int psiz, int max);The function `tcrdbfwmkeys2' is used in order to get forward matching string keys in a remote database object.
TCLIST *tcrdbfwmkeys2(TCRDB *rdb, const char *pstr, int max);The function `tcrdbaddint' is used in order to add an integer to a record in a remote database object.
int tcrdbaddint(TCRDB *rdb, const void *kbuf, int ksiz, int num);The function `tcrdbadddouble' is used in order to add a real number to a record in a remote database object.
double tcrdbadddouble(TCRDB *rdb, const void *kbuf, int ksiz, double num);The function `tcrdbext' is used in order to call a function of the script language extension.
void *tcrdbext(TCRDB *rdb, const char *name, int opts, const void *kbuf, int ksiz, const void *vbuf, int vsiz, int *sp);The function `tcrdbext2' is used in order to call a function of the script language extension with string parameters.
char *tcrdbext2(TCRDB *rdb, const char *name, int opts, const char *kstr, const char *vstr);The function `tcrdbsync' is used in order to synchronize updated contents of a remote database object with the file and the device.
bool tcrdbsync(TCRDB *rdb);The function `tcrdboptimize' is used in order to optimize the storage of a remove database object.
bool tcrdboptimize(TCRDB *rdb, const char *params);The function `tcrdbvanish' is used in order to remove all records of a remote database object.
bool tcrdbvanish(TCRDB *rdb);The function `tcrdbcopy' is used in order to copy the database file of a remote database object.
bool tcrdbcopy(TCRDB *rdb, const char *path);The function `tcrdbrestore' is used in order to restore the database file of a remote database object from the update log.
bool tcrdbrestore(TCRDB *rdb, const char *path, uint64_t ts, int opts);The function `tcrdbsetmst' is used in order to set the replication master of a remote database object.
bool tcrdbsetmst(TCRDB *rdb, const char *host, int port, uint64_t ts, int opts);The function `tcrdbsetmst2' is used in order to set the replication master of a remote database object with a simple server expression.
bool tcrdbsetmst2(TCRDB *rdb, const char *expr, uint64_t ts, int opts);The function `tcrdbrnum' is used in order to get the number of records of a remote database object.
uint64_t tcrdbrnum(TCRDB *rdb);The function `tcrdbsize' is used in order to get the size of the database of a remote database object.
uint64_t tcrdbsize(TCRDB *rdb);The function `tcrdbstat' is used in order to get the status string of the database of a remote database object.
char *tcrdbstat(TCRDB *rdb);The function `tcrdbmisc' is used in order to call a versatile function for miscellaneous operations of a remote database object.
TCLIST *tcrdbmisc(TCRDB *rdb, const char *name, int opts, const TCLIST *args);The function `tcrdbtblput' is used in order to store a record into a remote database object.
bool tcrdbtblput(TCRDB *rdb, const void *pkbuf, int pksiz, TCMAP *cols);The function `tcrdbtblputkeep' is used in order to store a new record into a remote database object.
bool tcrdbtblputkeep(TCRDB *rdb, const void *pkbuf, int pksiz, TCMAP *cols);The function `tcrdbtblputcat' is used in order to concatenate columns of the existing record in a remote database object.
bool tcrdbtblputcat(TCRDB *rdb, const void *pkbuf, int pksiz, TCMAP *cols);The function `tcrdbtblout' is used in order to remove a record of a remote database object.
bool tcrdbtblout(TCRDB *rdb, const void *pkbuf, int pksiz);The function `tcrdbtblget' is used in order to retrieve a record in a remote database object.
TCMAP *tcrdbtblget(TCRDB *rdb, const void *pkbuf, int pksiz);The function `tcrdbtblsetindex' is used in order to set a column index to a remote database object.
bool tcrdbtblsetindex(TCRDB *rdb, const char *name, int type);The function `tcrdbtblgenuid' is used in order to generate a unique ID number of a remote database object.
int64_t tcrdbtblgenuid(TCRDB *rdb);The function `tcrdbqrynew' is used in order to create a query object.
RDBQRY *tcrdbqrynew(TCRDB *rdb);The function `tcrdbqrydel' is used in order to delete a query object.
void tcrdbqrydel(RDBQRY *qry);The function `tcrdbqryaddcond' is used in order to add a narrowing condition to a query object.
void tcrdbqryaddcond(RDBQRY *qry, const char *name, int op, const char *expr);The function `tcrdbqrysetorder' is used in order to set the order of a query object.
void tcrdbqrysetorder(RDBQRY *qry, const char *name, int type);The function `tcrdbqrysetlimit' is used in order to set the limit number of records of the result of a query object.
void tcrdbqrysetlimit(RDBQRY *qry, int max, int skip);The function `tcrdbqrysearch' is used in order to execute the search of a query object.
TCLIST *tcrdbqrysearch(RDBQRY *qry);The function `tcrdbqrysearchout' is used in order to remove each record corresponding to a query object.
bool tcrdbqrysearchout(RDBQRY *qry);The function `tcrdbqrysearchget' is used in order to get records corresponding to the search of a query object.
TCLIST *tcrdbqrysearchget(RDBQRY *qry);The function `tcrdbqryrescols' is used in order to get columns of a record in a search result.
TCMAP *tcrdbqryrescols(TCLIST *res, int index);The function `tcrdbqrysearchcount' is used in order to get the count of corresponding records of a query object.
int tcrdbqrysearchcount(RDBQRY *qry);The function `tcrdbqryhint' is used in order to get the hint string of a query object.
const char *tcrdbqryhint(RDBQRY *qry);The function `tcrdbmetasearch' is used in order to retrieve records with multiple query objects and get the set of the result.
TCLIST *tcrdbmetasearch(RDBQRY **qrys, int num, int type);The function `tcrdbparasearch' is used in order to search records for multiple servers in parallel.
TCLIST *tcrdbparasearch(RDBQRY **qrys, int num);The following code is an example to use a remote database.
#include <tcrdb.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stdint.h>
int main(int argc, char **argv){
TCRDB *rdb;
int ecode;
char *value;
/* create the object */
rdb = tcrdbnew();
/* connect to the server */
if(!tcrdbopen(rdb, "localhost", 1978)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "open error: %s\n", tcrdberrmsg(ecode));
}
/* store records */
if(!tcrdbput2(rdb, "foo", "hop") ||
!tcrdbput2(rdb, "bar", "step") ||
!tcrdbput2(rdb, "baz", "jump")){
ecode = tcrdbecode(rdb);
fprintf(stderr, "put error: %s\n", tcrdberrmsg(ecode));
}
/* retrieve records */
value = tcrdbget2(rdb, "foo");
if(value){
printf("%s\n", value);
free(value);
} else {
ecode = tcrdbecode(rdb);
fprintf(stderr, "get error: %s\n", tcrdberrmsg(ecode));
}
/* close the connection */
if(!tcrdbclose(rdb)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "close error: %s\n", tcrdberrmsg(ecode));
}
/* delete the object */
tcrdbdel(rdb);
return 0;
}
The following code is an example to use a remote database with the table extension.
#include <tcrdb.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stdint.h>
int main(int argc, char **argv){
TCRDB *rdb;
int ecode, pksiz, i, rsiz;
char pkbuf[256];
const char *rbuf, *name;
TCMAP *cols;
RDBQRY *qry;
TCLIST *res;
/* create the object */
rdb = tcrdbnew();
/* connect to the server */
if(!tcrdbopen(rdb, "localhost", 1978)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "open error: %s\n", tcrdberrmsg(ecode));
}
/* store a record */
pksiz = sprintf(pkbuf, "%ld", (long)tcrdbtblgenuid(rdb));
cols = tcmapnew3("name", "mikio", "age", "30", "lang", "ja,en,c", NULL);
if(!tcrdbtblput(rdb, pkbuf, pksiz, cols)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "put error: %s\n", tcrdberrmsg(ecode));
}
tcmapdel(cols);
/* store a record in a naive way */
pksiz = sprintf(pkbuf, "12345");
cols = tcmapnew();
tcmapput2(cols, "name", "falcon");
tcmapput2(cols, "age", "31");
tcmapput2(cols, "lang", "ja");
if(!tcrdbtblput(rdb, pkbuf, pksiz, cols)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "put error: %s\n", tcrdberrmsg(ecode));
}
tcmapdel(cols);
/* search for records */
qry = tcrdbqrynew(rdb);
tcrdbqryaddcond(qry, "age", RDBQCNUMGE, "20");
tcrdbqryaddcond(qry, "lang", RDBQCSTROR, "ja,en");
tcrdbqrysetorder(qry, "name", RDBQOSTRASC);
tcrdbqrysetlimit(qry, 10, 0);
res = tcrdbqrysearch(qry);
for(i = 0; i < tclistnum(res); i++){
rbuf = tclistval(res, i, &rsiz);
cols = tcrdbtblget(rdb, rbuf, rsiz);
if(cols){
printf("%s", rbuf);
tcmapiterinit(cols);
while((name = tcmapiternext2(cols)) != NULL){
printf("\t%s\t%s", name, tcmapget2(cols, name));
}
printf("\n");
tcmapdel(cols);
}
}
tclistdel(res);
tcrdbqrydel(qry);
/* close the connection */
if(!tcrdbclose(rdb)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "close error: %s\n", tcrdberrmsg(ecode));
}
/* delete the object */
tcrdbdel(rdb);
return 0;
}
The API of C is available by programs conforming to the C89 (ANSI C) standard or the C99 standard. As the header files of Tokyo Tyrant are provided as `tcrdb.h', applications should include it to use the API. As the library is provided as `libtokyotyrant.a' and `libtokyotyrant.so' and they depend on `libtokyocabinet.so', `libz.so', `libbz2.so', `libresolv.so', `libnsl.so', `libdl.so', `librt.so', `libpthread.so', `libm.so', and `libc.so', linker options corresponding to them are required by the build command. The typical build command is the following.
gcc -I/usr/local/include tt_example.c -o tt_example \ -L/usr/local/lib -ltokyotyrant -ltokyocabinet -lz -lbz2 \ -lresolv -lnsl -ldl -lrt -lpthread -lm -lc
You can also use Tokyo Tyrant in programs written in C++. Because each header is wrapped in C linkage (`extern "C"' block), you can simply include them into your C++ programs.
The database server can starts reading a Lua script file by the `-ext' option. Clients can call functions defined in the script file by the `tcrdbext' function of the remote database API.
You can define some arbitrary functions in the script file. Each function receives two string parameters of the key and the value. The return value is sent back to the client. If the function returns `nil', the server send the error code to the client.
Two kinds of locking options are provided to `tcrdbext'. One is global locking which means that only one thread can operate the function at the same time. The other is record locking which means that only one thread can operate the record of the specified key at the same time.
Note that instances of Lua interpreter are handled separately by each native thread. Because global variables of Lua are not useful to share some data among native threads or sessions, shared data should be handled in the database or by the stash functions.
The following build-in functions for database operations are available in user defined functions. The type of `key' and `value' parameters should be string or number. If number is given, it is converted as decimal string.
_eval(chunk)_log(message, level)_put(key, value)_putkeep(key, value)_putcat(key, value)_out(key)_get(key)_vsiz(key)_iterinit()_iternext()_fwmkeys(prefix, max)_addint(key, value)_adddouble(key, value)_vanish()_rnum()_size()_misc(name, args, ...)_foreach(func)_mapreduce(mapper, reducer, keys)_stashput(key, value)_stashout(key)_stashget(key)_stashvanish()_stashforeach(func)_lock(key)_unlock(key)_pack(format, ary, ...)_unpack(format, str)_split(str, delims)_codec(mode, str)_hash(mode, str)_bit(mode, num, aux)_strstr(str, pattern, alt)_regex(str, pattern, alt)_ucs(data)_dist(astr, bstr, isutf)_isect(ary, ...)_union(ary, ...)_time()_sleep(sec)_stat(path)_glob(pattern)_remove(path)_mkdir(path)Built-in functions, whose names start with "_", cannot be called directly by clients. When the server starts, the function `_begin' is called implicitly if it has been defined. When the server starts, the function `_end' is called implicitly if it has been defined.
The global variable `_version' contains the version information of the server. The global variable `_pid' contains the process ID. The global variable `_sid' contains the server ID. The global variable `_thnum' contains the number of native threads. The global variable `_thid' contains the ID number of each native thread.
The following code is an example to increment the record value and store as a decimal number string. This function should be called with the record locking option to ensure atomicity.
function incr(key, value)
value = tonumber(value)
if not value then
return nil
end
local old = tonumber(_get(key))
if old then
value = value + old
end
if not _put(key, value) then
return nil
end
return value
end
The protocol between the server and clients stands on TCP/IP. By default, the service port is bound to every address of the local host and the port number is 1978. Each session of the server and a client is composed of a request and a response. The server speaks three protocols at the same port.
In the original binary protocol, requests are classified into the following commands. Structure of request and response is determined by the command. The byte order of integer in request and response is big endian.
put: for the function `tcrdbput'[magic:2][ksiz:4][vsiz:4][kbuf:*][vbuf:*][code:1]putkeep: for the function `tcrdbputkeep'[magic:2][ksiz:4][vsiz:4][kbuf:*][vbuf:*][code:1]putcat: for the function `tcrdbputcat'[magic:2][ksiz:4][vsiz:4][kbuf:*][vbuf:*][code:1]putshl: for the function `tcrdbputshl'[magic:2][ksiz:4][vsiz:4][width:4][kbuf:*][vbuf:*][code:1]putnr: for the function `tcrdbputnr'[magic:2][ksiz:4][vsiz:4][kbuf:*][vbuf:*]out: for the function `tcrdbout'[magic:2][ksiz:4][kbuf:*][code:1]get: for the function `tcrdbget'[magic:2][ksiz:4][kbuf:*][code:1]([vsiz:4][vbuf:*])mget: for the function `tcrdbget3'[magic:2][rnum:4][{[ksiz:4][kbuf:*]}:*][code:1][rnum:4][{[ksiz:4][vsiz:4][kbuf:*][vbuf:*]}:*]vsiz: for the function `tcrdbvsiz'[magic:2][ksiz:4][kbuf:*][code:1]([vsiz:4])iterinit: for the function `tcrdbiterinit'[magic:2][code:1]iternext: for the function `tcrdbiternext'[magic:2][code:1]([ksiz:4][kbuf:*])fwmkeys: for the function `tcrdbfwmkeys'[magic:2][psiz:4][max:4][pbuf:*][code:1][knum:4][{[ksiz:4][kbuf:*]}:*]addint: for the function `tcrdbaddint'[magic:2][ksiz:4][num:4][kbuf:*][code:1]([sum:4])adddouble: for the function `tcrdbadddouble'[magic:2][ksiz:4][integ:8][fract:8][kbuf:*][code:1]([integ:8][fract:8])ext: for the function `tcrdbext'[magic:2][nsiz:4][opts:4][ksiz:4][vsiz:4][nbuf:*][kbuf:*][vbuf:*][code:1]([rsiz:4][rbuf:*])sync: for the function `tcrdbsync'[magic:2][code:1]optimize: for the function `tcrdboptimize'[magic:2][psiz:4][pbuf:*][code:1]vanish: for the function `tcrdbvanish'[magic:2][code:1]copy: for the function `tcrdbcopy'[magic:2][psiz:4][pbuf:*][code:1]restore: for the function `tcrdbrestore'[magic:2][psiz:4][ts:8][opts:4][pbuf:*][code:1]setmst: for the function `tcrdbsetmst'[magic:2][hsiz:4][port:4][ts:8][opts:4][host:*][code:1]rnum: for the function `tcrdbrnum'[magic:2][code:1][rnum:8]size: for the function `tcrdbsize'[magic:2][code:1][rnum:8]stat: for the function `tcrdbstat'[magic:2][code:1][ssiz:4][sbuf:*]misc: for the function `tcrdbmisc'[magic:2][nsiz:4][opts:4][rnum:4][nbuf:*][{[asiz:4][abuf:*]}:*][code:1][rnum:4][{[esiz:4][ebuf:*]}:*]To finish the session, the client can shutdown and close the socket at any time. If not closed, the connection can be reused for the next session. If protocol violation or some fatal error occurs, the server immediately breaks the session and closes the connection.
As for the memcached (ASCII) compatible protocol, the server implements the following commands; "set", "add", "replace", "get", "delete", "incr", "decr", "stats", "flush_all", "version", and "quit". "noreply" options of update commands are also supported. However, "flags", "exptime", and "cas unique" parameters are ignored.
As for the HTTP (1.1) compatible protocol, the server implements the following commands; "GET" (relevant to `tcrdbget'), "HEAD" (relevant to `tcrdbvsiz'), "PUT" (relevant to `tcrdbput'), "POST" (relevant to `tcrdbext'), "DELETE" (relevant to `tcrdbout'), and "OPTIONS" (relevant to `tcrdbstat'). The URI of each request is treated as the key encoded by the URL encoding. And the entity body is treated as the value. However, headers except for "Connection" and "Content-Length" are ignored. "PUT" can have the header "X-TT-PDMODE" whose value is either of 1 (relevant to `tcrdbputkeep'), 2 (relevant to `tcrdbputcat'), or else (relevant to `tcrdbput').
"POST" should have one of the header "X-TT-XNAME" or the header "X-TT-MNAME". "X-TT-XNAME" is relevant to `tcrdbext' and specifies the function name. The header "X-TT-XOPTS" stands for bitwise-or options of 1 (record locking) and 2 (global locking). The URI of each request is treated as the key encoded by the URL encoding. And the entity body is treated as the value. The result is expressed as the entity body of the response. "X-TT-MNAME" is relevant to `tcrdbmisc' and specifies the function name. The header "X-TT-MOPTS" stands for bitwise-or options of 1 (omission of the update log). The request parameters are expressed as the entity body in the "application/x-www-form-urlencoded" format. The names are ignored and the values are treated as a list of the parameters. The result is expressed as the entity body of the response in the "application/x-www-form-urlencoded" format.
After installation of Tokyo Tyrant, you can start the server immediately by executing the command `ttserver' in the terminal. By default, the server listens to the port 1978 and serves as the accessor of an on-memory hash database, which is useful to store cache data.
[terminal-1]$ ttserver
To test storing operations, execute the following commands in another terminal. `tcrmgr put' calls the function `tcrdbput'.
[terminal-2]$ tcrmgr put localhost one first [terminal-2]$ tcrmgr put localhost two second [terminal-2]$ tcrmgr put localhost three third
To test retrieving operations, execute the following commands in another terminal. `tcrmgr get' calls the function `tcrdbget'.
[terminal-2]$ tcrmgr get localhost one [terminal-2]$ tcrmgr get localhost two [terminal-2]$ tcrmgr get localhost three
To retrieve multiple records at once, execute the following command. `tcrmgr mget' calls the function `tcrdbget3'.
[terminal-2]$ tcrmgr mget localhost one two three
To terminate the server, input Ctrl-C in the terminal of the server.
Next, let's run the server that handles a hash database, by specifying the file name whose suffix is `.tch'.
[terminal-1]$ ttserver casket.tch
Store some records.
[terminal-2]$ tcrmgr put localhost one first [terminal-2]$ tcrmgr put localhost two second [terminal-2]$ tcrmgr put localhost three third
Terminate the server by Ctrl-C, and then restart the server.
[terminal-1]$ ttserver casket.tch
Check consistency of stored records.
[terminal-2]$ tcrmgr mget localhost one two three
Terminate the server by Ctrl-C and remove the database, for the successive tutorial.
[terminal-1]$ rm casket.tch
To run the server as a daemon process, specify the option `-dmn'. Moreover, the option `-pid' should be specified to record the process ID into a file. Note that the current directory of the daemon process is changed to the root directory. So, the file path parameter should be expressed as the absolute path.
[terminal-1]$ ttserver -dmn -pid /tmp/ttserver.pid /tmp/casket.tch
To terminate the daemonized server, check the process ID from the file specified by `-pid' and send the SIGTERM signal to the process.
[terminal-1]$ kill -TERM `cat /tmp/ttserver.pid`
To run the server by the RC script of the operating system, use `ttservctl'. As for most Linux distribution, append the following line to `/etc/rc.local'.
/usr/local/sbin/ttservctl start
By default, the database file and the related files are placed under `/var/ttserver'. Because `ttservctl' is a tiny shell script, copy and edit it for your purpose. Also, it is suitable to install the modified script into `/etc/init.d' and set symbolic links from `/etc/rc3.d/S98ttserver' and `/etc/rc5.d/S98ttserver'.
Let's run the server again to continue this tutorial.
[terminal-1]$ ttserver casket.tch
Store some records.
[terminal-2]$ tcrmgr put localhost one first [terminal-2]$ tcrmgr put localhost two second [terminal-2]$ tcrmgr put localhost three third
To back up the database file, indicate the destination path to the server by the command `tcrmgr copy'. Note that the backup file is created on the local file system of the server (not on the client side).
[terminal-2]$ tcrmgr copy localhost backup.tch
Terminate the server by Ctrl-C and remove the database.
[terminal-1]$ rm casket.tch
Recover the database from the backup file and restart the server.
[terminal-1]$ cp backup.tch casket.tch [terminal-1]$ ttserver casket.tch
Check consistency of stored records.
[terminal-2]$ tcrmgr mget localhost one two three
Terminate the server by Ctrl-C and remove the databases, for the successive tutorial.
[terminal-1]$ rm casket.tch backup.tch
Let's run the server with update logging enabled. The option `-ulog' specifies the directory to contain the update log files.
[terminal-1]$ mkdir ulog [terminal-1]$ ttserver -ulog ulog casket.tch
Store some records.
[terminal-2]$ tcrmgr put localhost one first [terminal-2]$ tcrmgr put localhost two second [terminal-2]$ tcrmgr put localhost three third
Terminate the server by Ctrl-C and remove the database.
[terminal-1]$ rm casket.tch
Escape the update log directoty and restart the server.
[terminal-1]$ mv ulog ulog-back [terminal-1]$ mkdir ulog [terminal-1]$ ttserver -ulog ulog casket.tch
Restore the database from the escaped update log, by the command `tcrmgr restore' on the client side.
[terminal-2]$ tcrmgr restore localhost ulog-back
Check consistency of stored records.
[terminal-2]$ tcrmgr mget localhost one two three
Terminate the server by Ctrl-C and remove the database, for the successive tutorial.
[terminal-1]$ rm -rf casket.tch ulog ulog-back
Replication is a mechanism to synchronize two or more database servers for high availability and high integrity. The replication source server is called "master" and each destination server is called "slave". Replication requires the following preconditions.
This section describes how to set up one master (at port 1978) and one slave (at port 1979) replication. First, let's run the master server.
[terminal-1]$ mkdir ulog-1 [terminal-1]$ ttserver -port 1978 -ulog ulog-1 -sid 1 casket-1.tch
Next, let's run the slave server in another terminal.
[terminal-2]$ mkdir ulog-2
[terminal-2]$ ttserver -port 1979 -ulog ulog-2 -sid 2 \
-mhost localhost -mport 1978 -rts 2.rts casket-2.tch
Store some records into the master.
[terminal-3]$ tcrmgr put -port 1978 localhost one first [terminal-3]$ tcrmgr put -port 1978 localhost two second [terminal-3]$ tcrmgr put -port 1978 localhost three third
Check consistency of stored records in the master and the slave.
[terminal-2]$ tcrmgr mget -port 1978 localhost one two three [terminal-2]$ tcrmgr mget -port 1979 localhost one two three
Let's simulate the case that the master is crashed. Terminate the master by Ctrl-C and remove the database file.
[terminal-1]$ rm casket-1.tch
Terminate the slave by Ctrl-C and restart it as the new master.
[terminal-2]$ ttserver -port 1979 -ulog ulog-2 -sid 2 casket-2.tch
Add the new slave (at port 1980).
[terminal-1]$ mkdir ulog-3
[terminal-1]$ ttserver -port 1980 -ulog ulog-3 -sid 3 \
-mhost localhost -mport 1979 -rts 3.rts casket-3.tch
Check consistency of stored records in the new master and the new slave.
[terminal-2]$ tcrmgr mget -port 1979 localhost one two three [terminal-2]$ tcrmgr mget -port 1980 localhost one two three
Terminate the two servers by Ctrl-C and remove the database and related files.
[terminal-1]$ rm -rf casket-1.tch ulog-1 1.rts [terminal-2]$ rm -rf casket-2.tch ulog-2 2.rts [terminal-1]$ rm -rf casket-3.tch ulog-3 3.rts
Tokyo Tyrant supports "dual master" replication which realizes higher availability. To do it, run two servers which replicate each other. Note that updating both of the masters at the same time may cause inconsistency of their databases. By default, the servers do not complain even if inconsistency is detected. The option `-rcc' make them check the consistency and stop replication when inconsistency is detected.
You can set replication of the running database service without any downtime. First, prepare the following script for backup operation and save it as "ttbackup.sh" with executable permission (0755).
#! /bin/sh srcpath="$1" destpath="$1.$2" rm -f "$destpath" cp -f "$srcpath" "$destpath"
Next, let's run the master with update log enabled.
[terminal-1]$ mkdir ulog-1 [terminal-1]$ ttserver -port 1978 -ulog ulog-1 -sid 1 casket-1.tch
Store a volume of records into the master.
[terminal-2]$ tcrtest write -port 1978 localhost 10000
Check consistency of stored records.
[terminal-2]$ tcrmgr list -port 1978 -pv localhost
Backup the database.
[terminal-2]$ tcrmgr copy -port 1978 localhost '@./ttbackup.sh'
Confirm that the backup file was saved as "casket-1.tch.xxxxx" ("xxxxx" stands for the time stamp of the backup file). Then, run the slave with the backup file.
[terminal-2]$ ls [terminal-2]$ cp casket-1.tch.xxxxx casket-2.tch [terminal-2]$ echo xxxxx > 2.rts [terminal-2]$ mkdir ulog-2 [terminal-2]$ ttserver -port 1979 -ulog ulog-2 -sid 2 -rts 2.rts casket-2.tch
Note that the above operation did not specify the master server to the slave. For tutorial, let's simulate that some records are stored into the master by users while you are setting replication.
[terminal-3]$ tcrmgr put -port 1978 localhost one first [terminal-3]$ tcrmgr put -port 1978 localhost two second [terminal-3]$ tcrmgr put -port 1978 localhost three third
Check the difference between the master and the slave.
[terminal-3]$ tcrmgr inform -port 1978 localhost [terminal-3]$ tcrmgr inform -port 1979 localhost
Specify the master to the slave so that replication will start and the difference will be resolved.
[terminal-3]$ tcrmgr setmst -port 1979 -mport 1978 localhost localhost
Confirm that the slave knows the master and the difference has been resolved.
[terminal-3]$ tcrmgr inform -port 1979 -st localhost
Terminate the two servers by Ctrl-C and remove the database and related files.
[terminal-1]$ rm -rf casket-1.tch casket-1.tch.* ulog-1 1.rts ttbackup.sh [terminal-2]$ rm -rf casket-2.tch ulog-2 2.rts
If you use a hash database, set the tuning parameter "#bnum=xxx" to improve performance. It specifies the bucket number and should be more than the number of record to be stored.
If you use a B+ tree database, set the tuning parameters "#lcnum=xxx#bnum=yyy" to improve performance. The former specifies the maximum number of leaf nodes to be cached. It should be larger as long as the capacity of RAM on the system allows. The latter specifies the bucket number and should be more than 1/128 of the number of records to be stored.
If huge number of clients access the server, make sure the limit number of file descriptors per process is cleared. By default on most systems, it is set as 1024. If so, use `ulimit' to clear it.
In order to deal with rushing queries at the peak time of your service, replication combining the on-memory hash/tree database and the file hash/tree database is useful. The master server handles the on-memory database and it can come through rushing queries at the peak time. Though the on-memory database can not assure the data persistence, the slave of replication compensates the shortage by storing records in the file database.
If you want more complex database operations than existing ones, use the Lua extension. For example, prepare the following script and save it as "test.lua". There is a function "fibonacci" which returns the Fibonacci number of a number of the key.
function fibonacci(key, value) local num = tonumber(key) local large = math.pow((1 + math.sqrt(5)) / 2, num) local small = math.pow((1 - math.sqrt(5)) / 2, num) return (large - small) / math.sqrt(5) end
Let's start the server by making it read the script file.
[terminal-1]$ ttserver -ext test.lua
Call the function from the client command.
[terminal-2]$ tcrmgr ext localhost fibonacci 1 [terminal-2]$ tcrmgr ext localhost fibonacci 2 [terminal-2]$ tcrmgr ext localhost fibonacci 3 [terminal-2]$ tcrmgr ext localhost fibonacci 4 [terminal-2]$ tcrmgr ext localhost fibonacci 5 [terminal-2]$ tcrmgr ext localhost fibonacci 6
Fibonacci numbers can be generated by another algorithm, which is naive and stateful. Add the following script to "test.lua". There is a function "fibnext" which returns the next Fibonacci number from the database. The state information are stored in the database.
function fibnext(key, value)
local cur = tonumber(_get("fibcur"))
if not cur then
_put("fibold", 0)
_put("fibcur", 1)
return 1
end
local old = tonumber(_get("fibold"))
_put("fibold", cur)
cur = old + cur
_put("fibcur", cur)
return cur
end
Then, restart the server and test the new algorithm.
[terminal-2]$ tcrmgr ext localhost fibnext [terminal-2]$ tcrmgr ext localhost fibnext [terminal-2]$ tcrmgr ext localhost fibnext [terminal-2]$ tcrmgr ext localhost fibnext [terminal-2]$ tcrmgr ext localhost fibnext [terminal-2]$ tcrmgr ext localhost fibnext
As you see, the called function receives two string parameters of the key and the value. The return value is sent back to the client. You can use such built-in functions for database operations as "_put", "_out", "_get", and so on. There is a sample file `ext/senatus.lua'.
This section describes how to use a memcached client library of Perl (Cache::Memcached) with Tokyo Tyrant. Run the server of Tokyo Tyrant as usual. And, the following script is a typical example.
use Cache::Memcached;
my $memd = Cache::Memcached->new();
$memd->set_servers(['localhost:1978']);
$memd->set('one', 'first');
$memd->set('two', 'second');
$memd->set('three', 'third');
my $val = $memd->get('one');
printf("one: %s\n", $val);
$val = $memd->get_multi('one', 'two', 'three');
printf("one: %s\n", $val->{one});
printf("two: %s\n", $val->{two});
printf("three: %s\n", $val->{three});
$memd->delete('one');
This section describes how to use an HTTP client library of Perl (LWP::UserAgent) with Tokyo Tyrant. Run the server of Tokyo Tyrant as usual. And, the following script is a typical example.
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(keep_alive => 1);
my $baseurl = 'http://localhost:1978/';
my $req;
$req = HTTP::Request->new(PUT => $baseurl . 'one', [], 'first');
$ua->request($req);
$req = HTTP::Request->new(PUT => $baseurl . 'two', ["X-TT-PDMODE" => 1], 'second');
$ua->request($req);
$req = HTTP::Request->new(PUT => $baseurl . 'three', ["X-TT-PDMODE" => 2], 'third');
$ua->request($req);
$req = HTTP::Request->new(GET => $baseurl . 'one');
my $res = $ua->request($req);
if($res->is_success()){
printf("%s\n", $res->content());
}
$req = HTTP::Request->new(DELETE => $baseurl . 'one');
$res = $ua->request($req);
$req = HTTP::Request->new(POST => $baseurl . 'foo',
["X-TT-XNAME" => "echo", "X-TT-XOPTS" => 1], 'bar');
$res = $ua->request($req);
if($res->is_success()){
printf("%s\n", $res->content());
}
If you want to cache data like session information for your Web application but want to avoid data loss because of the server crash, using Tokyo Tyrant can be the solution, that is to say, "persistent" but expirable cache. It requires the following preconditions.
First, prepare the following script for expiration and save it as "ttexpire.lua". It will expire records where the value of the "x" column exceeds the current date.
function expire()
local args = {}
local cdate = string.format("%d", _time())
table.insert(args, "addcond\0x\0NUMLE\0" .. cdate)
table.insert(args, "out")
local res = _misc("search", args)
if not res then
_log("expiration was failed", 2)
end
end
Start the server by opening a table database which has the index for the "x" column, and by scheduling it to call the expiration function per second.
[terminal-1]$ ttserver -ext ttexpire.lua -extpc expire 1.0 "casket.tct#idx=x:dec#dfunit=8"
Store test records from another terminal.
[terminal-2]$ now=`date +%s` for((i=1;i<=60;i++)); do tcrmgr put -sep '|' localhost "$i" "x|$((now+i))" done
You can confirm that the records are being removed by expiration.
[terminal-2]$ tcrmgr list -pv -sep '|' localhost
Tokyo Tyrant is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License or any later version.
Tokyo Tyrant is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with Tokyo Tyrant (See the file `COPYING'); if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
Tokyo Tyrant was written by FAL Labs. You can contact the author by e-mail to `info@fallabs.com'.