Kyoto Tycoon is a lightweight database server with automatic expiration mechanism, which is useful to handle cache data and persistent data of various applications. Kyoto Tycoon is also a package of network interface to the DBM called Kyoto Cabinet. Although the DBM has high performance and high concurrency, you might bother in case that multiple processes share the same database, or remote processes access the database. Thus, Kyoto Tycoon is provided for concurrent and remote connections to Kyoto Cabinet. Kyoto Tycoon is composed of the server process managing multiple databases and its access library for client applications.
The server features high concurrency due to thread-pool modeled implementation and the epoll/kqueue mechanism of the modern Linux/*BSD kernel. It can handle more than 10 thousand connections at the same time. Because such system-specific features as epoll/kqueue are encapsulated and abstracted as the same interface, Kyoto Tycoon has high portability and works almost all UNIX-like systems and Windows.
The server provides hot backup so that you can make backup data without stopping the server while copying the database files. Update logging is also supported and it compensates for the difference between the contents of backup files and the current database. Moreover, the server implements asynchronous replication. A server sends update logs to other servers, which evaluate the logs immediately and keep their databases catching up to the master database.
The server and its clients communicate with each other by HTTP. So, you can write client applications and client libraries in almost all popular languages. Both of RESTful-style interface by the GET, HEAD, PUT, DELETE methods and RPC-style inteface by the POST method are supported. The RPC-style interface is based on the protocol called TSV-RPC. The entity bodies of the request and the response are text data formatted as tab-separated-values so that parsing them is very easy. In addition, several operations are available in an efficient binary protocol.
The server can embed Lua, a lightweight script language. Even if you cannot find any built-in operation matching your requirement among the client API of Kyoto Tycoon, you can define arbitrary operations by defining functions in Lua. The API for Lua scripts provides full set of database operations of Kyoto Cabinet including visitor, cursor, and transaction mechanisms.
The server can load a shared library file which implements a pluggable server mechanism. It can run arbitrary network services and operate database objects shared by the main server. It is useful to implement other network protocols. An implementation to support the memcached protocol is included in the source package. In a similar way, the server can load a shared library file which implements a pluggable database mechanism. It is useful to use other database libraries except for Kyoto Cabinet.
The server program of Kyoto Tycoon is written in the C++ language. It is available on platforms which have API conforming to C++03 with the TR1 library extensions. Kyoto Tycoon is a free software licensed under the GNU General Public License. You can write client applications which are not under control of our license, by making them just communicate with the server by HTTP without using the core library.
This section describes how to install Kyoto Tycoon with the source package. As for a binary package, see its installation manual.
Kyoto Tycoon is available on UNIX-like systems. At least, the following environments are supported. Development for other platworms including Windows is now work-in-progress.
gcc (GNU Compiler Collection) 4.2 or later and make (GNU Make) are required to install Kyoto Tycoon with the source package. They are installed by default on Linux, FreeBSD and so on.
As Kyoto Tycoon depends on the following libraries, install them beforehand.
When an archive file of Kyoto Tycoon is extracted, change the current working directory to the generated directory and perform installation.
Run the configuration script.
$ ./configure
Build programs.
$ make
Perform self-diagnostic test. This takes a while.
$ make check
Install programs. This operation must be carried out by the root user.
# make install
When a series of work finishes, the following files will be installed.
/usr/local/include/ktcommon.h /usr/local/include/ktutil.h /usr/local/include/ktsocket.h /usr/local/include/ktthserv.h /usr/local/include/kthttp.h /usr/local/include/ktrpc.h /usr/local/include/ktulog.h /usr/local/include/ktshlib.h /usr/local/include/kttimeddb.h /usr/local/include/ktremotedb.h /usr/local/include/ktplugserv.h /usr/local/include/ktplugdb.h /usr/local/lib/libkyototycoon.a /usr/local/lib/libkyototycoon.so.x.y.z /usr/local/lib/libkyototycoon.so.x /usr/local/lib/libkyototycoon.so /usr/local/lib/pkgconfig/kyototycoon.pc /usr/local/libexec/ktplugservmemc.so /usr/local/libexec/ktplugdbvoid.so /usr/local/bin/ktutiltest /usr/local/bin/ktutilmgr /usr/local/bin/ktutilserv /usr/local/bin/kttimedtest /usr/local/bin/kttimedmgr /usr/local/bin/ktserver /usr/local/bin/ktremotetest /usr/local/bin/ktremotemgr /usr/local/man/man1/... /usr/local/share/doc/kyototycoon/...
The following options can be specified with `./configure'.
`--prefix' and other options are also available as with usual UNIX software packages. If you want to install Kyoto Tycoon under `/usr' not `/usr/local', specify `--prefix=/usr'. As well, the library search path does not include `/usr/local/lib', it is necessary to set the environment variable `LD_LIBRARY_PATH' to include `/usr/local/lib' before running applications of Kyoto Tycoon.
Kyoto Tycoon provides API of the C++ language and it is available by programs conforming to the C++03 standard. As the header files of Kyoto Tycoon are provided as `ktutil.h', `ktremotedb.h', and so on, applications should include one or more of them accordingly to use the API. As the library is provided as `libkyototycoon.a' and `libkyototycoon.so' and they depends on underlying system libraries, linker options corresponding to them are required by the build command. The typical build command is the following.
$ g++ -I/usr/local/include example.cc -o example \ -L/usr/local/lib -lkyototycoon -lkyotocabinet -lz -lstdc++ \ -lresolv -lnsl -ldl -lrt -lpthread -lm -lc
If you don't use the core library of C++ but an HTTP library in another language, you don't have to know the above messy rules.
This section describes how to use Kyoto Tycoon with the command line utilities and some sample application programs.
To begin with, let's run the database server program. Simply execute the following commnad. Some log messages are printed on the terminal.
$ ktserver 2010-10-03T16:24:38.467252+09:00: [SYSTEM]: ================ [START]: pid=19069 2010-10-03T16:24:38.467473+09:00: [SYSTEM]: opening a database: path=: 2010-10-03T16:24:38.467645+09:00: [SYSTEM]: starting the server: expr=:1978 2010-10-03T16:24:38.467751+09:00: [SYSTEM]: server socket opened: expr=:1978 timeout=30.0 2010-10-03T16:24:38.467775+09:00: [SYSTEM]: listening server socket started: fd=3
The command `ktserver' starts network service accepting commands from clients on local or remote machines. By default, an unnamed on-memory database is opened and managed by the server through the port 1978. To finish the server, input `Ctrl-C' on the terminal or send such termination signals as `SIGINT' or `SIGTERM' from another terminal.
Next, insert some records into the database. Execute the follwoing command on another terminal. Corresponding access logs will be printed on the server terminal.
$ ktremotemgr set japan tokyo $ ktremotemgr set korea seoul $ ktremotemgr set china beijing
The command `ktremotemgr' is a tool kit to manage the database as a client. The sub command "set" is to set a record. The first argument next to the sub command name is the key of a record and the second argument is the value of the record.
Retrieve the records by the key of each record using the sub command "get".
$ ktremotemgr get japan tokyo $ ktremotemgr get korea seoul $ ktremotemgr get china beijing
Remove a record by the key using the sub command "remove".
$ ktremotemgr remove japan
Print the keys of all records using the sub command "list".
$ ktremotemgr list korea china
That's all for the fundamental operations. The KVS family have been improving performance thanks to discarding the functionality. See the specification of `ktserver' and `ktremotemgr' for more details.
Because every operation of the database is called via HTTP, you can use any HTTP client utility such as `curl' to operate the database.
# setting records $ curl "http://localhost:1978/rpc/set?key=japan&value=tokyo" # retrieving records $ curl "http://localhost:1978/rpc/get?key=japan" value tokyo # removing records $ curl "http://localhost:1978/rpc/remove?key=japan"
RESTful-style interface is also supported in addition to the above RPC-style interface.
# setting records $ echo -n tokyo | curl -X PUT -T - "http://localhost:1978/japan" # retrieving records $ curl "http://localhost:1978/japan" tokyo # removing records $ curl -X DELETE "http://localhost:1978/japan"
Of cource, you can use your favorite scripting languages and libraries for more complex use cases. Because Kyoto Tycoon supports keep-alive connection mechanism, using libraries which support keep-alive is strongly suggested for performance reason.
One of the most important features of Kyoto Tycoon is expiration mechanism of records. That is, you can specify the expiration time when inserting a record. The record is automatically deleted after the current time exceeds the expiration time.
To insert a record which will be expired one minute after, execute the following command.
$ ktremotemgr set -xt 60 japan tokyo
Check the record immediately before the expiration. The "-pt" option shows the expiration time in seconds from the epoch.
$ ktremotemgr get -pt japan tokyo 1286108387
Wait for more than one minute, and then retrieve the record again. It will be missing.
$ ktremotemgr get -pt japan ktremotemgr: DB::get failed: :1978: 2: logical inconsistency: DB: 7: no record: no record
You can do the same things with an arbitrary HTTP client by specifiying the "xt" parameter.
$ curl "http://localhost:1978/rpc/set?key=japan&value=tokyo&xt=60" $ curl "http://localhost:1978/rpc/get?key=japan" value tokyo xt 1286109204 $ sleep 60 $ curl "http://localhost:1978/rpc/get?key=japan" ERROR DB: 7: no record: no record
Leaving command line interface, let's write a sample application program handling a remote database. See the following source code.
#include <ktremotedb.h>
using namespace std;
using namespace kyototycoon;
// main routine
int main(int argc, char** argv) {
// create the database object
RemoteDB db;
// open the database
if (!db.open()) {
cerr << "open error: " << db.error().name() << endl;
}
// store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")) {
cerr << "set error: " << db.error().name() << endl;
}
// retrieve a record
string value;
if (db.get("foo", &value)) {
cout << value << endl;
} else {
cerr << "get error: " << db.error().name() << endl;
}
// traverse records
RemoteDB::Cursor* cur = db.cursor();
cur->jump();
string ckey, cvalue;
while (cur->get(&ckey, &cvalue, NULL, true)) {
cout << ckey << ":" << cvalue << endl;
}
delete cur;
// close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
}
return 0;
}
Save the above code as a file "example.cc". Then, perform the following command line. The command `kcutilmgr conf' prints the building configuration.
$ g++ `ktutilmgr conf -i` -o example example.cc `ktutilmgr conf -l`
Execute the application program built by the above. Of course, run the database server on another terminal beforehand.
$ ./example hop foo:hop bar:step baz:jump
The API of the remote database is defined in the header `ktremote.h'. So, include the header near the front of a source file. All symbols of Kyoto Tycoon are packaged in the name space `kyototycoon'. You can use them without any prefix by importing the name space.
#include <ktremotedb.h> using namespace kyototycoon;
The class `RemoteDB' contains all functionality of the remote database and each instance expresses a remote database file.
RemoteDB db;
Each connection must be opened by the `open' method before any database operation. Although it takes three parameters, all of them can be omitted. The first parameter specifies the host name of the server and the default value is the name of the local host. The second parameter specifies the port number and the default value is 1978 which is the same as the default port of the database server. The third parameter specifies the timeout of network operation in seconds and the default value is no timeout.
db.open();
Every opened connection should be closed by the `close' method when it is no longer in use.
db.close();
To store a record, use the `set' method with the key and the value.
db.put("foo", "hop");
To retrieve the value of a record, use the `get' method with the key. On success, the return value is true and the result is assigned into the string object pointed to by the second parameter.
string value;
if (db.get("foo", &value)) {
cout << value << endl;
}
Except for `set' and `get', there are other methods; `add', `replace', `append', `remove', `increment', and `cas'. Each method has two versions; for `std::string' parameters and for `char*' and `size_t' parameters.
Traversing records is a bit complicated task. It needs a cursor object, which expresses the current position in the sequence of all records in the database. Each cursor is created by the `cursor' method of the database object. Each cursor should be initialized by `jump' method before actual record operations.
RemoteDB::Cursor* cur = db.cursor(); cur->jump();
The cursor class has such methods against the record at the current position as `set_value', `remove', `get_key', `get_value', and `get'. Most methods have an optional stepping parameter to shift the current position to the next record atomically. Therefore, iterating such methods with the stepping parameter results in that all records are visited.
string ckey, cvalue;
while (cur->get(&ckey, &cvalue, NULL, true)) {
cout << ckey << ":" << cvalue << endl;
}
Please see the the API documents for details. Writing your own sample application is the best way to learn this library.
If you configured Kyoto Tycoon with the "--enable-lua" option, the scripting extension by Lua is available. The following is an example code to define a simple operation to store a record.
kt = __kyototycoon__
db = kt.db
function set(inmap, outmap)
local key = inmap.key
local value = inmap.value
if not key or not value then
return kt.RVEINVALID
end
local xt = inmap.xt
if not db:set(key, value, xt) then
return kt.RVEINTERNAL
end
return kt.RVSUCCESS
end
If you save the above code as "test.lua", you can embed the operation by stating the server as the following.
$ ktserver -scr test.lua
To call the operation, execute the following command.
$ ktremotemgr script set key japan value japan xt 60
See the specification of the scripting extension for more details. If you edit the script file and want to make the server reload it, send the `SIGUSR1' signal to the server. If the server is a daemon, sending the `SIGHUP' signal causes the same effect.
This section describes tips and hacks to use Kyoto Tycoon.
On the assumption that you runs the server of Kyoto Tycoon on an machine with 16GB main memory and stores 10 million records into a file hash database, the following setting is suggested.
$ ktserver -port 1978 -tout 10 \ -log /var/data/ktserver.log -ls \ -dmn -pid /var/data/ktserver.pid \ '/var/data/casket.kch#opts=l#bnum=20000000#msiz=12g#dfunit=8'
To improve performance, the bucket number of the hash table by the "bnum" parameter should be two times or more of the number of records. The size of mapped memory by the "msiz" parameter should be largest as far as the main memory is available. The unit number of automatic defragmentation by the "dfunit" parameter should be about 8 which means every eight detected fragmentations causes a series of automatic defragmentation steps. If you want higher availability at the cost of performance, using automatic transaction by the "-oat" option is a good idea. For details about database tuning, see the tips of Kyoto Cabinet.
The option "-dmn" switches the process into the background, which is called daemon. To stop or restart a daemon process, the PID files should be specified by the "-pid" option. The PID file contains the process ID by which you can send signals.
By default, verbose log messages are output and printed into the standard output. For usual use case, the "-ls" option which filters them is suggested. The "-log" option specifies the file to store log messages.
To stop the above daemon process, execute the following command.
$ kill -TERM `cat /var/data/ktserver.pid`
To rotate the log file of the above daemon process, execute the following command.
$ mv -f /var/data/ktserver.log /var/data/ktserver.log.`date '+%Y%M%d%H%M%S'` $ kill -HUP `cat /var/data/ktserver.pid`
You can use Kyoto Tycoon as a volatile (not persistent) cache server with an on-memory database. In most cases, space efficiency of Kyoto Tycoon is better than that of memcached. If it is expected that 10 million records are cached in the database using 10GB memory, the following setting is suggested.
$ ktserver ':#bnum=20000000#ktcapsiz=10g'
Note that the capacity tuning parameter limits the total size of memory allocation in the "user land" layer. So, actual memory usage will be probably higher than the limit size. For peaceful operation, the limit should be up to 65% of the total memory size of the machine.
In addition, automatic deletion by the capacity limit is performed at random. In that case, fresh records may also be deleted soon. So, setting effectual expiration time not to reach the limit is very important. If you cannot calculate effectual expiration time beforehand, use the cache hash database instead of the default stash database. The following setting is suggested.
$ ktserver '*#bnum=20000000#capsiz=8g'
Note that the space effiency of the cache hash database is worse than that of the stash database. The limit should be up to 50% of the total memory size of the machine. However, automatic deletion by the "capsiz" parameter (not "ktcapsiz") of the cache hash database is based on LRU algorithm, which prevents fresh records from sudden deletion.
Kyoto Tycoon supports the "background snapshot" mechanism for on-memory databases. This mechanism is similar to the one in Redis. If background snapshot is enabled, the server saves all records in on-memory databases into files in a directory periodically. Because snapshot operations are performed in background by child processes forked from the server process, any foreground operation called by clients is not blocked. Due to the copy-on-write memory mechanism by the operating system, each snapshot operation is performed atomically in logical view.
Snapshot files are saved in a directory specified when the server starts. Each snapshot operation runs periodically by the given interval and additionally when the server finishes. If there are snapshot files when the server starts, they are read and all records are restored. The following setting is an example.
$ ktserver -bgs mysnap -bgsi 30
Insert some records and wait for 30 seconds. Then you'll see a snapshot file named "00000000.ktss" in the "mysnap" directory.
$ ktremotemgr set france paris $ ktremotemgr set germany berlin
Terminate the server by Ctrl-C and restart the server.
$ ktserver -bgs mysnap -bgsi 30
Make sure that the records before termination were restored.
$ ktremotemgr list -pv germany berlin france paris
Because snapshot data are serialized in an compact format and the IO operations are in sequential order, the IO loading of the underlying storage device is much lower than those in file databases. In order to lower the IO loading at the cost of CPU time, you can use a compression algorithm by the "-bgsc" option. The supported compression algorithms are "zlib", "lzo" and , "lzma", although the latter two are optional support when building Kyoto Cabinet. Probably, using "lzo" is a good choice for most use cases.
The timed database is a database type of wrapper of the polymorphic database of Kyoto Cabinet. The value of each record in the timed database has the 5 bytes prefix to contain the expiration time in seconds from the epoch. When a database operation accesses a record, the current time and the expiration time of the record is compared. If the former is larger, the record is regarded as expired.
In order to eliminate or reuse regions of expired records, the timed database has an implicit cursor called "GC cursor". Every several updating operations have the GC cursor scan the database gradually. If a scanned record has been expired, the record is removed and its region is registered to the reusable list. When the GC cursor reaches the end of the database, the tail end unused regions are discarded and the size of the database file is reduced. Thus, meta data such as "count" and "size" have latency from the current status.
The above algorithm means that the GC cursor does not work while the database is not being updated. However, while the server is idle, the GC cursor works implicitly so that the database keeps compact. Moreover, If you want to cause the full GC operation explicitly, you can call the "vacuum" procedure.
$ ktremotemgr vacuum
If you use Kyoto Tycoon not for cache but for data storage, create the database with the persistent option by "#ktopts=p" parameter to disable the GC cursor and omit the time stamp region of each record, for the sake of time and space efficiency. Note that the persistent option works when creating a database and it does not work for existing databases.
If you want to manage a database without the persistent option directly on a local machine, use the `kttimedmgr' command. Do not use commands of Kyoto Cabinet such as `kchashmgr' and `kcpolymgr' for that purpose. If you managed Tycoon's "timed" database by Cabinet's command, you would see 5-byte garbage at the beginning of each record. If you managed Cabinet's "normal" database by Tycoon's command, you couldn't see the first 5-bytes data.
The signal mechanism is useful to monitor a record to be updated. Every RPC call can wait for a signal before its operation and can send a signal after its operation. Each signal is binded to a named condition variable. If you want to retrieve the new value of a record "hello" immediately after a signal is sent to "hello", execute the following code. Although you can use a different name of the condition variable from the key of a record to be monigored, using the same strings is straightforward.
db.set_signal_waiting("hello", 60);
string value;
db.get("hello", &value);
You can do the same thing by the following command line.
$ ktremotemgr get -swname hello -swtime 60 hello
If you want to update the record "hello" and then send a signal to "hello", execute the following code.
db.set_signal_sending("hello", true);
db.set("hello", "world");
You can do the same thing by the following command line.
$ ktremotemgr set -ssname hello -ssbrd hello world
The signal mechanism is useful to realize a job queue as well. In that case, you should choose one of B+ tree databases as its storage and store job data whose keys are time stamps so that records are always sorted in the accending order of time stamps. Task producers are to store task data and send a signal for each operation. Each task workers is to wait for a signal and then retrieve the first record in the queue. If you want a single queue on a server, use an empty string as the signal name. If you want plural queues on a server, open plural databases to match the queues and use different signal names to distinguish task kinds.
Note that an RPC call waiting for a signal occupies the thread during its operation. So, it should be avoided that an unspecified number of clients wait for signals. The signal mechanism should be used only for monitoring records by a few specified clients.
You can make backup files while the server is running, which is called "hot backup". To do it, prepare the following shell script beforehand and save it as "/ktbin/dbbackup". It must have executable permissions.
#! /bin/sh srcfile="$1" destfile="$1.$2" cp -f "$srcfile" "$destfile"
Run the server specifying the command search path "/ktbin".
$ ktserver -cmd /ktbin casket.kch
When you want to make a backup file, execute the following command to call the backup script.
$ ktremotemgr sync -cmd dbbackup
The "sync" sub command of the client utility makes the database synchronize, which means dirty buffers on memory are written into the database file. If the "-cmd" option is specified, an outer command is executed with two arguments. The first argument is the path of the database file. The second argument is the current time stamp. You can call arbitrary scripts other than the above sample. Using "snapshot" mechanism of the underlying operating system is a good idea to shorten the time to make a backup file.
Even if you take hot backup a day, update operations in 24 hours at most might be lost when an accident occurred. "update log" is provided to compensate for such data loss. It keeps track of every update operation in "record-base" model, which is called "row-base" in the context of RDBMS. You can recover the database entirely by applying update logs to the latest backup database file.
Run the server with update logging enabled. The "-ulog" option specifies the directory to contain the update log files. The "-sid" option specifies the server ID between 0 to 65535. The "-cmd" option enables hot backup by the above instruction.
$ ktserver -ulog 0001-ulog -sid 1 casket.kch -cmd /ktbin casket.kch
Insert some records.
$ ktremotemgr set one first $ ktremotemgr set two second
Take a backup file.
$ ktremotemgr sync -cmd dbbackup
Insert more records.
$ ktremotemgr set three third $ ktremotemgr set four fourth
Terminate the server by Ctrl-C and remove the database to simulate database crash.
$ rm casket.kch
Recover the database by applying update logs. "xxx..." means the time stamp of the backup database file. The "-ts" option specifies the time stamp until which old update logs are skipped.
$ cp casket.kch.xxxxxxxxxxxxxxxxxxxx casket.kch $ kttimedmgr recover -ts xxxxxxxxxxxxxxxxxxxx casket.kch 0001-ulog .. (2)
Confirm the records in the recovered database.
$ kttimedmgr list -pv casket.kch one first two second three third four fourth
Update logs are saved in respective files by constant size, which is 256MB by default. After you take a backup database file, you can remove older update logs than the time stamp of the backup database file. The command "ktremotemgr slave -uf" prints the size and the maximum time stamp in each update log file.
$ ktremotemgr slave -uf 0001-ulog/0000000001.ulog 268459706 1293335993793000167 0001-ulog/0000000002.ulog 268466346 1293336035697000149 0001-ulog/0000000003.ulog 268477224 1293336080897000091 0001-ulog/0000000004.ulog 268456392 1293336132156000021 0001-ulog/0000000005.ulog 101109851 1293336145396000029
You can remove old log files by the command "ktremotemgr slave -ur". The "-ts" option specifies the maximum time stamp of disposable logs. If the time stamp is not specified, all log files except for the current one are removed.
$ ktremotemgr slave -ur -ts 1293336080897000091
Even if you take hot backup and update logs, all data might be lost when the storage device or the computer itself broke down. Keep in mind that nothing is eternal. To overcome such critical situation, data replication is supported. It is a mechanism to synchronize two or more database servers for high availability and high integrity. The replication source server is called "master" and each destination server is called "slave". Replication requires the following preconditions.
This section describes how to set up one master at port 1978 and one slave at port 1979 replication. First, run the master server.
$ ktserver -port 1978 -ulog 0001-ulog -sid 1 casket-0001.kch
Next, run the slave server.
$ ktserver -port 1979 -ulog 0002-ulog -sid 2 \ -mhost localhost -mport 1978 -rts 0002.rts casket-0002.kch
Insert some records into the master.
$ ktremotemgr set -port 1978 one first $ ktremotemgr set -port 1978 two second
Check consistency of stored records in the master and the slave.
$ ktremotemgr list -port 1978 -pv one first two second $ ktremotemgr list -port 1979 -pv one first two second
Simulate the case that the master is crashed. Terminate the master by Ctrl-C and remove the database file.
$ rm casket-0001.kch
Terminate the slave by Ctrl-C and restart it as the new master.
$ ktserver -port 1979 -ulog 0002-ulog -sid 2 casket-0002.kch
Add a new slave at port 1980.
$ ktserver -port 1980 -ulog 0003-ulog -sid 3 \ -mhost localhost -mport 1979 -rts 0003.rts casket-0003.kch
Check consistency of stored records in the new master and the new slave.
$ ktremotemgr list -port 1979 -pv one first two second $ ktremotemgr list -port 1980 -pv one first two second
Because the master server uses a dedicated worker thread for each connection to slave, the number of worker threads should be increased if you set several slave servers.
Kyoto Tycoon supports "dual master" replication topology which realizes higher availability. It means that two servers replicate each other so that you don't have to restart the survivor when one of them crashed.
This section describes how to set up two masters called A and B which replicate each other. First, run the server A.
$ ktserver -port 10001 -ulog 0001-ulog -sid 1 \ -mhost localhost -mport 10002 -rts 0001.rts casket-0001.kch
Next, run the server B.
$ ktserver -port 10002 -ulog 0002-ulog -sid 2 \ -mhost localhost -mport 10001 -rts 0002.rts casket-0002.kch
Insert some records into the server A.
$ ktremotemgr set -port 10001 one first $ ktremotemgr set -port 10001 two second
Insert some records into the server B.
$ ktremotemgr set -port 10002 three third $ ktremotemgr set -port 10002 four fourth
Check consistency of stored records in the two servers.
$ ktremotemgr list -port 10001 -pv one first two second three third four fourth $ ktremotemgr list -port 10002 -pv one first two second three third four fourth
Simulate the case that the server A is crashed. Terminate the server A by Ctrl-C and remove the database file.
$ rm casket-0001.kch
Run a new server called C which replicates the server B.
$ ktserver -port 10003 -ulog 0003-ulog -sid 3 \ -mhost localhost -mport 10002 -rts 0003.rts casket-0003.kch
Modify the replication configuration of the server B to replicate the server C.
$ ktremotemgr tunerepl -port 10002 -mport 10003 localhost
Insert some records into the server C.
$ ktremotemgr set -port 10003 five fifth $ ktremotemgr set -port 10003 six sixth
Check consistency of stored records in the current servers.
$ ktremotemgr list -port 10002 -pv one first two second three third four fourth five fifth six sixth $ ktremotemgr list -port 10003 -pv one first two second three third four fourth five fifth six sixth
Of course, using a backup database file to apply "difference recovery" instead of "full recovery" when adding a new server is a good idea. To apply difference recovery, specify the time stamp of the backup database file with the "-ts" option of the "ktremotemgr tunerepl" command.
Note that updating both of the servers at the same time might cause inconsistency of their databases. That is, you should use one master as a "active master" and the other as a "standby master".
Snapshot files of on-memory databases can be used as backup files. And, you use them in order to set up replication slave servers. First, run a master server handling an on-memory database.
$ ktserver -port 10001 -ulog 0001-ulog -sid 1 \ -bgs 0001-bgs -bgsi 60 ':#bnum=10000000#ktopts=p'
Insert some records into the master server.
$ ktremotemgr setbulk -port 10001 one first two second
Wait for 60 seconds and then make sure that the above two records are saved in the snapshot file. The second field expresses the time stamp of the snapshot. The third field expresses the number of records.
$ kttimedmgr bgsinform 0001-bgs 0 1295607805662000000 2 94906353
Copy the snapshot directory as another one. Then, extract the time stamp and generate the RTS file for the slave server.
$ cp -r 0001-bgs 0002-bgs $ kttimedmgr bgsinform 0001-bgs | cut -f 2 < 0002.rts
Start the slave server.
$ ktserver -port 10002 -sid 2 \ -bgs 0002-bgs -bgsi 60 -mhost localhost -mport 10001 -rts 0002.rts \ ':#bnum=10000000#ktopts=p'
Of cource, you can also set up dual master replicatoin of on-memory databases. It realizes extremely fast and enough robust database solution.
If you want other protocols except for HTTP, the "pluggable server" mechanism is useful. The server can load a shared library which implements arbitrary network services and operate database objects shared by the main server. The shared library "ktplugservmemc.so" installed under "/usr/local/libexec" implements the memcached protocol. The following command starts the normal server at the default port 1978 and starts the pluggable server for the memcached protocol at another port 2001 simultaneously. If your application doesn't access the main server, the number of the main thread should be set 1 by specifying "-th 1".
$ ktserver -th 1 -plsv /usr/local/libexec/ktplugservmemc.so -plex 'port=2010'
Confirm the functionality by the telnet command.
$ telnet localhost 2010 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. set japan 0 0 5 tokyo STORED get japan VALUE japan 0 5 tokyo END quit Connection closed by foreign host.
This implementation supports a portion of commands supported by the original memcached server. As for now, "set", "add", "replace", "get", "delete", "incr", "decr", "stats", "flush_all", and "quit" are available. For the sake of interoperability with HTTP, the "flags" option of some update commands are ignored by default. If you want to handle the flags, enable the mechanism by adding "#opts=f" in the argument of the "-plex" option. The mechanism of "cas unique" is not implemented.
The configuration expression by the "-plex" option is composed of named parameters separated by "#". Each parameter is composed of the name and the value separated by "=". The supported parameter names are "host", "port", "tout", "th", and "opts". The default port is 11211, which is the same as the original memcached server. Don't forget "#opts=f" if your client library uses "flags" to determine the data type of each record. By default, "flags" are disabled for compatibility of the record format to HTTP.
The most important advantage of the pluggable server mechanism is to share database objects with the main server. Update operations by a pluggable server is reflected to the main server immediately and vice versa. Moreover, update operations are written as update logs and picked up by slave servers of replication. You can implement your own pluggable server. See the source code in "ktplugservmemc.cc" for details.
If you want other database implementations except for Kyoto Cabinet, the "pluggable database" mechanism is useful. The server can load a shared library which implements arbitrary storage services conforming to the interface of associative array. The shared library "ktplugdbvoid.so" installed under "/usr/local/libexec" implements the "void database" mechanism, which is similar to "blackhole storage engine" of MySQL. It is so-called "no-op" database and do nothing except for writing update logs. It is useful to get rid of the load of the master server and level the load in the time axis. The following command starts the replication master server opening a void database.
$ ktserver -port 1978 -ulog 0001-ulog -sid 1 -pldb /usr/local/libexec/ktplugdbvoid.so
The following command starts the replication slave server opening a file hash database.
$ ktserver -port 1979 -sid 2 -mhost localhost -mport 1978 -rts 2.rts casket.kch
Store a record into the master server and confirm that the record is discarded.
$ ktremotemgr set -port 1978 foo bar $ ktremotemgr get -port 1978 foo ktremotemgr: DB::get failed: :1978: 3: logical inconsistency: DB: 7: no record: no record
Confirm that the record exists in the database of the slave server.
$ ktremotemgr get -port 1979 foo bar
The most important advantage of the pluggable database mechanism is to give networking functionality to other DBM-style database libraries such as BerkeleyDB and GDBM. You can implement your own pluggable database. See the source code in "ktplugdbvoid.cc" for details.
If you want to replicate update operations from a server of Kyoto Tycoon to another database system like RDBMS, it is useful to pull update logs by the command "ktremotemgr slave" and process each log by your own script command. The master server setting is not particular.
$ ktserver -ulog 0001-ulog -sid 1 casket.kch
To generate update logs in the master server, perform some opdate operations.
$ ktremotemgr clear $ ktremotemgr set one first $ ktremotemgr set two second $ ktremotemgr remove one
Pull the above update logs.
$ ktremotemgr slave 1293014011013000000 1 0 clear 1293014017613000000 1 0 set b25l //////9maXJzdA== 1293014022052000000 1 0 set dHdv //////9zZWNvbmQ= 1293014027262000000 1 0 remove b25l
The output is in TSV format. Each line expresses each update operation. The fields are the time stamp of the update operation, the server ID number, the database ID number, the command name, and Base64-encoded arguments of the command. The first 5 bytes in the value of the "set" command specifies the expiration time. You can give the output to another command for post processing via pipe.
$ ktremotemgr slave -ts "`cat sometime.rts`" -uw | dosomething
The file "sometime.rts" should contain the time stamp of the last pulled update log, which should be recorded by the post processing command. The "-uw" option is to wait for future update logs forever even when the reader has reached the end of the log files. A sample implementation of a post processing command in Ruby is included as "example/ktreplprint.rb" in the source package.
The memcached pluggable server can work as a message queue server. If the database is one of tree databases (GrassDB, TreeDB, ForestDB) and the "#opts=q" option is specified, behaviors of "set", "get", and "delete" are overwritten. The default timeout of "get" is 10 seconds and it can be modified by the "qtout" option. The following is a typical configuration.
$ ktserver -th 1 -plsv /usr/local/libexec/ktplugservmemc.so \ -plex 'port=11211#opts=fq#qtout=5' 'casket.kct#ktopts=p'
Some clients can store messages with arbitrary keys. Messages are organized by names in the accending order of time stamps. Some clients can fetch messages with arbitrary keys. Fetched messages are hidden atomically in order to avoid race condition. So, multiple clients can monitor the same key simultaneously. If a client has done with a message, he should delete the message explicitly. If a client closes the connection witout deleting messages he fetched, those message are registered into the database implicitly. So, even if a client dies unexpectedly, no message won't be lost.
A typical client to write messages, called "job producer", is implemented like the following pseudo-code.
memc = Memcached::connect("localhost", 11211)
memc.set("foo", jobdata)
A typical client to read messages, called "job worker", is implemented like the following pseudo-code.
memc = Memcached::connect("localhost", 11211)
while (true) {
value = memc.get("foo")
if (value) {
if (do_something(value)) {
memc.delete("foo")
}
}
}
"set" and "get" operations are synchronized by the singal mechanism. When there is no message corresponding to the specified key, the "get" method blocks for 10 seconds or until a corresponding record is stored. So, the event loop of job workers does not cause useless network traffic or latency as in busy "polling" model.
This section describes detail specifications of the protocol of Kyoto Tycoon.
The server program of Kyoto Cabinet communicates with clients by HTTP. Although the default service port number is 1978, it can be changed by setting. The server understands requests of HTTP/1.0 and HTTP/1.1 and sends responses corresponding to each request.
If the request is HTTP/1.1 and the value of the "Content-Type" header is not "close", the server try to perform keep-alive connection. Keep-alive connection is strongly suggested to access the server for performance reason. However, usual intermissive connection is also allowed for legacy clients.
Kyoto Cabinet uses an RPC model called TSV-RPC, which is similar to XML-RPC but uses TSV rather than XML. Although TSV is inferior to XML in terms of expressive ability, TSV is superior to XML in terms of simplicity, space efficiency, and processing effectiveness.
TSV-RPC is a client-server modeled synchronous RPC protocol over HTTP. The following pseudo code is the common interface of each procedure. TSV is used to serialize the input data and the output data.
int call(String name, StringMap inmap, StringMap outmap);
Each procedure has its entry point under the path "/rpc/" of URL. For example the entry point of the command "set" is "/rpc/set". Clients call each command by the POST method specifying the entry point. Each procedure receives the input data of an associative array composed of key/value string records. The associative array is expressed by the entity body formatted in TSV text. Each line expresses a record. The first column is the key and the second column is the value. The value of the "Content-Type" header must be "text/tab-separated-values". The following is a typical request message.
POST /rpc/set HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values Content-Length: 22 key japan value tokyo
If a column includes one or more special characters such as tab, line-feed, and control characters, every column must be encoded by one of the following algorithms.
The encoding name must be described as an attribute of the value of the Content-Type header. The following is a typical request message with encoded TSV data.
POST /rpc/hello HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values; colenc=B Content-Length: 24 aWQ= MTIzNDU= YWdl MzE=
As a facile method, clients can send the input data by the entity body in HTML-form format. That is, the key and the value of a record are encoded respectively in URL encoding and they are concatenated with the "=" character. All serialized record expressions are concatenated with the "&" character. The value of the Content-Type header is "application/x-www-form-urlencoded". The following is a typical request with HTML-form data.
POST /rpc/goodbye HTTP/1.1 Host: localhost:1978 Content-Type: application/x-www-form-urlencoded Content-Length: 31 id=1234&name=%e5%b9%b9%e9%9b%84
As a more facile method, clients can use the GET method and specify the input data as the query string with the target path. The following is an example.
GET /rpc/paint?color=red&x=30&y=20 HTTP/1.1 Host: localhost:1978
Clients can select one of TSV, HTML-form, and query string arbitrarily. However, not that the query string has limit of length, which is 8192 bytes as for Kyoto Tycoon. TSV with Base64 encoding is recommended for arbitrary string and binary data because its space efficiency is the best of the three formats.
After receiving a request, the server send the result of computation in the response. The status code of HTTP is one of the follwoing.
200 and 450 is in normality. 400 indicates bugs in the client side. 500 indicates bugs in the server side. 501 is in the literature.
If the status code is 200, the output data of the procedure is sent as the entity body of the response message. The output data is also an associative array composed of key/value string records. The format rules are the same as the ones of the input data. The following is a typical response message.
HTTP/1.1 200 OK Content-Type: text/tab-separated-values; colenc=U Content-Length: 25 value %e5%b9%b3%e6%9e%97
If the status code is other than 200, error information is sent as the entity body of the response message. It is an assiciative array including one record whose key is "ERROR" and value is the error message. The following is an example.
HTTP/1.1 501 Not Implemented Content-Type: text/tab-separated-values Content-Length: 24 ERROR no such procedure
The server select the best encoding by scanning the output data. That is, raw data is selected if encoding is not needed. URL encoding is selected if ASCII characters are relatively many. Base64 is selected in the other cases. Because Quoted-printable is never selected by the server, clients don't have to implement it.
The input data of each procedure can be regarded as named paramters. Although each procedure needs various parameters different with each other, there are common arguments used by several procedures.
Remember that the server can handle multiple databases at the same time. Opened databases are distinguished by their ID numbers from 0 to N-1 in the order of the command line arguments. By default, the first database whose ID number is 0 is selected. The "DB" parameter specifies the identifier of the target database of each operation. If the identifier starts with a number, it is treated as the ID number. Otherwise, it is treated as a database name. The name of each database is made of the file name after the last "/" character of the path. The following example stores a record into the database named "phone_db.kch".
POST /rpc/set HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values Content-Length: 47 DB phone_db.kch key 09012345678 value John Doe
While keep-alive connection, clients can use cursor objects. Each cursor has the ID number from 0 to 2^63. Because each connection has a separate name space with each other, the same ID number can be used among different connections. The cursor ID is specified by the "CUR" parameter. If an unknown ID is specified, a new cursor with the ID numger is created. The following example creates a cursor whose ID is 8 and sets its position to the record whose key is "mikio".
POST /rpc/cur_jump HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values Content-Length: 32 DB staff_db.kch CUR 8 key mikio
Cursur objects can be reused in the same connection by specifying the ID number. When each connection is closed, cursor objects created while the connection are destroyed automatically and related resources are cleaned-up.
Every procedure can wait for a signal by a named condition variable before the procedure is performed. If the "WAIT" parameter is specified, the thread waits for a condition variable whose name is the value of the parameter. Although the default timeout is 30 second, it can be modified by the "WAITTIME" parameter. If any signal is sent within the given time, the procedure is aborted. The following example waits for updating a record for up to 10 seconds and then get the new value.
GET /rpc/get HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values Content-Length: 34 WAIT summer WAITTIME 8 key summer
Moreover, every procedure can send a signal to a thread waiting by a named condition variable after the procedure is performed. If the "SIGNAL" parameter is specified, a signal is sent to a thread waiting by a named condition variable whose name is the value of the parameter. If the "SIGNALBROAD" parameter is specified at the same time, the signal is sent to every corresponding thread. If a signal is sent, the response includes the paramter "SIGNALED", which expresses the number of threads waiting for the signal. The following example stores a record and then sends a signal to every thread waiting for a condition variable.
GET /rpc/set HTTP/1.1 Host: localhost:1978 Content-Type: text/tab-separated-values Content-Length: 52 SIGNAL summer SIGNALBROAD key summer value sailing
The server provides the following procedures.
/rpc/void/rpc/echo/rpc/report/rpc/play_scriptname: the name of the procedure to call./rpc/tune_replicationhost: (optional): the name or the address of the master server. If it is omitted, replication is disabled.port: (optional): the port numger of the server. If it is omitted, the default port is specified.ts: (optional): the maximum time stamp of already read logs. If it is omitted, the current setting is not modified. If it is "now", the current time is specified.iv: (optional): the interval of each replication operation in milliseconds. If it is omitted, the current setting is not modified./rpc/statusDB: (optional): the database identifier.count: the number of records.size: the size of the database file./rpc/clearDB: (optional): the database identifier./rpc/synchronizeDB: (optional): the database identifier.hard: (optional): for physical synchronization with the device.command: (optional): the command name to process the database file./rpc/setDB: (optional): the database identifier.key: the key of the record.value: the value of the record.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/addDB: (optional): the database identifier.key: the key of the record.value: the value of the record.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/replaceDB: (optional): the database identifier.key: the key of the record.value: the value of the record.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/appendDB: (optional): the database identifier.key: the key of the record.value: the value of the record.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/incrementDB: (optional): the database identifier.key: the key of the record.num: the additional number.orig: (optional): the origin number. If it is omitted, 0 is specified. "try" means INT64MIN. "set" means INT64MAX.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified.num: the result value./rpc/increment_doubleDB: (optional): the database identifier.key: the key of the record.num: the additional number.orig: (optional): the origin number. If it is omitted, 0 is specified. "try" means negative infinity. "set" means positive infinity.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified.num: the result value./rpc/casDB: (optional): the database identifier.key: the key of the record.oval: (optional): the old value. If it is omittted, no record is meant.nval: (optional): the new value. If it is omittted, the record is removed.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/removeDB: (optional): the database identifier.key: the key of the record./rpc/getDB: (optional): the database identifier.key: the key of the record.value: (optional): the value of the record.xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time./rpc/checkDB: (optional): the database identifier.key: the key of the record.vsiz: (optional): the size of the value of the record.xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time./rpc/seizeDB: (optional): the database identifier.key: the key of the record.value: (optional): the value of the record.xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time./rpc/set_bulkDB: (optional): the database identifier.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified.atomic: (optional): to perform all operations atomically. If it is omitted, non-atomic operations are performed.num: the number of stored reocrds./rpc/remove_bulkDB: (optional): the database identifier.atomic: (optional): to perform all operations atomically. If it is omitted, non-atomic operations are performed.num: the number of removed reocrds./rpc/get_bulkDB: (optional): the database identifier.atomic: (optional): to perform all operations atomically. If it is omitted, non-atomic operations are performed.num: the number of retrieved reocrds./rpc/vacuumDB: (optional): the database identifier.step: (optional): the number of steps. If it is omitted or not more than 0, the whole region is scanned./rpc/match_prefixDB: (optional): the database identifier.prefix: the prefix string.max: (optional): the maximum number to retrieve. If it is omitted or negative, no limit is specified.num: the number of retrieved keys./rpc/match_regexDB: (optional): the database identifier.regex: the regular expression string.max: (optional): the maximum number to retrieve. If it is omitted or negative, no limit is specified.num: the number of retrieved keys./rpc/match_similarDB: (optional): the database identifier.origin: the origin string.range: (optional): the maximum distance of keys to adopt. If it is omitted or negative, 1 is specified.utf: (optional): flag to treat keys as UTF-8 strings. If it is omitted, false is specified.max: (optional): the maximum number to retrieve. If it is omitted or negative, no limit is specified.num: the number of retrieved keys./rpc/cur_jumpDB: (optional): the database identifier.CUR: the cursor identifier.key: (optional): the key of the destination record. If it is omitted, the first record is specified./rpc/cur_jump_backDB: (optional): the database identifier.CUR: the cursor identifier.key: (optional): the key of the destination record. If it is omitted, the last record is specified./rpc/cur_stepCUR: the cursor identifier./rpc/cur_step_backCUR: the cursor identifier./rpc/cur_set_valueCUR: the cursor identifier.value: the value of the record.step: (optional): to move the cursor to the next record. If it is omitted, the cursor stays at the current record.xt: (optional): the expiration time from now in seconds. If it is negative, the absolute value is treated as the epoch time. If it is omitted, no expiration time is specified./rpc/cur_removeCUR: the cursor identifier./rpc/cur_get_keyCUR: the cursor identifier.step: (optional): to move the cursor to the next record. If it is omitted, the cursor stays at the current record./rpc/cur_get_valueCUR: the cursor identifier.step: (optional): to move the cursor to the next record. If it is omitted, the cursor stays at the current record./rpc/cur_getCUR: the cursor identifier.step: (optional): to move the cursor to the next record. If it is omitted, the cursor stays at the current record.xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time./rpc/cur_seizeCUR: the cursor identifier.xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time./rpc/cur_deleteCUR: the cursor identifier.While the procedure name is described in the URL in the above RPC-style interface, it is specified by the kind of method such as GET, HEAD, PUT, DELETE in the RESTful-style interface. And, the objective record is described in the URL. Because a lot of users prefers RESTful-style interface to RPC-style one, Kyoto Tycoon supports the both. The following methods are used in the RESTful-style interface.
GETContent-Length: the size of the value.X-Kt-Xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time.HEADContent-Length: the size of the value.X-Kt-Xt: (optional): the absolute expiration time. If it is omitted, there is no expiration time.PUTContent-Length: the size of the value.X-Kt-Mode: (optional): the method mode. "set", "add", and "replace" are supported. If it is omitted, "set" is specified.X-Kt-Xt: (optional): the absolute expiration time. If it is omitted, no expiration time is specified.DELETEThe space efficiency of RESTful-style interface is superior to the one of RPC-style interface because the entity body does not have any data structure and no encoding is needed.
The path of the URL in the request line must be encoded by URL encoding. If the path begins with "/", the character is ignored and the trailing string is decoded. Moreover, if the path includes "/" in the middle, the segment before the middle "/" is treated as the database identifier and the next segment is decoded as the key. For example, the record whose key is "I love you" in the database "words.kch" is expressed "/words.kch/I%20love%20you" in the request line.
The format of date strings by the "X-Kt-Xt" header is the RFC 1123 date format of GMT. The server understands the W3CDTF format, the RFC 822 (1123) format, and the decimal integer of seconds from the epoch.
In order to realize the best performance, several commands in an efficient binary protocol are supported. As they are available at the same port as other HTTP commands, they can be identified by the first one byte of each request. Every numeric value are expressed in big-endian order. If some error occurred in the server, the magic data of the output would be 0xBF and no data trails.
replicationmagic: (uint8_t): 0xB1: identifier.flags: (uint32_t): flags of bitwise-or. 0x01 for the while SID option.ts: (uint64_t): the maximum time stamp of already read logs.sid: (uint16_t): the server ID number.magic: (uint8_t): 0xB1: identifier.ts: (uint64_t): (iteration): the time stamp of the log.size: (uint32_t): (iteration): the size of the log.log: (variable): (iteration): the data of the log.0xB0. It means that the log reader has reached the current status and been waiting for the next update. In this case, the current time stamp of uint64_t trails the magic data. Clients must respond and send a one-byte message of 0xB1.play_scriptmagic: (uint8_t): 0xB4: identifier.flags: (uint32_t): flags of bitwise-or. 0x01 for the no-reply option.nsiz: (uint32_t): the size of the procedure name.rnum: (uint32_t): the number of the records in the request.name: (variable): the data of the procedure name.ksiz: (uint32_t): (iteration): the size of the key.vsiz: (uint32_t): (iteration): the size of the value.key: (variable): (iteration): the data of the key.value: (variable): (iteration): the data of the value.magic: (uint8_t): 0xB4: identifier.rnum: (uint32_t): the number of the result records.ksiz: (uint32_t): (iteration): the size of the key.vsiz: (uint32_t): (iteration): the size of the value.key: (variable): (iteration): the data of the key.value: (variable): (iteration): the data of the value.set_bulkmagic: (uint8_t): 0xB8: identifier.flags: (uint32_t): flags of bitwise-or. 0x01 for the no-reply option.rnum: (uint32_t): the number of records in the request.dbidx: (uint16_t): (iteration): the index of the target database.ksiz: (uint32_t): (iteration): the size of the key.vsiz: (uint32_t): (iteration): the size of the value.xt: (int64_t): (iteration): the expiration time.key: (variable): (iteration): the data of the key.value: (variable): (iteration): the data of the value.magic: (uint8_t): 0xB8: identifier.hits: (uint32_t): the number of stored records.remove_bulkmagic: (uint8_t): 0xB9: identifier.flags: (uint32_t): flags of bitwise-or. 0x01 for the no-reply option.rnum: (uint32_t): the number of records in the request.dbidx: (uint16_t): (iteration): the index of the target database.ksiz: (uint32_t): (iteration): the size of the key.key: (variable): (iteration): the data of the key.magic: (uint8_t): 0xB9: identifier.hits: (uint32_t): the number of removed records.get_bulkmagic: (uint8_t): 0xBA: identifier.flags: (uint32_t): reserved and not used now. It should be 0.rnum: (uint32_t): the number of records in the request.dbidx: (uint16_t): (iteration): the index of the target database.ksiz: (uint32_t): (iteration): the size of the key.key: (variable): (iteration): the data of the key.magic: (uint8_t): 0xBA: identifier.hits: (uint32_t): the number of retrieved records.dbidx: (uint16_t): (iteration): the index of the target database.ksiz: (uint32_t): (iteration): the size of the key.vsiz: (uint32_t): (iteration): the size of the value.xt: (int64_t): (iteration): the expiration time.key: (variable): (iteration): the data of the key.value: (variable): (iteration): the data of the value.If there is no client library for Kyoto Tycoon in your favorite language, you have to write it by yourself or use the memcached protocol by the pluggable memcached server module. However, it is very easy to implement your own client library for the RESTful interface.
In Python3, the `http.client' module is useful. Use the `httplib' instead in Python2.
import time
import urllib
import http.client
# RESTful interface of Kyoto Tycoon
class KyotoTycoon:
# connect to the server
def open(self, host = "127.0.0.1", port = 1978, timeout = 30):
self.ua = http.client.HTTPConnection(host, port, False, timeout)
# close the connection
def close(self):
self.ua.close()
# store a record
def set(self, key, value, xt = None):
if isinstance(key, str): key = key.encode("UTF-8")
if isinstance(value, str): value = value.encode("UTF-8")
key = "/" + urllib.parse.quote(key)
headers = {}
if xt != None:
xt = int(time.time()) + xt
headers["X-Kt-Xt"] = str(xt)
self.ua.request("PUT", key, value, headers)
res = self.ua.getresponse()
body = res.read()
return res.status == 201
# remove a record
def remove(self, key):
if isinstance(key, str): key = key.encode("UTF-8")
key = "/" + urllib.parse.quote(key)
self.ua.request("DELETE", key)
res = self.ua.getresponse()
body = res.read()
return res.status == 204
# retrieve the value of a record
def get(self, key):
if isinstance(key, str): key = key.encode("UTF-8")
key = "/" + urllib.parse.quote(key)
self.ua.request("GET", key)
res = self.ua.getresponse()
body = res.read()
if res.status != 200: return None
return body
# sample usage
kt = KyotoTycoon()
kt.open("localhost", 1978)
kt.set("japan", "tokyo", 60)
print(kt.get("japan"))
kt.remove("japan")
kt.close()
In Ruby, the `Net::HTTP' module is useful.
require 'uri'
require 'net/http'
# RESTful interface of Kyoto Tycoon
class KyotoTycoon
# connect to the server
def open(host = "127.0.0.1", port = 1978, timeout = 30)
@ua = Net::HTTP::new(host, port)
@ua.read_timeout = timeout
@ua.start
end
# close the connection
def close
@ua.finish
end
# store a record
def set(key, value, xt = nil)
key = "/" + URI::encode(key)
req = Net::HTTP::Put::new(key)
if xt
xt = Time::now.to_i + xt
req.add_field("X-Kt-Xt", xt)
end
res = @ua.request(req, value)
res.code.to_i == 201
end
# remove a record
def remove(key)
key = "/" + URI::encode(key)
req = Net::HTTP::Delete::new(key)
res = @ua.request(req)
res.code.to_i == 204
end
# retrieve the value of a record
def get(key)
key = "/" + URI::encode(key)
req = Net::HTTP::Get::new(key)
res = @ua.request(req)
return nil if res.code.to_i != 200
res.body
end
end
# sample usage
kt = KyotoTycoon::new
kt.open("localhost", 1978)
kt.set("japan", "tokyo", 60)
printf("%s\n", kt.get("japan"))
kt.remove("japan")
kt.close
Kyoto Tycoon is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.
Kyoto Tycoon is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see `http://www.gnu.org/licenses/'.
The FOSS License Exception and the Specific FOSS Library Linking Exception are also provided in order to accommodate products under other free and open source licenses. See the documents of Kyoto Cabinet for details.
Kyoto Tycoon was written and is maintained by FAL Labs. You can contact the author by e-mail to `info@fallabs.com'.