class Tkrzw::DBM

Polymorphic database manager. All operations except for “open” and “close” are thread-safe; Multiple threads can access the same database concurrently. You can specify a data structure when you call the “open” method. Every opened database must be closed explicitly by the “close” method to avoid data corruption.

Constants

ANY_DATA

The special bytes value for no-operation or any data.

Public Class Methods

new() click to toggle source

Does nothing especially.

# File tkrzw.rb, line 265
def initialize()
  # (native code)
end
restore_database(old_file_path, new_file_path, class_name="", end_offset=-1, cipher_key=nil) click to toggle source

Restores a broken database as a new healthy database.

  • @param old_file_path The path of the broken database.

  • @param new_file_path The path of the new database to be created.

  • @param class_name The name of the database class. If it is nil or empty, the class is guessed from the file extension.

  • @param end_offset The exclusive end offset of records to read. Negative means unlimited. 0 means the size when the database is synched or closed properly. Using a positive value is not meaningful if the number of shards is more than one.

  • @param cipher_key The encryption key for cipher compressors. If it is nil, an empty key is used.

  • @return The result status.

# File tkrzw.rb, line 677
def self.restore_database(old_file_path, new_file_path, class_name="",
                          end_offset=-1, cipher_key=nil)
  # (native code)
end

Public Instance Methods

[](key) click to toggle source

Gets the value of a record, to enable the [] operator.

  • @param key The key of the record.

  • @return The value of the matching record or nil on failure.

# File tkrzw.rb, line 703
def [](key)
  # (native code)
end
[]=(key, value) click to toggle source

Sets a record of a key and a value, to enable the []= operator.

  • @param key The key of the record.

  • @param value The value of the record.

  • @return The new value of the record or nil on failure.

# File tkrzw.rb, line 711
def []=(key, value)
  # (native code)
end
append(key, value, delim="") click to toggle source

Appends data at the end of a record of a key.

  • @param key The key of the record.

  • @param value The value to append.

  • @param delim The delimiter to put after the existing record.

  • @return The result status.

If there's no existing record, the value is set without the delimiter.

# File tkrzw.rb, line 436
def append(key, value, delim="")
  # (native code)
end
append_multi(delim="", **records) click to toggle source

Appends data to multiple records of the keyword arguments.

  • @param delim The delimiter to put after the existing record.

  • @param records Records to append, specified as keyword parameters.

  • @return The result status.

# File tkrzw.rb, line 444
def append_multi(delim="", **records)
  # (native code)
end
clear() click to toggle source

Removes all records.

  • @return The result status.

# File tkrzw.rb, line 556
def clear()
  # (native code)
end
close() click to toggle source

Closes the database file.

  • @return The result status.

# File tkrzw.rb, line 347
def close()
  # (native code)
end
compare_exchange(key, expected, desired) click to toggle source

Compares the value of a record and exchanges if the condition meets.

  • @param key The key of the record.

  • @param expected The expected value. If it is nil, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted.

  • @param desired The desired value. If it is nil, the record is to be removed. If it is ANY_DATA, no update is done.

  • @return The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.

# File tkrzw.rb, line 453
def compare_exchange(key, expected, desired)
  # (native code)
end
compare_exchange_and_get(key, expected, desired) click to toggle source

Does compare-and-exchange and/or gets the old value of the record.

  • @param key The key of the record.

  • @param expected The expected value. If it is nil, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted.

  • @param desired The desired value. If it is nil, the record is to be removed. If it is ANY_DATA, no update is done.

  • @return A pair of the result status and the.old value of the record. If the condition doesn't meet, the state is INFEASIBLE_ERROR. If there's no existing record, the value is nil.

# File tkrzw.rb, line 462
def compare_exchange_and_get(key, expected, desired)
  # (native code)
end
compare_exchange_multi(expected, desired) click to toggle source

Compares the values of records and exchanges if the condition meets.

  • @param expected An array of pairs of the record keys and their expected values. If the value is nil, no existing record is expected. If the value is ANY_DATA, an existing record with any value is expacted.

  • @param desired An array of pairs of the record keys and their desired values. If the value is nil, the record is to be removed.

  • @return The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.

# File tkrzw.rb, line 490
def compare_exchange_multi(expected, desired)
  # (native code)
end
copy_file_data(dest_path, sync_hard=false) click to toggle source

Copies the content of the database file to another file.

  • @param dest_path A path to the destination file.

  • @param sync_hard True to do physical synchronization with the hardware.

  • @return The result status.

# File tkrzw.rb, line 590
def copy_file_data(dest_path, sync_hard=false)
  # (native code)
end
count() click to toggle source

Gets the number of records.

  • @return The number of records on success, or nil on failure.

# File tkrzw.rb, line 532
def count()
  # (native code)
end
delete(key) click to toggle source

Removes a record of a key.

  • @param key The key of the record.

  • @return The old value of the record or nil on failure.

# File tkrzw.rb, line 718
def delete(key)
  # (native code)
end
destruct() click to toggle source

Releases the resource explicitly. The database is closed implicitly if it has not been closed. As long as you close the database explicitly, you don't have to call this method.

# File tkrzw.rb, line 271
def destruct()
  # (native code)
end
each(&block) click to toggle source

Calls the given block with the key and the value of each record

# File tkrzw.rb, line 723
def each(&block)
  # (native code)
end
export(dest_dbm) click to toggle source

Exports all records to another database.

  • @param dest_dbm The destination database.

  • @return The result status.

# File tkrzw.rb, line 597
def export(dest_dbm)
  # (native code)
end
export_keys_as_lines(dest_file) click to toggle source

Exports the keys of all records as lines to a text file.

  • @param dest_file The file object to write keys in.

  • @return The result status.

As the exported text file is smaller than the database file, scanning the text file by the search method is often faster than scanning the whole database.

# File tkrzw.rb, line 620
def export_keys_as_lines(dest_file)
  # (native code)
end
export_to_flat_records(dest_file) click to toggle source

Exports all records of a database to a flat record file.

  • @param dest_file The file object to write records in.

  • @return The result status.

A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration.

# File tkrzw.rb, line 605
def export_to_flat_records(dest_file)
  # (native code)
end
file_path() click to toggle source

Gets the path of the database file.

  • @return The file path of the database, or nil on failure.

# File tkrzw.rb, line 544
def file_path()
  # (native code)
end
file_size() click to toggle source

Gets the current file size of the database.

  • @return The current file size of the database, or nil on failure.

# File tkrzw.rb, line 538
def file_size()
  # (native code)
end
get(key, status=nil) click to toggle source

Gets the value of a record of a key.

  • @param key The key of the record.

  • @param status A status object to which the result status is assigned. It can be omitted.

  • @return The value of the matching record or nil on failure.

# File tkrzw.rb, line 372
def get(key, status=nil)
  # (native code)
end
get_multi(*keys) click to toggle source

Gets the values of multiple records of keys.

  • @param keys The keys of records to retrieve.

  • @return A map of retrieved records. Keys which don't match existing records are ignored.

# File tkrzw.rb, line 379
def get_multi(*keys)
  # (native code)
end
healthy?() click to toggle source

Checks whether the database condition is healthy.

  • @return True if the database condition is healthy, or false if not.

# File tkrzw.rb, line 644
def healthy?()
  # (native code)
end
import_from_flat_records(src_file) click to toggle source

Imports records to a database from a flat record file.

  • @param src_file The file object to read records from.

  • @return The result status.

# File tkrzw.rb, line 612
def import_from_flat_records(src_file)
  # (native code)
end
include?(key) click to toggle source

Checks if a record exists or not.

  • @param key The key of the record.

  • @return True if the record exists, or false if not.

# File tkrzw.rb, line 364
def include?(key)
  # (native code)
end
increment(key, inc=1, init=0, status=nil) click to toggle source

Increments the numeric value of a record.

  • @param key The key of the record.

  • @param inc The incremental value. If it is Utility::INT64MIN, the current value is not changed and a new record is not created.

  • @param init The initial value.

  • @param status A status object to which the result status is assigned. It can be omitted.

  • @return The current value, or nil on failure.

The record value is stored as an 8-byte big-endian integer. Negative is also supported.

# File tkrzw.rb, line 473
def increment(key, inc=1, init=0, status=nil)
  # (native code)
end
inspect() click to toggle source

Returns a string representation of the object.

  • @return The string representation of the object.

# File tkrzw.rb, line 696
def inspect()
  # (native code)
end
inspect_details() click to toggle source

Inspects the database.

  • @return A hash of property names and their values.

# File tkrzw.rb, line 626
def inspect_details()
  # (native code)
end
make_iterator() click to toggle source

Makes an iterator for each record.

  • @return The iterator for each record.

Every iterator should be destructed explicitly by the “destruct” method.

# File tkrzw.rb, line 666
def make_iterator()
  # (native code)
end
open(path, writable, **params) click to toggle source

Opens a database file.

  • @param path A path of the file.

  • @param writable If true, the file is writable. If false, it is read-only.

  • @param params Optional keyword parameters.

  • @return The result status.

The extension of the path indicates the type of the database.

  • .tkh : File hash database (HashDBM)

  • .tkt : File tree database (TreeDBM)

  • .tks : File skip database (SkipDBM)

  • .tkmt : On-memory hash database (TinyDBM)

  • .tkmb : On-memory tree database (BabyDBM)

  • .tkmc : On-memory cache database (CacheDBM)

  • .tksh : On-memory STL hash database (StdHashDBM)

  • .tkst : On-memory STL tree database (StdTreeDBM)

The optional parameters can include an option for the concurrency tuning. By default, database operatins are done under the GVL (Global Virtual-machine Lock), which means that database operations are not done concurrently even if you use multiple threads. If the “concurrent” parameter is true, database operations are done outside the GVL, which means that database operations can be done concurrently if you use multiple threads. However, the downside is that swapping thread data is costly so the actual throughput is often worse in the concurrent mode than in the normal mode. Therefore, the concurrent mode should be used only if the database is huge and it can cause blocking of threads in multi-thread usage.
By default, the encoding of retrieved record data by the “get” method is implicitly set as “ASCII-8BIT”. If you want to change the implicit encoding to “UTF-8” or others, set the encoding name as the value of the “encoding” parameter. The optional parameters can include options for the file opening operation.

  • truncate (bool): True to truncate the file.

  • no_create (bool): True to omit file creation.

  • no_wait (bool): True to fail if the file is locked by another process.

  • no_lock (bool): True to omit file locking.

  • sync_hard (bool): True to do physical synchronization when closing.

The optional parameter “dbm” supercedes the decision of the database type by the extension. The value is the type name: “HashDBM”, “TreeDBM”, “SkipDBM”, “TinyDBM”, “BabyDBM”, “CacheDBM”, “StdHashDBM”, “StdTreeDBM”.
The optional parameter “file” specifies the internal file implementation class. The default file class is “MemoryMapAtomicFile”. The other supported classes are “StdFile”, “MemoryMapAtomicFile”, “PositionalParallelFile”, and “PositionalAtomicFile”.
For HashDBM, these optional parameters are supported.

  • update_mode (string): How to update the database file: “UPDATE_IN_PLACE” for the in-palce or “UPDATE_APPENDING” for the appending mode.

  • record_crc_mode (string): How to add the CRC data to the record: “RECORD_CRC_NONE” to add no CRC to each record, “RECORD_CRC_8” to add CRC-8 to each record, “RECORD_CRC_16” to add CRC-16 to each record, or “RECORD_CRC_32” to add CRC-32 to each record.

  • record_comp_mode (string): How to compress the record data: “RECORD_COMP_NONE” to do no compression, “RECORD_COMP_ZLIB” to compress with ZLib, “RECORD_COMP_ZSTD” to compress with ZStd, “RECORD_COMP_LZ4” to compress with LZ4, “RECORD_COMP_LZMA” to compress with LZMA, “RECORD_COMP_RC4” to cipher with RC4, “RECORD_COMP_AES” to cipher with AES.

  • offset_width (int): The width to represent the offset of records.

  • align_pow (int): The power to align records.

  • num_buckets (int): The number of buckets for hashing.

  • restore_mode (string): How to restore the database file: “RESTORE_SYNC” to restore to the last synchronized state, “RESTORE_READ_ONLY” to make the database read-only, or “RESTORE_NOOP” to do nothing. By default, as many records as possible are restored.

  • fbp_capacity (int): The capacity of the free block pool.

  • min_read_size (int): The minimum reading size to read a record.

  • cache_buckets (bool): True to cache the hash buckets on memory.

  • cipher_key (string): The encryption key for cipher compressors.

For TreeDBM, all optional parameters for HashDBM are available. In addition, these optional parameters are supported.

  • max_page_size (int): The maximum size of a page.

  • max_branches (int): The maximum number of branches each inner node can have.

  • max_cached_pages (int): The maximum number of cached pages.

  • page_update_mode (string): What to do when each page is updated: “PAGE_UPDATE_NONE” is to do no operation or “PAGE_UPDATE_WRITE” is to write immediately.

  • key_comparator (string): The comparator of record keys: “LexicalKeyComparator” for the lexical order, “LexicalCaseKeyComparator” for the lexical order ignoring case, “DecimalKeyComparator” for the order of the decimal integer numeric expressions, “HexadecimalKeyComparator” for the order of the hexadecimal integer numeric expressions, “RealNumberKeyComparator” for the order of the decimal real number expressions.

For SkipDBM, these optional parameters are supported.

  • offset_width (int): The width to represent the offset of records.

  • step_unit (int): The step unit of the skip list.

  • max_level (int): The maximum level of the skip list.

  • restore_mode (string): How to restore the database file: “RESTORE_SYNC” to restore to the last synchronized state or “RESTORE_NOOP” to do nothing make the database read-only. By default, as many records as possible are restored.

  • sort_mem_size (int): The memory size used for sorting to build the database in the at-random mode.

  • insert_in_order (bool): If true, records are assumed to be inserted in ascending order of the key.

  • max_cached_records (int): The maximum number of cached records.

For TinyDBM, these optional parameters are supported.

  • num_buckets (int): The number of buckets for hashing.

For BabyDBM, these optional parameters are supported.

  • key_comparator (string): The comparator of record keys. The same ones as TreeDBM.

For CacheDBM, these optional parameters are supported.

  • cap_rec_num (int): The maximum number of records.

  • cap_mem_size (int): The total memory size to use.

All databases support taking update logs into files. It is enabled by setting the prefix of update log files.

  • ulog_prefix (str): The prefix of the update log files.

  • ulog_max_file_size (num): The maximum file size of each update log file. By default, it is 1GiB.

  • ulog_server_id (num): The server ID attached to each log. By default, it is 0.

  • ulog_dbm_index (num): The DBM index attached to each log. By default, it is 0.

For the file “PositionalParallelFile” and “PositionalAtomicFile”, these optional parameters are supported.

  • block_size (int): The block size to which all blocks should be aligned.

  • access_options (str): Values separated by colon. “direct” for direct I/O. “sync” for synchrnizing I/O, “padding” for file size alignment by padding, “pagecache” for the mini page cache in the process.

If the optional parameter “num_shards” is set, the database is sharded into multiple shard files. Each file has a suffix like “-00003-of-00015”. If the value is 0, the number of shards is set by patterns of the existing files, or 1 if they doesn't exist.

# File tkrzw.rb, line 341
def open(path, writable, **params)
  # (native code)
end
open?() click to toggle source

Checks whether the database is open.

  • @return True if the database is open, or false if not.

# File tkrzw.rb, line 632
def open?()
  # (native code)
end
ordered?() click to toggle source

Checks whether ordered operations are supported.

  • @return True if ordered operations are supported, or false if not.

# File tkrzw.rb, line 650
def ordered?()
  # (native code)
end
pop_first(status=nil) click to toggle source

Gets the first record and removes it.

  • @param status A status object to which the result status is assigned. It can be omitted.

  • @return A tuple of The key and the value of the first record. On failure, nil is returned.

# File tkrzw.rb, line 508
def pop_first(status=nil)
  # (native code)
end
process(key, writable) click to toggle source

Processes a record with an arbitrary block.

  • @param key The key of the record.

  • @param writable True if the processor can edit the record.

  • @block The block to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or nil if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is nil, the record is not modified. If the return value is false (not a false value but the false object), the record is removed.

  • @return The result status.

This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.

# File tkrzw.rb, line 357
def process(key, writable)
  # (native code)
end
process_each(writable) click to toggle source

Processes each and every record in the database with an arbitrary block.

  • @block The block to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or nil if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is nil, the record is not modified. If the return value is false (not a false value but the false object), the record is removed.

  • @param writable True if the processor can edit the record.

  • @return The result status.

The block function is called repeatedly for each record. It is also called once before the iteration and once after the iteration with both the key and the value being nil. This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.

# File tkrzw.rb, line 526
def process_each(writable)
  # (native code)
end
process_multi(keys, writable) click to toggle source

Processes multiple records with an arbitrary block.

  • @param keys A list of record keys.

  • @block The block to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or nil if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is nil, the record is not modified. If the return value is false (not a false value but the false object), the record is removed.

  • @return The result status.

This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.

# File tkrzw.rb, line 482
def process_multi(keys, writable)
  # (native code)
end
push_last(value, wtime=nil) click to toggle source

Adds a record with a key of the current timestamp.

  • @param value The value of the record.

  • @param wtime The current wall time used to generate the key. If it is nil, the system clock is used.

  • @return The result status.

The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds.

# File tkrzw.rb, line 517
def push_last(value, wtime=nil)
  # (native code)
end
rebuild(**params) click to toggle source

Rebuilds the entire database.

  • @param params Optional keyword parameters.

  • @return The result status.

The optional parameters are the same as the “open” method. Omitted tuning parameters are kept the same or implicitly optimized.
In addition, HashDBM, TreeDBM, and SkipDBM supports the following parameters.

  • skip_broken_records (bool): If true, the operation continues even if there are broken records which can be skipped.

  • sync_hard (bool): If true, physical synchronization with the hardware is done before finishing the rebuilt file.

# File tkrzw.rb, line 567
def rebuild(**params)
  # (native code)
end
rekey(old_key, new_key, overwrite=true, copying=false) click to toggle source

Changes the key of a record.

  • @param old_key The old key of the record.

  • @param new_key The new key of the record.

  • @param overwrite Whether to overwrite the existing record of the new key.

  • @param copying Whether to retain the record of the old key.

  • @return The result status. If there's no matching record to the old key, NOT_FOUND_ERROR is returned. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is returned.

This method is done atomically. The other threads observe that the record has either the old key or the new key. No intermediate states are observed.

# File tkrzw.rb, line 501
def rekey(old_key, new_key, overwrite=true, copying=false)
  # (native code)
end
remove(key) click to toggle source

Removes a record of a key.

  • @param key The key of the record.

  • @return The result status. If there's no matching record, NOT_FOUND_ERROR is returned.

# File tkrzw.rb, line 412
def remove(key)
  # (native code)
end
remove_and_get(key) click to toggle source

Removes a record and get the value.

  • @param key The key of the record.

  • @return A pair of the result status and the record value. If the record does not exist, nil is assigned as the value.

# File tkrzw.rb, line 426
def remove_and_get(key)
  # (native code)
end
remove_multi(*keys) click to toggle source

Removes records of keys.

  • @param keys The keys of the records.

  • @return The result status. If there are missing records, NOT_FOUND_ERROR is returned.

# File tkrzw.rb, line 419
def remove_multi(*keys)
  # (native code)
end
set(key, value, overwrite=true) click to toggle source

Sets a record of a key and a value.

  • @param key The key of the record.

  • @param value The value of the record.

  • @param overwrite Whether to overwrite the existing value. It can be omitted and then false is set.

  • @return The result status. If overwriting is abandoned, DUPLICATION_ERROR is returned.

# File tkrzw.rb, line 388
def set(key, value, overwrite=true)
  # (native code)
end
set_and_get(key, value, overwrite=true) click to toggle source

Sets a record and get the old value.

  • @param key The key of the record.

  • @param value The value of the record.

  • @param overwrite Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.

  • @return A pair of the result status and the old value. If the record has not existed when inserting the new record, nil is assigned as the value.

# File tkrzw.rb, line 405
def set_and_get(key, value, overwrite=true)
  # (native code)
end
set_multi(overwrite=true, **records) click to toggle source

Sets multiple records of the keyword arguments.

  • @param overwrite Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.

  • @param records Records to store, specified as keyword parameters.

  • @return The result status. If there are records avoiding overwriting, DUPLICATION_ERROR is returned.

# File tkrzw.rb, line 396
def set_multi(overwrite=true, **records)
  # (native code)
end
should_be_rebuilt?() click to toggle source

Checks whether the database should be rebuilt.

  • @return True to be optimized or False with no necessity.

# File tkrzw.rb, line 573
def should_be_rebuilt?()
  # (native code)
end
synchronize(hard, **params) click to toggle source

Synchronizes the content of the database to the file system.

  • @param hard True to do physical synchronization with the hardware or false to do only logical synchronization with the file system.

  • @param params Optional keyword parameters.

  • @return The result status.

Only SkipDBM uses the optional parameters. The “merge” parameter specifies paths of databases to merge, separated by colon. The “reducer” parameter specifies the reducer to apply to records of the same key. “ReduceToFirst”, “ReduceToSecond”, “ReduceToLast”, etc are supported.

# File tkrzw.rb, line 582
def synchronize(hard, **params)
  # (native code)
end
timestamp() click to toggle source

Gets the timestamp in seconds of the last modified time.

  • @return The timestamp of the last modified time, or nil on failure.

# File tkrzw.rb, line 550
def timestamp()
  # (native code)
end
to_i() click to toggle source

Gets the number of records.

  • @return The number of records on success, or -1 on failure.

# File tkrzw.rb, line 690
def to_i()
  # (native code)
end
to_s() click to toggle source

Returns a string representation of the content.

  • @return The string representation of the content.

# File tkrzw.rb, line 684
def to_s()
  # (native code)
end
writable?() click to toggle source

Checks whether the database is writable.

  • @return True if the database is writable, or false if not.

# File tkrzw.rb, line 638
def writable?()
  # (native code)
end