Tkrzw
|
Asynchronous database manager adapter. More...
#include <tkrzw_dbm_async.h>
Classes | |
class | CommonPostprocessor |
Interface of a common post processor for a record. More... | |
class | RecordProcessor |
Interface of asynchronous processor for a record. More... | |
Public Types | |
typedef std::function< void(const char *, const Status &)> | CommonPostLambdaType |
Lambda function type to postprocess a database operation. More... | |
Public Member Functions | |
AsyncDBM (DBM *dbm, int32_t num_worker_threads) | |
Constructor. More... | |
~AsyncDBM () | |
Destructor. More... | |
AsyncDBM (const AsyncDBM &rhs)=delete | |
Copy and assignment are disabled. More... | |
AsyncDBM & | operator= (const AsyncDBM &rhs)=delete |
void | SetCommonPostprocessor (std::unique_ptr< CommonPostprocessor > proc) |
Set the common post processor. More... | |
void | SetCommonPostprocessor (CommonPostLambdaType post_lambda) |
Set the common post processor with a lambda function. More... | |
template<typename PROC > | |
std::future< std::pair< Status, std::unique_ptr< PROC > > > | Process (std::string_view key, std::unique_ptr< PROC > proc, bool writable) |
Processes a record with a processor. More... | |
std::future< Status > | Process (std::string_view key, DBM::RecordLambdaType rec_lambda, bool writable) |
Processes a record with a lambda function. More... | |
std::future< std::pair< Status, std::string > > | Get (std::string_view key) |
Gets the value of a record of a key. More... | |
std::future< std::pair< Status, std::map< std::string, std::string > > > | GetMulti (const std::vector< std::string_view > &keys) |
Gets the values of multiple records of keys, with a string view vector. More... | |
std::future< std::pair< Status, std::map< std::string, std::string > > > | GetMulti (const std::vector< std::string > &keys) |
Gets the values of multiple records of keys, with a string vector. More... | |
std::future< Status > | Set (std::string_view key, std::string_view value, bool overwrite=true) |
Sets a record of a key and a value. More... | |
std::future< Status > | SetMulti (const std::map< std::string_view, std::string_view > &records, bool overwrite=true) |
Sets multiple records, with a map of string views. More... | |
std::future< Status > | SetMulti (const std::map< std::string, std::string > &records, bool overwrite=true) |
Sets multiple records, with a map of strings. More... | |
std::future< Status > | Remove (std::string_view key) |
Removes a record of a key. More... | |
std::future< Status > | RemoveMulti (const std::vector< std::string_view > &keys) |
Removes records of keys, with a string view vector. More... | |
std::future< Status > | RemoveMulti (const std::vector< std::string > &keys) |
Removes records of keys, with a string vector. More... | |
std::future< Status > | Append (std::string_view key, std::string_view value, std::string_view delim="") |
Appends data at the end of a record of a key. More... | |
std::future< Status > | AppendMulti (const std::map< std::string_view, std::string_view > &records, std::string_view delim="") |
Appends data to multiple records, with a map of string views. More... | |
std::future< Status > | AppendMulti (const std::map< std::string, std::string > &records, std::string_view delim="") |
Appends data to multiple records, with a map of strings. More... | |
std::future< Status > | CompareExchange (std::string_view key, std::string_view expected, std::string_view desired) |
Compares the value of a record and exchanges if the condition meets. More... | |
std::future< Status > | CompareExchangeMulti (const std::vector< std::pair< std::string_view, std::string_view >> &expected, const std::vector< std::pair< std::string_view, std::string_view >> &desired) |
Compares the values of records and exchanges if the condition meets. More... | |
std::future< std::pair< Status, int64_t > > | Increment (std::string_view key, int64_t increment=1, int64_t initial=0) |
Increments the numeric value of a record. More... | |
std::future< Status > | Rekey (std::string_view old_key, std::string_view new_key, bool overwrite=true, bool copying=false) |
Changes the key of a record. More... | |
template<typename PROC > | |
std::future< std::pair< Status, std::unique_ptr< PROC > > > | ProcessFirst (std::unique_ptr< PROC > proc, bool writable) |
Processes the first record with a processor. More... | |
std::future< Status > | ProcessFirst (DBM::RecordLambdaType rec_lambda, bool writable) |
Processes the first record with a lambda function. More... | |
std::future< std::pair< Status, std::pair< std::string, std::string > > > | PopFirst () |
Gets the first record and removes it. More... | |
std::future< Status > | PushLast (std::string_view value, double wtime=-1) |
Adds a record with a key of the current timestamp. More... | |
template<typename PROC > | |
std::future< std::pair< Status, std::vector< std::shared_ptr< PROC > > > > | ProcessMulti (const std::vector< std::pair< std::string_view, std::shared_ptr< PROC >>> &key_proc_pairs, bool writable) |
Processes multiple records with processors. More... | |
std::future< Status > | ProcessMulti (const std::vector< std::pair< std::string_view, DBM::RecordLambdaType >> &key_lambda_pairs, bool writable) |
Processes multiple records with lambda functions. More... | |
template<typename PROC > | |
std::future< std::pair< Status, std::unique_ptr< PROC > > > | ProcessEach (std::unique_ptr< PROC > proc, bool writable) |
Processes each and every record in the database with a processor. More... | |
std::future< Status > | ProcessEach (DBM::RecordLambdaType rec_lambda, bool writable) |
Processes each and every record in the database with a lambda function. More... | |
std::future< Status > | Clear () |
Removes all records. More... | |
std::future< Status > | Rebuild (const std::map< std::string, std::string > ¶ms={}) |
Rebuilds the entire database. More... | |
std::future< Status > | Synchronize (bool hard, std::unique_ptr< DBM::FileProcessor > proc=nullptr, const std::map< std::string, std::string > ¶ms={}) |
Synchronizes the content of the database to the file system. More... | |
std::future< Status > | CopyFileData (const std::string &dest_path, bool sync_hard=false) |
Copies the content of the database file to another file. More... | |
std::future< Status > | Export (DBM *dbm) |
Exports all records to another database. More... | |
std::future< Status > | ExportToFlatRecords (File *dest_file) |
Exports all records of a database to a flat record file. More... | |
std::future< Status > | ImportFromFlatRecords (File *src_file) |
Imports records to a database from a flat record file. More... | |
std::future< std::pair< Status, std::vector< std::string > > > | SearchModal (std::string_view mode, std::string_view pattern, size_t capacity=0) |
Searches the database and get keys which match a pattern, according to a mode expression. More... | |
TaskQueue * | GetTaskQueue () |
Gets the internal task queue. More... | |
Asynchronous database manager adapter.
This class is a wrapper of DBM for asynchronous operations. A task queue with a thread pool is used inside. Every method except for the constructor and the destructor is run by a thread in the thread pool and the result is set in the future oject of the return value. The caller can ignore the future object if it is not necessary. The destructor of this asynchronous database manager waits for all tasks to be done. Therefore, the destructor should be called before the database is closed.
typedef std::function<void(const char*, const Status&)> tkrzw::AsyncDBM::CommonPostLambdaType |
Lambda function type to postprocess a database operation.
The first parameter is the method name of the database operation. The second parameter is the status of the database operation.
tkrzw::AsyncDBM::AsyncDBM | ( | DBM * | dbm, |
int32_t | num_worker_threads | ||
) |
Constructor.
dbm | A database object which has been opened. The ownership is not taken. |
num_worker_threads | The number of threads in the internal thread pool. |
tkrzw::AsyncDBM::~AsyncDBM | ( | ) |
Destructor.
|
explicitdelete |
Copy and assignment are disabled.
void tkrzw::AsyncDBM::SetCommonPostprocessor | ( | std::unique_ptr< CommonPostprocessor > | proc | ) |
Set the common post processor.
proc | the common post processor. If it is nullptr, no postprocess is done. The ownership is taken. |
By default or if nullptr is set, no postprocess is done. If a processor is set, its Postprocess method is called every time in parallel after every method is called.
void tkrzw::AsyncDBM::SetCommonPostprocessor | ( | CommonPostLambdaType | post_lambda | ) |
Set the common post processor with a lambda function.
post_lambda | the lambda function of the common post processor. |
By default, no postprocess is done. If a processor is set, it is called every time in parallel after every method is called.
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::Process | ( | std::string_view | key, |
std::unique_ptr< PROC > | proc, | ||
bool | writable | ||
) |
Processes a record with a processor.
key | The key of the record. |
proc | The processor object derived from RecordProcessor. The ownership is taken. |
writable | True if the processor can edit the record. |
If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.
std::future< Status > tkrzw::AsyncDBM::Process | ( | std::string_view | key, |
DBM::RecordLambdaType | rec_lambda, | ||
bool | writable | ||
) |
Processes a record with a lambda function.
key | The key of the record. |
rec_lambda | The lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value. |
writable | True if the processor can edit the record. |
std::future<std::pair<Status, std::string> > tkrzw::AsyncDBM::Get | ( | std::string_view | key | ) |
Gets the value of a record of a key.
key | The key of the record. |
std::future<std::pair<Status, std::map<std::string, std::string> > > tkrzw::AsyncDBM::GetMulti | ( | const std::vector< std::string_view > & | keys | ) |
Gets the values of multiple records of keys, with a string view vector.
keys | The keys of records to retrieve. |
std::future<std::pair<Status, std::map<std::string, std::string> > > tkrzw::AsyncDBM::GetMulti | ( | const std::vector< std::string > & | keys | ) |
Gets the values of multiple records of keys, with a string vector.
keys | The keys of records to retrieve. |
std::future<Status> tkrzw::AsyncDBM::Set | ( | std::string_view | key, |
std::string_view | value, | ||
bool | overwrite = true |
||
) |
Sets a record of a key and a value.
key | The key of the record. |
value | The value of the record. |
overwrite | Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. |
std::future<Status> tkrzw::AsyncDBM::SetMulti | ( | const std::map< std::string_view, std::string_view > & | records, |
bool | overwrite = true |
||
) |
Sets multiple records, with a map of string views.
records | The records to store. |
overwrite | Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. |
std::future<Status> tkrzw::AsyncDBM::SetMulti | ( | const std::map< std::string, std::string > & | records, |
bool | overwrite = true |
||
) |
Sets multiple records, with a map of strings.
records | The records to store. |
overwrite | Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. |
std::future<Status> tkrzw::AsyncDBM::Remove | ( | std::string_view | key | ) |
Removes a record of a key.
key | The key of the record. |
std::future<Status> tkrzw::AsyncDBM::RemoveMulti | ( | const std::vector< std::string_view > & | keys | ) |
Removes records of keys, with a string view vector.
keys | The keys of records to remove. |
std::future<Status> tkrzw::AsyncDBM::RemoveMulti | ( | const std::vector< std::string > & | keys | ) |
Removes records of keys, with a string vector.
keys | The keys of records to remove. |
std::future<Status> tkrzw::AsyncDBM::Append | ( | std::string_view | key, |
std::string_view | value, | ||
std::string_view | delim = "" |
||
) |
Appends data at the end of a record of a key.
key | The key of the record. |
value | The value to append. |
delim | The delimiter to put after the existing record. |
If there's no existing record, the value is set without the delimiter.
std::future<Status> tkrzw::AsyncDBM::AppendMulti | ( | const std::map< std::string_view, std::string_view > & | records, |
std::string_view | delim = "" |
||
) |
Appends data to multiple records, with a map of string views.
records | The records to append. |
delim | The delimiter to put after the existing record. |
If there's no existing record, the value is set without the delimiter.
std::future<Status> tkrzw::AsyncDBM::AppendMulti | ( | const std::map< std::string, std::string > & | records, |
std::string_view | delim = "" |
||
) |
Appends data to multiple records, with a map of strings.
records | The records to append. |
delim | The delimiter to put after the existing record. |
If there's no existing record, the value is set without the delimiter.
std::future<Status> tkrzw::AsyncDBM::CompareExchange | ( | std::string_view | key, |
std::string_view | expected, | ||
std::string_view | desired | ||
) |
Compares the value of a record and exchanges if the condition meets.
key | The key of the record. |
expected | The expected value. If the data is nullptr, no existing record is expected. If it is DBM::ANY_DATA, an existing record with any value is expacted. |
desired | The desired value. If the data is nullptr, the record is to be removed. If it is DBM::ANY_DATA, no update is done. |
std::future<Status> tkrzw::AsyncDBM::CompareExchangeMulti | ( | const std::vector< std::pair< std::string_view, std::string_view >> & | expected, |
const std::vector< std::pair< std::string_view, std::string_view >> & | desired | ||
) |
Compares the values of records and exchanges if the condition meets.
expected | The record keys and their expected values. If the value is nullptr, no existing record is expected. If the value is DBM::ANY_DATA, an existing record with any value is expacted. |
desired | The record keys and their desired values. If the value is nullptr, the record is to be removed. |
std::future<std::pair<Status, int64_t> > tkrzw::AsyncDBM::Increment | ( | std::string_view | key, |
int64_t | increment = 1 , |
||
int64_t | initial = 0 |
||
) |
Increments the numeric value of a record.
key | The key of the record. |
increment | The incremental value. If it is INT64MIN, the current value is not changed and a new record is not created. |
initial | The initial value. |
The record value is stored as an 8-byte big-endian integer. Negative is also supported.
std::future<Status> tkrzw::AsyncDBM::Rekey | ( | std::string_view | old_key, |
std::string_view | new_key, | ||
bool | overwrite = true , |
||
bool | copying = false |
||
) |
Changes the key of a record.
old_key | The old key of the record. |
new_key | The new key of the record. |
overwrite | Whether to overwrite the existing record of the new key. |
copying | Whether to retain the record of the old key. |
This method is done atomically by ProcessMulti. The other threads observe that the record has either the old key or the new value. No intermediate states are observed.
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::ProcessFirst | ( | std::unique_ptr< PROC > | proc, |
bool | writable | ||
) |
Processes the first record with a processor.
proc | The processor object derived from RecordProcessor. The ownership is taken. |
writable | True if the processor can edit the record. |
If the first record exists, the ProcessFull of the processor is called. Otherwise, this method fails and no method of the processor is called. Whereas ordered databases have efficient implementations of this method, unordered databases have inefficient implementations.
std::future< Status > tkrzw::AsyncDBM::ProcessFirst | ( | DBM::RecordLambdaType | rec_lambda, |
bool | writable | ||
) |
Processes the first record with a lambda function.
rec_lambda | The lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the record. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value. |
writable | True if the processor can edit the record. |
std::future<std::pair<Status, std::pair<std::string, std::string> > > tkrzw::AsyncDBM::PopFirst | ( | ) |
Gets the first record and removes it.
key | The pointer to a string object to contain the key of the first record. If it is nullptr, it is ignored. |
value | The pointer to a string object to contain the value of the first record. If it is nullptr, it is ignored. |
std::future<Status> tkrzw::AsyncDBM::PushLast | ( | std::string_view | value, |
double | wtime = -1 |
||
) |
Adds a record with a key of the current timestamp.
value | The value of the record. |
wtime | The current wall time used to generate the key. If it is negative, the system clock is used. |
The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds.
std::future< std::pair< Status, std::vector< std::shared_ptr< PROC > > > > tkrzw::AsyncDBM::ProcessMulti | ( | const std::vector< std::pair< std::string_view, std::shared_ptr< PROC >>> & | key_proc_pairs, |
bool | writable | ||
) |
Processes multiple records with processors.
key_proc_pairs | Pairs of the keys and their processor objects derived from RecordProcessor. The ownership is taken. |
writable | True if the processors can edit the records. |
If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.
std::future< Status > tkrzw::AsyncDBM::ProcessMulti | ( | const std::vector< std::pair< std::string_view, DBM::RecordLambdaType >> & | key_lambda_pairs, |
bool | writable | ||
) |
Processes multiple records with lambda functions.
key_lambda_pairs | Pairs of the keys and their lambda functions. The first parameter of the lambda functions is the key of the record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value. |
writable | True if the processors can edit the records. |
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::ProcessEach | ( | std::unique_ptr< PROC > | proc, |
bool | writable | ||
) |
Processes each and every record in the database with a processor.
proc | The processor object derived from RecordProcessor. The ownership is taken. |
writable | True if the processor can edit the record. |
The ProcessFull of the processor is called repeatedly for each record. The ProcessEmpty of the processor is called once before the iteration and once after the iteration.
std::future< Status > tkrzw::AsyncDBM::ProcessEach | ( | DBM::RecordLambdaType | rec_lambda, |
bool | writable | ||
) |
Processes each and every record in the database with a lambda function.
rec_lambda | The lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value. |
writable | True if the processor can edit the record. |
The lambda function is called repeatedly for each record. It is also called once before the iteration and once after the iteration with both the key and the value being RecordProcessor::NOOP.
std::future<Status> tkrzw::AsyncDBM::Clear | ( | ) |
Removes all records.
std::future<Status> tkrzw::AsyncDBM::Rebuild | ( | const std::map< std::string, std::string > & | params = {} | ) |
Rebuilds the entire database.
params | Optional parameters. |
The parameters work in the same way as with PolyDBM::RebuildAdvanced.
std::future<Status> tkrzw::AsyncDBM::Synchronize | ( | bool | hard, |
std::unique_ptr< DBM::FileProcessor > | proc = nullptr , |
||
const std::map< std::string, std::string > & | params = {} |
||
) |
Synchronizes the content of the database to the file system.
hard | True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. |
proc | The file processor object, whose Process method is called while the content of the file is synchronized. The ownership is taken. If it is nullptr, it is ignored. If it is nullptr, it is not used. |
params | Optional parameters. |
The parameters work in the same way as with PolyDBM::Synchronize.
std::future<Status> tkrzw::AsyncDBM::CopyFileData | ( | const std::string & | dest_path, |
bool | sync_hard = false |
||
) |
Copies the content of the database file to another file.
dest_path | A path to the destination file. |
sync_hard | True to do physical synchronization with the hardware. |
Copying is done while the content is synchronized and stable. So, this method is suitable for making a backup file while running a database service.
Exports all records to another database.
dbm | The pointer to the destination database. The lefetime of the database object must last until the task finishes. |
Exports all records of a database to a flat record file.
dest_file | The file object to write records in. The lefetime of the file object must last until the task finishes. |
A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration.
Imports records to a database from a flat record file.
src_file | The file object to read records from. The lefetime of the file object must last until the task finishes. |
std::future<std::pair<Status, std::vector<std::string> > > tkrzw::AsyncDBM::SearchModal | ( | std::string_view | mode, |
std::string_view | pattern, | ||
size_t | capacity = 0 |
||
) |
Searches the database and get keys which match a pattern, according to a mode expression.
mode | The search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least. Ordered databases support "upper" and "lower" which extract keys whose positions are upper/lower than the pattern. "upperinc" and "lowerinc" are their inclusive versions. |
pattern | The pattern for matching. |
capacity | The maximum records to obtain. 0 means unlimited. |
TaskQueue* tkrzw::AsyncDBM::GetTaskQueue | ( | ) |
Gets the internal task queue.