Tkrzw
Classes | Public Types | Public Member Functions | List of all members
tkrzw::AsyncDBM Class Referencefinal

Asynchronous database manager adapter. More...

#include <tkrzw_dbm_async.h>

Classes

class  CommonPostprocessor
 Interface of a common post processor for a record. More...
 
class  RecordProcessor
 Interface of asynchronous processor for a record. More...
 

Public Types

typedef std::function< void(const char *, const Status &)> CommonPostLambdaType
 Lambda function type to postprocess a database operation. More...
 

Public Member Functions

 AsyncDBM (DBM *dbm, int32_t num_worker_threads)
 Constructor. More...
 
 ~AsyncDBM ()
 Destructor. More...
 
 AsyncDBM (const AsyncDBM &rhs)=delete
 Copy and assignment are disabled. More...
 
AsyncDBMoperator= (const AsyncDBM &rhs)=delete
 
void SetCommonPostprocessor (std::unique_ptr< CommonPostprocessor > proc)
 Set the common post processor. More...
 
void SetCommonPostprocessor (CommonPostLambdaType post_lambda)
 Set the common post processor with a lambda function. More...
 
template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > Process (std::string_view key, std::unique_ptr< PROC > proc, bool writable)
 Processes a record with a processor. More...
 
std::future< StatusProcess (std::string_view key, DBM::RecordLambdaType rec_lambda, bool writable)
 Processes a record with a lambda function. More...
 
std::future< std::pair< Status, std::string > > Get (std::string_view key)
 Gets the value of a record of a key. More...
 
std::future< std::pair< Status, std::map< std::string, std::string > > > GetMulti (const std::vector< std::string_view > &keys)
 Gets the values of multiple records of keys, with a string view vector. More...
 
std::future< std::pair< Status, std::map< std::string, std::string > > > GetMulti (const std::vector< std::string > &keys)
 Gets the values of multiple records of keys, with a string vector. More...
 
std::future< StatusSet (std::string_view key, std::string_view value, bool overwrite=true)
 Sets a record of a key and a value. More...
 
std::future< StatusSetMulti (const std::map< std::string_view, std::string_view > &records, bool overwrite=true)
 Sets multiple records, with a map of string views. More...
 
std::future< StatusSetMulti (const std::map< std::string, std::string > &records, bool overwrite=true)
 Sets multiple records, with a map of strings. More...
 
std::future< StatusRemove (std::string_view key)
 Removes a record of a key. More...
 
std::future< StatusRemoveMulti (const std::vector< std::string_view > &keys)
 Removes records of keys, with a string view vector. More...
 
std::future< StatusRemoveMulti (const std::vector< std::string > &keys)
 Removes records of keys, with a string vector. More...
 
std::future< StatusAppend (std::string_view key, std::string_view value, std::string_view delim="")
 Appends data at the end of a record of a key. More...
 
std::future< StatusAppendMulti (const std::map< std::string_view, std::string_view > &records, std::string_view delim="")
 Appends data to multiple records, with a map of string views. More...
 
std::future< StatusAppendMulti (const std::map< std::string, std::string > &records, std::string_view delim="")
 Appends data to multiple records, with a map of strings. More...
 
std::future< StatusCompareExchange (std::string_view key, std::string_view expected, std::string_view desired)
 Compares the value of a record and exchanges if the condition meets. More...
 
std::future< StatusCompareExchangeMulti (const std::vector< std::pair< std::string_view, std::string_view >> &expected, const std::vector< std::pair< std::string_view, std::string_view >> &desired)
 Compares the values of records and exchanges if the condition meets. More...
 
std::future< std::pair< Status, int64_t > > Increment (std::string_view key, int64_t increment=1, int64_t initial=0)
 Increments the numeric value of a record. More...
 
std::future< StatusRekey (std::string_view old_key, std::string_view new_key, bool overwrite=true, bool copying=false)
 Changes the key of a record. More...
 
template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > ProcessFirst (std::unique_ptr< PROC > proc, bool writable)
 Processes the first record with a processor. More...
 
std::future< StatusProcessFirst (DBM::RecordLambdaType rec_lambda, bool writable)
 Processes the first record with a lambda function. More...
 
std::future< std::pair< Status, std::pair< std::string, std::string > > > PopFirst ()
 Gets the first record and removes it. More...
 
std::future< StatusPushLast (std::string_view value, double wtime=-1)
 Adds a record with a key of the current timestamp. More...
 
template<typename PROC >
std::future< std::pair< Status, std::vector< std::shared_ptr< PROC > > > > ProcessMulti (const std::vector< std::pair< std::string_view, std::shared_ptr< PROC >>> &key_proc_pairs, bool writable)
 Processes multiple records with processors. More...
 
std::future< StatusProcessMulti (const std::vector< std::pair< std::string_view, DBM::RecordLambdaType >> &key_lambda_pairs, bool writable)
 Processes multiple records with lambda functions. More...
 
template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > ProcessEach (std::unique_ptr< PROC > proc, bool writable)
 Processes each and every record in the database with a processor. More...
 
std::future< StatusProcessEach (DBM::RecordLambdaType rec_lambda, bool writable)
 Processes each and every record in the database with a lambda function. More...
 
std::future< StatusClear ()
 Removes all records. More...
 
std::future< StatusRebuild (const std::map< std::string, std::string > &params={})
 Rebuilds the entire database. More...
 
std::future< StatusSynchronize (bool hard, std::unique_ptr< DBM::FileProcessor > proc=nullptr, const std::map< std::string, std::string > &params={})
 Synchronizes the content of the database to the file system. More...
 
std::future< StatusCopyFileData (const std::string &dest_path, bool sync_hard=false)
 Copies the content of the database file to another file. More...
 
std::future< StatusExport (DBM *dbm)
 Exports all records to another database. More...
 
std::future< StatusExportToFlatRecords (File *dest_file)
 Exports all records of a database to a flat record file. More...
 
std::future< StatusImportFromFlatRecords (File *src_file)
 Imports records to a database from a flat record file. More...
 
std::future< std::pair< Status, std::vector< std::string > > > SearchModal (std::string_view mode, std::string_view pattern, size_t capacity=0)
 Searches the database and get keys which match a pattern, according to a mode expression. More...
 
TaskQueueGetTaskQueue ()
 Gets the internal task queue. More...
 

Detailed Description

Asynchronous database manager adapter.

This class is a wrapper of DBM for asynchronous operations. A task queue with a thread pool is used inside. Every method except for the constructor and the destructor is run by a thread in the thread pool and the result is set in the future oject of the return value. The caller can ignore the future object if it is not necessary. The destructor of this asynchronous database manager waits for all tasks to be done. Therefore, the destructor should be called before the database is closed.

Member Typedef Documentation

◆ CommonPostLambdaType

typedef std::function<void(const char*, const Status&)> tkrzw::AsyncDBM::CommonPostLambdaType

Lambda function type to postprocess a database operation.

The first parameter is the method name of the database operation. The second parameter is the status of the database operation.

Constructor & Destructor Documentation

◆ AsyncDBM() [1/2]

tkrzw::AsyncDBM::AsyncDBM ( DBM dbm,
int32_t  num_worker_threads 
)

Constructor.

Parameters
dbmA database object which has been opened. The ownership is not taken.
num_worker_threadsThe number of threads in the internal thread pool.

◆ ~AsyncDBM()

tkrzw::AsyncDBM::~AsyncDBM ( )

Destructor.

◆ AsyncDBM() [2/2]

tkrzw::AsyncDBM::AsyncDBM ( const AsyncDBM rhs)
explicitdelete

Copy and assignment are disabled.

Member Function Documentation

◆ SetCommonPostprocessor() [1/2]

void tkrzw::AsyncDBM::SetCommonPostprocessor ( std::unique_ptr< CommonPostprocessor proc)

Set the common post processor.

Parameters
procthe common post processor. If it is nullptr, no postprocess is done. The ownership is taken.

By default or if nullptr is set, no postprocess is done. If a processor is set, its Postprocess method is called every time in parallel after every method is called.

◆ SetCommonPostprocessor() [2/2]

void tkrzw::AsyncDBM::SetCommonPostprocessor ( CommonPostLambdaType  post_lambda)

Set the common post processor with a lambda function.

Parameters
post_lambdathe lambda function of the common post processor.

By default, no postprocess is done. If a processor is set, it is called every time in parallel after every method is called.

◆ Process() [1/2]

template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::Process ( std::string_view  key,
std::unique_ptr< PROC >  proc,
bool  writable 
)

Processes a record with a processor.

Parameters
keyThe key of the record.
procThe processor object derived from RecordProcessor. The ownership is taken.
writableTrue if the processor can edit the record.
Returns
The result status and the same processor object as the parameter.

If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.

◆ Process() [2/2]

std::future< Status > tkrzw::AsyncDBM::Process ( std::string_view  key,
DBM::RecordLambdaType  rec_lambda,
bool  writable 
)

Processes a record with a lambda function.

Parameters
keyThe key of the record.
rec_lambdaThe lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value.
writableTrue if the processor can edit the record.
Returns
The result status.

◆ Get()

std::future<std::pair<Status, std::string> > tkrzw::AsyncDBM::Get ( std::string_view  key)

Gets the value of a record of a key.

Parameters
keyThe key of the record.
Returns
The result status and the result value. If there's no matching record, NOT_FOUND_ERROR is returned.

◆ GetMulti() [1/2]

std::future<std::pair<Status, std::map<std::string, std::string> > > tkrzw::AsyncDBM::GetMulti ( const std::vector< std::string_view > &  keys)

Gets the values of multiple records of keys, with a string view vector.

Parameters
keysThe keys of records to retrieve.
Returns
The result status and a map of retrieved records. Keys which don't match existing records are ignored. If all records of the given keys are found, SUCCESS is returned. If one or more records are missing, NOT_FOUND_ERROR is returned. Thus, even with an error code, the result map can have elements.

◆ GetMulti() [2/2]

std::future<std::pair<Status, std::map<std::string, std::string> > > tkrzw::AsyncDBM::GetMulti ( const std::vector< std::string > &  keys)

Gets the values of multiple records of keys, with a string vector.

Parameters
keysThe keys of records to retrieve.
Returns
The result status and a map of retrieved records. Keys which don't match existing records are ignored. If all records of the given keys are found, SUCCESS is returned. If one or more records are missing, NOT_FOUND_ERROR is returned. Thus, even with an error code, the result map can have elements.

◆ Set()

std::future<Status> tkrzw::AsyncDBM::Set ( std::string_view  key,
std::string_view  value,
bool  overwrite = true 
)

Sets a record of a key and a value.

Parameters
keyThe key of the record.
valueThe value of the record.
overwriteWhether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
Returns
The result status. If overwriting is abandoned, DUPLICATION_ERROR is returned.

◆ SetMulti() [1/2]

std::future<Status> tkrzw::AsyncDBM::SetMulti ( const std::map< std::string_view, std::string_view > &  records,
bool  overwrite = true 
)

Sets multiple records, with a map of string views.

Parameters
recordsThe records to store.
overwriteWhether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
Returns
The result status.

◆ SetMulti() [2/2]

std::future<Status> tkrzw::AsyncDBM::SetMulti ( const std::map< std::string, std::string > &  records,
bool  overwrite = true 
)

Sets multiple records, with a map of strings.

Parameters
recordsThe records to store.
overwriteWhether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
Returns
The result status.

◆ Remove()

std::future<Status> tkrzw::AsyncDBM::Remove ( std::string_view  key)

Removes a record of a key.

Parameters
keyThe key of the record.
Returns
The result status. If there's no matching record, NOT_FOUND_ERROR is returned.

◆ RemoveMulti() [1/2]

std::future<Status> tkrzw::AsyncDBM::RemoveMulti ( const std::vector< std::string_view > &  keys)

Removes records of keys, with a string view vector.

Parameters
keysThe keys of records to remove.
Returns
The result status. If there are missing records, NOT_FOUND_ERROR is returned.

◆ RemoveMulti() [2/2]

std::future<Status> tkrzw::AsyncDBM::RemoveMulti ( const std::vector< std::string > &  keys)

Removes records of keys, with a string vector.

Parameters
keysThe keys of records to remove.
Returns
The result status. If there are missing records, NOT_FOUND_ERROR is returned.

◆ Append()

std::future<Status> tkrzw::AsyncDBM::Append ( std::string_view  key,
std::string_view  value,
std::string_view  delim = "" 
)

Appends data at the end of a record of a key.

Parameters
keyThe key of the record.
valueThe value to append.
delimThe delimiter to put after the existing record.
Returns
The result status.

If there's no existing record, the value is set without the delimiter.

◆ AppendMulti() [1/2]

std::future<Status> tkrzw::AsyncDBM::AppendMulti ( const std::map< std::string_view, std::string_view > &  records,
std::string_view  delim = "" 
)

Appends data to multiple records, with a map of string views.

Parameters
recordsThe records to append.
delimThe delimiter to put after the existing record.
Returns
The result status.

If there's no existing record, the value is set without the delimiter.

◆ AppendMulti() [2/2]

std::future<Status> tkrzw::AsyncDBM::AppendMulti ( const std::map< std::string, std::string > &  records,
std::string_view  delim = "" 
)

Appends data to multiple records, with a map of strings.

Parameters
recordsThe records to append.
delimThe delimiter to put after the existing record.
Returns
The result status.

If there's no existing record, the value is set without the delimiter.

◆ CompareExchange()

std::future<Status> tkrzw::AsyncDBM::CompareExchange ( std::string_view  key,
std::string_view  expected,
std::string_view  desired 
)

Compares the value of a record and exchanges if the condition meets.

Parameters
keyThe key of the record.
expectedThe expected value. If the data is nullptr, no existing record is expected. If it is DBM::ANY_DATA, an existing record with any value is expacted.
desiredThe desired value. If the data is nullptr, the record is to be removed. If it is DBM::ANY_DATA, no update is done.
Returns
The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.

◆ CompareExchangeMulti()

std::future<Status> tkrzw::AsyncDBM::CompareExchangeMulti ( const std::vector< std::pair< std::string_view, std::string_view >> &  expected,
const std::vector< std::pair< std::string_view, std::string_view >> &  desired 
)

Compares the values of records and exchanges if the condition meets.

Parameters
expectedThe record keys and their expected values. If the value is nullptr, no existing record is expected. If the value is DBM::ANY_DATA, an existing record with any value is expacted.
desiredThe record keys and their desired values. If the value is nullptr, the record is to be removed.
Returns
The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.

◆ Increment()

std::future<std::pair<Status, int64_t> > tkrzw::AsyncDBM::Increment ( std::string_view  key,
int64_t  increment = 1,
int64_t  initial = 0 
)

Increments the numeric value of a record.

Parameters
keyThe key of the record.
incrementThe incremental value. If it is INT64MIN, the current value is not changed and a new record is not created.
initialThe initial value.
Returns
The result status and the current value.

The record value is stored as an 8-byte big-endian integer. Negative is also supported.

◆ Rekey()

std::future<Status> tkrzw::AsyncDBM::Rekey ( std::string_view  old_key,
std::string_view  new_key,
bool  overwrite = true,
bool  copying = false 
)

Changes the key of a record.

Parameters
old_keyThe old key of the record.
new_keyThe new key of the record.
overwriteWhether to overwrite the existing record of the new key.
copyingWhether to retain the record of the old key.
Returns
The result status. If there's no matching record to the old key, NOT_FOUND_ERROR is returned. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is returned.

This method is done atomically by ProcessMulti. The other threads observe that the record has either the old key or the new value. No intermediate states are observed.

◆ ProcessFirst() [1/2]

template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::ProcessFirst ( std::unique_ptr< PROC >  proc,
bool  writable 
)

Processes the first record with a processor.

Parameters
procThe processor object derived from RecordProcessor. The ownership is taken.
writableTrue if the processor can edit the record.
Returns
The result status and the same processor object as the parameter.

If the first record exists, the ProcessFull of the processor is called. Otherwise, this method fails and no method of the processor is called. Whereas ordered databases have efficient implementations of this method, unordered databases have inefficient implementations.

◆ ProcessFirst() [2/2]

std::future< Status > tkrzw::AsyncDBM::ProcessFirst ( DBM::RecordLambdaType  rec_lambda,
bool  writable 
)

Processes the first record with a lambda function.

Parameters
rec_lambdaThe lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the record. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value.
writableTrue if the processor can edit the record.
Returns
The result status.

◆ PopFirst()

std::future<std::pair<Status, std::pair<std::string, std::string> > > tkrzw::AsyncDBM::PopFirst ( )

Gets the first record and removes it.

Parameters
keyThe pointer to a string object to contain the key of the first record. If it is nullptr, it is ignored.
valueThe pointer to a string object to contain the value of the first record. If it is nullptr, it is ignored.
Returns
The result status and a pair of the key and the value of the first record.

◆ PushLast()

std::future<Status> tkrzw::AsyncDBM::PushLast ( std::string_view  value,
double  wtime = -1 
)

Adds a record with a key of the current timestamp.

Parameters
valueThe value of the record.
wtimeThe current wall time used to generate the key. If it is negative, the system clock is used.
Returns
The result status.

The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds.

◆ ProcessMulti() [1/2]

template<typename PROC >
std::future< std::pair< Status, std::vector< std::shared_ptr< PROC > > > > tkrzw::AsyncDBM::ProcessMulti ( const std::vector< std::pair< std::string_view, std::shared_ptr< PROC >>> &  key_proc_pairs,
bool  writable 
)

Processes multiple records with processors.

Parameters
key_proc_pairsPairs of the keys and their processor objects derived from RecordProcessor. The ownership is taken.
writableTrue if the processors can edit the records.
Returns
The result status and a vector of the same object as the parameter.

If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.

◆ ProcessMulti() [2/2]

std::future< Status > tkrzw::AsyncDBM::ProcessMulti ( const std::vector< std::pair< std::string_view, DBM::RecordLambdaType >> &  key_lambda_pairs,
bool  writable 
)

Processes multiple records with lambda functions.

Parameters
key_lambda_pairsPairs of the keys and their lambda functions. The first parameter of the lambda functions is the key of the record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value.
writableTrue if the processors can edit the records.
Returns
The result status.

◆ ProcessEach() [1/2]

template<typename PROC >
std::future< std::pair< Status, std::unique_ptr< PROC > > > tkrzw::AsyncDBM::ProcessEach ( std::unique_ptr< PROC >  proc,
bool  writable 
)

Processes each and every record in the database with a processor.

Parameters
procThe processor object derived from RecordProcessor. The ownership is taken.
writableTrue if the processor can edit the record.
Returns
The result status and the same processor object as the parameter.

The ProcessFull of the processor is called repeatedly for each record. The ProcessEmpty of the processor is called once before the iteration and once after the iteration.

◆ ProcessEach() [2/2]

std::future< Status > tkrzw::AsyncDBM::ProcessEach ( DBM::RecordLambdaType  rec_lambda,
bool  writable 
)

Processes each and every record in the database with a lambda function.

Parameters
rec_lambdaThe lambda function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or RecordProcessor::NOOP if it the record doesn't exist. The return value is a string reference to RecordProcessor::NOOP, RecordProcessor::REMOVE, or the new record value.
writableTrue if the processor can edit the record.
Returns
The result status.

The lambda function is called repeatedly for each record. It is also called once before the iteration and once after the iteration with both the key and the value being RecordProcessor::NOOP.

◆ Clear()

std::future<Status> tkrzw::AsyncDBM::Clear ( )

Removes all records.

Returns
The result status.

◆ Rebuild()

std::future<Status> tkrzw::AsyncDBM::Rebuild ( const std::map< std::string, std::string > &  params = {})

Rebuilds the entire database.

Parameters
paramsOptional parameters.
Returns
The result status.

The parameters work in the same way as with PolyDBM::RebuildAdvanced.

◆ Synchronize()

std::future<Status> tkrzw::AsyncDBM::Synchronize ( bool  hard,
std::unique_ptr< DBM::FileProcessor proc = nullptr,
const std::map< std::string, std::string > &  params = {} 
)

Synchronizes the content of the database to the file system.

Parameters
hardTrue to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
procThe file processor object, whose Process method is called while the content of the file is synchronized. The ownership is taken. If it is nullptr, it is ignored. If it is nullptr, it is not used.
paramsOptional parameters.
Returns
The result status.

The parameters work in the same way as with PolyDBM::Synchronize.

◆ CopyFileData()

std::future<Status> tkrzw::AsyncDBM::CopyFileData ( const std::string &  dest_path,
bool  sync_hard = false 
)

Copies the content of the database file to another file.

Parameters
dest_pathA path to the destination file.
sync_hardTrue to do physical synchronization with the hardware.
Returns
The result status.

Copying is done while the content is synchronized and stable. So, this method is suitable for making a backup file while running a database service.

◆ Export()

std::future<Status> tkrzw::AsyncDBM::Export ( DBM dbm)

Exports all records to another database.

Parameters
dbmThe pointer to the destination database. The lefetime of the database object must last until the task finishes.
Returns
The result status.

◆ ExportToFlatRecords()

std::future<Status> tkrzw::AsyncDBM::ExportToFlatRecords ( File dest_file)

Exports all records of a database to a flat record file.

Parameters
dest_fileThe file object to write records in. The lefetime of the file object must last until the task finishes.
Returns
The result status.

A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration.

◆ ImportFromFlatRecords()

std::future<Status> tkrzw::AsyncDBM::ImportFromFlatRecords ( File src_file)

Imports records to a database from a flat record file.

Parameters
src_fileThe file object to read records from. The lefetime of the file object must last until the task finishes.
Returns
The result status.

◆ SearchModal()

std::future<std::pair<Status, std::vector<std::string> > > tkrzw::AsyncDBM::SearchModal ( std::string_view  mode,
std::string_view  pattern,
size_t  capacity = 0 
)

Searches the database and get keys which match a pattern, according to a mode expression.

Parameters
modeThe search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least. Ordered databases support "upper" and "lower" which extract keys whose positions are upper/lower than the pattern. "upperinc" and "lowerinc" are their inclusive versions.
patternThe pattern for matching.
capacityThe maximum records to obtain. 0 means unlimited.
Returns
The result status and the result keys.

◆ GetTaskQueue()

TaskQueue* tkrzw::AsyncDBM::GetTaskQueue ( )

Gets the internal task queue.

Returns
The pointer to the internal task queue. The ownership is not moved.