Tkrzw
Classes | Public Member Functions | Static Public Member Functions | List of all members
tkrzw::ShardDBM Class Referencefinal

Sharding database manager adapter. More...

#include <tkrzw_dbm_shard.h>

Classes

class  Iterator
 Iterator for each record. More...
 

Public Member Functions

 ShardDBM ()
 Default constructor. More...
 
virtual ~ShardDBM ()
 Destructor. More...
 
 ShardDBM (const ShardDBM &rhs)=delete
 Copy and assignment are disabled. More...
 
ShardDBMoperator= (const ShardDBM &rhs)=delete
 
Status Open (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT) override
 Opens a database file. More...
 
Status OpenAdvanced (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT, const std::map< std::string, std::string > &params={}) override
 Opens a database file, in an advanced way. More...
 
Status Close () override
 Closes the database file. More...
 
Status Process (std::string_view key, RecordProcessor *proc, bool writable) override
 Processes a record with a processor. More...
 
Status Get (std::string_view key, std::string *value=nullptr) override
 Gets the value of a record of a key. More...
 
Status Set (std::string_view key, std::string_view value, bool overwrite=true, std::string *old_value=nullptr) override
 Sets a record of a key and a value. More...
 
Status Remove (std::string_view key, std::string *old_value=nullptr) override
 Removes a record of a key. More...
 
Status Append (std::string_view key, std::string_view value, std::string_view delim="") override
 Appends data at the end of a record of a key. More...
 
Status ProcessFirst (RecordProcessor *proc, bool writable) override
 Processes the first record with a processor. More...
 
Status ProcessMulti (const std::vector< std::pair< std::string_view, DBM::RecordProcessor * >> &key_proc_pairs, bool writable) override
 Processes multiple records with processors. More...
 
Status CompareExchangeMulti (const std::vector< std::pair< std::string_view, std::string_view >> &expected, const std::vector< std::pair< std::string_view, std::string_view >> &desired) override
 Compares the values of records and exchanges if the condition meets. More...
 
Status ProcessEach (RecordProcessor *proc, bool writable) override
 Processes each and every record in the database with a processor. More...
 
Status Count (int64_t *count) override
 Gets the number of records. More...
 
Status GetFileSize (int64_t *size) override
 Gets the current file size of the database. More...
 
Status GetFilePath (std::string *path) override
 Gets the path of the database file. More...
 
Status GetTimestamp (double *timestamp) override
 Gets the timestamp in seconds of the last modified time. More...
 
Status Clear () override
 Removes all records. More...
 
Status Rebuild () override
 Rebuilds the entire database. More...
 
Status RebuildAdvanced (const std::map< std::string, std::string > &params={}) override
 Rebuilds the entire database, in an advanced way. More...
 
Status ShouldBeRebuilt (bool *tobe) override
 Checks whether the database should be rebuilt. More...
 
Status Synchronize (bool hard, FileProcessor *proc=nullptr) override
 Synchronizes the content of the database to the file system. More...
 
Status SynchronizeAdvanced (bool hard, FileProcessor *proc=nullptr, const std::map< std::string, std::string > &params={}) override
 Synchronizes the content of the database to the file system, in an advanced way. More...
 
Status CopyFileData (const std::string &dest_path, bool sync_hard=false) override
 Copies the content of the database files to other files. More...
 
std::vector< std::pair< std::string, std::string > > Inspect () override
 Inspects the database. More...
 
bool IsOpen () const override
 Checks whether the database is open. More...
 
bool IsWritable () const override
 Checks whether the database is writable. More...
 
bool IsHealthy () const override
 Checks whether the database condition is healthy. More...
 
bool IsOrdered () const override
 Checks whether ordered operations are supported. More...
 
std::unique_ptr< DBM::IteratorMakeIterator () override
 Makes an iterator for each record. More...
 
std::unique_ptr< DBMMakeDBM () const override
 Makes a new DBM object of the same concrete class. More...
 
UpdateLoggerGetUpdateLogger () const override
 Gets the logger to write all update operations. More...
 
void SetUpdateLogger (UpdateLogger *update_logger) override
 Sets the logger to write all update operations. More...
 
DBMGetInternalDBM () const
 Gets the pointer to the first internal database object. More...
 
- Public Member Functions inherited from tkrzw::ParamDBM
virtual ~ParamDBM ()=default
 Destructor. More...
 
- Public Member Functions inherited from tkrzw::DBM
virtual ~DBM ()=default
 Destructor. More...
 
virtual Status Process (std::string_view key, RecordLambdaType rec_lambda, bool writable)
 Processes a record with a lambda function. More...
 
virtual std::string GetSimple (std::string_view key, std::string_view default_value="")
 Gets the value of a record of a key, in a simple way. More...
 
virtual Status GetMulti (const std::vector< std::string_view > &keys, std::map< std::string, std::string > *records)
 Gets the values of multiple records of keys, with a string view vector. More...
 
virtual Status GetMulti (const std::initializer_list< std::string_view > &keys, std::map< std::string, std::string > *records)
 Gets the values of multiple records of keys, with an initializer list. More...
 
virtual Status GetMulti (const std::vector< std::string > &keys, std::map< std::string, std::string > *records)
 Gets the values of multiple records of keys, with a string vector. More...
 
virtual Status SetMulti (const std::map< std::string_view, std::string_view > &records, bool overwrite=true)
 Sets multiple records, with a map of string views. More...
 
virtual Status SetMulti (const std::initializer_list< std::pair< std::string_view, std::string_view >> &records, bool overwrite=true)
 Sets multiple records, with an initializer list. More...
 
virtual Status SetMulti (const std::map< std::string, std::string > &records, bool overwrite=true)
 Sets multiple records, with a map of strings. More...
 
virtual Status RemoveMulti (const std::vector< std::string_view > &keys)
 Removes records of keys, with a string view vector. More...
 
virtual Status RemoveMulti (const std::initializer_list< std::string_view > &keys)
 Removes records of keys, with an initializer list. More...
 
virtual Status RemoveMulti (const std::vector< std::string > &keys)
 Removes records of keys, with a string vector. More...
 
virtual Status AppendMulti (const std::map< std::string_view, std::string_view > &records, std::string_view delim="")
 Appends data to multiple records, with a map of string views. More...
 
virtual Status AppendMulti (const std::initializer_list< std::pair< std::string_view, std::string_view >> &records, std::string_view delim="")
 Appends data to multiple records, with an initializer list. More...
 
virtual Status AppendMulti (const std::map< std::string, std::string > &records, std::string_view delim="")
 Appends data to multiple records, with a map of strings. More...
 
virtual Status CompareExchange (std::string_view key, std::string_view expected, std::string_view desired, std::string *actual=nullptr, bool *found=nullptr)
 Compares the value of a record and exchanges if the condition meets. More...
 
virtual Status Increment (std::string_view key, int64_t increment=1, int64_t *current=nullptr, int64_t initial=0)
 Increments the numeric value of a record. More...
 
virtual int64_t IncrementSimple (std::string_view key, int64_t increment=1, int64_t initial=0)
 Increments the numeric value of a record, in a simple way. More...
 
virtual Status ProcessMulti (const std::vector< std::pair< std::string_view, RecordProcessor * >> &key_proc_pairs, bool writable)=0
 Processes multiple records with processors. More...
 
virtual Status ProcessMulti (const std::vector< std::pair< std::string_view, RecordLambdaType >> &key_lambda_pairs, bool writable)
 Processes multiple records with lambda functions. More...
 
virtual Status Rekey (std::string_view old_key, std::string_view new_key, bool overwrite=true, bool copying=false, std::string *value=nullptr)
 Changes the key of a record. More...
 
virtual Status ProcessFirst (RecordLambdaType rec_lambda, bool writable)
 Processes the first record with a lambda function. More...
 
virtual Status PopFirst (std::string *key=nullptr, std::string *value=nullptr)
 Gets the first record and removes it. More...
 
virtual Status PushLast (std::string_view value, double wtime=-1, std::string *key=nullptr)
 Adds a record with a key of the current timestamp. More...
 
virtual Status ProcessEach (RecordLambdaType rec_lambda, bool writable)
 Processes each and every record in the database with a lambda function. More...
 
virtual int64_t CountSimple ()
 Gets the number of records, in a simple way. More...
 
virtual int64_t GetFileSizeSimple ()
 Gets the current file size of the database, in a simple way. More...
 
virtual std::string GetFilePathSimple ()
 Gets the path of the database file, in a simple way. More...
 
virtual double GetTimestampSimple ()
 Gets the timestamp of the last modified time, in a simple way. More...
 
virtual bool ShouldBeRebuiltSimple ()
 Checks whether the database should be rebuilt, in a simple way. More...
 
virtual Status Export (DBM *dest_dbm)
 Exports all records to another database. More...
 
const std::type_info & GetType () const
 Gets the type information of the actual class. More...
 

Static Public Member Functions

static Status GetNumberOfShards (const std::string &path, int32_t *num_shards)
 Gets the number of shards of a database. More...
 
static Status RestoreDatabase (const std::string &old_file_path, const std::string &new_file_path, const std::string &class_name="", int64_t end_offset=-1, std::string_view cipher_key="")
 Restores a broken database as a new healthy database. More...
 
static Status RenameDatabase (const std::string &old_file_path, const std::string &new_file_path)
 Renames all files of a database to ones of another name. More...
 

Additional Inherited Members

- Public Types inherited from tkrzw::DBM
typedef std::function< std::string_view(std::string_view, std::string_view)> RecordLambdaType
 Lambda function type to process a record. More...
 
- Static Public Attributes inherited from tkrzw::DBM
static const std::string_view ANY_DATA
 The special string_view value to represent any data. More...
 

Detailed Description

Sharding database manager adapter.

All operations except for Open and Close are thread-safe; Multiple threads can access the same database concurrently. Every opened database must be closed explicitly to avoid data corruption.

This class is a wrapper of PolyDBM for sharding the database into multiple instances.

Constructor & Destructor Documentation

◆ ShardDBM() [1/2]

tkrzw::ShardDBM::ShardDBM ( )
explicit

Default constructor.

◆ ~ShardDBM()

virtual tkrzw::ShardDBM::~ShardDBM ( )
virtual

Destructor.

◆ ShardDBM() [2/2]

tkrzw::ShardDBM::ShardDBM ( const ShardDBM rhs)
explicitdelete

Copy and assignment are disabled.

Member Function Documentation

◆ Open()

Status tkrzw::ShardDBM::Open ( const std::string &  path,
bool  writable,
int32_t  options = File::OPEN_DEFAULT 
)
overridevirtual

Opens a database file.

Parameters
pathA path of the file.
writableIf true, the file is writable. If false, it is read-only.
optionsBit-sum options of File::OpenOption enums for opening the file.
Returns
The result status.

Implements tkrzw::DBM.

◆ OpenAdvanced()

Status tkrzw::ShardDBM::OpenAdvanced ( const std::string &  path,
bool  writable,
int32_t  options = File::OPEN_DEFAULT,
const std::map< std::string, std::string > &  params = {} 
)
overridevirtual

Opens a database file, in an advanced way.

Parameters
pathA path of the file.
writableIf true, the file is writable. If false, it is read-only.
optionsBit-sum options for opening the file.
paramsOptional parameters. All parameters for PolyDBM::OpenAdvanced are supported. Moreover, the parameter "num_shards" specifies the number of shards. Each shard file has a suffix like "-00003-of-00015". If the number of shards is not specified and existing files match the path, it is implicitly specified. If there are no matching files, 1 is implicitly set.
Returns
The result status.

Implements tkrzw::ParamDBM.

◆ Close()

Status tkrzw::ShardDBM::Close ( )
overridevirtual

Closes the database file.

Returns
The result status.

Implements tkrzw::DBM.

◆ Process()

Status tkrzw::ShardDBM::Process ( std::string_view  key,
RecordProcessor proc,
bool  writable 
)
overridevirtual

Processes a record with a processor.

Parameters
keyThe key of the record.
procThe pointer to the processor object.
writableTrue if the processor can edit the record.
Returns
The result status.

If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.

Implements tkrzw::DBM.

◆ Get()

Status tkrzw::ShardDBM::Get ( std::string_view  key,
std::string *  value = nullptr 
)
overridevirtual

Gets the value of a record of a key.

Parameters
keyThe key of the record.
valueThe pointer to a string object to contain the result value. If it is nullptr, the value data is ignored.
Returns
The result status. If there's no matching record, NOT_FOUND_ERROR is returned.

Reimplemented from tkrzw::DBM.

◆ Set()

Status tkrzw::ShardDBM::Set ( std::string_view  key,
std::string_view  value,
bool  overwrite = true,
std::string *  old_value = nullptr 
)
overridevirtual

Sets a record of a key and a value.

Parameters
keyThe key of the record.
valueThe value of the record.
overwriteWhether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
old_valueThe pointer to a string object to contain the old value. Assignment is done even on the duplication error. If it is nullptr, it is ignored.
Returns
The result status. If overwriting is abandoned, DUPLICATION_ERROR is returned.

Reimplemented from tkrzw::DBM.

◆ Remove()

Status tkrzw::ShardDBM::Remove ( std::string_view  key,
std::string *  old_value = nullptr 
)
overridevirtual

Removes a record of a key.

Parameters
keyThe key of the record.
old_valueThe pointer to a string object to contain the old value. If it is nullptr, it is ignored.
Returns
The result status. If there's no matching record, NOT_FOUND_ERROR is returned.

Reimplemented from tkrzw::DBM.

◆ Append()

Status tkrzw::ShardDBM::Append ( std::string_view  key,
std::string_view  value,
std::string_view  delim = "" 
)
overridevirtual

Appends data at the end of a record of a key.

Parameters
keyThe key of the record.
valueThe value to append.
delimThe delimiter to put after the existing record.
Returns
The result status.

If there's no existing record, the value is set without the delimiter.

Reimplemented from tkrzw::DBM.

◆ ProcessFirst()

Status tkrzw::ShardDBM::ProcessFirst ( RecordProcessor proc,
bool  writable 
)
overridevirtual

Processes the first record with a processor.

Parameters
procThe pointer to the processor object.
writableTrue if the processor can edit the record.
Returns
The result status.

If the first record exists, the ProcessFull of the processor is called. Otherwise, this method fails and no method of the processor is called. Whereas ordered databases have efficient implementations of this method, unordered databases have inefficient implementations.

Implements tkrzw::DBM.

◆ ProcessMulti()

Status tkrzw::ShardDBM::ProcessMulti ( const std::vector< std::pair< std::string_view, DBM::RecordProcessor * >> &  key_proc_pairs,
bool  writable 
)
override

Processes multiple records with processors.

Parameters
key_proc_pairsPairs of the keys and their processor objects.
writableTrue if the processors can edit the records.
Returns
The result status.

Precondition: The database is opened. The writable parameter should be consistent to the open mode.

If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.

ShardDBM has a special implementation to assure isolation of transactions in multi-threading environments. The given keys are locked in the space of the first shard so that isolation is assured among ProcessMulti even if actual keys belongs to different shards. In other words, other methods such as Get, Set, and Remove are executed without exclusive control.

◆ CompareExchangeMulti()

Status tkrzw::ShardDBM::CompareExchangeMulti ( const std::vector< std::pair< std::string_view, std::string_view >> &  expected,
const std::vector< std::pair< std::string_view, std::string_view >> &  desired 
)
overridevirtual

Compares the values of records and exchanges if the condition meets.

Parameters
expectedThe record keys and their expected values. If the value is nullptr, no existing record is expected. If the value is ANY_DATA, an existing record with any value is expacted.
desiredThe record keys and their desired values. If the value is nullptr, the record is to be removed.
Returns
The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.

ShardDBM has a special implementation to assure isolation of transactions in multi-threading environments. In contrast to the ProcessMulti method, isolation of CompareExchangeMulti is assured among all kinds of methods.

Reimplemented from tkrzw::DBM.

◆ ProcessEach()

Status tkrzw::ShardDBM::ProcessEach ( RecordProcessor proc,
bool  writable 
)
overridevirtual

Processes each and every record in the database with a processor.

Parameters
procThe pointer to the processor object.
writableTrue if the processor can edit the record.
Returns
The result status.

The ProcessFull of the processor is called repeatedly for each record. The ProcessEmpty of the processor is called once before the iteration and once after the iteration.

Implements tkrzw::DBM.

◆ Count()

Status tkrzw::ShardDBM::Count ( int64_t *  count)
overridevirtual

Gets the number of records.

Parameters
countThe pointer to an integer object to contain the result count.
Returns
The result status.

Implements tkrzw::DBM.

◆ GetFileSize()

Status tkrzw::ShardDBM::GetFileSize ( int64_t *  size)
overridevirtual

Gets the current file size of the database.

Parameters
sizeThe pointer to an integer object to contain the result size.
Returns
The result status.

Implements tkrzw::DBM.

◆ GetFilePath()

Status tkrzw::ShardDBM::GetFilePath ( std::string *  path)
overridevirtual

Gets the path of the database file.

Parameters
pathThe pointer to a string object to contain the result path.
Returns
The result status.

Implements tkrzw::DBM.

◆ GetTimestamp()

Status tkrzw::ShardDBM::GetTimestamp ( double *  timestamp)
overridevirtual

Gets the timestamp in seconds of the last modified time.

Parameters
timestampThe pointer to a double object to contain the timestamp.
Returns
The result status.

Implements tkrzw::DBM.

◆ Clear()

Status tkrzw::ShardDBM::Clear ( )
overridevirtual

Removes all records.

Returns
The result status.

Implements tkrzw::DBM.

◆ Rebuild()

Status tkrzw::ShardDBM::Rebuild ( )
overridevirtual

Rebuilds the entire database.

Returns
The result status.

Implements tkrzw::DBM.

◆ RebuildAdvanced()

Status tkrzw::ShardDBM::RebuildAdvanced ( const std::map< std::string, std::string > &  params = {})
overridevirtual

Rebuilds the entire database, in an advanced way.

Parameters
paramsOptional parameters.
Returns
The result status.

The parameters work in the same way as with PolyDBM::RebuildAdvanced.

Implements tkrzw::ParamDBM.

◆ ShouldBeRebuilt()

Status tkrzw::ShardDBM::ShouldBeRebuilt ( bool *  tobe)
overridevirtual

Checks whether the database should be rebuilt.

Parameters
tobeThe pointer to a boolean object to contain the result decision.
Returns
The result status.

Implements tkrzw::DBM.

◆ Synchronize()

Status tkrzw::ShardDBM::Synchronize ( bool  hard,
FileProcessor proc = nullptr 
)
overridevirtual

Synchronizes the content of the database to the file system.

Parameters
hardTrue to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
procThe pointer to the file processor object, whose Process method is called while the content of the file is synchronized. If it is nullptr, it is ignored.
Returns
The result status.

Implements tkrzw::DBM.

◆ SynchronizeAdvanced()

Status tkrzw::ShardDBM::SynchronizeAdvanced ( bool  hard,
FileProcessor proc = nullptr,
const std::map< std::string, std::string > &  params = {} 
)
overridevirtual

Synchronizes the content of the database to the file system, in an advanced way.

Parameters
hardTrue to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
procThe pointer to the file processor object, whose Process method is called while the content of the file is synchronized. If it is nullptr, it is ignored.
paramsOptional parameters.
Returns
The result status.

The parameters work in the same way as with PolyDBM::OpenAdvanced.

Implements tkrzw::ParamDBM.

◆ CopyFileData()

Status tkrzw::ShardDBM::CopyFileData ( const std::string &  dest_path,
bool  sync_hard = false 
)
overridevirtual

Copies the content of the database files to other files.

Parameters
dest_pathA path prefix to the destination files.
sync_hardTrue to do physical synchronization with the hardware.
Returns
The result status.

Copying is done while the content is synchronized and stable. So, this method is suitable for making a backup file while running a database service. Each shard file is copied and the destination file also has the same suffix.

Reimplemented from tkrzw::DBM.

◆ Inspect()

std::vector<std::pair<std::string, std::string> > tkrzw::ShardDBM::Inspect ( )
overridevirtual

Inspects the database.

Returns
A vector of pairs of a property name and its value.

Implements tkrzw::DBM.

◆ IsOpen()

bool tkrzw::ShardDBM::IsOpen ( ) const
overridevirtual

Checks whether the database is open.

Returns
True if the database is open, or false if not.

Implements tkrzw::DBM.

◆ IsWritable()

bool tkrzw::ShardDBM::IsWritable ( ) const
overridevirtual

Checks whether the database is writable.

Returns
True if the database is writable, or false if not.

Implements tkrzw::DBM.

◆ IsHealthy()

bool tkrzw::ShardDBM::IsHealthy ( ) const
overridevirtual

Checks whether the database condition is healthy.

Returns
True if the database condition is healthy, or false if not.

Implements tkrzw::DBM.

◆ IsOrdered()

bool tkrzw::ShardDBM::IsOrdered ( ) const
overridevirtual

Checks whether ordered operations are supported.

Returns
True if ordered operations are supported, or false if not.

Implements tkrzw::DBM.

◆ MakeIterator()

std::unique_ptr<DBM::Iterator> tkrzw::ShardDBM::MakeIterator ( )
overridevirtual

Makes an iterator for each record.

Returns
The iterator for each record.

Implements tkrzw::DBM.

◆ MakeDBM()

std::unique_ptr<DBM> tkrzw::ShardDBM::MakeDBM ( ) const
overridevirtual

Makes a new DBM object of the same concrete class.

Returns
The new DBM object.

Implements tkrzw::DBM.

◆ GetUpdateLogger()

UpdateLogger* tkrzw::ShardDBM::GetUpdateLogger ( ) const
overridevirtual

Gets the logger to write all update operations.

Returns
The update logger if it has been set or nullptr if it hasn't.

Implements tkrzw::DBM.

◆ SetUpdateLogger()

void tkrzw::ShardDBM::SetUpdateLogger ( UpdateLogger update_logger)
overridevirtual

Sets the logger to write all update operations.

Parameters
update_loggerThe pointer to the update logger object. Ownership is not taken. If it is nullptr, no logger is used.

Implements tkrzw::DBM.

◆ GetInternalDBM()

DBM* tkrzw::ShardDBM::GetInternalDBM ( ) const

Gets the pointer to the first internal database object.

Returns
The pointer to the first internal database object, or nullptr on failure.

◆ GetNumberOfShards()

static Status tkrzw::ShardDBM::GetNumberOfShards ( const std::string &  path,
int32_t *  num_shards 
)
static

Gets the number of shards of a database.

Parameters
pathThe database path.
num_shardsThe pointer to a variable to store the result.
Returns
The result status.

If there are multiple candidates, this function report a failure.

◆ RestoreDatabase()

static Status tkrzw::ShardDBM::RestoreDatabase ( const std::string &  old_file_path,
const std::string &  new_file_path,
const std::string &  class_name = "",
int64_t  end_offset = -1,
std::string_view  cipher_key = "" 
)
static

Restores a broken database as a new healthy database.

Parameters
old_file_pathThe path of the broken database.
new_file_pathThe path of the new database to be created.
class_nameThe name of the database class. If it is empty, the class is guessed from the file extension.
end_offsetThe exclusive end offset of records to read. Negative means unlimited. 0 means the size when the database is synched or closed properly. Using a positive value is not meaningful if the number of shards is more than one.
cipher_keyThe encryption key for cipher compressors.
Returns
The result status.

◆ RenameDatabase()

static Status tkrzw::ShardDBM::RenameDatabase ( const std::string &  old_file_path,
const std::string &  new_file_path 
)
static

Renames all files of a database to ones of another name.

Parameters
old_file_pathThe path of the database.
new_file_pathThe path of the new database.
Returns
The result status.