Kyoto Tycoon
Classes | Public Types | Public Member Functions | Protected Member Functions
kyototycoon::MapReduce Class Reference

MapReduce framework. More...

#include <ktdbext.h>

List of all members.

Classes

class  FlushThread
 Cache flusher.
class  MapChecker
 Checker for the map process.
class  MapVisitor
 Visitor for the map process.
struct  MergeLine
 Front line of a merging list.
class  ReduceTaskQueue
 Task queue for parallel reducer.
class  ValueIterator
 Value iterator for the reducer. More...

Public Types

enum  Option {
  XNOLOCK = 1 << 0, XPARAMAP = 1 << 1, XPARARED = 1 << 2, XPARAFLS = 1 << 3,
  XNOCOMP = 1 << 8
}
 Execution options. More...

Public Member Functions

 MapReduce ()
 Default constructor.
virtual ~MapReduce ()
 Destructor.
virtual bool map (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)=0
 Map a record data.
virtual bool reduce (const char *kbuf, size_t ksiz, ValueIterator *iter)=0
 Reduce a record data.
virtual bool preprocess ()
 Preprocess the map operations.
virtual bool midprocess ()
 Mediate between the map and the reduce phases.
virtual bool postprocess ()
 Postprocess the reduce operations.
virtual bool log (const char *name, const char *message)
 Process a log message.
bool execute (TimedDB *db, const std::string &tmppath="", uint32_t opts=0)
 Execute the MapReduce process about a database.
void tune_storage (int32_t dbnum, int64_t clim, int64_t cbnum)
 Set the storage configurations.
void tune_thread (int32_t mapthnum, int32_t redthnum, int32_t flsthnum)
 Set the thread configurations.

Protected Member Functions

bool emit (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)
 Emit a record from the mapper.

Detailed Description

MapReduce framework.

Note:
Although this framework is not distributed or concurrent, it is useful for aggregate calculation with less CPU loading and less memory usage.

Member Enumeration Documentation

Execution options.

Enumerator:
XNOLOCK 

avoid locking against update operations

XPARAMAP 

run mappers in parallel

XPARARED 

run reducers in parallel

XPARAFLS 

run cache flushers in parallel

XNOCOMP 

avoid compression of temporary databases


Constructor & Destructor Documentation

Default constructor.

virtual kyototycoon::MapReduce::~MapReduce ( ) [virtual]

Destructor.


Member Function Documentation

virtual bool kyototycoon::MapReduce::map ( const char *  kbuf,
size_t  ksiz,
const char *  vbuf,
size_t  vsiz 
) [pure virtual]

Map a record data.

Parameters:
kbufthe pointer to the key region.
ksizthe size of the key region.
vbufthe pointer to the value region.
vsizthe size of the value region.
Returns:
true on success, or false on failure.
Note:
This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.
virtual bool kyototycoon::MapReduce::reduce ( const char *  kbuf,
size_t  ksiz,
ValueIterator iter 
) [pure virtual]

Reduce a record data.

Parameters:
kbufthe pointer to the key region.
ksizthe size of the key region.
iterthe iterator to get the values.
Returns:
true on success, or false on failure.
Note:
To avoid deadlock, any explicit database operation must not be performed in this function.
virtual bool kyototycoon::MapReduce::preprocess ( ) [virtual]

Preprocess the map operations.

Returns:
true on success, or false on failure.
Note:
This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.
virtual bool kyototycoon::MapReduce::midprocess ( ) [virtual]

Mediate between the map and the reduce phases.

Returns:
true on success, or false on failure.
Note:
This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.
virtual bool kyototycoon::MapReduce::postprocess ( ) [virtual]

Postprocess the reduce operations.

Returns:
true on success, or false on failure.
Note:
To avoid deadlock, any explicit database operation must not be performed in this function.
virtual bool kyototycoon::MapReduce::log ( const char *  name,
const char *  message 
) [virtual]

Process a log message.

Parameters:
namethe name of the event.
messagea supplement message.
Returns:
true on success, or false on failure.
bool kyototycoon::MapReduce::execute ( TimedDB db,
const std::string &  tmppath = "",
uint32_t  opts = 0 
)

Execute the MapReduce process about a database.

Parameters:
dbthe source database.
tmppaththe path of a directory for the temporary data storage. If it is an empty string, temporary data are handled on memory.
optsthe optional features by bitwise-or: MapReduce::XNOLOCK to avoid locking against update operations by other threads, MapReduce::XNOCOMP to avoid compression of temporary databases.
Returns:
true on success, or false on failure.
void kyototycoon::MapReduce::tune_storage ( int32_t  dbnum,
int64_t  clim,
int64_t  cbnum 
)

Set the storage configurations.

Parameters:
dbnumthe number of temporary databases.
climthe limit size of the internal cache.
cbnumthe bucket number of the internal cache.
void kyototycoon::MapReduce::tune_thread ( int32_t  mapthnum,
int32_t  redthnum,
int32_t  flsthnum 
)

Set the thread configurations.

Parameters:
mapthnumthe number of threads for the mapper.
redthnumthe number of threads for the reducer.
flsthnumthe number of threads for the internal flusher.
bool kyototycoon::MapReduce::emit ( const char *  kbuf,
size_t  ksiz,
const char *  vbuf,
size_t  vsiz 
) [protected]

Emit a record from the mapper.

Parameters:
kbufthe pointer to the key region.
ksizthe size of the key region.
vbufthe pointer to the value region.
vsizthe size of the value region.
Returns:
true on success, or false on failure.