kyotocabinet::MapReduce Class Reference

MapReduce framework. More...

#include <kcdbext.h>

List of all members.

Classes
class	FlushThread
	Cache flusher.
class	MapChecker
	Checker for the map process.
class	MapVisitor
	Visitor for the map process.
struct	MergeLine
	Front line of a merging list.
class	ReduceTaskQueue
	Task queue for parallel reducer.
class	ValueIterator
	Value iterator for the reducer. More...
Public Types
enum	Option {   XNOLOCK = 1 << 0, XPARAMAP = 1 << 1, XPARARED = 1 << 2, XPARAFLS = 1 << 3,   XNOCOMP = 1 << 8 }
	Execution options. More...
Public Member Functions
	MapReduce ()
	Default constructor.
virtual	~MapReduce ()
	Destructor.
virtual bool	map (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)=0
	Map a record data.
virtual bool	reduce (const char *kbuf, size_t ksiz, ValueIterator *iter)=0
	Reduce a record data.
virtual bool	preprocess ()
	Preprocess the map operations.
virtual bool	midprocess ()
	Mediate between the map and the reduce phases.
virtual bool	postprocess ()
	Postprocess the reduce operations.
virtual bool	log (const char *name, const char *message)
	Process a log message.
bool	execute (BasicDB *db, const std::string &tmppath="", uint32_t opts=0)
	Execute the MapReduce process about a database.
void	tune_storage (int32_t dbnum, int64_t clim, int64_t cbnum)
	Set the storage configurations.
void	tune_thread (int32_t mapthnum, int32_t redthnum, int32_t flsthnum)
	Set the thread configurations.
Protected Member Functions
bool	emit (const char *kbuf, size_t ksiz, const char *vbuf, size_t vsiz)
	Emit a record from the mapper.

---

Detailed Description

MapReduce framework.

Note:

Although this framework is not distributed or concurrent, it is useful for aggregate calculation with less CPU loading and less memory usage.

---

Member Enumeration Documentation

enum kyotocabinet::MapReduce::Option

Execution options.

Enumerator:

XNOLOCK	avoid locking against update operations
XPARAMAP	run mappers in parallel
XPARARED	run reducers in parallel
XPARAFLS	run cache flushers in parallel
XNOCOMP	avoid compression of temporary databases

---

Constructor & Destructor Documentation

kyotocabinet::MapReduce::MapReduce

(

)

[explicit]

Default constructor.

virtual kyotocabinet::MapReduce::~MapReduce

(

)

[virtual]

Destructor.

---

Member Function Documentation

virtual bool kyotocabinet::MapReduce::map	(	const char *	kbuf,
		size_t	ksiz,
		const char *	vbuf,
		size_t	vsiz
	)		[pure virtual]

Map a record data.

Parameters:

kbuf	the pointer to the key region.
ksiz	the size of the key region.
vbuf	the pointer to the value region.
vsiz	the size of the value region.

Returns:

true on success, or false on failure.

Note:

This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::reduce	(	const char *	kbuf,
		size_t	ksiz,
		ValueIterator *	iter
	)		[pure virtual]

Reduce a record data.

Parameters:

kbuf	the pointer to the key region.
ksiz	the size of the key region.
iter	the iterator to get the values.

Returns:

true on success, or false on failure.

Note:

To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::preprocess

(

)

[virtual]

Preprocess the map operations.

Returns:

true on success, or false on failure.

Note:

This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::midprocess

(

)

[virtual]

Mediate between the map and the reduce phases.

Returns:

true on success, or false on failure.

Note:

This function can call the MapReduce::emit method to emit a record. To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::postprocess

(

)

[virtual]

Postprocess the reduce operations.

Returns:

true on success, or false on failure.

Note:

To avoid deadlock, any explicit database operation must not be performed in this function.

virtual bool kyotocabinet::MapReduce::log	(	const char *	name,
		const char *	message
	)		[virtual]

Process a log message.

Parameters:

name	the name of the event.
message	a supplement message.

Returns:

true on success, or false on failure.

bool kyotocabinet::MapReduce::execute	(	BasicDB *	db,
		const std::string &	tmppath = "",
		uint32_t	opts = 0
	)

Execute the MapReduce process about a database.

Parameters:

db	the source database.
tmppath	the path of a directory for the temporary data storage. If it is an empty string, temporary data are handled on memory.
opts	the optional features by bitwise-or: MapReduce::XNOLOCK to avoid locking against update operations by other threads, MapReduce::XPARAMAP to run the mapper in parallel, MapReduce::XPARARED to run the reducer in parallel, MapReduce::XNOCOMP to avoid compression of temporary databases.

Returns:

true on success, or false on failure.

void kyotocabinet::MapReduce::tune_storage	(	int32_t	dbnum,
		int64_t	clim,
		int64_t	cbnum
	)

Set the storage configurations.

Parameters:

dbnum	the number of temporary databases.
clim	the limit size of the internal cache.
cbnum	the bucket number of the internal cache.

void kyotocabinet::MapReduce::tune_thread	(	int32_t	mapthnum,
		int32_t	redthnum,
		int32_t	flsthnum
	)

Set the thread configurations.

Parameters:

mapthnum	the number of threads for the mapper.
redthnum	the number of threads for the reducer.
flsthnum	the number of threads for the internal flusher.

bool kyotocabinet::MapReduce::emit	(	const char *	kbuf,
		size_t	ksiz,
		const char *	vbuf,
		size_t	vsiz
	)		[protected]

Emit a record from the mapper.

Parameters:

kbuf	the pointer to the key region.
ksiz	the size of the key region.
vbuf	the pointer to the value region.
vsiz	the size of the value region.

Returns:

true on success, or false on failure.