org.apache.storm.perf

Class KafkaClientHdfsTopo

java.lang.Object

org.apache.storm.perf.KafkaClientHdfsTopo

public class KafkaClientHdfsTopo
extends Object

This topo helps measure speed of reading from Kafka and writing to Hdfs. Spout Reads from Kafka. Bolt writes to Hdfs

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaClientHdfsTopo.LineWriter

Field Summary

Fields

Modifier and Type	Field and Description
static String	BOLT_ID
static String	BOLT_NUM
static int	DEFAULT_BOLT_NUM
static int	DEFAULT_HDFS_BATCH
static int	DEFAULT_SPOUT_NUM
static String	HDFS_BATCH
static String	HDFS_PATH
static String	HDFS_URI
static String	KAFKA_BOOTSTRAP_HOSTS
static String	KAFKA_TOPIC
static String	SPOUT_ID
static String	SPOUT_NUM
static String	TOPOLOGY_NAME

Constructor Summary

Constructors

Constructor and Description
KafkaClientHdfsTopo()

Method Summary

Modifier and Type	Method and Description
static int	getInt(Map map, Object key, int def)
static String	getStr(Map map, Object key)
static void	main(String[] args) Copies text file content from sourceDir to destinationDir.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPOUT_NUM

public static final String SPOUT_NUM

See Also:

Constant Field Values

BOLT_NUM

public static final String BOLT_NUM

See Also:

Constant Field Values

KAFKA_TOPIC

public static final String KAFKA_TOPIC

See Also:

Constant Field Values

KAFKA_BOOTSTRAP_HOSTS

public static final String KAFKA_BOOTSTRAP_HOSTS

See Also:

Constant Field Values

HDFS_URI

public static final String HDFS_URI

See Also:

Constant Field Values

HDFS_PATH

public static final String HDFS_PATH

See Also:

Constant Field Values

HDFS_BATCH

public static final String HDFS_BATCH

See Also:

Constant Field Values

DEFAULT_SPOUT_NUM

public static final int DEFAULT_SPOUT_NUM

See Also:

Constant Field Values

DEFAULT_BOLT_NUM

public static final int DEFAULT_BOLT_NUM

See Also:

Constant Field Values

DEFAULT_HDFS_BATCH

public static final int DEFAULT_HDFS_BATCH

See Also:

Constant Field Values

TOPOLOGY_NAME

public static final String TOPOLOGY_NAME

See Also:

Constant Field Values

SPOUT_ID

public static final String SPOUT_ID

See Also:

Constant Field Values

BOLT_ID

public static final String BOLT_ID

See Also:

Constant Field Values

Constructor Detail

KafkaClientHdfsTopo

public KafkaClientHdfsTopo()

Method Detail

getInt

public static int getInt(Map map,
                         Object key,
                         int def)

getStr

public static String getStr(Map map,
                            Object key)

main

public static void main(String[] args)
                 throws Exception

Copies text file content from sourceDir to destinationDir. Moves source files into sourceDir after its done consuming

Throws:

Exception