org.apache.storm.hdfs.bolt

Class AbstractHdfsBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

org.apache.storm.hdfs.bolt.AbstractHdfsBolt

All Implemented Interfaces:

Serializable, IBolt, IComponent, IRichBolt

Direct Known Subclasses:

AvroGenericRecordBolt, HdfsBolt, SequenceFileBolt

public abstract class AbstractHdfsBolt
extends BaseRichBolt

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected OutputCollector	collector
protected String	configKey
protected FileNameFormat	fileNameFormat
protected Integer	fileRetryCount
protected org.apache.hadoop.fs.FileSystem	fs
protected String	fsUrl
protected org.apache.hadoop.conf.Configuration	hdfsConfig
protected Integer	maxOpenFiles
protected long	offset
protected Partitioner	partitioner
protected List<RotationAction>	rotationActions
protected Map<String,Integer>	rotationCounterMap
protected FileRotationPolicy	rotationPolicy
protected Timer	rotationTimer
protected SyncPolicy	syncPolicy
protected Integer	tickTupleInterval
protected Object	writeLock
protected Map<String,Writer>	writers

Constructor Summary

Constructors

Constructor and Description
AbstractHdfsBolt()

Method Summary

Modifier and Type	Method and Description
void	cleanup() Called when an IBolt is going to be shutdown.
void	declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) Declare the output schema for all the streams of this topology.
protected abstract void	doPrepare(Map<String,Object> conf, TopologyContext topologyContext, OutputCollector collector)
void	execute(Tuple tuple) Process a single tuple of input.
protected org.apache.hadoop.fs.Path	getBasePathForNextFile(Tuple tuple)
Map<String,Object>	getComponentConfiguration() Declare configuration specific to this component.
protected abstract String	getWriterKey(Tuple tuple)
protected abstract Writer	makeNewWriter(org.apache.hadoop.fs.Path path, Tuple tuple)
void	prepare(Map<String,Object> conf, TopologyContext topologyContext, OutputCollector collector) Marked as final to prevent override.
protected void	rotateOutputFile(Writer writer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

writers

protected Map<String,Writer> writers

rotationCounterMap

protected Map<String,Integer> rotationCounterMap

rotationActions

protected List<RotationAction> rotationActions

collector

protected OutputCollector collector

fs

protected transient org.apache.hadoop.fs.FileSystem fs

syncPolicy

protected SyncPolicy syncPolicy

rotationPolicy

protected FileRotationPolicy rotationPolicy

fileNameFormat

protected FileNameFormat fileNameFormat

fsUrl

protected String fsUrl

configKey

protected String configKey

writeLock

protected transient Object writeLock

rotationTimer

protected transient Timer rotationTimer

offset

protected long offset

fileRetryCount

protected Integer fileRetryCount

tickTupleInterval

protected Integer tickTupleInterval

maxOpenFiles

protected Integer maxOpenFiles

partitioner

protected Partitioner partitioner

hdfsConfig

protected transient org.apache.hadoop.conf.Configuration hdfsConfig

Constructor Detail

AbstractHdfsBolt

public AbstractHdfsBolt()

Method Detail

rotateOutputFile

protected void rotateOutputFile(Writer writer)
                         throws IOException

Throws:

IOException

prepare

public final void prepare(Map<String,Object> conf,
                          TopologyContext topologyContext,
                          OutputCollector collector)

Marked as final to prevent override. Subclasses should implement the doPrepare() method.

Parameters:

conf - The Storm configuration for this bolt. This is the configuration provided to the topology merged in with cluster configuration on this machine.

topologyContext - This object can be used to get information about this task's place within the topology, including the task id and component id of this task, input and output information, etc.

collector - The collector is used to emit tuples from this bolt. Tuples can be emitted at any time, including the prepare and cleanup methods. The collector is thread-safe and should be saved as an instance variable of this bolt object.

execute

public final void execute(Tuple tuple)

Description copied from interface: IBolt

Process a single tuple of input. The Tuple object contains metadata on it about which component/stream/task it came from. The values of the Tuple can be accessed using Tuple#getValue. The IBolt does not have to process the Tuple immediately. It is perfectly fine to hang onto a tuple and process it later (for instance, to do an aggregation or join).

Tuples should be emitted using the OutputCollector provided through the prepare method. It is required that all input tuples are acked or failed at some point using the OutputCollector. Otherwise, Storm will be unable to determine when tuples coming off the spouts have been completed.

For the common case of acking an input tuple at the end of the execute method, see IBasicBolt which automates this.

Parameters:

tuple - The input tuple to be processed.

getComponentConfiguration

public Map<String,Object> getComponentConfiguration()

Description copied from interface: IComponent

Declare configuration specific to this component. Only a subset of the "topology.*" configs can be overridden. The component configuration can be further overridden when constructing the topology using TopologyBuilder

Specified by:

getComponentConfiguration in interface IComponent

Overrides:

getComponentConfiguration in class BaseComponent

declareOutputFields

public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer)

Description copied from interface: IComponent

Declare the output schema for all the streams of this topology.

Parameters:

outputFieldsDeclarer - this is used to declare output stream ids, output fields, and whether or not each output stream is a direct stream

cleanup

public void cleanup()

Description copied from interface: IBolt

Called when an IBolt is going to be shutdown. Storm will make a best-effort attempt to call this if the worker shutdown is orderly. The Config.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS setting controls how long orderly shutdown is allowed to take. There is no guarantee that cleanup will be called if shutdown is not orderly, or if the shutdown exceeds the time limit.

The one context where cleanup is guaranteed to be called is when a topology is killed when running Storm in local mode.

Specified by:

cleanup in interface IBolt

Overrides:

cleanup in class BaseRichBolt

getBasePathForNextFile

protected org.apache.hadoop.fs.Path getBasePathForNextFile(Tuple tuple)

doPrepare

protected abstract void doPrepare(Map<String,Object> conf,
                                  TopologyContext topologyContext,
                                  OutputCollector collector)
                           throws IOException

Throws:

IOException

getWriterKey

protected abstract String getWriterKey(Tuple tuple)

makeNewWriter

protected abstract Writer makeNewWriter(org.apache.hadoop.fs.Path path,
                                        Tuple tuple)
                                 throws IOException

Throws:

IOException