org.apache.storm.kafka.spout

Class KafkaSpoutConfig.Builder<K,V>

java.lang.Object

org.apache.storm.kafka.spout.internal.CommonKafkaSpoutConfig.Builder<K,V,KafkaSpoutConfig.Builder<K,V>>

org.apache.storm.kafka.spout.KafkaSpoutConfig.Builder<K,V>

Enclosing class:

KafkaSpoutConfig<K,V>

public static class KafkaSpoutConfig.Builder<K,V>
extends CommonKafkaSpoutConfig.Builder<K,V,KafkaSpoutConfig.Builder<K,V>>

Constructor Summary

Constructors

Constructor and Description
Builder(String bootstrapServers, Pattern topics)
Builder(String bootstrapServers, Set<String> topics)
Builder(String bootstrapServers, String... topics)
Builder(String bootstrapServers, TopicFilter topicFilter, ManualPartitioner topicPartitioner) Create a KafkaSpoutConfig builder with default property values and no key/value deserializers.

Method Summary

Modifier and Type	Method and Description
KafkaSpoutConfig<K,V>	build()
KafkaSpoutConfig.Builder<K,V>	setEmitNullTuples(boolean emitNullTuples) Specifies if the spout should emit null tuples to the component downstream, or rather not emit and directly ack them.
KafkaSpoutConfig.Builder<K,V>	setMaxUncommittedOffsets(int maxUncommittedOffsets) Defines the max number of polled offsets (records) that can be pending commit, before another poll can take place.
KafkaSpoutConfig.Builder<K,V>	setMetricsTimeBucketSizeInSecs(int metricsTimeBucketSizeInSecs) The time period that metrics data in bucketed into.
KafkaSpoutConfig.Builder<K,V>	setOffsetCommitPeriodMs(long offsetCommitPeriodMs) Specifies the period, in milliseconds, the offset commit task is periodically called.
KafkaSpoutConfig.Builder<K,V>	setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee processingGuarantee) Specifies which processing guarantee the spout should offer.
KafkaSpoutConfig.Builder<K,V>	setRetry(KafkaSpoutRetryService retryService) Sets the retry service for the spout to use.
KafkaSpoutConfig.Builder<K,V>	setTupleListener(KafkaTupleListener tupleListener) Sets the tuple listener for the spout to use.
KafkaSpoutConfig.Builder<K,V>	setTupleTrackingEnforced(boolean tupleTrackingEnforced) Specifies whether the spout should require Storm to track emitted tuples when using a KafkaSpoutConfig.ProcessingGuarantee other than KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE.

Methods inherited from class org.apache.storm.kafka.spout.internal.CommonKafkaSpoutConfig.Builder

getKafkaProps, setFirstPollOffsetStrategy, setPartitionRefreshPeriodMs, setPollTimeoutMs, setProp, setProp, setProp, setRecordTranslator, setRecordTranslator, setRecordTranslator, setStartTimeStamp

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

public Builder(String bootstrapServers,
               String... topics)

Builder

public Builder(String bootstrapServers,
               Set<String> topics)

Builder

public Builder(String bootstrapServers,
               Pattern topics)

Builder

public Builder(String bootstrapServers,
               TopicFilter topicFilter,
               ManualPartitioner topicPartitioner)

Create a KafkaSpoutConfig builder with default property values and no key/value deserializers.

Parameters:

bootstrapServers - The bootstrap servers the consumer will use

topicFilter - The topic filter defining which topics and partitions the spout will read

topicPartitioner - The topic partitioner defining which topics and partitions are assinged to each spout task

Method Detail

setOffsetCommitPeriodMs

public KafkaSpoutConfig.Builder<K,V> setOffsetCommitPeriodMs(long offsetCommitPeriodMs)

Specifies the period, in milliseconds, the offset commit task is periodically called. Default is 15s.

This setting only has an effect if the configured KafkaSpoutConfig.ProcessingGuarantee is KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE or KafkaSpoutConfig.ProcessingGuarantee.NO_GUARANTEE.

Parameters:

offsetCommitPeriodMs - time in ms

setMaxUncommittedOffsets

public KafkaSpoutConfig.Builder<K,V> setMaxUncommittedOffsets(int maxUncommittedOffsets)

Defines the max number of polled offsets (records) that can be pending commit, before another poll can take place. Once this limit is reached, no more offsets (records) can be polled until the next successful commit(s) sets the number of pending offsets below the threshold. The default is KafkaSpoutConfig.DEFAULT_MAX_UNCOMMITTED_OFFSETS. This limit is per partition and may in some cases be exceeded, but each partition cannot exceed this limit by more than maxPollRecords - 1.

This setting only has an effect if the configured KafkaSpoutConfig.ProcessingGuarantee is KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE.

Parameters:

maxUncommittedOffsets - max number of records that can be be pending commit

setRetry

public KafkaSpoutConfig.Builder<K,V> setRetry(KafkaSpoutRetryService retryService)

Sets the retry service for the spout to use.

This setting only has an effect if the configured KafkaSpoutConfig.ProcessingGuarantee is KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE.

Parameters:

retryService - the new retry service

Returns:

the builder (this).

setTupleListener

public KafkaSpoutConfig.Builder<K,V> setTupleListener(KafkaTupleListener tupleListener)

Sets the tuple listener for the spout to use.

Parameters:

tupleListener - the tuple listener

Returns:

the builder (this).

setEmitNullTuples

public KafkaSpoutConfig.Builder<K,V> setEmitNullTuples(boolean emitNullTuples)

Specifies if the spout should emit null tuples to the component downstream, or rather not emit and directly ack them. By default this parameter is set to false, which means that null tuples are not emitted.

Parameters:

emitNullTuples - sets if null tuples should or not be emitted downstream

setProcessingGuarantee

public KafkaSpoutConfig.Builder<K,V> setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee processingGuarantee)

Specifies which processing guarantee the spout should offer. Refer to the documentation for KafkaSpoutConfig.ProcessingGuarantee.

Parameters:

processingGuarantee - The processing guarantee the spout should offer.

setTupleTrackingEnforced

public KafkaSpoutConfig.Builder<K,V> setTupleTrackingEnforced(boolean tupleTrackingEnforced)

Specifies whether the spout should require Storm to track emitted tuples when using a KafkaSpoutConfig.ProcessingGuarantee other than KafkaSpoutConfig.ProcessingGuarantee.AT_LEAST_ONCE. The spout will always track emitted tuples when offering at-least-once guarantees regardless of this setting. This setting is false by default.

Enabling tracking can be useful even in cases where reliability is not a concern, because it allows Config.TOPOLOGY_MAX_SPOUT_PENDING to have an effect, and enables some spout metrics (e.g. complete-latency) that would otherwise be disabled.

Parameters:

tupleTrackingEnforced - true if Storm should track emitted tuples, false otherwise

setMetricsTimeBucketSizeInSecs

public KafkaSpoutConfig.Builder<K,V> setMetricsTimeBucketSizeInSecs(int metricsTimeBucketSizeInSecs)

The time period that metrics data in bucketed into.

Parameters:

metricsTimeBucketSizeInSecs - time in seconds

build

public KafkaSpoutConfig<K,V> build()

Specified by:

build in class CommonKafkaSpoutConfig.Builder<K,V,KafkaSpoutConfig.Builder<K,V>>