Package org.apache.storm

Class DaemonConfig

java.lang.Object
org.apache.storm.DaemonConfig
All Implemented Interfaces:
Validated
public class DaemonConfig extends Object implements Validated
Storm configs are specified as a plain old map. This class provides constants for all the configurations possible on a Storm cluster. Each constant is paired with an annotation that defines the validity criterion of the corresponding field. Default values for these configs can be found in defaults.yaml.

This class extends Config for supporting Storm Daemons.

  • Field Details

    • STORM_NETTY_FLUSH_CHECK_INTERVAL_MS

      public static final String STORM_NETTY_FLUSH_CHECK_INTERVAL_MS
      We check with this interval that whether the Netty channel is writable and try to write pending messages.
      See Also:

    • STORM_DAEMON_METRICS_REPORTER_PLUGINS

      public static final String STORM_DAEMON_METRICS_REPORTER_PLUGINS
      A list of daemon metrics reporter plugin class names. These plugins must implement PreparableReporter interface.
      See Also:

    • STORM_DAEMON_METRICS_REPORTER_PLUGIN_DOMAIN

      public static final String STORM_DAEMON_METRICS_REPORTER_PLUGIN_DOMAIN
      Specify the domain for daemon metrics reporter plugin to limit reporting to specific domain.
      See Also:

    • STORM_DAEMON_METRICS_REPORTER_INTERVAL_SECS

      public static final String STORM_DAEMON_METRICS_REPORTER_INTERVAL_SECS
      We report the metrics with this interval period.
      See Also:

    • STORM_DAEMON_METRICS_REPORTER_CSV_LOG_DIR

      public static final String STORM_DAEMON_METRICS_REPORTER_CSV_LOG_DIR
      Specify the csv reporter directory for CvsPreparableReporter daemon metrics reporter.
      See Also:

    • STORM_LOG4J2_CONF_DIR

      public static final String STORM_LOG4J2_CONF_DIR
      A directory that holds configuration files for log4j2. It can be either a relative or an absolute directory. If relative, it is relative to the storm's home directory.
      See Also:

    • STORM_SCHEDULER

      public static final String STORM_SCHEDULER
      A global task scheduler used to assign topologies's tasks to supervisors' workers.

      If this is not set, a default system scheduler will be used.

      See Also:

    • SCHEDULING_TIMEOUT_SECONDS_PER_TOPOLOGY

      public static final String SCHEDULING_TIMEOUT_SECONDS_PER_TOPOLOGY
      Max time to attempt to schedule one topology. The default is 60 seconds
      See Also:

    • BLACKLIST_SCHEDULER_TOLERANCE_TIME

      public static final String BLACKLIST_SCHEDULER_TOLERANCE_TIME
      The number of seconds that the blacklist scheduler will concern of bad slots or supervisors.
      See Also:

    • BLACKLIST_SCHEDULER_TOLERANCE_COUNT

      public static final String BLACKLIST_SCHEDULER_TOLERANCE_COUNT
      The number of hit count that will trigger blacklist in tolerance time.
      See Also:

    • BLACKLIST_SCHEDULER_RESUME_TIME

      public static final String BLACKLIST_SCHEDULER_RESUME_TIME
      The number of seconds that the blacklisted slots or supervisor will be resumed.
      See Also:

    • BLACKLIST_SCHEDULER_ENABLE_SEND_ASSIGNMENT_FAILURES

      public static final String BLACKLIST_SCHEDULER_ENABLE_SEND_ASSIGNMENT_FAILURES
      Enables blacklisting support for supervisors with failed send assignment calls.
      See Also:

    • BLACKLIST_SCHEDULER_REPORTER

      public static final String BLACKLIST_SCHEDULER_REPORTER
      The class that the blacklist scheduler will report the blacklist.
      See Also:

    • BLACKLIST_SCHEDULER_STRATEGY

      public static final String BLACKLIST_SCHEDULER_STRATEGY
      The class that specifies the eviction strategy to use in blacklist scheduler. If you are using the RAS scheduler please set this to "org.apache.storm.scheduler.blacklist.strategies.RasBlacklistStrategy" or you may get odd behavior when the cluster is full and there are blacklisted nodes.
      See Also:

    • BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_BAD_BASED_ON_BAD_SLOT

      public static final String BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_BAD_BASED_ON_BAD_SLOT
      Whether BlacklistScheduler will assume the supervisor is bad based on bad slots or not. A bad slot indicates the situation where the nimbus doesn't receive heartbeat from the worker in time, it's hard to differentiate if it's because of the supervisor node or the worker itself. If this is set to true, the scheduler will consider a supervisor is bad when seeing bad slots in it. Otherwise, the scheduler will assume a supervisor is bad only when it does not receive supervisor heartbeat in time.
      See Also:

    • SCHEDULER_DISPLAY_RESOURCE

      public static final String SCHEDULER_DISPLAY_RESOURCE
      Whether we want to display all the resource capacity and scheduled usage on the UI page. You MUST have this variable set if you are using any kind of resource-related scheduler.

      If this is not set, we will not display resource capacity and usage on the UI.

      See Also:

    • STORM_HEALTH_CHECK_DIR

      public static final String STORM_HEALTH_CHECK_DIR
      The directory where storm's health scripts go.
      See Also:

    • STORM_HEALTH_CHECK_TIMEOUT_MS

      public static final String STORM_HEALTH_CHECK_TIMEOUT_MS
      The time to allow any given healthcheck script to run before it is marked failed due to timeout.
      See Also:

    • STORM_HEALTH_CHECK_FAIL_ON_TIMEOUTS

      public static final String STORM_HEALTH_CHECK_FAIL_ON_TIMEOUTS
      Boolean setting to configure if health checks should fail when timeouts occur or not.
      See Also:

    • NIMBUS_DAEMON_USER

      public static final String NIMBUS_DAEMON_USER
      This is the user that the Nimbus daemon process is running as. May be used when security is enabled to authorize actions in the cluster.
      See Also:

    • NIMBUS_CHILDOPTS

      public static final String NIMBUS_CHILDOPTS
      This parameter is used by the storm-deploy project to configure the jvm options for the nimbus daemon.
      See Also:

    • NIMBUS_TASK_TIMEOUT_SECS

      public static final String NIMBUS_TASK_TIMEOUT_SECS
      How long without heartbeating a task can go before nimbus will consider the task dead and reassign it to another location. Can be exceeded when Config.TOPOLOGY_WORKER_TIMEOUT_SECS is set.
      See Also:

    • NIMBUS_MONITOR_FREQ_SECS

      public static final String NIMBUS_MONITOR_FREQ_SECS
      How often nimbus should wake up to check heartbeats and do reassignments. Note that if a machine ever goes down Nimbus will immediately wake up and take action. This parameter is for checking for failures when there's no explicit event like that occurring.
      See Also:

    • NIMBUS_CLEANUP_INBOX_FREQ_SECS

      public static final String NIMBUS_CLEANUP_INBOX_FREQ_SECS
      How often nimbus should wake the cleanup thread to clean the inbox.
      See Also:

    • NIMBUS_INBOX_JAR_EXPIRATION_SECS

      public static final String NIMBUS_INBOX_JAR_EXPIRATION_SECS
      The length of time a jar file lives in the inbox before being deleted by the cleanup thread.

      Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS. Note that the time it takes to delete an inbox jar file is going to be somewhat more than NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS is set to).

      See Also:

    • NIMBUS_SUPERVISOR_TIMEOUT_SECS

      public static final String NIMBUS_SUPERVISOR_TIMEOUT_SECS
      How long before a supervisor can go without heartbeating before nimbus considers it dead and stops assigning new work to it.
      See Also:

    • NIMBUS_TASK_LAUNCH_SECS

      public static final String NIMBUS_TASK_LAUNCH_SECS
      A special timeout used when a task is initially launched. During launch, this is the timeout used until the first heartbeat, overriding nimbus.task.timeout.secs.

      A separate timeout exists for launch because there can be quite a bit of overhead to launching new JVM's and configuring them.

      Can be exceeded when Config.TOPOLOGY_WORKER_TIMEOUT_SECS is set.
      See Also:

    • NIMBUS_FILE_COPY_EXPIRATION_SECS

      public static final String NIMBUS_FILE_COPY_EXPIRATION_SECS
      During upload/download with the master, how long an upload or download connection is idle before nimbus considers it dead and drops the connection.
      See Also:

    • NIMBUS_TOPOLOGY_VALIDATOR

      public static final String NIMBUS_TOPOLOGY_VALIDATOR
      A custom class that implements ITopologyValidator that is run whenever a topology is submitted. Can be used to provide business-specific logic for whether topologies are allowed to run or not.
      See Also:

    • NIMBUS_AUTHORIZER

      public static final String NIMBUS_AUTHORIZER
      Class name for authorization plugin for Nimbus.
      See Also:

    • SUPERVISOR_AUTHORIZER

      public static final String SUPERVISOR_AUTHORIZER
      Class name for authorization plugin for supervisor.
      See Also:

    • NIMBUS_IMPERSONATION_AUTHORIZER

      public static final String NIMBUS_IMPERSONATION_AUTHORIZER
      Impersonation user ACL config entries.
      See Also:

    • NIMBUS_CREDENTIAL_RENEW_FREQ_SECS

      public static final String NIMBUS_CREDENTIAL_RENEW_FREQ_SECS
      How often nimbus should wake up to renew credentials if needed.
      See Also:

    • NIMBUS_TOPOLOGY_ACTION_NOTIFIER_PLUGIN

      public static final String NIMBUS_TOPOLOGY_ACTION_NOTIFIER_PLUGIN
      FQCN of a class that implements I @see org.apache.storm.nimbus.ITopologyActionNotifierPlugin for details.
      See Also:

    • NIMBUS_ASSIGNMENTS_SERVICE_THREADS

      public static final String NIMBUS_ASSIGNMENTS_SERVICE_THREADS
      This controls the number of working threads for distributing master assignments to supervisors.
      See Also:

    • NIMBUS_ASSIGNMENTS_SERVICE_THREAD_QUEUE_SIZE

      public static final String NIMBUS_ASSIGNMENTS_SERVICE_THREAD_QUEUE_SIZE
      This controls the number of working thread queue size of assignment service.
      See Also:

    • NIMBUS_WORKER_HEARTBEATS_RECOVERY_STRATEGY_CLASS

      public static final String NIMBUS_WORKER_HEARTBEATS_RECOVERY_STRATEGY_CLASS
      class controls heartbeats recovery strategy.
      See Also:

    • NIMBUS_TOPOLOGY_BLOBSTORE_DELETION_DELAY_MS

      public static final String NIMBUS_TOPOLOGY_BLOBSTORE_DELETION_DELAY_MS
      This controls the number of milliseconds nimbus will wait before deleting a topology blobstore once detected it is able to delete.
      See Also:

    • UI_HOST

      public static final String UI_HOST
      Storm UI binds to this host/interface.
      See Also:

    • UI_PORT

      public static final String UI_PORT
      Storm UI binds to this port.
      See Also:

    • UI_TITLE

      public static final String UI_TITLE
      Storm UI's title.
      See Also:

    • UI_DISABLE_HTTP_BINDING

      public static final String UI_DISABLE_HTTP_BINDING
      This controls wheather Storm UI should bind to http port even if ui.port is > 0.
      See Also:

    • UI_DISABLE_SPOUT_LAG_MONITORING

      public static final String UI_DISABLE_SPOUT_LAG_MONITORING
      This controls whether Storm UI displays spout lag for the Kafka spout.
      See Also:

    • LOGVIEWER_DISABLE_HTTP_BINDING

      public static final String LOGVIEWER_DISABLE_HTTP_BINDING
      This controls wheather Storm Logviewer should bind to http port even if logviewer.port is > 0.
      See Also:

    • DRPC_DISABLE_HTTP_BINDING

      public static final String DRPC_DISABLE_HTTP_BINDING
      This controls wheather Storm DRPC should bind to http port even if drpc.http.port is > 0.
      See Also:

    • UI_PROJECT_BUGTRACKER_URL

      public static final String UI_PROJECT_BUGTRACKER_URL
      Storm UI Project BUGTRACKER Link for reporting issue.
      See Also:

    • UI_CENTRAL_LOGGING_URL

      public static final String UI_CENTRAL_LOGGING_URL
      Storm UI Central Logging URL.
      See Also:

    • UI_PAGINATION

      public static final String UI_PAGINATION
      Storm UI drop-down pagination value. Set ui.pagination to be a positive integer or -1 (displays all entries). Valid values: -1, 10, 20, 25 etc.
      See Also:

    • LOGVIEWER_PORT

      public static final String LOGVIEWER_PORT
      HTTP UI port for log viewer.
      See Also:

    • LOGVIEWER_CHILDOPTS

      public static final String LOGVIEWER_CHILDOPTS
      Childopts for log viewer java process.
      See Also:

    • LOGVIEWER_CLEANUP_INTERVAL_SECS

      public static final String LOGVIEWER_CLEANUP_INTERVAL_SECS
      How often to clean up old log files.
      See Also:

    • LOGVIEWER_CLEANUP_AGE_MINS

      public static final String LOGVIEWER_CLEANUP_AGE_MINS
      How many minutes since a log was last modified for the log to be considered for clean-up.
      See Also:

    • LOGVIEWER_MAX_SUM_WORKER_LOGS_SIZE_MB

      public static final String LOGVIEWER_MAX_SUM_WORKER_LOGS_SIZE_MB
      The maximum number of bytes all worker log files can take up in MB.
      See Also:

    • LOGVIEWER_MAX_PER_WORKER_LOGS_SIZE_MB

      public static final String LOGVIEWER_MAX_PER_WORKER_LOGS_SIZE_MB
      The maximum number of bytes per worker's files can take up in MB.
      See Also:

    • LOGVIEWER_HTTPS_PORT

      public static final String LOGVIEWER_HTTPS_PORT
      Storm Logviewer HTTPS port. Logviewer must use HTTPS if Storm UI is using HTTPS.
      See Also:

    • LOGVIEWER_HTTPS_KEYSTORE_PATH

      public static final String LOGVIEWER_HTTPS_KEYSTORE_PATH
      Path to the keystore containing the certs used by Storm Logviewer for HTTPS communications.
      See Also:

    • LOGVIEWER_HTTPS_KEYSTORE_PASSWORD

      public static final String LOGVIEWER_HTTPS_KEYSTORE_PASSWORD
      Password for the keystore for HTTPS for Storm Logviewer.
      See Also:

    • LOGVIEWER_HTTPS_KEYSTORE_TYPE

      public static final String LOGVIEWER_HTTPS_KEYSTORE_TYPE
      Type of the keystore for HTTPS for Storm Logviewer. see http://docs.oracle.com/javase/8/docs/api/java/security/KeyStore.html for more details.
      See Also:

    • LOGVIEWER_HTTPS_KEY_PASSWORD

      public static final String LOGVIEWER_HTTPS_KEY_PASSWORD
      Password to the private key in the keystore for setting up HTTPS (SSL).
      See Also:

    • LOGVIEWER_HTTPS_TRUSTSTORE_PATH

      public static final String LOGVIEWER_HTTPS_TRUSTSTORE_PATH
      Path to the truststore containing the certs used by Storm Logviewer for HTTPS communications.
      See Also:

    • LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD

      public static final String LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD
      Password for the truststore for HTTPS for Storm Logviewer.
      See Also:

    • LOGVIEWER_HTTPS_TRUSTSTORE_TYPE

      public static final String LOGVIEWER_HTTPS_TRUSTSTORE_TYPE
      Type of the truststore for HTTPS for Storm Logviewer. see http://docs.oracle.com/javase/8/docs/api/java/security/Truststore.html for more details.
      See Also:

    • LOGVIEWER_HTTPS_WANT_CLIENT_AUTH

      public static final String LOGVIEWER_HTTPS_WANT_CLIENT_AUTH
      Password to the truststore used by Storm Logviewer setting up HTTPS (SSL).
      See Also:

    • LOGVIEWER_HTTPS_NEED_CLIENT_AUTH

      public static final String LOGVIEWER_HTTPS_NEED_CLIENT_AUTH
      See Also:

    • LOGVIEWER_HTTPS_ENABLE_SSL_RELOAD

      public static final String LOGVIEWER_HTTPS_ENABLE_SSL_RELOAD
      If set to true, keystore and truststore for Logviewer will be automatically reloaded when modified.
      See Also:

    • LOGS_USERS

      public static final String LOGS_USERS
      A list of users allowed to view logs via the Log Viewer.
      See Also:

    • LOGS_GROUPS

      public static final String LOGS_GROUPS
      A list of groups allowed to view logs via the Log Viewer.
      See Also:

    • LOGVIEWER_APPENDER_NAME

      public static final String LOGVIEWER_APPENDER_NAME
      Appender name used by log viewer to determine log directory.
      See Also:

    • LOGVIEWER_FILTER

      public static final String LOGVIEWER_FILTER
      A class implementing javax.servlet.Filter for authenticating/filtering Logviewer requests.
      See Also:

    • LOGVIEWER_FILTER_PARAMS

      public static final String LOGVIEWER_FILTER_PARAMS
      Initialization parameters for the javax.servlet.Filter for Logviewer.
      See Also:

    • UI_CHILDOPTS

      public static final String UI_CHILDOPTS
      Childopts for Storm UI Java process.
      See Also:

    • UI_FILTER

      public static final String UI_FILTER
      A class implementing javax.servlet.Filter for authenticating/filtering UI requests.
      See Also:

    • UI_FILTER_PARAMS

      public static final String UI_FILTER_PARAMS
      Initialization parameters for the javax.servlet.Filter for UI.
      See Also:

    • UI_HEADER_BUFFER_BYTES

      public static final String UI_HEADER_BUFFER_BYTES
      The size of the header buffer for the UI in bytes.
      See Also:

    • UI_HTTPS_PORT

      public static final String UI_HTTPS_PORT
      This port is used by Storm UI for receiving HTTPS (SSL) requests from clients.
      See Also:

    • UI_HTTPS_KEYSTORE_PATH

      public static final String UI_HTTPS_KEYSTORE_PATH
      Path to the keystore used by Storm UI for setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_KEYSTORE_PASSWORD

      public static final String UI_HTTPS_KEYSTORE_PASSWORD
      Password to the keystore used by Storm UI for setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_KEYSTORE_TYPE

      public static final String UI_HTTPS_KEYSTORE_TYPE
      Type of keystore used by Storm UI for setting up HTTPS (SSL). see http://docs.oracle.com/javase/7/docs/api/java/security/KeyStore .html for more details.
      See Also:

    • UI_HTTPS_KEY_PASSWORD

      public static final String UI_HTTPS_KEY_PASSWORD
      Password to the private key in the keystore for setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_TRUSTSTORE_PATH

      public static final String UI_HTTPS_TRUSTSTORE_PATH
      Path to the truststore used by Storm UI setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_TRUSTSTORE_PASSWORD

      public static final String UI_HTTPS_TRUSTSTORE_PASSWORD
      Password to the truststore used by Storm UI setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_TRUSTSTORE_TYPE

      public static final String UI_HTTPS_TRUSTSTORE_TYPE
      Type of truststore used by Storm UI for setting up HTTPS (SSL). see http://docs.oracle .com/javase/7/docs/api/java/security/KeyStore.html for more details.
      See Also:

    • UI_HTTPS_WANT_CLIENT_AUTH

      public static final String UI_HTTPS_WANT_CLIENT_AUTH
      Password to the truststore used by Storm DRPC setting up HTTPS (SSL).
      See Also:

    • UI_HTTPS_NEED_CLIENT_AUTH

      public static final String UI_HTTPS_NEED_CLIENT_AUTH
      See Also:

    • UI_HTTPS_ENABLE_SSL_RELOAD

      public static final String UI_HTTPS_ENABLE_SSL_RELOAD
      If set to true, keystore and truststore for UI will be automatically reloaded when modified.
      See Also:

    • PACEMAKER_MAX_THREADS

      public static final String PACEMAKER_MAX_THREADS
      The maximum number of threads that should be used by the Pacemaker. When Pacemaker gets loaded it will spawn new threads, up to this many total, to handle the load.
      See Also:

    • PACEMAKER_CHILDOPTS

      public static final String PACEMAKER_CHILDOPTS
      This parameter is used by the storm-deploy project to configure the jvm options for the pacemaker daemon.
      See Also:

    • DRPC_HTTP_PORT

      public static final String DRPC_HTTP_PORT
      This port is used by Storm DRPC for receiving HTTP DPRC requests from clients.
      See Also:

    • DRPC_HTTPS_PORT

      public static final String DRPC_HTTPS_PORT
      This port is used by Storm DRPC for receiving HTTPS (SSL) DPRC requests from clients.
      See Also:

    • DRPC_HTTPS_KEYSTORE_PATH

      public static final String DRPC_HTTPS_KEYSTORE_PATH
      Path to the keystore used by Storm DRPC for setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_KEYSTORE_PASSWORD

      public static final String DRPC_HTTPS_KEYSTORE_PASSWORD
      Password to the keystore used by Storm DRPC for setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_KEYSTORE_TYPE

      public static final String DRPC_HTTPS_KEYSTORE_TYPE
      Type of keystore used by Storm DRPC for setting up HTTPS (SSL). see http://docs.oracle .com/javase/7/docs/api/java/security/KeyStore.html for more details.
      See Also:

    • DRPC_HTTPS_KEY_PASSWORD

      public static final String DRPC_HTTPS_KEY_PASSWORD
      Password to the private key in the keystore for setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_TRUSTSTORE_PATH

      public static final String DRPC_HTTPS_TRUSTSTORE_PATH
      Path to the truststore used by Storm DRPC setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_TRUSTSTORE_PASSWORD

      public static final String DRPC_HTTPS_TRUSTSTORE_PASSWORD
      Password to the truststore used by Storm DRPC setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_TRUSTSTORE_TYPE

      public static final String DRPC_HTTPS_TRUSTSTORE_TYPE
      Type of truststore used by Storm DRPC for setting up HTTPS (SSL). see http://docs.oracle .com/javase/7/docs/api/java/security/KeyStore.html for more details.
      See Also:

    • DRPC_HTTPS_WANT_CLIENT_AUTH

      public static final String DRPC_HTTPS_WANT_CLIENT_AUTH
      Password to the truststore used by Storm DRPC setting up HTTPS (SSL).
      See Also:

    • DRPC_HTTPS_NEED_CLIENT_AUTH

      public static final String DRPC_HTTPS_NEED_CLIENT_AUTH
      See Also:

    • DRPC_HTTPS_ENABLE_SSL_RELOAD

      public static final String DRPC_HTTPS_ENABLE_SSL_RELOAD
      If set to true, keystore and truststore for DRPC Server will be automatically reloaded when modified.
      See Also:

    • DRPC_AUTHORIZER

      public static final String DRPC_AUTHORIZER
      Class name for authorization plugin for DRPC client.
      See Also:

    • DRPC_REQUEST_TIMEOUT_SECS

      public static final String DRPC_REQUEST_TIMEOUT_SECS
      The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also timeout based on the socket timeout on the DRPC client, and separately based on the topology message timeout for the topology implementing the DRPC function.
      See Also:

    • DRPC_CHILDOPTS

      public static final String DRPC_CHILDOPTS
      Childopts for Storm DRPC Java process.
      See Also:

    • SUPERVISOR_SCHEDULER_META

      public static final String SUPERVISOR_SCHEDULER_META
      the metadata configured on the supervisor.
      See Also:

    • SUPERVISOR_SLOTS_PORTS

      public static final String SUPERVISOR_SLOTS_PORTS
      A list of ports that can run workers on this supervisor. Each worker uses one port, and the supervisor will only run one worker per port. Use this configuration to tune how many workers run on each machine.
      See Also:

    • SUPERVISOR_BLOBSTORE

      public static final String SUPERVISOR_BLOBSTORE
      What blobstore implementation the supervisor should use.
      See Also:

    • SUPERVISOR_LOCALIZER_CACHE_TARGET_SIZE_MB

      public static final String SUPERVISOR_LOCALIZER_CACHE_TARGET_SIZE_MB
      The distributed cache target size in MB. This is a soft limit to the size of the distributed cache contents.
      See Also:

    • SUPERVISOR_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS

      public static final String SUPERVISOR_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS
      The distributed cache cleanup interval. Controls how often it scans to attempt to cleanup anything over the cache target size.
      See Also:

    • SUPERVISOR_LOCALIZER_UPDATE_BLOB_INTERVAL_SECS

      public static final String SUPERVISOR_LOCALIZER_UPDATE_BLOB_INTERVAL_SECS
      The distributed cache interval for checking for blobs to update.
      See Also:

    • SUPERVISOR_BLOBSTORE_DOWNLOAD_THREAD_COUNT

      public static final String SUPERVISOR_BLOBSTORE_DOWNLOAD_THREAD_COUNT
      What blobstore download parallelism the supervisor should use.
      See Also:

    • SUPERVISOR_BLOBSTORE_DOWNLOAD_MAX_RETRIES

      public static final String SUPERVISOR_BLOBSTORE_DOWNLOAD_MAX_RETRIES
      Maximum number of retries a supervisor is allowed to make for downloading a blob.
      See Also:

    • SUPERVISOR_NUMA_META

      public static final String SUPERVISOR_NUMA_META
      A map with keys mapped to each NUMA Node on the supervisor that will be used by scheduler. CPUs, memory and ports available on each NUMA node will be provided. Each supervisor will have different map of NUMAs. Example: "supervisor.numa.meta": { "0": { "numa.memory.mb": 122880, "numa.cores": [ 0, 12, 1, 13, 2, 14, 3, 15, 4, 16, 5, 17], "numa.ports": [6700, 6701]}, "1" : {"numa.memory.mb": 122880, "numa.cores": [ 6, 18, 7, 19, 8, 20, 9, 21, 10, 22, 11, 23], "numa.ports": [6702, 6703], "numa.generic.resources.map": {"gpu.count" : 1}} }
      See Also:

    • NIMBUS_BLOBSTORE

      public static final String NIMBUS_BLOBSTORE
      What blobstore implementation nimbus should use.
      See Also:

    • NIMBUS_BLOBSTORE_EXPIRATION_SECS

      public static final String NIMBUS_BLOBSTORE_EXPIRATION_SECS
      During operations with the blob store, via master, how long a connection is idle before nimbus considers it dead and drops the session and any associated connections.
      See Also:

    • NIMBUS_SLOTS_PER_TOPOLOGY

      public static final String NIMBUS_SLOTS_PER_TOPOLOGY
      A number representing the maximum number of workers any single topology can acquire. This will be ignored if the Resource Aware Scheduler is used.
      See Also:

    • DRPC_HTTP_FILTER

      public static final String DRPC_HTTP_FILTER
      A class implementing javax.servlet.Filter for DRPC HTTP requests.
      See Also:

    • DRPC_HTTP_FILTER_PARAMS

      public static final String DRPC_HTTP_FILTER_PARAMS
      Initialization parameters for the javax.servlet.Filter of the DRPC HTTP service.
      See Also:

    • NIMBUS_EXECUTORS_PER_TOPOLOGY

      public static final String NIMBUS_EXECUTORS_PER_TOPOLOGY
      A number representing the maximum number of executors any single topology can acquire.
      See Also:

    • SUPERVISOR_CHILDOPTS

      public static final String SUPERVISOR_CHILDOPTS
      This parameter is used by the storm-deploy project to configure the jvm options for the supervisor daemon.
      See Also:

    • SUPERVISOR_WORKER_START_TIMEOUT_SECS

      public static final String SUPERVISOR_WORKER_START_TIMEOUT_SECS
      How long a worker can go without heartbeating during the initial launch before the supervisor tries to restart the worker process. This value override supervisor.worker.timeout.secs during launch because there is additional overhead to starting and configuring the JVM on launch. Can be exceeded when Config.TOPOLOGY_WORKER_TIMEOUT_SECS is set.
      See Also:

    • SUPERVISOR_ENABLE

      public static final String SUPERVISOR_ENABLE
      Whether or not the supervisor should launch workers assigned to it. Defaults to true -- and you should probably never change this value. This configuration is used in the Storm unit tests.
      See Also:

    • SUPERVISOR_HEARTBEAT_FREQUENCY_SECS

      public static final String SUPERVISOR_HEARTBEAT_FREQUENCY_SECS
      how often the supervisor sends a heartbeat to the master.
      See Also:

    • SUPERVISOR_MONITOR_FREQUENCY_SECS

      public static final String SUPERVISOR_MONITOR_FREQUENCY_SECS
      How often the supervisor checks the worker heartbeats to see if any of them need to be restarted.
      See Also:

    • WORKER_PROFILER_CHILDOPTS

      public static final String WORKER_PROFILER_CHILDOPTS
      The jvm profiler opts provided to workers launched by this supervisor.
      See Also:

    • WORKER_PROFILER_ENABLED

      public static final String WORKER_PROFILER_ENABLED
      Enable profiling of worker JVMs using Oracle's Java Flight Recorder. Unlocking commercial features requires a special license from Oracle. See http://www.oracle.com/technetwork/java/javase/terms/products/index.html
      See Also:

    • WORKER_PROFILER_COMMAND

      public static final String WORKER_PROFILER_COMMAND
      The command launched supervisor with worker arguments pid, action and [target_directory] Where action is - start profile, stop profile, jstack, heapdump and kill against pid.
      See Also:

    • STORM_CLUSTER_METRICS_CONSUMER_REGISTER

      public static final String STORM_CLUSTER_METRICS_CONSUMER_REGISTER
      A list of classes implementing IClusterMetricsConsumer (See storm.yaml.example for exact config format). Each listed class will be routed cluster related metrics data. Each listed class maps 1:1 to a ClusterMetricsConsumerExecutor and they're executed in Nimbus. Only consumers which run in leader Nimbus receives metrics data.
      See Also:

    • STORM_CLUSTER_METRICS_CONSUMER_PUBLISH_INTERVAL_SECS

      public static final String STORM_CLUSTER_METRICS_CONSUMER_PUBLISH_INTERVAL_SECS
      How often cluster metrics data is published to metrics consumer.
      See Also:

    • STORM_TOPOLOGY_CLASSPATH_BEGINNING_ENABLED

      public static final String STORM_TOPOLOGY_CLASSPATH_BEGINNING_ENABLED
      Enables user-first classpath. See topology.classpath.beginning.
      See Also:

    • JAVA_LIBRARY_PATH

      public static final String JAVA_LIBRARY_PATH
      This value is passed to spawned JVMs (e.g., Nimbus, Supervisor, and Workers) for the java.library.path value. java.library.path tells the JVM where to look for native libraries. It is necessary to set this config correctly since Storm uses the ZeroMQ and JZMQ native libs.
      See Also:

    • DEV_ZOOKEEPER_PATH

      public static final String DEV_ZOOKEEPER_PATH
      The path to use as the zookeeper dir when running a zookeeper server via "storm dev-zookeeper". This zookeeper instance is only intended for development; it is not a production grade zookeeper setup.
      See Also:

    • ISOLATION_SCHEDULER_MACHINES

      public static final String ISOLATION_SCHEDULER_MACHINES
      A map from topology name to the number of machines that should be dedicated for that topology. Set storm.scheduler to org.apache.storm.scheduler.IsolationScheduler to make use of the isolation scheduler.
      See Also:

    • SCHEDULER_CONFIG_CACHE_EXPIRATION_SECS

      public static final String SCHEDULER_CONFIG_CACHE_EXPIRATION_SECS
      How long before a scheduler considers its config cache expired.
      See Also:

    • SCHEDULER_CONFIG_LOADER_URI

      public static final String SCHEDULER_CONFIG_LOADER_URI
      For ArtifactoryConfigLoader, this can either be a reference to an individual file in Artifactory or to a directory. If it is a directory, the file with the largest lexographic name will be returned. Users need to add "artifactory+" to the beginning of the real URI to use ArtifactoryConfigLoader. For FileConfigLoader, this is the URI pointing to a file.
      See Also:

    • SCHEDULER_CONFIG_LOADER_POLLTIME_SECS

      public static final String SCHEDULER_CONFIG_LOADER_POLLTIME_SECS
      It is the frequency at which the plugin will call out to artifactory instead of returning the most recently cached result. Currently it's only used in ArtifactoryConfigLoader.
      See Also:

    • SCHEDULER_CONFIG_LOADER_TIMEOUT_SECS

      public static final String SCHEDULER_CONFIG_LOADER_TIMEOUT_SECS
      It is the amount of time an http connection to the artifactory server will wait before timing out. Currently it's only used in ArtifactoryConfigLoader.
      See Also:

    • SCHEDULER_CONFIG_LOADER_ARTIFACTORY_BASE_DIRECTORY

      public static final String SCHEDULER_CONFIG_LOADER_ARTIFACTORY_BASE_DIRECTORY
      It is the part of the uri, configurable in Artifactory, which represents the top of the directory tree. It's only used in ArtifactoryConfigLoader.
      See Also:

    • MULTITENANT_SCHEDULER_USER_POOLS

      public static final String MULTITENANT_SCHEDULER_USER_POOLS
      A map from the user name to the number of machines that should that user is allowed to use. Set storm.scheduler to org.apache.storm.scheduler.multitenant.MultitenantScheduler
      See Also:

    • RESOURCE_AWARE_SCHEDULER_USER_POOLS

      public static final String RESOURCE_AWARE_SCHEDULER_USER_POOLS
      A map of users to another map of the resource guarantees of the user. Used by Resource Aware Scheduler to ensure per user resource guarantees.
      See Also:

    • RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY

      public static final String RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY
      the class that specifies the scheduling priority strategy to use in ResourceAwareScheduler.
      See Also:

    • RESOURCE_AWARE_SCHEDULER_MAX_TOPOLOGY_SCHEDULING_ATTEMPTS

      public static final String RESOURCE_AWARE_SCHEDULER_MAX_TOPOLOGY_SCHEDULING_ATTEMPTS
      The maximum number of times that the RAS will attempt to schedule a topology. The default is 5.
      See Also:

    • NIMBUS_CODE_SYNC_FREQ_SECS

      public static final String NIMBUS_CODE_SYNC_FREQ_SECS
      How often nimbus's background thread to sync code for missing topologies should run.
      See Also:

    • STORM_RESOURCE_ISOLATION_PLUGIN

      public static final String STORM_RESOURCE_ISOLATION_PLUGIN
      The plugin to be used for resource isolation.
      See Also:

    • STORM_CGROUP_RESOURCES

      public static final String STORM_CGROUP_RESOURCES
      resources to to be controlled by cgroups.
      See Also:

    • STORM_CGROUP_HIERARCHY_NAME

      public static final String STORM_CGROUP_HIERARCHY_NAME
      name for the cgroup hierarchy.
      See Also:

    • STORM_RESOURCE_ISOLATION_PLUGIN_ENABLE

      public static final String STORM_RESOURCE_ISOLATION_PLUGIN_ENABLE
      flag to determine whether to use a resource isolation plugin Also determines whether the unit tests for cgroup runs. If storm.resource.isolation.plugin.enable is set to false the unit tests for cgroups will not run
      See Also:

    • STORM_METRIC_STORE_CLASS

      public static final String STORM_METRIC_STORE_CLASS
      Class implementing MetricStore. Runs on Nimbus.
      See Also:

    • STORM_METRIC_PROCESSOR_CLASS

      public static final String STORM_METRIC_PROCESSOR_CLASS
      Class implementing WorkerMetricsProcessor. Runs on Supervisors.
      See Also:

    • STORM_ROCKSDB_LOCATION

      public static final String STORM_ROCKSDB_LOCATION
      RocksDB file location. This setting is specific to the org.apache.storm.metricstore.rocksdb.RocksDbStore implementation for the storm.metricstore.class.
      See Also:

    • STORM_ROCKSDB_CREATE_IF_MISSING

      public static final String STORM_ROCKSDB_CREATE_IF_MISSING
      RocksDB create if missing flag. This setting is specific to the org.apache.storm.metricstore.rocksdb.RocksDbStore implementation for the storm.metricstore.class.
      See Also:

    • STORM_ROCKSDB_METADATA_STRING_CACHE_CAPACITY

      public static final String STORM_ROCKSDB_METADATA_STRING_CACHE_CAPACITY
      RocksDB metadata cache capacity. This setting is specific to the org.apache.storm.metricstore.rocksdb.RocksDbStore implementation for the storm.metricstore.class.
      See Also:

    • STORM_ROCKSDB_METRIC_RETENTION_HOURS

      public static final String STORM_ROCKSDB_METRIC_RETENTION_HOURS
      RocksDB setting for length of metric retention. This setting is specific to the org.apache.storm.metricstore.rocksdb.RocksDbStore implementation for the storm.metricstore.class.
      See Also:

    • STORM_ROCKSDB_METRIC_DELETION_PERIOD_HOURS

      public static final String STORM_ROCKSDB_METRIC_DELETION_PERIOD_HOURS
      RocksDB setting for period of metric deletion thread. This setting is specific to the org.apache.storm.metricstore.rocksdb .RocksDbStore implementation for the storm.metricstore.class.
      See Also:

    • STORM_NIMBUS_ZOOKEEPER_ACLS_CHECK

      public static final String STORM_NIMBUS_ZOOKEEPER_ACLS_CHECK
      In nimbus on startup check if all of the zookeeper ACLs are correct before starting. If not don't start nimbus.
      See Also:

    • STORM_NIMBUS_ZOOKEEPER_ACLS_FIXUP

      public static final String STORM_NIMBUS_ZOOKEEPER_ACLS_FIXUP
      In nimbus on startup check if all of the zookeeper ACLs are correct before starting. If not do your best to fix them before nimbus starts, if it cannot fix them nimbus will not start. This overrides any value set for storm.nimbus.zookeeper.acls.check.
      See Also:

    • VALIDATE_TOPOLOGY_SCHEDULER_STRATEGY

      public static final String VALIDATE_TOPOLOGY_SCHEDULER_STRATEGY
      Server side validation that @{see Config#TOPOLOGY_SCHEDULER_STRATEGY} is set ot a subclass of IStrategy.
      See Also:

    • UI_HTTP_CREDS_PLUGIN

      public static final String UI_HTTP_CREDS_PLUGIN
      Class name of the HTTP credentials plugin for the UI.
      See Also:

    • DRPC_HTTP_CREDS_PLUGIN

      public static final String DRPC_HTTP_CREDS_PLUGIN
      Class name of the HTTP credentials plugin for DRPC.
      See Also:

    • STORM_SUPERVISOR_CGROUP_ROOTDIR

      public static String STORM_SUPERVISOR_CGROUP_ROOTDIR
      root directory for cgoups.

    • STORM_WORKER_CGROUP_MEMORY_MB_LIMIT

      public static String STORM_WORKER_CGROUP_MEMORY_MB_LIMIT
      the manually set memory limit (in MB) for each CGroup on supervisor node.

    • STORM_WORKER_CGROUP_CPU_LIMIT

      public static String STORM_WORKER_CGROUP_CPU_LIMIT
      the manually set cpu share for each CGroup on supervisor node.

    • STORM_CGROUP_CGEXEC_CMD

      public static String STORM_CGROUP_CGEXEC_CMD
      full path to cgexec command.

    • STORM_CGROUP_MEMORY_LIMIT_TOLERANCE_MARGIN_MB

      public static String STORM_CGROUP_MEMORY_LIMIT_TOLERANCE_MARGIN_MB
      Please use STORM_SUPERVISOR_MEMORY_LIMIT_TOLERANCE_MARGIN_MB instead. The amount of memory a worker can exceed its allocation before cgroup will kill it.

    • STORM_CGROUP_INHERIT_CPUSET_CONFIGS

      public static String STORM_CGROUP_INHERIT_CPUSET_CONFIGS
      To determine whether or not to cgroups should inherit cpuset.cpus and cpuset.mems config values form parent cgroup Note that cpuset.cpus and cpuset.mems configs in a cgroup must be initialized (i.e. contain a valid value) prior to being able to launch processes in that cgroup. The common use case for this config is when the linux distribution that is used does not support the cgroup.clone_children config.

    • STORM_CGROUP_MEMORY_ENFORCEMENT_ENABLE

      public static String STORM_CGROUP_MEMORY_ENFORCEMENT_ENABLE
      Java does not always play nicely with cgroups. It is coming but not fully implemented and not for the way storm uses cgroups. In the short term you can disable the hard memory enforcement by cgroups and let the supervisor handle shooting workers going over their limit in a kinder way.

    • STORM_SUPERVISOR_MEMORY_LIMIT_TOLERANCE_MARGIN_MB

      public static String STORM_SUPERVISOR_MEMORY_LIMIT_TOLERANCE_MARGIN_MB
      Memory given to each worker for free (because java and storm have some overhead). This is memory on the box that the workers can use. This should not be included in SUPERVISOR_MEMORY_CAPACITY_MB, as nimbus does not use this memory for scheduling.

    • STORM_SUPERVISOR_HARD_MEMORY_LIMIT_MULTIPLIER

      public static String STORM_SUPERVISOR_HARD_MEMORY_LIMIT_MULTIPLIER
      A multiplier for the memory limit of a worker that will have the supervisor shoot it immediately. 1.0 means shoot the worker as soon as it goes over. 2.0 means shoot the worker if its usage is double what was requested. This value is combined with STORM_SUPERVISOR_HARD_MEMORY_LIMIT_OVERAGE and which ever is greater is used for enforcement. This allows small workers to not be shot.

    • STORM_SUPERVISOR_HARD_LIMIT_MEMORY_OVERAGE_MB

      public static String STORM_SUPERVISOR_HARD_LIMIT_MEMORY_OVERAGE_MB
      If the memory usage of a worker goes over its limit by this value is it shot immediately. This value is combined with STORM_SUPERVISOR_HARD_LIMIT_MEMORY_MULTIPLIER and which ever is greater is used for enforcement. This allows small workers to not be shot.

    • STORM_SUPERVISOR_LOW_MEMORY_THRESHOLD_MB

      public static String STORM_SUPERVISOR_LOW_MEMORY_THRESHOLD_MB
      If the amount of memory that is free in the system (either on the box or in the supervisor's cgroup) is below this number (in MB) consider the system to be in low memory mode and start shooting workers if they are over their limit.

    • STORM_SUPERVISOR_MEDIUM_MEMORY_THRESHOLD_MB

      public static String STORM_SUPERVISOR_MEDIUM_MEMORY_THRESHOLD_MB
      If the amount of memory that is free in the system (either on the box or in the supervisor's cgroup) is below this number (in MB) consider the system to be a little low on memory and start shooting workers if they are over their limit for a given grace period STORM_SUPERVISOR_MEDIUM_MEMORY_GRACE_PERIOD_MS.

    • STORM_SUPERVISOR_MEDIUM_MEMORY_GRACE_PERIOD_MS

      public static String STORM_SUPERVISOR_MEDIUM_MEMORY_GRACE_PERIOD_MS
      The number of milliseconds that a worker is allowed to be over their limit when there is a medium amount of memory free in the system.

    • STORM_WORKER_MIN_CPU_PCORE_PERCENT

      public static String STORM_WORKER_MIN_CPU_PCORE_PERCENT
      The config indicates the minimum percentage of cpu for a core that a worker will use. Assuming the a core value to be 100, a value of 10 indicates 10% of the core. The P in PCORE represents the term "physical". A default value will be set for this config if user does not override.

      Workers in containers or cgroups may require a minimum amount of CPU in order to launch within the supervisor timeout. This setting allows configuring this to occur.

    • STORM_WORKER_TOKEN_LIFE_TIME_HOURS

      public static String STORM_WORKER_TOKEN_LIFE_TIME_HOURS
      The number of hours a worker token is valid for. This also sets how frequently worker tokens will be renewed.

    • STORM_OCI_NSCD_DIR

      public static String STORM_OCI_NSCD_DIR
      The directory of nscd - name service cache daemon, e.g. "/var/run/nscd/". nscd must be running so that profiling can work properly.

    • STORM_OCI_READONLY_BINDMOUNTS

      public static String STORM_OCI_READONLY_BINDMOUNTS
      A list of read only bind mounted directories.

    • STORM_OCI_READWRITE_BINDMOUNTS

      public static String STORM_OCI_READWRITE_BINDMOUNTS
      A list of read-write bind mounted directories.

    • STORM_OCI_CGROUP_PARENT

      public static String STORM_OCI_CGROUP_PARENT
      The cgroup root for oci container. (Also a --cgroup-parent config for docker command) Must follow the constraints of the docker command. The path will be made as absolute path if it's a relative path because we saw some weird bugs (the cgroup memory directory disappears after a while) when a relative path is used. Note that we only support cgroupfs cgroup driver because of some issues with systemd; restricting to `cgroupfs` also makes cgroup paths simple.

    • STORM_OCI_IMAGE

      public static String STORM_OCI_IMAGE
      Default oci image to use if the topology doesn't specify which oci image to use.

    • STORM_OCI_ALLOWED_IMAGES

      public static String STORM_OCI_ALLOWED_IMAGES
      A list of oci image that are allowed. A special entry of asterisk(*) means any image is allowed, but the image has to pass other checks. Storm currently assumes OCI container is not supported on the cluster if this is not configured.

    • STORM_OCI_SECCOMP_PROFILE

      public static String STORM_OCI_SECCOMP_PROFILE
      Specify the seccomp Json file to be used as a seccomp filter.

    • STORM_OCI_IMAGE_HDFS_TOPLEVEL_DIR

      public static String STORM_OCI_IMAGE_HDFS_TOPLEVEL_DIR
      The HDFS location under which the oci image manifests, layers, and configs directories exist.

    • STORM_OCI_IMAGE_TAG_TO_MANIFEST_PLUGIN

      public static final String STORM_OCI_IMAGE_TAG_TO_MANIFEST_PLUGIN
      The plugin to be used to get the image-tag to manifest mappings.
      See Also:

    • STORM_OCI_MANIFEST_TO_RESOURCES_PLUGIN

      public static final String STORM_OCI_MANIFEST_TO_RESOURCES_PLUGIN
      The plugin to be used to get oci resource according to the manifest.
      See Also:

    • STORM_OCI_RESOURCES_LOCALIZER

      public static final String STORM_OCI_RESOURCES_LOCALIZER
      The plugin to use for oci resources localization.
      See Also:

    • STORM_OCI_RESOURCES_LOCAL_DIR

      public static final String STORM_OCI_RESOURCES_LOCAL_DIR
      The local directory for localized oci resources.
      See Also:

    • STORM_OCI_LAYER_MOUNTS_TO_KEEP

      public static final String STORM_OCI_LAYER_MOUNTS_TO_KEEP
      Target count of OCI layer mounts that we should keep on disk at one time.
      See Also:

  • Constructor Details

    • DaemonConfig

      public DaemonConfig()

  • Method Details

    • getCgroupRootDir

      public static String getCgroupRootDir(Map<String,Object> conf)

    • getCgroupStormHierarchyDir

      public static String getCgroupStormHierarchyDir(Map<String,Object> conf)

    • getCgroupStormResources

      public static ArrayList<String> getCgroupStormResources(Map<String,Object> conf)
      Get the cgroup resources from the conf.
      Parameters:
      conf - the config to read
      Returns:
      the resources.

    • getCgroupStormHierarchyName

      public static String getCgroupStormHierarchyName(Map<String,Object> conf)