A metric store (MetricStore
) interface was added
to Nimbus to allow storing metric information (Metric
)
to a database. The default implementation
(RocksDbStore
) is using RocksDB,
a key-value store.
As metrics are stored in RocksDB, their string values (for topology ID and executor ID, etc.) are converted to unique integer IDs, and these strings are also stored to the database as metadata indexed by the integer ID. When a metric is stored, it is also aggregated with any existing metric within the same 1, 10, and 60 minute timeframe.
The FilterOptions
class provides an interface
to select which options can be used to scan the metrics.
The following configuation options exist:
storm.metricstore.class: "org.apache.storm.metricstore.rocksdb.RocksDbStore"
storm.metricprocessor.class: "org.apache.storm.metricstore.NimbusMetricProcessor"
storm.metricstore.rocksdb.location: "storm_rocks"
storm.metricstore.rocksdb.create_if_missing: true
storm.metricstore.rocksdb.metadata_string_cache_capacity: 4000
storm.metricstore.rocksdb.retention_hours: 240
MetricStore
).WorkerMetricsProcessor
).The RocksDB Key (represented by RocksDbKey
)
fields are as follows:
Field | Size | Offset | Description |
---|---|---|---|
Type | 1 | 0 | The type maps to the KeyType enum, specifying a metric or various types of metadata strings |
Aggregation Level | 1 | 1 | The aggregation level for a metric (see AggLevel enum). Set to 0 for metadata. |
Topology Id | 4 | 2 | The metadata string Id representing a topologyId for a metric, or the unique string Id for a metadata string |
Timestamp | 8 | 6 | The timestamp for a metric, unused for metadata |
Metric Id | 4 | 14 | The metadata string Id for the metric name |
Component Id | 4 | 18 | The metadata string Id for the component Id |
Executor Id | 4 | 22 | The metadata string Id for the executor Id |
Host Id | 4 | 26 | The metadata string Id for the host Id |
Port | 4 | 30 | The port number |
Stream Id | 4 | 34 | The metadata string Id for the stream Id |
The RocksDB Value fields for metadata strings (represented by
RocksDbValue
) are as follows:
Field | Size | Offset | Description |
---|---|---|---|
Version | 1 | 0 | The current metadata version - allows migrating if the format changes in the future |
Timestamp | 8 | 1 | The time when the metadata was last used by a metric. Allows deleting of old metadata. |
Metadata String | any | 9 | The metadata string |
RocksDB Value fields for metric data are as follows:
Field | Size | Offset | Description |
---|---|---|---|
Version | 1 | 0 | The current metric version - allows migrating if the format changes in the future |
Value | 8 | 1 | The metric value |
Count | 8 | 9 | The metric count |
Min | 8 | 17 | The minimum metric value |
Max | 8 | 25 | The maximum metric value |
Sum | 8 | 33 | The sum of the metric values |