The Apache Storm community is pleased to announce that version 2.0.0 has been released and is available from the downloads page. This release represents a major milestone and accomplishment by the Apache Storm community.
Apache Storm 2.0.0 includes significant improvements in terms of performance, new features, and integration with external systems. In the coming weeks members will post a series of deep dive articles covering new features improvements. In this post we'll highlight some of the key features and changes in this release.
The full list of changes in this release can be found here.
In previous releases a large part of Apache Storm's core functionality was implemented in Clojure. Apache Storm 2.0.0 has been rearchitected with it's core functionality implemented in pure Java. The new Java-based implementation has improved performance significantly, and made Apache Storm's internal APIs more maintainable and extensible. While Apache Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Apache Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.
Apache Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Apache Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
Apache Storm 2.0.0 introduces a new typed API for expressing streaming computations more easily using functional style operations. It builds on top of the Apache Storm's core spouts and bolt APIs and automatically fuses multiple operations together to optimize the pipeline.
For more details and examples see the Stream API documentation.
Apache Storm 2.0.0's Windowing API can save/restore the window state to the configured state backend so that larger continuous windows can be supported. The window boundaries can now be accessed via the APIs.
For more details see stateful windowing documentation.
The most significant change to Apache Storm's Kafka integration since 1.x, is that storm-kafka has been removed. The module was deprecated a while back, due to Kafka's deprecation of the underlying client library. Users will have to move to the storm-kafka-client module, which uses Kafka's ´kafka-clients´ library for integration.
For the most part, the migration to storm-kafka-client is straightforward. The documentation for storm-kafka-client contains a helpful mapping between the old and new spout configurations. If you are using any of the storm-kafka spouts, you will need to migrate offset checkpoints to the new spout, to avoid the new spout starting from scratch on your partitions. Apache Storm provides a helper tool to do this which can be found here.
When performing a migration, you should stop your topology, run the migration tool, then redeploy your topology with the storm-kafka-client spout.
Storm-kafka-client in Apache Storm 1.x allowed you to use Kafka's own mechanism to manage which spout tasks were responsible for which partitions. This mechanism was a poor fit for Apache Storm, and was deprecated in 1.2.0. It has been removed entirely in 2.0.
The storm-kafka-client Subscription interface has also been removed. It offered too limited control over the subscription behavior. It has been replaced with the TopicFilter and ManualPartitioner interfaces. Unless you were using a custom Subscription implementation, this will likely not affect you. If you were using a custom Subscription, the storm-kafka-client documentation describes how to customize assignment.
With the release of 2.0.0 the 1.0.x version line will no longer be maintained. 1.0.x users are strongly encouraged to upgrade to a more recent release.
Java 7 support has been dropped, and Apache Storm 2.0.0 requires Java 8.
The storm-core artifact has been split into client and server-facing parts. Topology jars should depend on the following artifact as of Apache Storm 2.0.0:
<groupId>org.apache.storm</groupId>
<artifactId>storm-client</artifactId>
<version>2.0.0</version>
<scope>provided</scope>
Projects using LocalCluster for testing will additionally need to depend on the Apache Storm server jar:
<groupId>org.apache.storm</groupId>
<artifactId>storm-server</artifactId>
<version>2.0.0</version>
<scope>test</scope>
Keep an eye on the Apache Storm blog for additional posts by Apache Storm contributors for more in-depth discussions of new features in Apache Storm 2.0.0 including:
Special thanks are due to all those who have contributed to Apache Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated.