Apache Storm Course

Apache Storm is a real-time distributed framework for real-time processing of Big Data. It allows you to perform real-time analytics of a wide variety of streamed data. Apache Storm is written using Java and Clojure. It is designed to process vast amounts of data in a fault-tolerant and horizontally scalable method. Apache Storm continues to be the leader in real-time analytics. It is easy to install, set up and operate. Apache Storm is used in many fields to deal with large data.

Benefits of Apache Storm

Here is a list of all the benefits of Apache Storm:

• Apache Storm is open-source, highly scalable and user-friendly
• It is fault-tolerant and flexible
• Apache Storm is reliable and it can support any programming language
• Apache Storm allows real-time stream processing
• It is very quick and processes the data very fast that is millions of tuples per second per node.
• Storm provides guaranteed data processing even if any of the nodes are lost in the cluster messages
• Storm can be used with any programming language and it is fun to use Storm

Use Cases of Apache Storm

Apache Storm is suitable in various use cases and a few are listed below:

• Stream Processing
• Continuous Computation
• Distributed RPC
• Real-Time Analytics
• Online Machine Learning

Course Objectives

At the end of this course you will be able to:

• Master the fundamentals and architecture of Apache Storm
• Understand where to Apache storm for real-time analytics
• Setup Apache storm cluster on your computer
• Understand the basics of storm interfaces with Java and others
• Learn about storm technology stack and groupings
• Implement Spouts and Bolts
• Work on projects using Apache Strom

Apache Storm Course Description:

Section 1: History

Apache Storm was originally created by Nathan Marz while he was working at BackType. Nathan discovered Storm because of the cumbersome and brittle system of distributed queues and workers faced in another real-time component. Storm was the first to introduce the concept of stream which is a fault tolerant and reliable model. Storm is now acquired and open-sourced by Twitter. In a very short period of time, Apache Storm has become a more popular and leading real-time processing system. This chapter contains a brief introduction to Apache and explains the history of Apache Storm.

Section 2: Features

Apache Storm has a lot of good features than other real-time processing systems. The below-mentioned are the top features of Apache Storm

• Simple programming model – Topology, Spouts, Bolts
• Programming language agnostic – Clojure, Java, Ruby, Python
• Fault-tolerant
• Horizontally scalable
• Guaranteed message processing
• Very fast
• Local mode

The architecture of Apache Storm

One of the main advantages of Apache Storm is its fault-tolerant and no single point of failure. In this chapter, a pictorial representation of the architecture of Apache Storm is given for easy understanding of the cluster design of the storm. Apache Storm has two types of nodes, the Nimbus which is the Master node and the Supervisor which is the Worker node. The goal of Nimbus is to run the storm topology and the work of the supervisor is to delegate the tasks to the worker processes. 

The below-mentioned components are explained in detail in this chapter:

• Nimbus
• Supervisor
• Worker Process
• Executor
• Task
• Zookeeper framework

Architecture Explanation in Detail:

Storm has an architecture that helps it to process real-time data in the best possible and quickest way. It contains a monitoring tool called monit which helps the process to restart if there is any failure. Storm has an advanced topology called Trident Topology which provides a high-level API like Pig. All these features are discussed in detail in this chapter.

Topology

Topology is a graph of operators and streams. Topology is a combination of Spouts and Bolts. Topology helps to define a streaming application in Storm. The node in a topology contains processing logic. The links in topology determine how the data should be run through the nodes. The process of running a topology is straightforward. Apache Storm’s main objective is to run the topology as many times until the topology is killed.  The topology command is explained in detail in this lesson.

Topology Creation

The topology in Apache Storm is a thrift structure. Topology Builder has simple and easy methods to create topologies. Topology Builder has a Create Topology syntax to create a new topology. The code to create a topology is given in this chapter.

Trident

Apache Storm has an advanced topology called Trident Topology. Trident has functions, filters, joins, grouping and aggregation. These components are explained in detail in this chapter. 

The other topics included in this chapter are listed below:

• Trident Tuples
• Trident Spout
• Trident Operations
• State Maintenance
• Distributed RPC
• When Trident should be used
• Example of Trident
• Formatting the call information
• CSV Split
• Log Analyzer

Spouts

A topology usually starts with Spouts. These are the sources of streams in a topology that are used for data creation. Spout reads tuples from a messaging framework and transfers them to one or more bolts. A tuple is a named list of values in Apache Storm.

Spout Creation

Spout will implement an “IRichSpout” interface which has the following components

• Open – conf, context, collector
• nextTuple – contains the signature of the nextTuple method
• close – signature of the close component is mentioned under this topic
• declare-output fields – this is used to specify the output schema of the tuple
• ack – ensures that the specific tuple is processed
• fail – this method informs if there is any failure in the processing of the tuple
• Fake Call Log Reader Spout – The call log contains caller number, receiver number and duration

Bolt

Bolt is considered to be a node in a topology. Bolts help to process the input stream and produce a new stream. Bolts have the smallest processing logic. The output of one bolt can be used as input for another bolt.

Bolt Creation

Bolt is a component that takes a tuple as input, processes it and produces a new tuple or tuples as output. This implements the “IRichBolt” interface. The operations are carried out using two classes CallLog CreaterBolt and CallLog CounterBolt. The interface in bolt has the following methods.

• prepare – conf, context and collector.
• execute – this method processes a single tuple at a time. Multiple tuples can also be processed but it produces a single output tuple as the output
• cleanup – a signature of the cleanup method is given here.
• declare-output fields – the parameter declarer is used to declare output stream IDs, output fields and others.
• Call Log Creator Bolt – this receives the call log tuple and it has the caller number, receiver number and call duration. this topic gives the complete code of CallLog Creator Bolt.
• Call log counter bolt – this method receives the call and its duration as a tuple. This bolt method creates a dictionary object in the prepare method. The coding of the Call log counter bolt is given in this section.

Stream

A stream is an unordered sequence of tuples that is processed by the application. Apache storm reads raw stream data from one end and this data goes through a sequence of processing units which produces the output at the other end. Streams of data flow from spouts to bolts and from one bolt to another. The stream concept in Storm is discussed in this section with examples.

Stream Grouping

Stream grouping helps to control the route of the tuples in topology and helps to understand the workflow of the topology. There are four in-built groupings as explained in this chapter

• Shuffle Grouping – In this grouping equal number of tuples are distributed randomly to all the workers who are executing the bolts
• Field Grouping – In this grouping, the fields with the same values are grouped together. Such field values are sent to the same worker who is executing the bolts.
• Global Grouping – Under this grouping, all the streams are grouped and sent to a single bolt usually to the bolt with the lowest ID.
• All Grouping – This grouping sends a copy of each tuple to all the bolts. This is used for joint operations.

Section 3: Installation Process

Apache Storm can be installed in your system using three steps

Installation of Java – The steps of Java installation are listed below

• Download JDK
• Extract files
• Move to the opt directory
• Set path
• Java Alternatives
• Commands to check whether Java is installed

Installation of Zookeeper framework

Below are the steps to install the Zookeeper framework:

• Download Zookeeper
• Extract tar file
• Create configuration file
• Start Zookeeper Server
• Start CLI
• Stop Zookeeper Server

Installation of Apache Storm framework:

• Download Storm
• Extract tar file
• Open configuration file
• Start the nimbus
• Start the Supervisor
• Start the UI

Target Audience for this course:

This course is meant for professionals who are willing to start their career in Big data analytics using the Apache Storm framework. The others include Software professionals, Data scientists, ETL developers and Project managers.