Skip to Content

How to monitor Amazon Kinesis

Posted on October 25, 2022 by

Categories: AWS

Tags:

Every day, the world we live in gets more interconnected. Public cloud hosts like Amazon Web Services (AWS) offer platforms with a broad range of capabilities, which scale swiftly in response to demand. As a result, there has been an explosion of new services and apps that are improving our daily lives. A crucial part of each of these systems is data. Large volumes of data can be ingested, processed, transformed, and sent.

Platforms for event streaming, like Amazon Kinesis, offer a reliable solution to manage these enormous data streams. This essay will examine Kinesis, go through its features, and describe how to successfully control it. We’ll go over how Kinesis functions and how you can use it to ensure that your organization’s data streams are effectively handled and used according to its needs for effectiveness and dependability.

How and when do you utilize Amazon Kinesis?

AWS offers a managed streaming solution called Amazon Kinesis as a service. Video streams, IoT data, and logging events from hundreds of sources are just a few types of data that Kinesis can collect, buffer, and analyze in real-time. Kinesis may send data and events to various locations, including business intelligence, data analytics, and machine learning systems.

You don’t have to worry about deploying hardware or managing changes in the amount or frequency of data because Kinesis is managed and built on top of AWS infrastructure. Numerous apps, some of which you could use daily, employ Amazon Kinesis (like Zillow, Netflix, and Lyft). Kinesis may be used whenever a company wants to combine data from many sources.

Setting up Kinesis to meet your needs.

Users may set up Kinesis to meet their pricing and performance needs, much like other managed services offered by AWS. Based on the quantity and volume of data the stream anticipates processing and how long records must be accessible for processing by the stream’s consumers, you may specify the stream’s capacity and data retention period.

Kinesis data stream creation in AWS

A Kinesis data stream comprises several shards the user specifies when they build the stream. The basic unit of capacity inside a Kinesis stream is a Kinesis shard, which supports up to a thousand PUT events per second. One megabyte of input data and two megabytes of output data can be sent and received simultaneously by a single shard. When the stream is initially constructed, the number of shards is specified, and it may be changed through a procedure called resharding.

You may buffer events using Kinesis streams, preventing downstream processors from overloading and enabling various clients to retrieve events as soon as they become available. A Kinesis stream’s default data retention duration is 24 hours, but the stream’s owner has the option to extend it for up to 365 days. Raising the data retention duration has the same financial consequences as increasing the stream’s capacity or several shards.

surveillance and observability

When establishing a new Kinesis stream in the AWS UI, you may use several calculators to determine the number of shards and related charges. Even the most precisely set-up streams require monitoring to ensure they are operating at their best. Processing delays can be caused by insufficient capacity, while financial problems can be caused by having too much capacity.

Two layers of insights are provided by AWS monitoring. The first is the metrics compilation at the stream level, which is written to CloudWatch once per minute. This comprises the metrics listed below (gathered over a specific period):

  • Records from the stream that have been read and counted.
  • The amount of data requested from the stream and received (in bytes).
  • The period (time) that records were in the stream before being read (in milliseconds).
  • The time it takes to read one record from the stream after another.
  • Counts of read and write attempts from the stream, both successful and unsuccessful.
  • Write to and read from the stream-related rate exceptions and throttling incidents.

The two metrics collection comprises shard-level metrics that are delivered to CloudWatch at one-minute intervals and can be activated for increased monitoring (for an extra fee). These metrics (which are gathered for each shard during a predetermined timeframe) consist of the following:

The volume of data is retrieved and read (in bytes).

Understanding the health and effectiveness of the stream’s setup requires understanding the capacity of the data traveling through your stream, how long it takes, and if the records indicate any throttling or exceptions throughout their interactions with the stream. Having said that, you shouldn’t spend time manually monitoring Kinesis stream metrics.

You may set alerts in CloudWatch to notify you when performance declines or your Kinesis stream fills with data. Although helpful, CloudWatch is a general tool that lacks user-friendliness and intuitiveness regarding the subtleties of Kinesis stream performance and optimization. Let’s look at how you can automate stream monitoring in Kinesis to get actionable observability of your streams’ behavior.

Using Sumo Logic and Amazon Kinesis

You may get a preconfigured dashboard that combines metrics and event information into precise, actionable data by combining the simplicity and scalability of AWS Kinesis with an analytics platform like Sumo Logic.