Amazon Kinesis is a family of services provided by Amazon Web Services (AWS) for processing and analyzing real-time streaming data at a large scale. Launched in November 2013, it offers developers the ability to build applications that can consume and process data from multiple sources simultaneously.[2] Kinesis supports multiple use cases, including real-time analytics, log and event data collection, and real-time processing of data generated by IoT devices.
| Developer(s) | Amazon Web Services |
|---|---|
| Initial release | November 2013[1] |
| Operating system | Amazon Web Services |
| Platform | Cloud computing |
| Type | Big data and Streaming data |
| License | Proprietary software |
| Website | aws |
History
editAmazon Kinesis was launched by Amazon Web Services (AWS) in November 2013 as a managed service for processing and analyzing real-time streaming data at a large scale.[3] The service was introduced to address the growing need for businesses to process and analyze data as it was generated, rather than in batches, allowing for real-time insights and decision-making.
Since its launch, the Amazon Kinesis family of services has expanded to include four main components: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.[4] Each of these components serves a specific purpose in the processing and analysis of real-time streaming data.
In August 2015, AWS announced the availability of Kinesis Data Firehose, a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, and Amazon Elasticsearch.[5] A year later in August 2016, AWS launched Kinesis Data Analytics, enabling customers to analyze streaming data in real time using standard SQL queries.[6]
AWS introduced Kinesis Video Streams, a fully managed service for securely capturing, processing, and storing video streams for analytics and machine learning applications, was introduced by AWS in November 2017.[7]
Components
editAmazon Kinesis is composed of four main services: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.[4]
Kinesis Data Streams
editKinesis Data Streams is a scalable and durable real-time data streaming service that captures and processes gigabytes of data per second from multiple sources.[8] It enables the storage and processing of data in real time, making it useful for applications that require immediate insights, such as monitoring and alerting.
Kinesis Data Firehose
editKinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon Elasticsearch, and AWS-partner data stores.[9] With Data Firehose, users can configure and scale data delivery without manual intervention.
Kinesis Data Analytics
editKinesis Data Analytics enables the analysis of streaming data in real time using standard SQL or Apache Flink.
Kinesis Video Streams
editKinesis Video Streams is a fully managed service for securely capturing, processing, and storing video streams for analytics and machine learning.[10] It supports multiple video codecs and streaming protocols, making it suitable for various use cases, such as security and surveillance, video-enabled IoT devices, and live event broadcasting.
Integration
editAmazon Kinesis can be easily integrated with other AWS services, such as AWS Lambda, Amazon S3, Amazon Redshift, and Amazon OpenSearch. This integration enables developers to build end-to-end streaming data processing applications, taking advantage of the extensive AWS ecosystem.[11]
Use cases
editSome common use cases for Amazon Kinesis include:[2]
- Real-time analytics: Analyzing streaming data in real time to provide immediate insights and make data-driven decisions.
- Log and event data collection: Collecting, processing, and analyzing log and event data generated by applications, infrastructure, and devices.[12]
- IoT data processing: Processing and analyzing large volumes of data generated by IoT devices in real time.[13]
- Machine learning: Ingesting and processing video streams for machine learning applications, such as object recognition, facial recognition, and sentiment analysis.
Pricing
editAmazon Kinesis follows a pay-as-you-go pricing model, with costs depending on the chosen service, data volume, and processing power required.[14] AWS provides a free tier for Kinesis Data Streams and Kinesis Data Firehose, allowing users to get started with the services at no cost.[15]
See also
edit- Apache Kafka
- Google Cloud Pub/Sub
- Microsoft Azure Event Hubs
- Stream processing
References
edit- ^ Jeff Barr (2013-11-14). "Amazon Kinesis - Real-Time Stream Processing". Retrieved 2023-03-24.
- ^ a b Makota, Tarik; Maguire, Brian; Gagne, Danny; Chakrabarti, Rajeev (2021-03-31). Scalable Data Streaming with Amazon Kinesis: Design and secure highly available, cost-effective data streaming applications with Amazon Kinesis. Packt Publishing Ltd. ISBN 978-1-80056-433-6.
- ^ Jeff Barr (2013-11-14). "Amazon Kinesis - Real-Time Stream Processing". Retrieved 2023-03-24.
- ^ a b "Amazon Kinesis". Amazon Web Services. Retrieved 2023-03-24.
- ^ Jeff Barr (2015-08-05). "Amazon Kinesis Firehose – Simple & Highly Scalable Data Ingestion". Retrieved 2023-03-24.
- ^ Jeff Barr (2016-08-11). "Amazon Kinesis Analytics – Process Streaming Data in Real Time with SQL". Retrieved 2023-03-24.
- ^ Jeff Barr (2017-11-27). "Amazon Kinesis Video Streams – Serverless Video Ingestion and Storage for Vision-Enabled Apps". Retrieved 2023-03-24.
- ^ "AWS Launches Amazon Kinesis Data Streams On-Demand". InfoQ. Retrieved 2023-03-25.
- ^ "Amazon Releases Kinesis Firehose". InfoQ. Retrieved 2023-03-25.
- ^ "Amazon Kinesis Video Streams". Amazon Web Services. Retrieved 2023-03-24.
- ^ "Amazon Kinesis Integrations". Amazon Web Services. Retrieved 2023-03-24.
- ^ Srivastava, Mayank; Yadav, Pradduman (2021-10-22). "Build a Log Analytic Solution on AWS". 2021 5th International Conference on Information Systems and Computer Networks (ISCON). pp. 1–5. doi:10.1109/ISCON52037.2021.9702374. ISBN 978-1-6654-0341-2. S2CID 246870198.
- ^ Quadri, Nasreen Sultana; Yadav, Kusum (2018-04-25). "Efficient Data Classification for IoT Devices using AWS Kinesis Platform". 2018 21st Saudi Computer Society National Computer Conference (NCC). pp. 1–5. doi:10.1109/NCG.2018.8593105. ISBN 978-1-5386-4110-1. S2CID 57364493.
- ^ "Amazon Kinesis Pricing". Amazon Web Services. Retrieved 2023-03-24.
- ^ "AWS Free Tier". Amazon Web Services. Retrieved 2023-03-24.