In this crisp article, I will show you how to design an end-to-end serverless ETL application that is event-driven and orchestrated using managed services only. TL;DR As soon a file is uploaded in an S3 bucket, an event is triggered to an SQS queue SQS queue is subscribed by a…

As Apache Flink continues to pique my interest due to its rapidly growing popularity in Big Data/Hadoop ecosystem, I toyed with its stream processing capability over the holidays. In this article, I will highlight how Flink can be used for distributed real-time stream processing of unbounded data stream using Kafka…

If you have been following big-data trends recently, you might have heard of the new kid on the block called Apache Flink. In this article, I’ll introduce you to how you can use Apache Flink to implement simple batch processing algorithm on a YARN cluster using AWS EMR. What is…

To me, error analysis is the sweet spot for improvement. — Donald Norman Preface AWS EMR and Hadoop both produce hundreds of log files that report status on the cluster. AWS EMR writes step, bootstrap action, and instance state logs. Apache Hadoop writes logs to report the processing of YARN…

Systems that Run Forever Self-heal and Scale — Joe Armstrong Preface This post complements my previous article on event-driven serverless architecture and explains how we’ve implemented monitoring and alerting using the same principles of Event sourcing — Workflow modeling using loosely coupled stateless components such that monitoring and alerting are…

This post describes the problem statement, implementation, and the learnings of how we solved an AWS limitation using AWS services. I’ll write another follow-up post to describe how we implemented CI/CD, monitoring and alerting for this system. We heavily use AWS DataPipeline to orchestrate our data-processing. All our batch jobs…