Greek Architectures of Data Processing
Three popular Data processing architecure for big and small data on cloud. These cover various scenarios for both batch, realtime, small and big data. The links take to the dedicated blog for each architecture
Data Engineering
Processing Real-time streams in Databricks – Part 2
In this section we go Azure Databricks and create the cluster and notebook to ingest the data in real-time and process and visualize the stream
Processing Real-Time Streams in Databricks – Part 1
Databricks is becoming the new normal in data processing technologies in cloud, both Azure and AWS. This is step by step guide to get started on Realtime (streaming) analytics using spark streaming on Databricks
Introduction to Delta Architecture
Delta architecture processes any new streaming records like delta (incremental) records and data lake is no longer immutable data structure
Redesigning Enterprise Analytics Teams
Organizations need leaner and agile structure to focus on the business outcome. Following are the key roles that should be part of the analytics organization which is focused solely on delivering business value
Kappa Architecture – Another way of Data Processing
Imagine a scenario where we can maintain an immutable persistent stream of data and instead of processing the data twice, we can use the stream to replay the data for a different time using the code. That is the premise of Kappa architecture
Lambda Architecture using Databricks
From technology point of view Databricks is becoming the new normal in data processing technologies, in both Azure and AWS. This post provides a view of lambda architecture and uses Databricks at front and center. Databricks has capabilities to replace multiple tools and those are described in bit detail below
5 Things to get best out of Azure Databricks
Databricks has become the new normal in the data processing in cloud. If you are using or plan to use Azure Databricks, this post is will guide you on some interesting things that you can plan to investigate as you start.