In my previous blogs I introduced Kappa and Lambda Architectures. These are big data architectures designed to support massive amounts of data both in real time and at rest. The key difference between those two architectures is the presence of a data lake/data hub to consolidate all data in one …

We have all heard about SQL databases and have used SQL Server, Oracle, MySQL, etc. for storing or modeling data. The SQL family of databases stores data in relational, tabular structures. With the advent of new channels of data generation and consumption, SQL databases sometimes don’t provide …

Cloud technologies are changing the way we operate and do business. They enable us to focus the time and energy of resources on developing applications rather than spending time maintaining infrastructure. While cloud technologies enable business to take the leap, it is vital to understand how these …

In Part 1 I wrote about the analytics maturity curve and the gap/chasm that makes it difficult for enterprises to move from traditional BI/descriptive analytics to more modern, value-driven analytics.

Following are the key reasons that make it difficult for organizations to move from the bottom of …

Data solutions and data architectures encompass key areas in data management and provide guidance on how we ingest, process, store, maintain, consume, secure, and manage data. That is the lifecycle of most data analytics projects and systems.

Microsoft is fully invested in making Azure one of the …

For details about what Lambda architecture is, read Introduction to Lambda Architecture.

From a technology point of view, Databricks is becoming the new normal in data processing technologies in both Azure and AWS. This post provides a view of lambda architecture with Databricks at front and center. …

Kappa architecture was proposed by Jay Kreps (co-creator of Apache Kafka) as a simplification of the Lambda architecture. The core idea: remove the batch layer entirely and treat everything as a stream.

The Core Concept

In Lambda architecture, you maintain two separate processing paths — batch and …

Lambda architecture is a data processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods. The architecture was introduced by Nathan Marz and is based on three layers: the Batch Layer, the Speed Layer, and the Serving Layer. …

Azure Cost Management helps in monitoring, optimizing, and controlling costs of Azure resources in subscriptions and resource groups. It shows organizational cost and usage patterns with advanced analytics.

There are three main components of Cost Management:

1. Cost Analysis — Explore and analyze …