The first part of this post can be found at Processing Real-Time Streams in Databricks – Part 1.

This is the continuation of Part 1 — we won’t repeat the architecture and setup. In this section we go into Azure Databricks and create the cluster and notebook to ingest data in real-time, …

Databricks is becoming the new normal in data processing technologies in cloud, both Azure and AWS. This is a step-by-step guide to get started on real-time (streaming) analytics using Spark Streaming on Databricks.

Architecture

The demo was built to show the speed layer (hot path) of a typical …

Data solutions and data architectures encompass key areas in data management and provide guidance on how we ingest, process, store, maintain, consume, secure, and manage data. That is the lifecycle of most data analytics projects and systems.

Microsoft is fully invested in making Azure one of the …

For details about what Lambda architecture is, read Introduction to Lambda Architecture.

From a technology point of view, Databricks is becoming the new normal in data processing technologies in both Azure and AWS. This post provides a view of lambda architecture with Databricks at front and center. …

Azure Cost Management helps in monitoring, optimizing, and controlling costs of Azure resources in subscriptions and resource groups. It shows organizational cost and usage patterns with advanced analytics.

There are three main components of Cost Management:

1. Cost Analysis — Explore and analyze …

A data lake is a framework, concept, and guidance on where to place data (Microsoft named their product Azure Data Lake, but the concept is broader). From a technology point of view, it suggests storing all data in object-oriented or hierarchical storage. This is the concept of data locality — data …

Most of the customers I talk to are directly or indirectly asking to scale their workloads and use Databricks. It has become the new normal in data processing in cloud. If you are using or plan to use Azure Databricks, this post will guide you on some interesting things to investigate as you start. …