Databricks
Databricks releases, Lakeflow, Unity Catalog and Spark pipelines on the Databricks platform.
DatabricksConnect Claude, Cursor, Codex and ChatGPT to Databricks MCPSince 27 October 2026, Claude Code, Claude, Codex, Cursor and ChatGPT connect to Databricks MCP servers with a ready-made OAuth client ID, so no admin has to create an OAuth app first. The address each one needs, the exact command or setting for each client, what you must already have, and what to check when sign-in fails.31 October 2026 · 7 min read
DatabricksDatabricks agent memory: how managed memory, sessions and memory pipelines workDatabricks now runs an agent's memory for you, in Lakebase, as 3 parts: a session store for the conversation, a memory store for what lasts, and a memory pipeline that distills one into the other. What each holds, the code to wire it into an existing agent, what a pipeline keeps and skips, why a run can write nothing, and the one security rule to get right.31 October 2026 · 9 min read
DatabricksDatabricks Zerobus Ingest vs Kafka: when to skip the message busZerobus Ingest lets producers write records straight into a Unity Catalog Delta table, with no Kafka cluster in between. It replaces Kafka when Databricks is the only reader of the stream; it cannot when other systems read it too, because it is write-only. What it is, how a Kafka producer switches to it, what you give up, its limits, and how it is billed.30 October 2026 · 8 min read
DatabricksDatabricks Lakebase: Postgres next to the lakehouseLakebase is a managed Postgres database inside Databricks, for the app, agent and feature lookups a Delta table is too slow for. How it differs from the lakehouse, what it costs per CU-hour, why the default scale-to-zero timeout saves little for an office-hours app, and how data moves between the two.29 October 2026 · 9 min read
DatabricksDatabricks AI functions cost: ai_query, ai_parse_documentA SQL query that calls ai_parse_document, ai_extract or ai_query is billed twice: once for the warehouse or cluster running it, and once for the model inference behind each row. What each function adds, a worked month for 50,000 invoices, the 4 settings that cut the inference line, and the query that finds it in system.billing.usage.28 October 2026 · 8 min read
DatabricksDatabricks Unity Gateway: govern models, agents and MCPUnity Gateway is the door that model calls, MCP tool calls and coding agents go through on Databricks: Unity Catalog grants, guardrail policies, rate limits, budgets, fallbacks and a log of who called what. Follow one request through it, see what the caller gets back at each step, and what the gateway itself costs.28 October 2026 · 8 min read
DatabricksUnity Catalog vs Hive metastore: the differences, and how to moveThe Hive metastore is Databricks' legacy, per-workspace table catalog; Unity Catalog is the account-level catalog shared by every workspace in a region, with lineage, auditing and access control built in. What changes in names, permissions and storage, and which of SYNC, DEEP CLONE, the upgrade wizard or Hive metastore federation moves each kind of table.28 October 2026 · 8 min read
DatabricksDelta Lake time travel: how far back you can go, and how to keep moreOn Databricks, a Delta table's time travel reaches back 7 days by default, even though DESCRIBE HISTORY lists 30. The 2 table properties that decide it, why both must move together, what a longer history costs in storage, and when a monthly deep clone is the better way to keep the past.27 October 2026 · 7 min read
DatabricksDatabricks vs Azure Databricks: same platform, different contractAzure Databricks is the Databricks platform sold, billed and supported as a first-party Azure service; Databricks on AWS and Google Cloud is bought from Databricks or a cloud marketplace. The notebooks, Spark, Delta Lake and Unity Catalog are the same. What differs is the bill: the tiers on offer, the price of a DBU, the DBUs each machine uses, and how commitments work. A worked month on each cloud.27 October 2026 · 7 min read
DatabricksDatabricks serverless vs classic compute: cost, limits and when to switchServerless compute on Databricks costs more per DBU than classic jobs compute but includes the machines, starts faster and keeps idle time low. When it is cheaper, what it cannot run (R, RDDs, init scripts, cache(), time-based streaming triggers), and a worked break-even for a nightly job.26 October 2026 · 8 min read
DatabricksMulti-statement transactions in Databricks: BEGIN ATOMIC for Delta and Iceberg tablesSince 14 October 2026, Databricks commits several SQL statements across several tables as one transaction: all of them or none. Write them between BEGIN ATOMIC and END, or BEGIN TRANSACTION and COMMIT on a SQL warehouse. Tables you write must be Unity Catalog managed Delta or Iceberg with catalog commits. What breaks without a transaction, the code with one, and the limits.25 October 2026 · 8 min read
DatabricksDatabricks Knowledge Assistant and Supervisor Agent are legacy: what replaces themIn October 2026 Databricks marked Knowledge Assistant, Supervisor Agent, AI Playground, legacy agent memory and agents on Model Serving as legacy. The no-code path is now Genie Agents; the code path is the Agent Bricks CLI, which scaffolds a LangGraph or OpenAI Agents SDK project, runs it locally and deploys it to the agent runtime. The full old-to-new map and the first 5 commands.25 October 2026 · 7 min read
DatabricksDatabricks Spark Developer Associate exam: what to studyThe Databricks Certified Associate Developer for Apache Spark exam is 45 scored multiple-choice questions, plus up to 10 unscored, on Spark and the PySpark DataFrame API. The pairs of methods it expects you to tell apart, the 20% in streaming, Spark Connect and the pandas API, and how to practise for free.24 October 2026 · 6 min read
DatabricksDatabricks Context Engineer Associate: how to prepareThe Databricks Context Engineer Associate exam hands you a broken agent and asks what to change. How the 45 questions are built, the Databricks parts you need hands-on, the ideas that come from outside Databricks, and the traps in the guide's own samples.24 October 2026 · 7 min read
DatabricksDP-750 vs DP-700: Azure Databricks or Fabric data engineer exam?Microsoft now has 2 data engineering certifications: DP-750 for Azure Databricks and DP-700 for Microsoft Fabric. Same price, same pass mark, same free yearly renewal, very different skills. What each one tests, line by line, how much of each your current job already covers, and where Databricks' own Data Engineer Associate fits.24 October 2026 · 8 min read
DatabricksDatabricks Generative AI Engineer Associate: how to prepareThe Databricks Generative AI Engineer Associate exam is 45 questions in 90 minutes, and over half of them are about building and deploying an LLM app. A study plan built around one RAG project, the product names that changed since the guide, and the topics people miss.24 October 2026 · 8 min read
DatabricksDatabricks Data Engineer Professional exam: 2026 study guideThe Databricks Data Engineer Professional exam changed on 9 October 2026: 60 questions in 9 sections, with Python and SQL code at 23% and cost and performance at 15%. What moved, the topics older material misses, and a study plan of 4 builds.24 October 2026 · 8 min read
DatabricksWhich Databricks certification? All 8 compared for 2026Databricks has 8 certification exams in 2026, from Data Engineer Associate to the new Context Engineer and Apache Spark Developer exams. What each one covers, how long it is, what it costs to keep, and which one to take first for the work you do.24 October 2026 · 9 min read
DatabricksWhy is my Databricks bill so high? Find the cause2 fixable things often raise a Databricks bill: scheduled work running on all-purpose clusters, which cost more per DBU and sit idle until they time out, and plain Python running on a multi-node cluster whose workers do nothing. Find them with one query on the billing system tables, then fix them with job compute, single-node clusters and a cluster policy.23 October 2026 · 8 min read
DatabricksDatabricks renamed products: every old name and its new nameDLT is Lakeflow Spark Declarative Pipelines, Asset Bundles are Declarative Automation Bundles, the Assistant is Genie Code, Vector Search is AI Search and Delta Sharing is OpenSharing. Every Databricks rename in one table, with the date and what, if anything, changes in your code.22 October 2026 · 7 min read
DatabricksDatabricks metric views: examples and limitationsA metric view defines a business number once in Unity Catalog, as YAML, and computes it correctly at any grouping with MEASURE(). A complete example on the TPC-H sample, how to query it, how Power BI and Tableau use it, and the limits to know: no SELECT *, no query-time joins, runtime versions, and materialization.22 October 2026 · 8 min read
DatabricksGenie Ontology in Databricks: what it is and how it worksGenie Ontology is the shared business context behind Genie One and Genie Code: the metric views, domains and Pages you define in Unity Catalog, plus snippets Genie learns from your dashboards, queries and agents, ranked by authority and filtered by permissions. How it picks a definition, and how to make it pick yours.22 October 2026 · 6 min read
DatabricksDatabricks Genie One: pricing, connectors and setupGenie One, formerly Databricks One, is the chat-first home for business users on Databricks: questions in plain words, dashboards, apps, documents and scheduled briefings. What it costs, which access each person needs, the tools it connects to, and the 5 things an admin sets up first.22 October 2026 · 7 min read
DatabricksDatabricks Genie Code: what it does and what it costsGenie Code is Databricks' coding agent for notebooks, the SQL editor, pipelines, dashboards and MLflow. It plans, runs code and fixes its own errors within your Unity Catalog permissions. Since 8 July 2026 it is billed: 150 free DBUs per user a month, then about $0.07 a DBU in US East. How to see the bill and cap it.22 October 2026 · 7 min read
DatabricksDatabricks Genie vs Genie One vs Genie Code: what each one doesDatabricks Genie is now a family of 3: Genie One, the simple chat and dashboards for business users; Genie Agents, the curated spaces data teams set up (formerly Genie Spaces); and Genie Code, the coding agent in notebooks and pipelines. Who each is for, what it costs and how they share one context.22 October 2026 · 7 min readDatabricksDatabricks AI/BI in January 2026: Prompt Matching, LOD Calculations and Genie OneWhat changed in Databricks AI/BI in January 2026: Genie's value sampling features renamed to prompt matching, format assistance and entity matching, level-of-detail calculations in dashboards, pivot table and filter updates, and Genie One generally available. Plus what has changed since.17 January 2026 · 5 min readDatabricksSpark Declarative Pipelines: A Practical Guide, from Delta Live Tables to Apache SparkSpark Declarative Pipelines let you declare the tables a pipeline should produce and leave the order, checkpoints and retries to Spark. What they are, a pipeline you can run on a laptop, and what Databricks adds on top of the open-source version.11 January 2026 · 9 min readDatabricksDeletion Vectors in Delta Lake and Databricks: Faster DELETE, UPDATE and MERGEHow deletion vectors let Delta Lake mark rows as deleted instead of rewriting whole files, what that changed in a measured 100-row DELETE on 20 million rows, how to turn them on, which readers can still read the table, and how to apply the changes for good.9 July 2025 · 7 min readDatabricksDatabricks Free Edition: What You Get, the Limits, and How It Differs from the Free TrialDatabricks Free Edition is a permanent, no-cost Databricks workspace for learning. What it includes, its limits on compute, pipelines, model serving and apps, what happens when you hit the daily quota, and how it compares with the 14-day free trial.16 June 2025 · 5 min readDatabricksVACUUM LITE in Delta Lake and Databricks: Faster Cleanup, and When It FailsWhat VACUUM does to a Delta table, how VACUUM LITE finds old files from the transaction log instead of listing storage, the one case where it fails, how retention and time travel interact, and when you need neither because predictive optimization runs it for you.6 March 2025 · 5 min readDatabricksMedallion Architecture in Databricks: Bronze, Silver and Gold Done RightWhat each medallion layer is for, how to build bronze, silver and gold as one Databricks pipeline with streaming tables, AUTO CDC and materialized views, where the layers go in Unity Catalog, and how often to refresh them.2 March 2025 · 8 min read
DatabricksDatabricks Certified Data Engineer Associate: The Current Exam and How to PrepareWhat the Databricks Data Engineer Associate exam covers now (ingestion, transformation, Lakeflow Jobs, CI/CD, troubleshooting and Unity Catalog), the exam facts, and a hands-on plan to prepare in Databricks Free Edition.28 February 2025 · 7 min read
DatabricksDatabricks Performance Tuning: A Checklist That Starts with What Is AutomaticHow to make Databricks workloads faster and cheaper: predictive optimization, liquid clustering, Photon, right-sized compute, adaptive query execution, joins and skew, the disk cache and VACUUM, in the order that saves the most.22 February 2025 · 7 min read
DatabricksLiquid Clustering in Databricks: How It Replaces Partitioning and ZORDERWhat liquid clustering is, when to use it, how to turn it on for new, existing and partitioned Delta tables, how to choose clustering keys or let CLUSTER BY AUTO choose them, and what to check before switching.27 January 2025 · 7 min readWork it out
- BuilderCron expressions for Airflow, Databricks and SnowflakeA schedule in words in; the cron, Quartz and Snowflake expressions, the code and the next runs out.Cron expression builder
- CalculatorWhat a Databricks cluster costsDBUs and machines for jobs, all-purpose and pipeline clusters, serverless and SQL warehouses, on AWS, Azure and Google Cloud.Databricks cost calculator