Data engineering
Pipelines, warehouses, SQL and the tools data engineers use, how to choose between them, what they cost to run, and the certifications that test them.
Data engineeringSnowflake Business Ontology vs semantic views: what each layer doesSnowflake's Business Ontology stores one approved meaning for each business term, such as net revenue, and serves only approved definitions to agents. Semantic views still calculate the numbers, and Cortex Sense still guesses where nothing is modelled. How the 3 layers fit, how a Cortex Agent reads the ontology, and what to do while it is in private preview.10 November 2026 · 8 min read
Data engineeringAirflow 2 to Airflow 3: what breaks and how to moveAirflow 2 reached end of life on 22 April 2026, so it gets no more fixes. The move to Airflow 3 is mostly done on Airflow 2: fix your Dags with ruff's AIR rules, install the standard provider, then migrate the database. The risky part is not the errors you see but the schedules that change without one.8 November 2026 · 10 min read
Data engineeringMicrosoft Fabric IQ explained: business meaning for agentsFabric IQ is the Microsoft Fabric workload that gives AI agents your business's vocabulary: an ontology of entity types, relationships, rules and metrics, bound to the data where it already lives. What goes into one, the 4 ways to build it, which agents can read it, and the known issues that decide whether it works on your data today.31 October 2026 · 9 min read
Data engineeringDuckDB vs Spark: when one laptop is enoughDuckDB runs SQL inside your Python process on one machine; Spark spreads a job over a cluster. Use DuckDB when the data and its working set fit one machine's disk and memory and one process writes; use Spark when the data outgrows a machine, the job streams, or many writers share governed tables. What DuckDB 2.0 changed, where each one breaks, and the same job in both.31 October 2026 · 8 min read
Data engineeringBigQuery ML, BI Engine and Data Transfer pricingThree BigQuery meters a query estimate misses. BigQuery ML bills CREATE MODEL at $312.50 a TiB on demand, 50 times the query rate, and auto-ARIMA multiplies the bytes by every candidate model. BI Engine bills reserved memory at $0.0416 a GiB-hour whether or not a dashboard runs. Data Transfer Service is free for Google sources except Google Play, object storage and migration connectors, and bills slot-hours for the paid connectors such as Salesforce and the databases. The rules, worked sums and where each shows on the bill.31 October 2026 · 8 min read
Data engineeringFabric data agents: setup, limits and why they give wrong answersA Microsoft Fabric data agent turns plain-English questions into SQL, DAX or KQL over your lakehouses, warehouses, semantic models and KQL databases, and answers as the person asking. How to set one up, the limits that stop it (25 rows, 5 sources, English only, same region), how to make its answers right, and what it costs in capacity.30 October 2026 · 9 min read
Data engineeringSynapse to Fabric migration: what moves and what breaksMoving Azure Synapse to Microsoft Fabric is 3 migrations: dedicated SQL pools to a Fabric warehouse, Spark to Fabric notebooks and lakehouses, and pipelines to Fabric Data Factory. Which tool moves each part, what each tool leaves for you, the code and data types that break, and the order to do it in.28 October 2026 · 8 min read
Data engineeringS3 Tables vs your own Iceberg on S3: cost and trade-offsAmazon S3 Tables keeps Apache Iceberg tables in a table bucket that compacts, expires snapshots and deletes unused files for you. It costs about 15% more to store than your own Iceberg table in a general purpose bucket. What each costs on a real table, where the gap grows, the catches in S3 Tables, and the 3 jobs you run if you keep the table yourself.28 October 2026 · 8 min read
Data engineeringSpark AQE settings: what each one does and when to change itAdaptive query execution is on by default since Spark 3.2 and re-plans a query after each shuffle: it merges small partitions, switches a sort-merge join to a broadcast and splits skewed partitions. Every setting behind those 3, its default in Apache Spark and on Databricks, the few worth changing, and how to see in the plan that it worked.28 October 2026 · 8 min read
Data engineeringKafka max message size: the 2 limits that reject a recordKafka's default largest message is about 1 MB, set in 2 places that reject a record: the producer's max.request.size, checked before compression, and the broker or topic's message.max.bytes, checked after. The consumer and replica fetch sizes no longer block a large record. The defaults, the errors, a worked example and the settings to change.28 October 2026 · 7 min read
Data engineeringAzure Data Lake Storage disaster recovery: failover, step by stepAzure Data Lake Storage Gen2 accounts on geo-redundant storage can now fail over to their secondary region, planned or unplanned, with the same endpoint names. What each redundancy option survives, how much an unplanned failover can lose, the Last Sync Time check, the commands, failing back, and what geo-redundancy does not protect against.28 October 2026 · 8 min read
Data engineeringAmazon Athena query cost: pay for less data scannedAthena charges $5 for every terabyte a SQL query scans, with a 10 MB minimum a query, so its bill is set by how your files are laid out, not by how clever the SQL is. Parquet, compression, partitions and a scan limit cut the same query from $180 to about 12 cents. The rules, a worked example and the settings.27 October 2026 · 7 min read
Data engineeringBigQuery storage cost: logical vs physical billing, long-term and time travelBigQuery bills storage by the GiB-month, either on uncompressed (logical) bytes or on compressed (physical) bytes at twice the rate. Physical usually wins when data compresses more than 2 times, unless tables are rewritten often, because physical billing also charges for time travel and fail-safe. The prices, the rules, worked examples and the query that tells you which to pick.26 October 2026 · 8 min read
Data engineeringSCD type 2: keep history in a dimension, with MERGE, dbt and AUTO CDCSlowly changing dimension type 2 keeps every version of a record as its own row, with a surrogate key, valid-from and valid-to dates and a current flag, so past facts stay with the values that were true at the time. How it works, a MERGE for Delta Lake, a dbt snapshot, Databricks AUTO CDC, and the point-in-time join that reads it.26 October 2026 · 9 min read
Data engineeringMicrosoft Fabric vs Databricks: how they differ and how each billsMicrosoft Fabric is a software-as-a-service analytics suite billed as a fixed capacity shared by every workload, Power BI included. Azure Databricks is a lakehouse platform billed per use. What each is built for, how capacity units, smoothing and throttling work, and a worked month for the same team on both.26 October 2026 · 9 min read
Data engineeringBigQuery on-demand vs editions: when paying for slots is cheaperOn-demand BigQuery charges $6.25 per TiB scanned; editions charge per slot-hour of compute, from $0.04 on Standard. Which is cheaper depends on one number you can measure: slot-hours used per TiB scanned. Below 104 it favours Enterprise, below 156 Standard. The query that finds your number, and 2 teams whose answers come out opposite.25 October 2026 · 8 min read
Data engineeringSpark shuffle partitions: how to choose the numberSet spark.sql.shuffle.partitions so each partition holds at most about 128 MB and every core gets at least 2 tasks, rounded up to whole rounds: 200 GB on 64 cores is 1,600. Why a fixed default of 200 does not fit every job, what adaptive query execution fixes and what it cannot, and the Databricks auto setting.24 October 2026 · 6 min read
Data engineeringdbt cluster_by on Snowflake: the ORDER BY it addsSetting cluster_by on a dbt Snowflake model does 2 things: it sorts the whole result with an ORDER BY when the table is built, and it adds a clustering key. dbt documents the sort as intended, but on a large table model that rebuilds every run, or on a full refresh, the sort can add a lot of compute to the build. How to see it, find every model it touches, and decide per model.24 October 2026 · 6 min read
Data engineeringHow many Kafka partitions do you need?Take the topic's throughput a year or 2 ahead and divide it by what 1 partition takes from producers and by what 1 consumer reads from 1 partition; create the larger number. 50 MB/s planned to double, at 10 MB/s a partition in and 5 out, is 20 partitions. Why to plan ahead, and the limits per broker.23 October 2026 · 6 min read
Data engineeringdbt deps 500 Internal Server Error on dbt 2: the fixIf dbt 2 installed with pip fails dbt deps with a 500 from codeload.github.com, on a different package each run, you are probably hitting dbt issue 16620. The reported workaround is the native dbt binary, which passed every run in the report. How to check it is this bug, switch locally and in CI with a pinned version, and know when to switch back.21 October 2026 · 6 min read
Data engineeringDP-203 Is Retired: What DP-700 Tests and How to PrepareMicrosoft retired DP-203 on 31 March 2025. DP-700, on Microsoft Fabric, is Microsoft's data engineer exam for Fabric: how the two compare, what DP-700 tests, the exam facts and how to prepare.18 January 2025 · 5 min read
Data engineeringSQL Query Optimization: 8 Tips That Work, and 3 That Don'tHow to make a slow SQL query fast: read the plan, index what you filter on, keep indexed columns out of functions, use covering indexes and partitions. Measured on 2 million rows, with three popular tips that make no difference.13 November 2024 · 9 min read
Data engineeringThe Best Data Engineering Tools in 2026: the Top 10, by the Job Each DoesThe 10 data engineering tools worth knowing in 2026, grouped by the job they do: Fivetran and Kafka to move data, Spark, Databricks, Snowflake, BigQuery and Fabric to store and process it, dbt to transform it, Airflow and Prefect to run it.10 November 2024 · 8 min readWork it out
- BuilderCron expressions for Airflow, Databricks and SnowflakeA schedule in words in; the cron, Quartz and Snowflake expressions, the code and the next runs out.Cron expression builder
- CalculatorPartitions, brokers and disk for a Kafka topicMessages, their size, retention and readers in; the partitions, disk, network and brokers out.Kafka sizing calculator