Related articles

The Pelican comparison grid for Astra is pretty interesting

Achieving Extreme Efficiency through Specialized GPU Kernel Generation

Traditionally, production inference systems rely on generic kernels to handle diverse...

What Happens to Your Scaled Agile Ways of Working When You Adopt an Agentic SDLC?

Originally published on yuvalyeret.com. It's time to rethink your ways of working, not get rid of them Every week, I have several conversations with leaders whose teams are implementing an AI harness, adopting spec-driven development, and starting conversations about throwing agi…

Implementing SCD Type 2 in Microsoft Fabric – The Definitive Guide!

Slowly Changing Dimensions (SCD) are one of the fundamental concepts in dimensional modeling. If you are not sure what dimensional modeling is, I suggest you first check the series of articles I wrote on data modeling some time ago. And, when it comes to SCD in particular, SCD Ty…

OpenAI's rogue agents were caught communicating via public wikis

Five ways marketers can use Genie One

Every marketing team sits on a goldmine of data: campaign performance metrics, customer...

Níveis de Maturidade do Product Owner: Por Que Ninguém Começa Faixa Preta

No mercado corporativo, venderam uma ideia perigosa: a de que todo Product Owner já senta na cadeira como o “mini-CEO” da empresa, com autonomia total para ditar estratégias e gerir orçamentos milionários desde o primeiro dia. A realidade, porém, é bem diferente. Assim como nas a…

Teach Claude Your Power BI Standards, Custom Skills for the Power BI Modeling MCP Server

Can you teach Claude, or any LLM, to follow your standards and your guidelines, so that when it connects to a Power BI model, when it wants to build a semantic model or a report, it does it exactly as you do, following all the instructions that you would normally provide? Yes, it…

Daybreak for Frontline Defenders: $1B to protect essential services

OpenAI introduces Daybreak for Frontline Defenders. A $1 billion commitment expands access to frontier cyber AI, training, and support for essential services.

Governance beyond security: knowledge, context & ontology on the lakehouse

Ask most organizations what data governance for AI means, and you’ll hear a security...

O que sabemos agora que não sabíamos no início da Sprint?

Image Scrum é um framework de aprendizagem Quando pensamos em Scrum, é fácil pensar em Sprints, Product Backlogs, Daily Scrums, Sprint Reviews ou Sprint Retrospectives. Mas estes são mecanismos. A questão mais interessante é: para que existem? Uma forma útil de compreender Scrum…

Matrix Totals – The Whiteboard #13

Let’s talk about matrix totals in this episode of The Whiteboard! Special guest: the SQLBI Whiteboard! Learn abstract DAX concepts in a more interactive way with “The Whiteboard” series. Read more: https://www.sqlbi.com/blog/marco/2022/07/14/the-whiteboard-video-series-on-sqlbi-y…

Playco cut manual fixes 50% prototyping games with GPT-6 Astra

Using GPT-6 Astra, Playco built three themed game prototypes from one grey box foundation and reported 50% fewer manual fixes than with the previous model.

Building High-Quality and Trusted Data Products with Databricks

IntroductionOrganizations aiming to become AI and data-driven often need to provide...

If You Hate the Organization, You Can't Help it Change

He was right about everything, and it was helping nothing. Watch the agile coach who has the organization completely figured out. He can name every dysfunction in the building, accurately, usually before lunch. The fake empowerment. The theater dressed up as progress. He isn't wr…

Referencing Images And Other Files In Power BI Reports Using OneLake URLs

Several years ago I wrote a very popular blog post about how to store images for your reports inside your Power BI semantic model. It solved the problem of how you could use display images (for example of products) inside your reports without making those images available via a p…

Legora reviewed 41 documents in minutes with GPT-6 Astra

Legora used GPT-6 Astra to review 41 documents in minutes, find all four planted errors, and improve performance by nearly 40% in this financial-review workflow.

What is Data Transformation?

What is data transformation?Data transformation is the process of taking raw data...

Don't Redesign Your Process Yet. Change Who Writes the Artifacts.

Originally published on yuvalyeret.com. The first thing to change is who writes the artifacts, not how you work "We didn't change the process. We still have a quarterly planning and we have sprint planning and we have roadmaps and a list of features that we want to deliver in a q…

Fabric Apps & Rayfin Deep Dive: A Conversation with Sachin Patney | Fabric Insider Ep. 11

I have written and recorded about Rayfin before, what it is, how the Data App template works, and the cost story that makes it genuinely significant. But this conversation goes much deeper. I sat down with Sachin Patney, who leads Fabric App Development at Microsoft, and by exten…

GPT-6 Astra: A new generation of intelligence

Introducing GPT-6 Astra, our most intelligent and aligned model yet, with state-of-the-art capabilities across computer use, coding, cybersecurity, and science.

Southern Company’s SCOUT: Completing the Storm Intelligence Story

Completing the storm intelligence storyIn an earlier post, we explored how Southern...

Bliki: Paracelsus Maxim

The difference between a medicine and a poison is dosage. Often we talk about certain habits, in programming or life, are good or bad. But few things are simple binaries. Some vary with context: reading a book is a good thing sitting in my garden, but not while driving my car. Bu…

SQLBI Whiteboard

SQLBI Whiteboard is a free, open-source whiteboard for Windows 11 designed for teaching DAX, SQL, and data modeling, with pen ink, live application capture, and syntax-highlighted code on the same canvas. Pen-first ink — Low-latency, pressure-aware strokes with palm rejection,…

Safety overview: GPT-6 Astra

GPT-6 Astra is our most capable broadly deployed model and our first to reach the Critical level of cybersecurity capability under our Preparedness Framework.

Expanding Genie Agents: Deep analysis, file reasoning, and more

At Data and AI Summit, we announced the evolution of Genie Spaces to Genie Agents,...

Fostering Agency in Daily Leadership

This is the third and final part of the series on agency. Part one explained what agency actually is, and part two looked at what a lack of agency looks like in practice. This part could almost be short. Most of it is already captured in the fifth principle of the Agile Manifesto…

No-Code Power BI Report Authoring — GitHub Copilot + Fabric Skills + Power BI Desktop Bridge

AI can perform a lot of interesting tasks these days. It can check your email. It can draft an email response. And it can do many actions within the Power BI environment — including creating a complete Power BI report entirely from a prompt window, without touching anything insid…

Claude's new system prompt really doesn't want to reproduce song lyrics

Announcing the Databricks Big Book of AgentOps

What is AgentOps?AgentOps is the operating discipline for building, deploying and...

An Accidental Blackboard

Giles Edwards-Alexander reports that during an experiment to see how productive a team could be using fully agentic engineering practices, the team accidentally prompted the agents into creating a blackboard coordination system inside the git repository. more…

Testing DAX measures by using AI

How to use AI to identify relevant test cases in a semantic model and create repeatable DAX tests for a measure.

ATV Big Air Tour turned 3 days of work into 3 hours with ChatGPT

ATV Big Air Tour uses ChatGPT Work to speed up marketing, merchandising, and more. It even turned merchandise photos into an inventory website in 15 minutes.

Try DuckDB v2.0-alpha

The State of DuckDB v2.0 A few weeks ago, we published a preview of DuckDB v2.0. Today, we branched off v2.0-cyanoptera from the main development branch in the duckdb/duckdb repository. This marks a feature freeze – in the coming weeks, development work will focus on testing, bug…

Maybe We Shouldn't Be Reviewing All This Code

TL;DR Or, perhaps the problem isn't that AI has broken code review, maybe it’s that we've been using code review to solve the wrong problems I was on a panel recently with Brian Houck from DX at Code Remix, hosted by Moderne. It was one of the more interesting panels I’ve done, l…

Testing DAX measures by using AI

This article describes how to use AI to identify relevant test cases in a semantic model and create repeatable DAX tests for a measure. A DAX measure is correct only when it produces the expected result in every situation of…

Claude Fable 5.1 made me a really nice animated pelican

Context, Semantics, and Ontology: A Primer for the Agentic Era

A primer on the context layer, semantic layer, and ontology: what each term means, how they fit together, and why AI agents still need a human to decide what's true.

Everything Was Green. Then it Wasn't.

Image The status was green all quarter. Green in the weekly report, green in the steering deck, green in the note to the sponsor. Then, close to the deadline, it went red. Not drifted. Snapped. And in the post-mortem, the part that stings: one after another, people say they alrea…

Tools With Interactive UIs In Fabric Notebooks With Semantic Link Labs

There’s so much going on in the Fabric community that it can be hard to keep up with it all. Semantic Link Labs is a great example: in the six months or so since I last had a proper look at it my colleague Michael Kovalsky has done a whole load of cool things and … Continue readi…

How AI-native companies turn workflows into operating capability

Basis, Clay, and Exa Labs use AI agents to improve onboarding, account management, and developer integrations. See what enterprise leaders can apply.

Agents Don’t Query Like Humans Do

We analyzed a slice of MotherDuck query history and found agents ran 29 times more queries than humans last month, a gap that has roughly doubled every month. The average organization now has twice as many agents as human users. Agents are the dominant consumers of the data platf…

Fragments: September 1

Like many readers, I’m wary of AI generated prose. Simon Wilison has written an LLM cliché highlighter - paste in some text, or a URL, and it will flag various patterns common to LLMs. It references a wikipedia page of signs of AI writing. That page points out that: Humans are no…

Displaying The Output Of Detail Rows Expressions In Power BI Reports Using The Paginated Report Visual

In last week’s post I mentioned that while Power BI reports (unlike Excel PivotTables) do not support the Detail Rows Expression feature, it is possible to partially work around this limitation by using the paginated report visual. In this post I’ll show you how I was able to do…

Path to Astra: critical capabilities and frontier safeguards

Astra is the first OpenAI model to meet the Critical cybersecurity capability threshold under the Preparedness Framework, with stronger safeguards for release.

How the FDA is building a secure, AI-ready data foundation on Databricks for Government

Modernizing a federal data platform is a little like steering an aircraft carrier while rebuilding the engine mid-ocean...

What Leaders Do That Keep Scrum Teams Working Like Individuals

Scrum teams often work like individuals because the organization staffs, measures, interrupts, and rewards them that way. Leaders can change the conditions that make real teamwork possible.

The Next Evolution of Power BI — A Conversation with Mohammad Ali | Fabric Insider Ep. 10

What is the next evolution of Power BI? Will Power BI Desktop ever become a prompt-first UI? And how do you actually decide whether you need a semantic model, an ontology, or both? I sat down with Mohammad Ali — Director of Product Management for Power BI at Microsoft, overseeing…

Understanding ChatGPT Work

How we eliminated $1 million a year of wasted AI agent spend in one hour

Databricks engineers rely heavily on AI agents to streamline and accelerate their work. In turn...

[Episode 6] How to Build and Set up AI Agents for Scrum Teams

"AI-native Scrum Team" is a podcast series that shares how Scrum Teams can effectively take the steps in their AI Transformation journey to become AI-native Scrum Teams. In episode 6, Mehdi and Gizem discuss how to build and set up AI Agents for Scrum Teams. They share some servi…

Creating DAX functions with AI to remove duplicated code

How to leverage AI to refactor existing DAX code, remove duplicated parts, and centralize the common code in user-defined functions.

Conceptual integrity and counting lines of code

Data Engineering Weekly #285

The Weekly Data Engineering Newsletter

Scrum Masters — the World Needs You Now as Much as Ever

Image We all know too little planning is ineffective — and I think most of us know too much planning is wasteful in conditions of uncertainty, volatility, and disruptive competition. Bigger, longer, more predictive planning — thus more assumptive, guessworking, and speculative —…

Creating DAX functions with AI to remove duplicated code

Code written by AI can contain many duplicate parts. User-defined functions are a great tool to simplify complex measures; this article shows how to leverage AI to build better DAX code. AI is getting better and better at generating DAX…

Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things

Agentic SQL for Free with Qwen3.8 27B and DuckDB

If your laptop has 16GB of RAM, your agent can write SQL locally for free with Qwen3.8 27B. On the DABstep SQL benchmark, Qwen beat GPT 5.6 Luna and cost under 50 cents in electricity, over 17x less. Add DuckDB and you have a data analyst that runs on your own machine. This post…

Join the AI Delegation Audit Webinar, October 6, by Scrum.org

TL;DR: The AI Delegation Audit Webinar, October 6 Join me on October 6 for the AI Delegation Audit webinar. You will learn how to run a 45-to-60-minute recurring check that reveals whether delegated AI work still meets its required quality standard, still uses the right model, at…

Restoring alternating rows in Power BI matrices with Fluent 2

Microsoft recently introduced a new default visual design style for Power BI named Fluent 2. I personally disagree with a few choices in the new defaults, including the removal of alternating row colors from Table and Matrix visuals. However, the…

Now we have a timeline of the OpenAI accidental attack against Hugging Face

MotherDuck CLI: query, pipelines, and dashboards from your terminal

Meet the MotherDuck CLI: one install to query data, publish Python pipelines, and host dashboards from your terminal, for coding agents and CI alike.

How to Leverage Flow Metrics To Accelerate Your Agentic Development Lifecycle

Originally published on yuvalyeret.com. Isn't it surprising that 10x coding doesn't move the needle that much? Fully agentic software factories are of course all the rage these days. But more and more leaders I talk to acknowledge that while creating a fully agentic software deve…

Using Detail Rows Expressions To Drill To A Different Fact Table In Power BI

If you have a DirectQuery fact table in Power BI you can use user-defined aggregations to improve query performance; querying a smaller, summarised copy of your data in an Import mode aggregation table is always going to be faster than querying a large fact table containing all y…

Building an Operational Ontology: An E-Commerce Walkthrough

The write side of the ontology conversation: named actions, business rules, and write-back to the systems of record — a pattern already running at enterprise scale.

AI Tool Introduction – Gamma

Image Imagine you are a Scrum Master or Product Owner. When you want to deliver your work to your audience, you need a tool to present your work. In the past, we used to use PowerPoint presentations. But how can AI elevate it? Gamma is the AI-based solution for the new generation…

Rayfin Data App — The Future of Power BI Reporting

You can build an application whose frontend is TypeScript and ReactJS, and whose backend is a Power BI semantic model — meaning you get a fully interactive, visualized report inside your application without it being a Power BI report at all. Project Rayfin makes this possible. An…

DuckLabs to Join AWS, Projects to Remain Open Source

DuckLabs today has announced that they will be joining Amazon Web Services as a new subsidiary. The big news for DuckDB and related projects like DuckLake is that there are no changes for our projects’ roadmap, licensing, and governance model! DuckDB, DuckLake, Quack, and all the…

Cognitive Trap: Increment Misinterpretation

Increment Misconception Trap Most mistakes in Scrum aren’t because people don’t understand the framework—they come from applying reasonable thinking in the wrong context. Cognitive traps happen when decisions favor efficiency, control, or comfort over transparency, inspection, an…

How to check table sizes in a Microsoft Fabric warehouse

If you've tried to check how much storage your Fabric warehouse tables consume, you've probably discovered that the usual SQL Server approaches don't work. This post explains why, what else doesn't work, and the one approach that does The post How to check table sizes in a Micros…

DuckDB outgrows its nest

Today Duck Labs, the developers of DuckDB, announced they are being acquired by Amazon. This is big news in the duck-iverse, and many people are wondering what this will mean for everyone’s favorite duck-powered database, MotherDuck.

Ask Them to Define the Product Role. The Answer Describes Their Last Employer.

Image Two candidates, same question, same afternoon. What is the job of a product person? The first says it is owning the backlog and keeping the team supplied with clear, ready work. The second says it is working out which problem is worth solving, then finding a solution custom…

Data Integration, Migration & the Future of ETL in Microsoft Fabric — A Conversation with Wee Hyong Tok | Fabric Insider Ep. 9

Is ETL as we know it coming to an end? What is the real migration path from ADF, Synapse, or SSIS to Microsoft Fabric? And what does AI mean for the future of the ETL developer’s career? I sat down with Wee Hyong Tok — Partner Director of Product Management for Data Integration a…

How DuckDB Runs Recursive CTEs Faster

When I implemented DuckDB's first recursive CTE operator in 2020, correctness determined the design: evaluate the non-recursive term once, then evaluate the recursive term until the next working table is empty. That established the right semantic contract, but reusable runtime st…

At Some Point, not Knowing Becomes a Choice

Open the retro notes from a year ago. Then open last quarter's. The same problem is in both, in almost the same words. The dependency that strangles every release. The hand-off that loses a week. The "we should really talk about the roadmap" that never becomes a conversation. It'…

Analysing Power BI Data Using Copilot In Excel

This week there was another big Copilot-related announcement for Power BI – but on the Excel blog. You can now analyse data from Power BI semantic models using Copilot in Excel. Full details on how to enable this are in the docs here, but what does it look like? Using the same se…

DuckDB Table Functions in Java

In a large organization, data is spread across many systems, including relational databases, document stores, message queues, data lakes, and cloud data warehouses. Some of these systems can be reached only through a vendor SDK, usually provided in Java, or a SOAP endpoint, and m…

Making Your Data Ready for Agentic AI

Lots of organizations are excited about what AI can do to streamline their processes, save money, and juice margins. But AI's capabilities are founded on the data that AI accesses, and for many organizations that foundation is little more than sand. Pramod Sadalage and Prem Chand…

(Don’t) Stop materializing your silver layer!

Why is your silver layer a set of tables? Challenging the fully materialized medallion pattern - with benchmark numbers and reproducible demo scripts. The post (Don’t) Stop materializing your silver layer! appeared first on Data Mozart.

MotherDuck Acquires Tower: Agents Can Answer. Now They Can Build.

Agility Might Have Been Waiting for AI

Originally published on yuvalyeret.com. AI is forcing everybody to be very agile about who they are, what they do, what impact can they make, and how they're perceived, including myself. I've pivoted from Scaling with Agility to Scaling AI from Activity to Impact. This article ex…

Fabric MCP Servers, CLI & AI Agents — A Conversation with Hasan Abo-Shally | Fabric Insider Ep. 8

Everyone is talking about MCP Servers. But most people still cannot clearly explain what one actually is, why it matters for Fabric, or how it differs from the APIs and skills they already use. In this episode of Fabric Insider, I sat down with Hasan Abo-Shally — Product Manager…

Data Engineering Weekly #284

The Weekly Data Engineering Newsletter

Critical Asked Questions (CAQ) About PSPO-AI Essentials Course

In this blog post, I answer critical questions you have before choosing a Professional Scrum Product Owner – AI Essentials class. Image 1- Why should I take the class when I can directly take the PSPO-AI Essentials exam and get the certificate without attending the class? Yes, yo…

Analyzing the performance impact of visual calculations

Visual calculations can either improve or degrade the performance of a report. Let’s see how we ensure visual calculations have a positive impact on performance.

Chunked Query Results in the DuckDB Java Driver

DuckDB is a columnar, vectorized database. Every operator inside the engine works on data chunks: batches of column vectors, 2,048 rows at a time. That is a big part of why DuckDB is fast: the engine amortizes interpretation overhead over thousands of values instead of paying it…

Fragments: August 24

I was listening to Ezra Klein’s interview with Helen Toner about the recent OpenAI hack of Hugging Face and the subsequent discovery that there were swarms of agents inside OpenAI doing unsanctioned activities. One of the points Klein made was that at no point did any of these (t…

Analyzing the performance impact of visual calculations

Visual calculations can either improve or degrade the performance of a report. In this article, we outline how we ensure visual calculations have a positive impact on performance. The goal of visual calculations is to simplify some reports and calculations,…

DuckDB v2.0: Your Database Deserves a Better Parser

At DuckDB, one of our goals is to make working with a database system as easy as possible. Users interact with the system through the widely understood Structured Query Language (SQL). Previous blog posts have covered DuckDB’s friendly SQL, including GROUP BY ALL and column selec…

Product Owner Interview Questions for the Age of AI

TL; DR: 10 New Product Owner Interview Questions on AI “How do you use AI in your work?” If that is still one of your Product Owner interview questions, you are screening for tooling fluency in a role that lives or dies on product judgment. Every candidate has an answer; LLMs mak…

Power BI, M365 Copilot And The Importance Of DAX UDFs

This was a big week for Power BI Copilot with the announcement of a new direction for the feature. If you haven’t done so already, read the two announcement blog posts here and here and check out this demo video. There are some pros and cons to this change (yes, you’re going to n…

AI Writes the Semantic Layer

We proved that AI agents could build a semantic layer. The more interesting result was that the semantic layer itself became disposable. Q&A pairs are unreasonably effective at capturing business intent and give agents something concrete to iterate with. If code can be regenerate…

[Episode 5] AI Effects When You Use It In Building Your Product

"AI-native Scrum Team" is a podcast series that shares how Scrum Teams can effectively take the steps of their AI Transformation journey to become AI-native Scrum Teams. In episode 5, we discussed what effects we will see in our processes when we use AI to build the product. (for…

What Is Fabric App or Rayfin and Why You Should Care

Imagine an application built entirely inside the Microsoft Fabric environment. Not an application that connects to Fabric. Not an application that reads from a Fabric data source. An application where the database is a Fabric SQL database, the semantic model is a Power BI semanti…

Reconciling JSON in DuckDB, One Patch at a Time

Guest blog post by Mustafa Khan (Atlan). DuckDB's JSON extension already covers reading JSON, extracting paths, applying RFC 7396 merge patches, and the usual scalar accessors. The upcoming v2.0 release will extend it with four new scalar functions: json_merge_patch_diff computes…

Citizens Build, Agents Execute, Experts Govern

TL;DR Why building an app over the weekend isn't the same as building enterprise software I’ve noticed an interesting gap opening up over the last six months. It isn’t really a gap in technology. It’s a gap in what different people think software engineering actually is. The conv…

Using Fabric Operations Agents And Workspace Monitoring With Power BI

This week, in the announcement about support for Fabric Pipelines in Workspace Monitoring, I noticed that it came with an Operations Agent that actively monitors and analyses Pipeline activity. And that got me thinking, since Workspace Monitoring also contains Power BI activity d…

Scalable Data Apps with MotherDuck & Replit

AI agents make building data apps easy, but what about deploying them or making sure they scale? Replit provides a way for just about anyone to build real applications, and MotherDuck is the easiest way to help them scale to larger data sizes. Load data with MotherDuck Flights an…

Practitioner Voice: The Writing Category Nobody has Named Yet

Jim Highsmith recognizes that effective writing from a practitioner is a style distinct from academic writing or thought-leadership content. It's a style that I advocate, and my contributors mostly follow. Jim decided it was important to give it a name, and identify what makes it…

Everything About the Microsoft MVP Program — A Conversation with Rie Merritt | Fabric Insider Ep. 7

This episode of Fabric Insider is a little different from the others. We are not talking about a product feature or a technical update. We are talking about a program that has shaped the Microsoft data community for over two decades — and one that a lot of people misunderstand. T…

Data Engineering Weekly #283

The Weekly Data Engineering Newsletter

Fragments: August 18

Part of the reason why I’m at Thoughtworks is because I’d like to see a software development organization founded on technical excellence as an example for the rest of the industry. The trouble is that I have little aptitude or inclination for the hard work of building such an or…

Dynamic formatting by hierarchy level with ISINSCOPE and ISATLEVEL

How to apply a different formatting rule at each level of a hierarchy using ISINSCOPE in a measure or ISATLEVEL in a visual calculation.

A Preview of DuckDB v2.0

DuckDB v2.0 will be named “Cyanoptera” after the cinnamon teal (Anas cyanoptera), a strikingly reddish-brown duck found in the western Americas. A major version bump is not something we do lightly, and it is not just ceremony: v2.0 ships a new SQL parser, a new default storage fo…

Why Your Scrum Team Still Works Like Individuals

Learn how handoffs, individual task ownership, and status-report daily scrums keep Scrum teams from collaborating—and what to change next sprint.

Dynamic formatting by hierarchy level with ISINSCOPE and ISATLEVEL

This article describes how to apply different formatting rules at each level of a hierarchy (one rule at the year level, another at the quarter level, another at the month level) using ISINSCOPE in a measure or ISATLEVEL in a…

What an Ontology for AI Agents Actually Needs

How to think about the semantic system that makes an agent coherent, governable, and useful.

TDD inside the agent loop - theater or actual value?

My colleagues at Thoughtworks tend to be big fans of Test-Driven Development, and many people in the industry advocate telling LLM agents to use TDD when building software. Birgitta Böckeler was curious if this really makes a difference, so conducted a few experiments. more…

The Benefits Of Using Direct Lake Mode In Power BI

This is a blog post I’ve been meaning to write for a long time. Since Fabric launched there has been a lot of focus on Direct Lake mode in Power BI and a lot of people used it because it was the cool new thing. Arguably, we at Microsoft have been guilty of telling people … Contin…

Vibe Code Data Apps with Vercel V0 & MotherDuck

Use an agent to build data apps with the combo of Vercel V0 and MotherDuck! V0 makes prototyping and deployment easy, while MotherDuck Flights provide seamless data ingestion and MotherDuck handles up to billions of rows. Your agent can even sign up for MotherDuck for you! See a…

Lack of Agency – When the system swallows the human

In my previous article, I explored the concept of agency — the capacity and freedom of individuals to act independently, make choices, and meaningfully influence their circumstances. But what does it look like when agency is entirely absent? I learned the hard way when I tried to…

No Code Dataflow Gen 2 in Microsoft Fabric Using GitHub Copilot and Fabric Skills

What if you could create a complete Dataflow Gen 2 in Microsoft Fabric — with data connections, merge transformations, and a fully built date table — without writing a single line of M code? Not as a theoretical future capability. Right now. Today. In this video, I show you exact…

DuckDB Ecosystem Newsletter : August 2026

The August 2026 DuckDB Ecosystem Newsletter: giving every agent its own embedded DuckDB, the ADBC extension reaching 30+ external databases, async I/O landing in v2.0, ISO GQL graph queries, Perspective 5.0 pushdown, plus a community spotlight on Kyle Cheung.

Fragments: August 4

There’s been a fair bit of publicity of the Open AI “rogue agent” that hacked into Hugging Face. This prompted Anthropic to check what their models were up to and, to my complete lack of surprise, discovered three incidents where models had gained unauthorized access to data in o…

Generative AI guidelines at SQLBI (2026 update)

3 years ago, we wrote our “Generative AI guidelines at SQLBI”. We think it’s time to update them. Over the past 3 years, the world has changed: it is simply no longer possible to avoid using generative AI services for…

Declarative Automation in Dagster: Smarter Data Orchestration Without Complex Schedules

Learn how Dagster's Declarative Automation simplifies data orchestration by defining desired asset state instead of managing schedules, sensors, and custom triggers.

The Conductor Developer

TL;DR Why I think software development is starting to feel a little more like conducting an orchestra. There’s a shift happening in software development that I don’t think we’re talking about clearly enough. For the last couple of years we’ve framed AI as a productivity tool. How…

Liar, liar! What to do when AI writes your pipelines confidently wrong?

Should we let AI write our pipelines? YES...But not the way you probably think. The post Liar, liar! What to do when AI writes your pipelines confidently wrong? appeared first on Data Mozart.

Creative Dagster Projects from the Community

From public dataset explorers to infrastructure monitoring and research automation, the Dagster community is building projects we never could have predicted. Here are a few creative community-built use cases that caught our attention.

The Economic Benefit of Refactoring

Giles Edwards-Alexander does an experiment to see if decomposing a large function helps reduce token costs, suggesting that is may now be possible to measure the economic benefit of refactoring more…

Understanding The “Database Was Evicted To Balance The CPU Load” Error In Power BI

A few months ago I wrote about a rare error – the “Maximum allowable memory allocation” error – that may occur when the physical machine, or node, that a semantic model is running on in the Power BI Service comes under memory pressure. Recently, someone I was working with who was…

Data Engineering Weekly #282

The Weekly Data Engineering Newsletter

Leading Agile Initiatives: How Leaders Help Agile Change Succeed

An agile initiative is not mainly a one-time rollout of Scrum, Kanban, SAFe, Jira, new job titles, or a different meeting calendar.

One-minute trap! How new Fabric Warehouse CU metering affect your workloads

Microsoft is moving Fabric Warehouse CU metering from per-query CPU time to per-workspace virtual-node time. That brings Fabric closer to Snowflake and Databricks, but it also changes the cost equation: dense workloads may benefit, while sparse, chatty workloads could get a surpr…

Agents can sign up for MotherDuck

AI agents can now sign up for MotherDuck with a single API call - no forms, no credentials, no human in the loop (unless you want to be, of course). Experiment faster, share data, or just give your agents a ducking fast set of tools for analytics.

The Orchestrator's Tax

Subagents get justified by time saved and parallel execution, but Rahul Garg explains that's not what matters most. Every token in the orchestrator's context is competing for its attention, and the real value of a subagent is what it keeps out of that context. Subagents should be…

Using REMOVEFILTERS in DAX user defined functions

How to implement a DAX function that removes filter-keep column filters from a calendar, using REMOVEFILTERS as the return value of the function.

On Benchmarking

Why a throughput number is not an architecture decision.

Why I’m Writing Rachel’s Ramblings

TL;DR I have ideas. I haven’t been writing them. That’s about to change. I promise… myself. I’ve been thinking a lot about talent. Actually, I’ve been thinking a lot about thinking. And writing. Or more specifically, not writing. This really hit me earlier this year at the Future…

Using REMOVEFILTERS in DAX user-defined functions

In this article, we implement a function that removes filter-keep column filters from a calendar, using REMOVEFILTERS as the return value of the function. A DAX user-defined function, also known as a UDF, is expected to return a scalar or…

Thank You for 40 000 Stars on GitHub

The duckdb/duckdb GitHub repository has just passed 40 000 stars! We’d like to use this milestone to stop for a moment and revisit recent developments in the “Duck Stack” ecosystem since last summer (2025), when we surpassed 30 000 stars. Releases Since the last milestone, the Du…

Fragments: July 21

With this post, I’ll wrap up my notes from the second Future of Software Development Retreat. But before I do, I should note that the full Thoughtworks report on the retreat is now available. They have five headline findings: Code generation is no longer the bottleneck — verifica…

Why data teams are emerging as leaders in AI agent adoption

New research from Microsoft and MIT Technology Review Insights reveals how organizations are building trust in AI agents—starting with structured data workflows and scaling through shared context and real-time intelligence. The post Why data teams are emerging as leaders in AI ag…

I let an agent sign itself up to a cloud data warehouse, then replayed every step

AI agents can now provision a Postgres database, an email inbox or a data warehouse with no signup and no credit card. Here is how MotherDuck's agent signup endpoint works, and how motherduck.com/try lets you inspect, share and replay everything the agent did.

The Archaeologist’s Copilot

When people think of legacy modernization, most folks aren't imagining the target environment will be Java 8. But this was the challenge facing Nik Malykhin when he needed to run a Java 1.5 codebase on today's hardware. His early use of LLMs gave plausible answers that did not ho…

Power BI DirectQuery Mode: A Better Choice Than You Might Think

For as long as I’ve been using Power BI – which has been from the beginning – the advice about which storage mode to choose has been the same: use Import mode unless you have a really, really good reason to use DirectQuery mode and even then you’re probably wrong and should use I…

Data Engineering Weekly #281

The Weekly Data Engineering Newsletter

DSLs Enable Reliable Use of LLMs

LLMs generate code incredibly fast, but to ensure they generate exactly what is intended, they need clear boundaries. Abstractions and Domain-Specific Languages (DSLs) provide a strong harness that guides LLMs right from the start. Unmesh Joshi describes how the example of Ticklo…

FabCon Europe 2026: The sessions we’re most excited to bring to Barcelona

Get a first look at FabCon Europe 2026 in Barcelona, with curated session picks across Microsoft Fabric, SQL, AI, real-time intelligence, and data engineering—plus insights into the innovations shaping the future of data. The post FabCon Europe 2026: The sessions we’re most excit…

OpenAI Just Made Analytics 10x Cheaper

Since OpenAI slashed the price of GPT 5.6 Luna by 80% this week, low latency AI-powered answers are finally feasible for less than half a penny per answer, AI and DB costs included. For data questions, GPT 5.6 Luna is intelligence too cheap to meter. Well documented context, rigo…

Fragments: July 13

Some more of my notes from Thoughtworks Future of Software Development Retreat. When we had our first retreat in Utah early this year, nobody had heard of Harness Engineering. This time we had a whole session on it. When comes to the guide side of harnesses, most of the discussio…

Variable libraries in Microsoft Fabric: One value to rule them all!

Hardcoded values are the cockroaches of a data platform: you think you got them all, then Test starts reading from Prod and nobody notices for three days. Variable libraries in Microsoft Fabric put your configuration in one versioned place and let each stage resolve its own value…

Asynchronous I/O in DuckDB: Work, Thread, Work

It doesn't matter how fast query operators are in a database system if we can't pull in the data quickly. For most of DuckDB's history, however, this problem was largely avoided by pruning data early. By pushing down filters and projections, we could ensure that we only read what…

Experiences with local models for coding

Birgitta Böckeler now reports on her recent experiences trying local LLMs for coding. She compares them using two standard tasks, and tries out the most promising model for day-to-day use. more…

Optional parameters in DAX user defined functions

How to define optional parameters in DAX user-defined functions and set default values for parameters not specified by the caller.

Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin

Maxime Beauchemin, creator of Airflow and Superset, shares how he uses AI agents for data engineering: orchestrating agents on a shared canvas with Agor, context layers and semantics, Okta for Agents security, and automating CEO work with AI assistants.

Focusing Where We Can Have the Most Impact

Mountain Goat Software is shifting its training and coaching focus to private client work, where teams and leaders can apply agile to their real challenges.

Optional parameters in DAX user-defined functions

This article describes how to define optional parameters in DAX user-defined functions and set default values for parameters not specified by the caller. When Microsoft announced that DAX User-defined functions (UDFs) are generally available (GA), another new feature was also…

Creative Dagster Projects from the Community

From public dataset explorers to infrastructure monitoring and research automation, the Dagster community is building projects we never could have predicted. Here are a few creative community-built use cases that caught our attention.

We Tried Agile and It Didn’t Work

“We tried agile and it didn’t work” is one of the most revealing things a leader can hear.

Data, AI, and the case for continuous learning

Microsoft Data Days brings together training, certification pathways, and more—helping you build practical skills and stay ahead in a rapidly evolving field. The post Data, AI, and the case for continuous learning appeared first on Microsoft Fabric Blog.

Context belongs in the warehouse

Introducing Guides, MotherDuck's context layer for AI agents. More accurate queries, lower token spend, and distributed context for your entire organization.

Why Scrum Isn’t Working Even Though You’re Doing Scrum

Story splitting helps teams turn large user stories into smaller, valuable pieces they can finish within a sprint without turning them into tasks.

Microsoft Build 2026: Building agentic apps with Microsoft Fabric and Microsoft Databases

Microsoft Build 2026 highlights advancements in app development with Microsoft Fabric and Microsoft Databases, emphasizing a unified data and AI platform for scalable, agentic applications. The post Microsoft Build 2026: Building agentic apps with Microsoft Fabric and Microsoft D…

Data Engineering Weekly #280

The Weekly Data Engineering Newsletter

MobAI: From Copilot to Team Pilot

AI can make individuals faster. For leaders, the bigger question is whether it helps teams build shared context, judgment, and better decisions. AI can make individuals faster. It can also make teams more fragmented. That is one of the risks I see in how many organizations approa…

ALL vs ALLSELECTED vs ALLEXCEPT vs REMOVEFILTERS

Learn the differences between similar but different DAX functions that remove filters from the filter context.

Data Engineering Weekly #279

The Weekly Data Engineering Newsletter

Beyond “Taking Responsibility”: Why Leading for Agency is the Future of Modern Management

In recent years, a new word has steadily climbed the ranks of leadership literature and corporate strategy sessions: Agency. While many managers intuitively grasp that it has something to do with employees taking ownership, the concept goes far deeper than simply “taking responsi…

ALL vs ALLSELECTED vs ALLEXCEPT vs REMOVEFILTERS

DAX offers many functions to remove filters from the filter context. In this article, we analyze the differences among the most commonly-used functions. Computing values in DAX is all about understanding how to manipulate the filter context to obtain the…

Announcing the Data Outpost Conference

Data Outpost is a new conference for data and AI builders, coming to San Francisco November 4-5, 2026. The next evolution of Small Data SF: two days of workshops, talks, and real time with the people building the data layer AI depends on.

Catalyst Leadership: You Cannot Optimize the Whole System if Your Inner System Optimizes for Safety

Most leaders are not held back by a lack of competence. They are held back by an inner system that was built for a different kind of world. According to Bob Anderson and Bill Adams at Leadership Circle, their research suggests that 70-80% of leaders lead reactively. That does not…

CLUSTER BY in Fabric Warehouse: when the magic shows up and when it doesn’t!

Microsoft's announcement post for data clustering in Fabric Warehouse promised 24x speedups. So I built a 100-million-row clickstream table and tested it myself. The result is more nuanced than the marketing - and more useful for anyone deciding whether to actually adopt this fea…

Developer Dad Does Disney: Vacationing in the Personal Software Era

Waiting in line is no fun! While standing in line at Disney World, a developer dad built a custom data app to pick the perfect next ride. It combines live APIs, historical wait times, and weather data into personal software with MotherDuck's MCP, Dives, and Flights.

When Planning Should Become A Shared Problem

Turn planning pressure into a shared conversation about options and tradeoffs.

Introducing user-aware calculated columns in Power BI

User-aware calculated columns are not materialized: we can use them as virtual calculated columns for localization and for custom security scenarios. This feature is available through the new Expression Context property of calculated columns in Power BI.

DuckDB Ecosystem Newsletter : July 2026

July's DuckDB Ecosystem Newsletter: a portable R2 + DuckLake analytics stack, why PostHog rebuilt on DuckDB over ClickHouse, DuckCon #7 and the 1.5.4 / 1.4.5 releases, a SedonaDB WASM build for spatial SQL in the browser, and more.

How the Story Critic AI Skill Helps Teams Write Better Backlog Items

See how AI coaching can help teams write clearer, smaller, more testable backlog items.

Introducing user-aware calculated columns in Power BI

This article describes the new Expression Context property of calculated columns in Power BI, explaining how user-aware calculated columns work, why they are not materialized, and how to use them as virtual calculated columns for localization and custom security scenarios.…

Self-hosting DuckDB: the road to production

A practical map of self-hosting DuckDB, from a laptop to a full data platform: the architectures people actually build at each stage, benchmarks that show the trade-offs, and the boxes you end up maintaining, so you can make an honest build-versus-buy call.

Why Smart Teams Overcommit And How Leaders Make It Worse

Understand how leader pressure can push smart teams into unrealistic commitments.

Hot, Cold, Gone: Caching and Retention in Microsoft Fabric Eventhouse

Two policies, three storage states, one rule connecting them. A newcomer's guide to caching, retention, and tiered storage in Microsoft Fabric Eventhouse. The post Hot, Cold, Gone: Caching and Retention in Microsoft Fabric Eventhouse appeared first on Data Mozart.

Data Engineering Weekly #278

The Weekly Data Engineering Newsletter

The Cost of Change Curve Is Outdated

The cost-of-change curve isn’t what it used to be—modern tools, agile practices, and AI have dramatically flattened it.

Materialized Lake Views in Microsoft Fabric: When your medallion fits in a SELECT statement

Five surfaces collapsed into one declarative layer. Here's the full story of Materialized Lake Views in Microsoft Fabric - from syntax to the new GA capabilities announced at FabCon Atlanta The post Materialized Lake Views in Microsoft Fabric: When your medallion fits in a SELECT…

Beyond Redaction: Anatomy of a Privacy-Safe Data Platform

Why privacy engineering is about governing data in use—not simply hiding it.

Leading Complexity 2025 — Just Two Weeks Away!

We’re just two weeks away from the launch of Leading Complexity 2025, and we couldn’t be more excited to welcome the amazing group of participants who’ve signed up from around the world. This year marks the fourth season of the program — and the theme feels more urgent and releva…

A whirlwind tour on User-context-aware calculated columns in Power BI!

Bye, bye translation headaches! With the new User-context-aware calculated columns, we can tackle certain challenges more convenient than ever The post A whirlwind tour on User-context-aware calculated columns in Power BI! appeared first on Data Mozart.

DuckLake Architecture Deep Dive

DuckLake is a next generation data lakehouse and open table format. It is significantly simpler and faster than Apache Iceberg or Delta Lake because it uses a SQL database for storing the catalog and metadata. This post covers an architecture deep dive with a detailed Q&A section…

How to Lead—and Not Be Led—by AI in Complexity

A conversation with long-time Crisp collaborator Jonathan ReamsSummary (Abstract) AI is accelerating our tools—and our context. In this Leading Complexity Podcast episode, Jonathan Reams argues that leaders must grow their inner capacity at least as fast as AI evolves, or risk be…

Highlights from FabCon and SQLCon 2026: Where databases and Fabric come together

From technical sessions to hallway conversations, FabCon and SQLCon 2026 showcased the momentum behind the Fabric and SQL ecosystems. The post Highlights from FabCon and SQLCon 2026: Where databases and Fabric come together appeared first on Microsoft Fabric Blog.

Data Engineering Weekly #277

The Weekly Data Engineering Newsletter

Trying Out Lovable – A New Way to Build Websites with AI

Recently, I explored a new tool called Lovable. It’s an AI-powered platform that helps you build websites and web apps just by describing what you want. You write a short description (called a “prompt”), and Lovable builds the code and design for you. As someone who works with Ag…

FabCon and SQLCon 2026: Unifying databases and Fabric on a single data platform

We're bring attendees together to share real experiences and solve challenges side-by-side. Only together can we move into meaningful results. The post FabCon and SQLCon 2026: Unifying databases and Fabric on a single data platform appeared first on Microsoft Fabric Blog.

Announcing leetdata.ai — A Practice Ground for Data Engineers

Software engineers got a gym. Data engineers got production. It's time to fix that.

Leveling Up Leadership: A 5-Year Journey Boosting Engagement at a Game Studio

How a long-term leadership program helped build stronger leaders, increase engagement, and boost collaboration across teams. The Challenge: When Engagement Slipped, Leadership Stepped Up When the executive team at a large game studio noticed a dip in employee engagement in their…

Advancing agentic AI with Microsoft databases across a unified data estate

Built on a consistent Microsoft SQL foundation from on premises to the cloud, Azure SQL brings AI capabilities directly into your database experience. The post Advancing agentic AI with Microsoft databases across a unified data estate appeared first on Microsoft Fabric Blog.

Creative Dagster Projects from the Community

From public dataset explorers to infrastructure monitoring and research automation, the Dagster community is building projects we never could have predicted. Here are a few creative community-built use cases that caught our attention.

Leading Complexity – Year 4: Navigating the Age of AI (and Everything Else)

Three years ago, we launched the Leading Complexity Program with a simple but urgent question: How do we lead when the world is changing at an exponential pace? Organizations everywhere were feeling the pressure of complexity—messy systems, rapid change, and no playbook for how t…

Unpacking your top questions on agentic AI: The Shift podcast

We're launching a new podcast called "The Shift," which does exactly that. Grounded in questions we heard from you after Ignite news, we're releasing eight episodes that bring engineering, product, and strategy perspectives together. The post Unpacking your top questions on agent…

Data Engineering Weekly #276

The Weekly Data Engineering Newsletter

Mob Programming with AI: Inside a High-Performing Team’s Journey

Introduction What happens when you combine a passionate development team, a dash of Tesla-inspired thinking, and the principles of Mob Programming, with AI as an active participant? At Traton-Scania, the AI-Enablement team is doing just that, transforming how they build software…

Top sessions you won’t want to miss at the Microsoft Fabric Community Conference 2026

The Microsoft Fabric Community Conference (FabCon) is returning stateside in just a few weeks, taking place at the Georgia World Congress Center in Atlanta, March 16 - 20, and it’s shaping up to be our biggest event yet. The post Top sessions you won’t want to miss at the Microso…

Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations

Three Strategies for Leading in Complexity

Summary of my talk from Agile Sverige 2025 A few years ago, I heard the quote: That felt true then—and it’s even more true now. After co-hosting the LeadingComplexity.com program for several years with over 30 world-class thought leaders on the topic of leading in complex environ…

FabCon and SQLCon: From workshops and keynotes to demos and deep dives

FabCon and SQLCon will be one of the year’s largest gatherings of data professionals from around the world, bringing together tech enthusiasts, innovators, and industry leaders to explore the future of data, SQL, analytics, business intelligence, and AI integration. The post FabC…

The Dagster Almanack: From Complexity to Composability

How we deploy confidently with Canary Deployment

When you push to production multiple times a day, the stakes are high — and manual rollbacks slow. Here’s my experiences from deploying services using Canary Deployments for an important backend service for a large global streaming service. Why Canary Deployments? We adopted Cana…

You Don't Graduate From Data Engineering: Why We Built aide for Continuous Learning

Introducing aidataengineer.io — an AI-powered learning platform built on 275+ editions of Data Engineering Weekly.

1-2-4-All and Meetings People Actually Want to Attend

Have you ever found yourself stuck in a meeting where one person does all the talking… and everyone else just tunes out? Or where it’s so open-ended that nothing really gets done? Even with smart people in the room, the best ideas often never get heard. That’s precisely the probl…

Data Engineering Weekly #275

The Weekly Data Engineering Newsletter

Unlock Your Team’s Full Potential: Dive into a Liberating Structures Immersion Workshop

Are you ready to revolutionize the way your team collaborates, innovates, and solves problems? Welcome to the world of Liberating Structures, where conventional meeting formats are transformed into dynamic, inclusive, and highly productive sessions. ## What Are Liberating Structu…

The Orchestration Maturity Model: Why Teams Move from Jobs to Assets

Most orchestration systems answer questions about jobs, but organizations need answers about data. This post introduces the Orchestration Maturity Model and explains why enterprises adopt Dagster to move from job-centric orchestration to asset-aware data platforms.

Align Your Strategies

In the past few weeks, I’ve received similar questions from two different companies, which I’d like to reflect on here. Both organizations were in the process of crafting strategies for critical areas: one for AI and the other for cybersecurity. Their question to me was whether i…

Data Engineering Weekly #274

The Weekly Data Engineering Newsletter

Why your teams aren’t really empowered (and how to fix It)

What actually empowers a team? Despite all the talk about empowered teams in tech, there's often confusion about what real empowerment looks like. At its core, an empowered product team is given significant customer or business problems to solve, rather than being handed features…

How to Orchestrate dbt with Dagster

With Dagster’s dbt integration, run and monitor dbt models as part of a larger, asset-driven pipeline for improved lineage and scheduling.

If You Lived in Germany in 1939, What Would You Do?

Imagine the following: You live in Germany in 1939. Or perhaps it’s 1938, or even as early as 1935. The Nazis are in power. If you’re an average German, your material conditions are probably quite good. Unemployment is low, and the Nazis have introduced extensive “welfare program…

Data Engineering Weekly #273

The Weekly Data Engineering Newsletter

How Brooklyn Data Uses Compass for Self-Service Analytics in Slack

Brooklyn Data shares how it uses Compass to make operational and financial data accessible in Slack, reducing time-to-answer, improving data discoverability, and enabling governed self-service analytics across the organization.

Data Engineering Weekly #272

The Weekly Data Engineering Newsletter

Snowflake Runs Your Data. Dagster Runs Everything Else

Snowflake powers modern data platforms, but orchestration still matters. Learn how Dagster complements Snowflake with SQL-native assets, Dynamic Table lineage, virtual assets, downstream automation, and Snowflake cost attribution.

We Tried ty for Performance. It Found Real Bugs

Dagster replaced Pyright with Astral’s new Python type checker, ty, and saw CI type checking drop from minutes to seconds. Along the way, ty surfaced real runtime bugs Pyright missed and improved developer feedback loops across a large Python monorepo.

Announcing the Dagster+ Terraform Provider

Manage Dagster+ deployments, permissions, and alerting as code with the new Terraform provider. Bring consistency, reviewability, and automation to your data platform.

Why AI Agents Need Operational Context — Not Just Semantic Definitions

Semantic context tells AI agents what data means. Operational context tells them whether to trust it. Learn how Dagster and Atlan combine both halves of the context layer.

Databricks Cross-Workspace Orchestration with Dagster

Lakeflow Jobs can't see across workspace boundaries. Learn how Dagster unifies multiple Databricks workspaces into a single asset graph with real dependencies.

Dagster 1.13: AI Skills, Partitioned Asset Checks, Virtual Assets & 20+ New Components

Dagster 1.13 makes the framework easier to adopt with AI coding skills, partition-aware asset checks, virtual assets for database views, and 20+ new integration components.

Monorepos, the hub-and-spoke model, and Copybara

How we use Copybara to build a hub-and-spoke model of public and private git repositories

Making Dagster Easier to Contribute to in an AI-Driven World

As AI makes it easier to generate pull requests, reviewing them has become the bottleneck. Learn how Dagster is improving contributor experience with better tooling, clearer guidelines, and a focus on high-impact contributions.

dbt Fusion Support in the Dagster dbt integration

How we added support for the dbt Fusion engine in the Dagster dbt integration