When I joined the Manufacturing Data Platform team at Microsoft, we were ingesting IoT data for customers with 330,000 nodes and 450,000 relationships. The pipeline worked — but it was slow, expensive, and had one major bottleneck hiding in plain sight: Azure Digital Twin (ADT).
This is the story of how I removed it, what I learned, and the numbers we ended up with.
The Problem: ADT Was Doing Too Much
Azure Digital Twin was sitting as an intermediate layer in the ingestion pipeline. Every node and relationship we ingested had to pass through ADT before landing in Azure Data Explorer (ADX), which was our actual query and analytics layer.
ADT was also the most expensive resource in the entire solution — customers were paying significantly for it. But more importantly, it was slowing us down in ways that were getting harder to ignore as customer datasets grew:
- Node ingestion for 330k nodes was taking 18 minutes
- Relationship ingestion for 450k relationships was taking 39 minutes
The Hypothesis
We had already been storing everything in ADX anyway. ADT was essentially a pass-through that added latency and cost without providing meaningful analytical value in our architecture. The hypothesis was simple: what if we wrote directly to ADX and skipped ADT entirely?
I'd done a prior POC showing ADX was 5–6x faster than ADT for graph queries. That gave us confidence the data model could live entirely in ADX. Now the question was whether we could redesign the ingestion flows to match.
The Work
There were three distinct ingestion flows to redesign: node creation, relationship creation, and OPC-UA data ingestion. Each had its own quirks.
Node Ingestion
Straightforward to redesign — instead of creating twins in ADT then syncing to ADX, we wrote directly to ADX property tables. The main work was mapping the DTDL schema correctly and ensuring the batching logic was preserved.
Relationship Ingestion — Where the Real Gain Was
This is where the 54% improvement came from. The original flow was processing relationships one batch at a time through ADT, which involved multiple round trips. The key fix was modifying the GROUP BY clause logic to batch-process multiple twins in a single ADX call.
Instead of: for each relationship → call ADT → sync to ADX
We did: group relationships → single batch call to ADX
This eliminated the relationship ingestion bottleneck almost entirely.
Infrastructure and Configuration
I also updated the Bicep IaC templates to reflect the new architecture — ADT resources were removed, ADX configuration was updated. And I tuned configuration parameters (RequestBatchSize, PollingInterval, partitionCount) based on benchmarking data I had collected earlier.
Validation
Before shipping, I validated correctness across all 330k nodes and 450k relationships — zero validation errors. The concern was always: does removing ADT break anything downstream? Answer: no, because ADX was already the source of truth for all queries.
Results
18 min → 11 min
39 min → 18 min
What I'd Tell Someone Doing This
- Benchmark first. The ADT vs ADX POC data was what gave us confidence to commit to the redesign. Without numbers, it's just an opinion.
- The bottleneck is usually in the batching. The 54% gain on relationships came from a GROUP BY change, not a fundamental architectural overhaul. Often the biggest wins are in how you batch calls, not which service you call.
- Question your intermediary layers. Every intermediate service adds latency, cost, and failure modes. If it's not providing unique value, it's a liability.
- IaC matters at the end. The Bicep templates had to reflect the new architecture cleanly — otherwise you end up with orphaned resources and customer confusion on the next deployment.