Back to the blog

Where Should Enrichment Run in a Fabric Pipeline?

Published Nov 19, 2026

The short answer

Enrichment that lands with the record in a Microsoft Fabric streaming pipeline can run in three places: inside the eventstream, at ingestion through an eventhouse update policy, or in a function behind the stream. The choice turns on how fresh the field has to be. The eventstream's reference data join (in preview as of 11-04-26) enriches the message before it lands, on a processor meter starting at 0.778 CU hours per hour at the Low throughput setting. An update policy joins each ingestion batch on eventhouse compute. A function outside Fabric takes longer and moves the join off your capacity.

A message from a device carries a reading and a device identifier, and almost nothing a person would recognize. Somewhere between the device and the dashboard, a field has to be joined onto it: a company, a site, a machine, an asset class. Where that join happens is a decision with a price attached, and it tends to get made in a sprint, long after the design review.

If you are standing up a real-time pipeline on Microsoft Fabric, the architecture questions that get answered first tend to be ingestion and storage. IoT Hub or Event Hubs. Eventstream or direct ingestion. Eventhouse, KQL database, lakehouse. Those are the questions a design document is built to answer, and they tend to be settled before the first message flows. The questions in this post are the ones that surface later, in the sprint where somebody actually has to write the join. They're easier to leave implicit, and an implicit answer to a cost question is a cost surprise waiting for the capacity report.

In the stream, at ingestion, or behind it: what actually differs

Inside the eventstream At ingestion in the eventhouse In a function behind the stream
When the field appears On the event, before it reaches a destination With the row: data isn't queryable until the policy runs After the message lands, so there's a window with no field
Where the compute is billed Eventstream Processor CU hours on your Fabric capacity Eventhouse UpTime on your Fabric capacity The join leaves Fabric capacity only if the function runs outside Fabric
Failure behavior An inner join drops unmatched events; a left outer join keeps them with empty fields A failed policy lands data in the source table only, unless the policy is transactional The stream keeps flowing; enrichment falls behind independently
Best fit Routing, alerting, or a live view depends on the field Every query needs the enriched row, and the reference table shares the eventhouse Analysis can wait seconds or longer for the field

None of the three is a best practice.

Should enrichment happen inside the Fabric eventstream?

Enrich inside the eventstream when the enriched field is needed to decide what happens to the message next. Routing and alerting fall in that category, as does anything a person is watching in near real time. The cost is that the join runs on the Fabric capacity carrying the stream, so it competes with everything else on that capacity.

Eventstreams now have a dedicated path for this: a reference data join, in preview as of 11-04-26 (Microsoft Learn). It joins the stream to a lakehouse Delta table that can reload on an interval you set. The reason to use it is timing: a field joined in the stream is present the moment the message lands downstream, with no second hop, and it's as fresh as the last reload.

Pick the join type on purpose: an inner join drops any event with no match, and a left outer join keeps it with empty fields. Delta tables with deletion vectors or column mapping aren't supported as reference data.

The reason to hesitate is that capacity in Fabric is a shared, finite pool: its capacity units are shared across all workloads (Microsoft Learn). At the Low throughput setting, the eventstream processor's meter starts at 0.778 CU hours per hour and autoscales with throughput, logic complexity, and partition count (Microsoft Learn). Add the 0.222 flat charge and the floor is 1.0 CU hours per hour, which is 25% of an F4 and about 2% of an F64 for as long as data flows, before data traffic. Data from Event Hubs and IoT Hub already runs through that processor, so on those sources the real question is what the join adds.

When is a function behind the stream the cheaper option?

A function behind the stream can be cheaper when the enriched field serves analysis rather than action. If nothing downstream branches on the joined value, and no human is watching a live tile that depends on it, then the message can land unenriched and a separate process can fill the field in afterward.

This buys two things. The join's compute moves off the Fabric capacity and onto the function's own consumption, which can be a different budget line and a different scaling profile. And the stream stops caring whether the lookup is available: if the enrichment process falls over, telemetry keeps landing, and the backlog gets filled in when it recovers.

Check where the function runs, though. Fabric user data functions bill to your capacity as a User Data Functions Execution operation (Microsoft Learn), so only a function hosted outside Fabric, like Azure Functions, moves the join off the pool (writing the field back still lands on your capacity, Microsoft Learn).

The cost is the window. Between landing and enrichment, the record is in the store and incomplete. Anything querying in that window sees a null where the business key should be. That's fine if every consumer knows about it, and a defect for any consumer that doesn't.

Can the eventhouse do the join at ingestion instead?

Yes, and a two-way comparison leaves it out. An update policy runs a query when new data lands in a table and writes the result to a target table, and Microsoft lists enrichment with dimension tables among its uses (Microsoft Learn). The compute stays on Fabric capacity, billed as Eventhouse UpTime.

The draw is that the null window goes away: data isn't queryable in the source or target table until the policies run. It's probably the strongest default for a field nothing needs in flight, with conditions (Microsoft Learn):

  • Streaming ingestion. It's on by default, and a policy whose function uses the join operator needs it turned off, which means queued ingestion batched for up to 5 minutes by default (Microsoft Learn).
  • Failure handling. A failed policy still lands data in the source table by default. Microsoft recommends IsTransactional: true in production, which fails the ingestion.
  • Timing. The join sees the dimension table as it stands, so a device registered after its first message misses its enrichment (Microsoft Learn).
  • Scope. Source and target tables must share a database, and the policy query can't reach across eventhouses.

How do you size Fabric capacity before you know the workload?

You do not, and designing as though you do is the actual mistake. The workable approach is to start on the capacity you already have, design the workspace layout so workspaces can be moved between capacities without rework, instrument utilization from the first day of live data, and split a workspace out into its own capacity when usage starts pressing against the ceiling.

For a first number, Microsoft offers the Fabric SKU Estimator (in preview) or a trial capacity watched in the Capacity Metrics app (Microsoft Learn).

Fabric supports the split directly: workspaces are assigned to a capacity and can be reassigned (Microsoft Learn). That reassignability is the design property worth protecting. If the workspace boundaries are drawn so that the noisy workload can be lifted out on its own, capacity planning stays a configuration change. If they are drawn so that lifting one thing out means lifting five, it becomes a migration.

A move cancels all jobs tied to items in the workspace, though, and it only stays a configuration change inside one region: eventstreams, eventhouses, and lakehouses aren't on the list of items that can move between regions.

Watch utilization as the signal. Message volume tells you how much is arriving; utilization tells you what it costs to handle, which is the number the decision actually depends on.

Can you join to a table in a different capacity or region?

Yes for an eventstream join or a shortcut read (an update policy can't query across eventhouses), and the blocker people expect turns out to be the wrong one. OneLake authorizes a shortcut read with the caller's identity (Microsoft Learn), so the question is whether the service principal or account doing the reading has read permission at the target, directly or through a security group. The two workspaces sharing a capacity has nothing to do with it.

This matters because the instinct when a needed dimension table turns out to live somewhere else is to assume the architecture is wrong and to plan a copy. The better first step is to check the permission. A copy has to be kept in sync from then on, which a permission grant avoids. A OneLake shortcut can finish the job, and an eventstream can use a shortcut-based lakehouse table as reference data.

There's a second gate now. With workspace outbound access protection on, a lakehouse shortcut to another workspace needs a managed private endpoint or a connector allow list (Microsoft Learn), and, in preview, an eventstream can't send data to items in other workspaces (Microsoft Learn).

Region is a separate consideration, and it lands mostly on data residency and, for external storage, cross-region latency and egress pricing (Microsoft Learn). If the data has a residency requirement, that requirement governs regardless of what the permission model allows.

Should the payload be shaped at the device or after it lands?

Shaping at the device means annotating the message before it is ever published: marking which fields change constantly and which change rarely, so the pipeline can branch on the annotation without inspecting and reshaping every message after the fact. When the device firmware allows it, this can be the cheapest place to solve the problem, because the work happens once per message on hardware you are not billed for by the unit.

The caveat belongs in the same breath as the idea. What can be changed at the device depends entirely on what the device vendor's libraries expose, and that varies by platform. Treat it as a question to answer with a small investigation before it is written into a design, and keep a downstream path that works if the answer comes back no.

Do you need a landing zone between telemetry and the analytics store?

Not always, and this is a step that can end up in a design because it was in the last design. Telemetry can be routed from a Fabric eventstream straight into a KQL database in an eventhouse (Microsoft Learn). Where that works, the intermediate staging layer is a hop that adds cost and latency and buys nothing.

A landing zone earns its place when raw messages have to be retained in original form for replay or audit, or when more than one consumer needs the unprocessed stream. Check those reasons first: one eventstream can feed multiple destinations and retain events for up to 90 days (Microsoft Learn), though keeping them past a day bills as OneLake storage (Microsoft Learn). The test is whether anyone can name what the landing zone is for. If the answer is that it is how these things are built, it is a candidate for removal.

The decisions to settle before the first message flows

Six decisions, none of which requires a running pipeline to answer:

  1. Which enriched fields are needed for action, and which are needed only for analysis. This single split decides which fields have to be enriched in the stream.
  2. Whether anything downstream branches on a joined value.
  3. Whether the workspace layout allows a single workload to be lifted onto its own capacity in the same region without moving anything else.
  4. Whether the service principal already has read permission on every reference table, and whether outbound access protection is on anywhere in the path.
  5. What the device firmware actually permits in the outbound payload.
  6. What a landing zone would be for, in one sentence, with a name attached.

There is a seventh: where the query surface lives. Once the data is landed and modeled, something still has to answer requests from an application, and the options are an endpoint inside Fabric, a function, or a backend hosted elsewhere. That decision sits outside the data architecture and can still be open when the data architecture is considered finished.

If your team is in the middle of this and the capacity question is the one without a good answer, a data readiness assessment is designed to settle exactly that class of question before a build commits to a shape.


Sources and references

All Microsoft Learn, accessed 11-04-26.

Building something similar?

Talk it through with the engineers who write about it.

Contact us