The Enterprise AI Stack Will Be Heterogeneous by Design

A company wants to help employees find internal guidance, resolve customer requests, and support technicians working without reliable connectivity. All three initiatives involve AI. They do not present the same architecture decision.

The knowledge service must retrieve current information without crossing access boundaries. The customer workflow must distinguish interpreting a request from authorizing a change to an account. The technician's tool must remain useful when a remote service cannot be reached. Choosing a preferred model leaves most of these differences unresolved.

Enterprise leaders understandably seek standardization. One platform promises fewer contracts, a common security review, clearer ownership, and a smaller skills burden. Those benefits are real. They should influence architecture decisions.

Normatic's view is that mature enterprise AI will nevertheless be heterogeneous by design: different workloads will use different combinations of models, conventional software, data, deployment environments, and human decisions. The important management question is where that variation is justified and how it remains governable.

This is broader than a multi-model strategy. An enterprise can operate a varied architecture largely within one platform. It can also buy from many vendors while reproducing the same unsuitable pattern everywhere. Supplier count is a poor measure of architectural maturity.

Standardization should have a defined scope

The strongest case for a common platform is operational. Procurement can consolidate commitments. Security teams can understand a smaller set of services. Engineers can reuse identity, monitoring, deployment, and support arrangements. Business teams can draw on familiar tools rather than starting each initiative from scratch.

For an organization with similar workloads and limited operating capacity, one platform may remain the right answer. Building alternatives before a material need exists consumes resources and introduces new failure points. A preferred implementation should therefore be a serious default, supported by investment and expertise.

The difficulty arises when the default becomes a requirement to make every task behave alike. A document-drafting service may tolerate a response time that makes a live interaction unusable. A workflow that proposes an action has different obligations from one that executes it. A process involving restricted information may need a boundary that a general employee assistant does not.

Standardize where consistency creates more value than choice. Allow variation where a material workload requirement justifies its full operating cost.

That rule separates platform consolidation from architectural uniformity. It permits common infrastructure while allowing different models, access policies, deployment configurations, and review arrangements. It also permits an exception outside the preferred platform when the business case and operating responsibility are clear.

Define the work before defining its technology

A workload is more than a prompt. It includes the information used, the decisions or actions produced, the people affected, and the conditions under which the service must operate. Quality, latency, confidentiality, integration, and recoverability belong in that definition.

“High quality” also needs interpretation. For an extraction task, it might mean accurate fields and reliable identification of missing information. For decision support, it might mean faithfully presenting conflicting evidence. For a transaction, the decisive requirement may be that the correct action occurs once, within the user's authority, against current records.

These requirements can point toward a hosted frontier model, a smaller model, a specialist model, or an open-weight model in an eligible environment. The categories overlap: an open-weight model can be large or small, specialized or general, and locally or remotely hosted. None of those labels establishes suitability without evaluation.

Enterprises need not resolve every architecture from first principles. They can define broad workload classes with approved patterns, then check whether each proposed workflow fits. This reduces repeated design work without assuming that every use case is identical.

Exhibit 1. Workload classes can guide architecture without prescribing it

Source: Normatic analysis. These classes overlap and are illustrative, not a fixed classification system or automatic technology assignment. A productivity task involving sensitive information must also meet the relevant information requirements.

An approved pattern describes the conditions under which it may be reused. If a knowledge assistant gains the ability to change records, its earlier approval should not silently expand with it. The move from information to action changes the workload and the evidence required for release.

The model is one component of the operating system

The term “AI stack” can draw attention toward model access and the tools around it. Enterprise architecture reaches further. It includes the employee or customer experience, application logic, coordination of steps, authorized context, enterprise connections, and the decisions reserved for people.

Some elements run through the whole system. Identity and permissions determine who can see and do what. Evaluation establishes whether the combined workflow meets its requirements. Monitoring makes failures and changes visible. Governance assigns ownership and determines how releases, incidents, and exceptions are handled.

These are responsibilities to design, not necessarily separate products to buy. A platform may provide several of them. Existing enterprise applications may already perform others. Adding another orchestration product is unnecessary if the current workflow software can reliably coordinate the required steps.

Consider an illustrative customer-request workflow. AI interprets an unstructured message and identifies information that may be relevant. Retrieval supplies current policy and account context within the user's permissions. Conventional software checks required fields, performs calculations, and validates whether a proposed action is allowed. A person resolves ambiguity or authorizes an exception where the consequences require judgment.

The architecture is already heterogeneous even if it uses one model. Its strength depends on allocating work well across these components. Deterministic software provides repeatable behavior, but an incorrect rule can repeat the wrong decision perfectly. Human review adds value only when the reviewer has the evidence, time, expertise, and authority to intervene.

This develops the argument in Why Model Agnostic Wins. Freedom to select a model is one part of a wider freedom to choose how the work is performed. Some improvements require a different model; others require better information, conventional automation, or a changed decision boundary.

Exhibit 2. Different implementations can share an enterprise control foundation

The foundation does not have to mean one physical service processing every workload. Common access rules may need enforcement in both connected and disconnected environments. Monitoring may use different tools while preserving the evidence needed for accountability. Coherence comes from clear obligations and tested behavior across the boundaries.

Deployment is another dimension of fit

A vendor-hosted model, an enterprise cloud deployment, an organization-exclusive environment, and on-premises execution place different responsibilities on the enterprise. They may also overlap. A private environment can be operated by an external provider; open weights can be deployed in public cloud infrastructure.

The relevant decision concerns where information is processed and retained, who can access or administer the service, which capabilities are available, and who can sustain the operation. Locality alone does not establish privacy or resilience. A local component can still depend on remote retrieval, licensing, monitoring, or updates.

Current provider documentation demonstrates that variation exists within platforms. Microsoft's Foundry documentation distinguishes global, data-zone, and geography-based processing arrangements, alongside different capacity and billing options. It also states that model availability varies by deployment type. These are product-specific choices, not a universal recommendation or assurance that a particular requirement is satisfied. Microsoft Foundry deployment types.

The earlier Perspective on sovereign, local, and private AI examined those requirements in detail. The architectural implication is that one deployment policy may legitimately produce several operating arrangements across an enterprise.

Variation must remain affordable to operate. Local execution can introduce hardware, updating, security, and support obligations. A cloud option can introduce service, capacity, and commercial dependencies. The organization should compare the whole cost of a reliably completed unit of work, including review and recovery, rather than only a model's usage price.

Designed heterogeneity needs boundaries

Fragmentation often begins with reasonable local decisions. One team selects a tool that works well for its documents. Another chooses a different service because procurement is faster. A third builds its own connector. Each project advances; the enterprise accumulates duplicated reviews, inconsistent permissions, and components with unclear support.

Designed heterogeneity begins with a shared default and a reason to depart from it. The reason might be a processing constraint, demonstrated task performance, disconnected operation, or economics at meaningful scale. A preference for a new tool is insufficient without a material benefit.

Exhibit 3. Deliberate variation leaves a different operating footprint

Source: Normatic analysis. The distinction concerns decision quality and operating discipline, not the number of suppliers or products.

The practical objective is a small enough set of patterns to understand, support, and improve. That requires an exception process with an owner, evidence, an operating budget, and a review point. A successful exception can become a reusable pattern. An exception whose original justification disappears can be retired or moved back to the standard approach.

This is also a portfolio decision. A local performance improvement must be weighed against the ongoing cost of another platform, another support arrangement, and another set of dependencies. Shared services can make variation cheaper, but only if they meet the participating workloads' requirements without becoming an unmanageable bottleneck.

Routing should earn its place

Using different models for different workloads does not require selecting a new model for every request. A stable assignment can be simpler: one approved model for a defined extraction process and another for complex analysis. Dynamic routing adds a further decision about which model should handle a particular input.

Routing may be valuable when task difficulty varies, candidate costs differ materially, or an approved fallback supports continuity. It also introduces selection errors, additional processing, configuration changes, and a wider evaluation burden. A model that is strong for a task provides no benefit if the router repeatedly sends that task elsewhere.

Commercial routing is a current capability with defined boundaries. Amazon Bedrock documents intelligent prompt routing among supported models within the same family, using predicted response quality and configured criteria. That is one product implementation; it should not be read as unrestricted routing across all providers or proof of savings for a particular workflow. Amazon Bedrock intelligent prompt routing.

Recent research reinforces the need to test the benefit. The January 2026 LLMRouterBench preprint finds complementary strengths across models, but several evaluated routing approaches fail to reliably outperform its best-single-model baseline. Adding models shows diminishing returns. The benchmark does not cover every routing method, specialist domain, long-context setting, or multimodal task, and its latency estimates are approximate. These results challenge automatic claims for routing; they do not establish that routing cannot work. Li and colleagues, LLMRouterBench.

The enterprise decision should compare routing with a credible simpler option under its own conditions. That comparison includes the routing decision, response quality, processing boundaries, latency, retries, review effort, and operating cost. A fallback must meet the workload's permissions and constraints before it can receive information. If no eligible alternative is available, a controlled pause or human route may be preferable to continuing with an unsuitable service.

More providers do not automatically create resilience. Several models may depend on the same infrastructure, gateway, identity service, or enterprise connector. Continuity should be demonstrated through the relevant failure scenarios, not inferred from the number of names in a diagram.

Evaluation gives variation a common basis

A heterogeneous environment needs a shared evaluation discipline. It does not need one universal score that ranks every application. A useful knowledge response and a correctly executed transaction require different evidence.

The common discipline starts with defined outcomes, representative cases, consequential failure scenarios, and explicit acceptance conditions. It tests the full workflow, including its information, permissions, actions, and human responsibilities. Alternatives for the same workload can then be compared on a consistent basis.

That evidence can support several decisions: whether a smaller model is sufficient, whether local execution meets the quality requirement, whether deterministic software performs a step more reliably, or whether further automation would be appropriate. Reducing human review requires evidence about the resulting workflow and a clear allocation of responsibility, not simply a higher benchmark score.

Evaluation must also cross component boundaries. A retrieval service may return an accurate document that the receiving user is not authorized to see. A model may propose a valid action against a record that has since changed. A fallback may preserve answer quality while violating the approved processing boundary. Testing components separately will not resolve those failures.

Operating evidence should keep the comparison current. Changes to source information, policies, models, traffic, or user behavior can invalidate an earlier decision. A shared record of configurations, evaluation results, incidents, and changes makes it possible to distinguish a new opportunity from a regression.

Common governance can permit different implementations

Enterprise governance should establish the obligations that apply across the environment: authorized data use, security, proportionate logging, evaluation, release approval, incident handling, and named ownership. It should specify what evidence is needed to show that those obligations are met.

NIST's AI Risk Management Framework provides a voluntary reference for incorporating trustworthiness into the design, development, use, and evaluation of AI systems. It is governance guidance, not a prescribed enterprise stack or a certification of any implementation. NIST AI Risk Management Framework.

Normatic's interpretation is that common requirements should permit different implementations where equivalent obligations can be demonstrated. This does not mean accepting broad principles without operational detail. A requirement to make consequential actions traceable must translate into usable records and assigned access, retention, and review responsibilities.

Business owners remain accountable for outcomes and the authority granted to the workflow. Platform and engineering teams own the technical services within their scope. Security and governance functions establish and assess the relevant controls. The interfaces between these responsibilities matter more as implementations diversify.

Human judgment is part of this design. A reviewer should receive the evidence needed for a decision, understand the system's limits, and be able to reject or redirect the proposed action. “Human in the loop” is not an adequate operating specification if no one knows what that person must decide.

Build for evolution without making change the objective

The architecture should support a deliberate response when capability, cost, operating requirements, or provider arrangements change. Useful options depend on understandable boundaries between components, access to relevant records and evaluation assets, and a realistic account of replacement work.

Portability is rarely complete. A new model can alter tool behavior. A new deployment environment can change monitoring and support. A shared interface can reduce integration work without preserving every feature or control. These dependencies should be visible when commitments are made.

The enterprise need not replace a working system whenever a newer option appears. It should compare the expected improvement with migration cost, disruption, re-evaluation, and the continuing cost of operating the alternative. Retirement matters as much as introduction: an architecture that can add components but never remove them will become progressively harder to govern.

For leaders, the test is whether the organization can explain both its common foundations and its deliberate differences. Which requirements justify each pattern? What evidence supports it? Who operates it? What would trigger consolidation, replacement, or retirement?

A mature enterprise may operate a portfolio of AI architectures, much of it supported by shared platforms and services. Its coherence will come from disciplined decisions about the work, the controls, and the evidence.

The goal is a governed combination of the right capabilities for the right workloads, with no more variation than the organization can justify and sustain.

Sources

Current technology capability

• Microsoft, Deployment types for Microsoft Foundry Models, updated August 12, 2026; checked September 18, 2026. Product-specific documentation on processing scope, deployment options, and availability. These conditions should be rechecked when making an actual deployment decision.
• Amazon Web Services, Understanding intelligent prompt routing in Amazon Bedrock, checked September 18, 2026. Documentation of a bounded commercial routing capability; not independent evidence that routing improves every workload.

Research and governance reference

• Hao Li and colleagues, LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing, January 2026 preprint. Comparative routing research within a specified benchmark and model set, with limits on task coverage and latency measurement. Findings are not a ranking of today's enterprise deployment options.
• NIST, AI Risk Management Framework. AI RMF 1.0 was released in 2023; NIST's current page states that it is being revised. Used here as a governance reference, not evidence about current model performance or a mandated technical architecture.

Normatic interpretation

The workload classes, three exhibits, enterprise examples, and proposed balance between standardization and variation are Normatic analysis. They illustrate architectural decisions without asserting measured client results, a universal reference architecture, or a requirement to adopt multiple providers.