Private AI routing
Model routing can improve resilience, quality, cost, and customer choice. It does not automatically improve privacy.
The wrong design sends a sensitive prompt to several providers or reconstructs it at a central router. That increases the number of entities that can see or correlate it.
The right design routes by a local privacy policy before any data leaves the customer’s device.
Direct answer on spreading requests
Fallback or cascade
Provider A or Provider B receives the whole prompt. This can improve availability, but every possible provider expands the vendor, contract, telemetry, breach, and lawful-access surface.
Semantic splitting
Sending names to one provider, a summary to another, and a final question to a third is not a cryptographic privacy system. Fragments can reveal the sensitive fact, and timing, account, IP, prompt structure, and final output can correlate them.
It can be local data minimization when carefully designed. It must not be marketed as anonymity or MPC.
Secret sharing
Cryptographic shares can protect an input only when the compute parties run an MPC or FHE-compatible inference protocol. Normal commercial LLM APIs cannot answer from unintelligible shares.
Central reconstruction
If the router reassembles the full prompt, the router becomes the main trust boundary. This merely moves the privacy problem.
The good routing architecture
The route selector runs locally:
flowchart TB Start([incoming task]) Classify[Classify task and data sensitivity] Retrieve[Retrieve documents and embeddings locally] Offer[Offer a local model first] Local[Local route and local receipt] Attest[Offer an attested confidential model] Confidential[Confidential route and attested receipt] Fail[Fail closed: never silently downgrade to cloud] External[External route with visible provider, region, retention] Receipt([local privacy receipt: route, model, attestation, egress policy]) Start --> Classify --> Retrieve --> Offer Offer -- accepted --> Local Offer -- user opts in for cloud --> Attest Attest -- attestation valid --> Confidential Attest -- attestation fails --> Fail Offer -- explicit external choice --> External Local --> Receipt Confidential --> Receipt External --> Receipt Fail --> Receipt
- Classify the task and data sensitivity.
- Retrieve documents and embeddings locally.
- Offer a local model first.
- If the user opts in, offer an attested confidential model.
- Offer contractual or external models only with visible provider, country, retention, and tool-egress information.
- Never silently fall back from local or confidential mode to ordinary cloud.
The output includes a local privacy receipt:
- selected route and model
- model and runtime version
- attestation verdict, where applicable
- whether web search, retrieval, file upload, or tool calls occurred
- declared provider region and policy category
- egress policy and available audit signal
The receipt should not retain prompt content by default. An attestation does not prove physical jurisdiction or complete absence of network egress; those are contractual, configuration, and audit claims that must be labeled accordingly.
Better trust splits
An independently operated Oblivious HTTP relay and gateway can separate the client network identity from request content. It does not hide content from the gateway and RFC 9458 warns that identifiers, cookies, configuration, and traffic analysis can still correlate requests.
Opaque entitlement tokens, such as Privacy Pass-style designs, can let a service check quota without exposing a central billing identifier to every model endpoint. Unlinked subscription architecture has the same design principle.
These splits reduce specific links. They do not remove payment, support, abuse, tax, or legal obligations.
Tool routing is higher risk than model routing
An agent that searches the web, uploads a document, sends email, calls an API, or invokes an MCP tool can expose more than the model itself.
The local policy layer should:
- default tools to off
- name the destination and data class before a tool receives input
- use read-only, short-lived, scoped credentials
- isolate untrusted retrieved content from system instructions
- require human confirmation for consequential actions
- keep observability and bug reports free of raw prompts by default
Venice’s restrictions in its confidential modes illustrate the honest product tradeoff: web search, file upload, or function calling can break a privacy boundary unless they are redesigned inside it.
Business conclusion
The differentiator is not a giant list of providers. It is a privacy policy engine that makes each remote disclosure intentional, visible, minimal, and verifiable where possible.
Private AI strategy and Private AI services apply this to a commercial product.