Field Notes

concept

Local AI

StatusStableReviewdue UpdatedStands on5 sourcesSupportsnotes

Local AI means model inference, conversation history, and relevant documents remain on the user’s device or on hardware controlled by the customer’s organization.

It is the best available privacy default because no remote model provider needs to receive raw content. Removing the provider removes one attacker, not the rest: the endpoint, backups, extensions, local logs, model downloads, and network exposure still matter.

Mature enough to start

In 2026, local models are commercially viable for many practical tasks:

  • drafting and rewriting
  • Swedish and English translation
  • summarization and explanation
  • structured extraction
  • local document retrieval and question answering
  • redaction assistance
  • transcription and basic coding support

They are not a universal substitute for the best frontier cloud model. Hardware, battery, memory, context window, multimodal needs, and model quality set the ceiling.

The right product is not “local only at all costs.” It is local by default, with an explicit, auditable cloud boost when a user decides it is worth the tradeoff.

Building blocks

llama.cpp supports quantized local inference across common CPU, GPU, Apple Silicon, and other hardware paths. It can also expose a local, OpenAI-compatible API, which Ollama packages into a simpler local model manager built on the same inference engine.

MLX is a strong technical path for Apple Silicon deployments. On-device platform models from Apple and Android show that local inference is becoming a normal product category, while their context and capability limits still require careful task selection.

Not every hardware story is a local one. The OfficeChai report on Taalas ChatJimmy describes model-specific silicon that embodies one model in custom hardware rather than running it as software on a general-purpose GPU. That can cut latency or operating cost, but a request to a remotely hosted instance still crosses a provider trust boundary; custom silicon changes where inference is fast, not where it happens.

Open WebUI and comparable self-hosted interfaces can make local deployments usable, but strict offline operation needs actual egress control. Disabling a visible cloud toggle is not enough if update checks, model downloads, OAuth, plugins, search, or remote retrieval remain active.

Secure local-workbench design

The first product should use:

  • a signed native client or appliance
  • a model manifest with hashes, license information, and update policy
  • a local encrypted history
  • a local vector index and embeddings by default
  • a local policy sidecar for secrets, sensitive fields, and connector approval
  • bind-to-loopback defaults for local APIs
  • no undisclosed remote-management account
  • customer-controlled recovery material
  • visible network and data-egress indicator

Treat embeddings, chunks, cache files, crash reports, and browser storage as sensitive data. A vector is not harmless merely because it is not readable prose.

Product roles

Personal workbench

Sell a supported private workstation for writers, developers, journalists, lawyers, and people with high exposure. The offer is setup, task benchmarking, model-update hygiene, recovery, and training, not a mysterious “unhackable AI box.”

Team appliance

For a small organization, deploy a customer-owned appliance or self-hosted tenant: local retrieval, approved models, role controls, update policy, and safe document ingestion.

The startup can support the system without holding the customer’s raw documents or a permanent administrator secret.

AI readiness service

Before hardware, offer a fixed-scope assessment: which tasks can move local today, which data must never leave, which cloud use needs consent, and what the organization must test before rollout.

Private AI services defines the commercial packages.

Limits and customer truth

Local AI does not protect:

  • an unlocked, seized, or compromised endpoint
  • careless copy-and-paste into an external chatbot
  • unvetted model files or extensions
  • unsafe local network exposure
  • a weak password, weak backup, or retained plaintext log
  • inaccurate or biased model output

Privacy threat modeling must guide which workload is appropriate for local mode. Private AI routing governs the point where the customer elects to send anything remotely.

Built on 5 sources (5 external).

Working out connections…