AI IntelligenceAI-AGENT.NEWS

AI Engineering Topics

Local Models and Inference

Local inference runs models on developer-controlled or self-hosted hardware instead of relying only on a managed remote API.

161 matching public evidence items

All Topics · Models and Ecosystem

How it works

A model is selected, compressed or quantised for available hardware, served through a local runtime and evaluated for the intended workloads.

  1. Select model
  2. Fit hardware
  3. Serve locally
  4. Measure quality

Key concepts

Quantisation
Reducing numerical precision to lower memory and compute use.
Inference server
Software that loads a model and serves generation requests.
On-device
Execution on the user's own workstation or edge device.

Engineering risks

  • Small or compressed models may lose important capability.
  • Hardware-specific results may not generalise.
  • Self-hosting transfers patching and security responsibility to the operator.

Aggregated from the site

Evidence and developments

Matching public material uses the same topic rules as AI Engineering.