How it works
A model is selected, compressed or quantised for available hardware, served through a local runtime and evaluated for the intended workloads.
- Select model
- Fit hardware
- Serve locally
- Measure quality
Key concepts
- Quantisation
- Reducing numerical precision to lower memory and compute use.
- Inference server
- Software that loads a model and serves generation requests.
- On-device
- Execution on the user's own workstation or edge device.
Engineering risks
- Small or compressed models may lose important capability.
- Hardware-specific results may not generalise.
- Self-hosting transfers patching and security responsibility to the operator.
Aggregated from the site
Evidence and developments
Matching public material uses the same topic rules as AI Engineering.
public GitHub discoveries
hwdsl2/embedcrate
public GitHub discoveries
felladrin/MiniSearch
public GitHub discoveries
lucasmacori/2048-ai-agent-benchmarks
public GitHub discoveries
medhiclb/HelixAI
public GitHub discoveries
CacinieP/SMAB
public GitHub discoveries
Whq02/MercuryCLI
public GitHub discoveries
Parasaran-Python/colab-agent-bridge
public GitHub discoveries
Muesli-HQ/muesli
public GitHub discoveries
flashback7766/FlashAgent
public GitHub discoveries
tphakala/birdnet-go
public GitHub discoveries
EssaAlshammari1/AI-Assistant-for-NWC-customers
public GitHub discoveries
huzjie/loopforge
public GitHub discoveries
casonk/crew-chief
public GitHub discoveries
philbert440/rivetOS
public GitHub discoveries
NEUIR/Vis-Skill
public GitHub discoveries
OmniJev/OneJev
public GitHub discoveries
k-k1/agent-fleet
public GitHub discoveries
WikiTeq/rag-of-all-trades
public GitHub discoveries
Cocolalilal/LastChat
public GitHub discoveries
tijs/local-model-bench
public GitHub discoveries
intellibitz/susi
public GitHub discoveries
bhakthan/tokn-dist
public GitHub discoveries
Fandrarista0/opencode-llama-local-agent
public GitHub discoveries
vybestack/llxprt-code
public GitHub discoveries
drowzeys/keys-ghost-machine-audio-visual-agent
public GitHub discoveries
BugraAkdemir/memo
public GitHub discoveries
chanto-sh/chanto
public GitHub discoveries
toasterbook88/Axis
public GitHub discoveries
kekzl/imp
public GitHub discoveries
araray/llmcore
public GitHub discoveries
openbrowse-ai/openbrowse
public GitHub discoveries
rl337/mechaharness
public GitHub discoveries
Ilyaberdar/local-cognitive-AI-system
public GitHub discoveries
laurentftech/pi-outpost
public GitHub discoveries
mikeholownych/nexus-code
public GitHub discoveries
nicedreamzapp/browser-agent
public YouTube video
This Free AI Memory Skips the LLM and Still Hits 96.6% Recall
public GitHub discoveries
luxdotdev/polaris
public GitHub discoveries
fengshihao/agent1
public GitHub discoveries
dohwi/pipaya
public GitHub discoveries
bharathvbcr/Manvi
public GitHub discoveries
StarTrail-org/LEANN
public GitHub discoveries
santhoshsathish94/clover-framework
public GitHub discoveries
kennethlaw325/awesome-llm-knowledge-systems
public GitHub discoveries
kyleslight/shun
public web article
Granite Speech 5.0: Transcribe, Translate, and Attribute more Accurately and Faster
public web article
Kimi K3.1 Pro launches at 4th on the Intelligence Index and will lead open weights models
public YouTube video
Introducing TorchTPU: Native PyTorch on Google Cloud TPUs
public GitHub discoveries
AmartyaRanjan/Draco
public GitHub discoveries
dflippojr/agent-harness
public GitHub discoveries
HSA-AI/HSAAI
public GitHub discoveries
T0mSIlver/working-set
public GitHub discoveries
DarkFoot101/EdgeMind
public GitHub discoveries
lusipad/browser-agent
public web article
Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
public GitHub discoveries
bigduu/Bamboo-agent
public GitHub discoveries
gerardogrisolini/ZenCODE
public GitHub discoveries
gonicolas12/My_OS
public GitHub discoveries
jonfairbanks/local-rag
public GitHub discoveries
iDiagoValeta/localOllamaRAG
public GitHub discoveries
chrisws/hali
public GitHub discoveries
devoidfury/hotdog
public GitHub discoveries
YamineRL/oc-fusion
public GitHub discoveries
xiaozhe7772222/tianxuan
public GitHub discoveries
AgentiLoop/Agent
public web article
Backpropagation on an Inference-Only NPU: Pretraining VKORE-46M-NPU From Scratch
public GitHub discoveries
rellytechgame/Reverse3D-tool
public GitHub discoveries
superlinked/sie
public GitHub discoveries
techrian/mixed-gpu-llm-lab
public GitHub discoveries
Star-Code-Prod/Star-Code
public GitHub discoveries
nedonatelli/sidecar
public GitHub discoveries
LimiNode/agent-memory-cpp
public GitHub discoveries
afjcjsbx/picoclaw
public GitHub discoveries
Jozkah/flint
public GitHub discoveries
iyulab/FluxImprover
public GitHub discoveries
anush008/fastembed-rs
public GitHub discoveries
Fortemi/fortemi
public GitHub discoveries
SneppX/sneppx-alg
public GitHub discoveries
gonicolas12/My_AI
public GitHub discoveries
alexvilelabah/bah-browser
public GitHub discoveries
bbvch-ai/aihub-core
public GitHub discoveries
MariaJastrzebska/edge-agent-evaluation-research
public GitHub discoveries
deskmind-ai/deskmind
public GitHub discoveries
bigduu/Bodhi-AI
public GitHub discoveries
mrveiss/AutoBot-AI
public GitHub discoveries
tinyhumansai/tinyinference
public GitHub discoveries
David2024patton/Mimir
public GitHub discoveries
sepehrtaji-dev/Nova-agent
public GitHub discoveries
mpieniak01/Venom
public GitHub discoveries
sydches/yuj
public GitHub discoveries
muneebkhan08/Cuse
public GitHub discoveries
tejas-sarnaik/ProPEX-RAG
public GitHub discoveries
LearningCircuit/local-deep-research
public GitHub discoveries
mojolearn/mojolearn
public GitHub discoveries
hrabanazviking/RuneForgeAI-Project-Aesir
public GitHub discoveries
c1t1zen1/markgitup-research-desk
public GitHub discoveries
guaardvark/guaardvark
public GitHub discoveries
savant0x/savant-code
public GitHub discoveries
TedHaley/pensieve
public GitHub discoveries
fswolf/luna-local-agent
public GitHub discoveries
xorbitsai/inference
public GitHub discoveries
rongxinzy/RongxinAI
public GitHub discoveries
LIXBON-FOUNDER/Lixbon
public web article
Reflection has released Beam, their first model launch
public web article
NVIDIA Releases Nemotron 3.5 Super VL and Nemotron 3.5 Bolt 74B A9B
public GitHub discoveries
Thisen-Ekanayake/Athena
public GitHub discoveries
SKaiNET-developers/SKaiNET
public GitHub discoveries
evanwtf/local-llm
public GitHub discoveries
Meusehorseradishroot8810/edge-browser-agent
public YouTube video
My Coding Agent Took 19 Seconds to Say Anything
public YouTube video
I Built My Own AI Coding Agent (100% Local & FREE)
public GitHub discoveries
lschaffer/tealkit
public YouTube video
Splash on Apple Silicon: 74 Tok/s, But Is Your Coding Agent Faster?
public GitHub discoveries
mohamedhabila/Kavi
public GitHub discoveries
sebastianrcnt/atto
public GitHub discoveries
majorcontext/harness
public GitHub discoveries
andyrewlee/awesome-system-one
public GitHub discoveries
Acekorneya/POK-Agent
public YouTube video
Local Coding Agent: Weak Model? No, Bad Plumbing
public GitHub discoveries
Peuqui/AIfred-Intelligence
public GitHub discoveries
thetahealth/mirobody
public GitHub discoveries
sonos/tract
public GitHub discoveries
hertz-ai/HARTOS
public GitHub discoveries
dm1try/samagotchi · v0.40.0
public GitHub discoveries
hertz-ai/HARTOS · nightly-b1d46f5-37699397483
public web article
Mistral has released Mistral Large 4, making France home to the most intelligent model outside the US and China
public GitHub discoveries
powerpuff-kitty/embedded-AI
public GitHub discoveries
dreamference/mightling
public GitHub discoveries
abhayzangir1/KIN
public GitHub discoveries
Mingbird/Mingbird-agent
public web article
Data-center AI, now on a laptop — POCKET-Darwin-180B
public web article
autotrust/JEV-27B: fast, calibrated decisions and full reasoning from one open model
public web article
Qwen3.8-27B-pi: Effort-Ordered Reasoning for Agentic Coding
public web article
Pre-training a 1.11B LLM on a 6 GB Laptop GPU — Measured, Not Claimed
public web article
Laya AI Model: How It Works, Run It Locally, and Evaluate It
public web article
The DGX Spark Handbook
public web article
AA-AgentPerf-Local: Benchmarking local AI agents on laptops and workstations
public YouTube video
Can a 16GB M6 Mac Mini Really Handle Local AI?
public web article
Jev ai vs djev vs Laya vs OpenJev vs SemIf: Which Decision Model Should You Use? sora-2 • 2 days ago • 7
public web article
Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization
public web article
BananaMind 2 Pro: We've (almost) matched SmolLM2 at 20x fewer tokens... Trained On a 5070 Ti Banaxi-Tech • 6 days ago • 6
public web article
Optimum-Intel v2.2.0 & OpenVINO GenAI 2026.4.0: What's New echarlaix • 6 days ago • 8
public GitHub discoveries
ashworks1706/SparkyAI · v0.3.0
public web article
Ant Group releases finance-focused Ling-3.0-flash-Fin
public web article
适配器即模型 —— ShadowPEFT 现已集成到 🤗 PEFT 库 shadow-llm • 5 days ago • 16
public web article
What If the Adaptation Were a Model? ShadowPEFT in 🤗 PEFT library shadow-llm • 6 days ago • 23
public web article
Same bytes, closer to the original: two lines of AutoRound we had wrong FINAL-Bench • 1 day ago • 12
public web article
TranslatePsy-AfriSLM: Optimized Machine Translation for African Languages qvac • 16 days ago • 14
public web article
KV Caching Explained: Optimizing Transformer Inference Efficiency not-lain • Jan 30, 2025 • 418
public web article
Making open-source AI weather forecasting models easy to run hugging-science • 16 days ago • 38
public web article
Per-tensor layout maps for GGUF quantization bartowski • 7 days ago • 37
public YouTube video
Run a $10,000 AI Model at Home, Here’s How
public YouTube video
Magnitude: Free Local Models for 8 AI Coding Agents
public YouTube video
Qwen3.8-27B: 17GB, One File, and Your OpenAI Invoice Is Cooked
public YouTube video
IBM Granite 4.2 Just Broke OpenAI's Pricing
public YouTube video
I Tested Ornith's New 35b MoE on Mac — Here's How It Goes
public YouTube video
Qwen 3.8 27B Is a Game Changer for Web Design
public YouTube video
Top 10 AI Repos You Should Know
public YouTube video
How to Run AI Locally in 2026 (Full Beginner Guide)
public YouTube video
Unsloth's New Qwen Quants Just Dropped, And...
public YouTube video