01 / Interfaces
◆LibreChat
◆Open WebUI
◆API clients
← Current Platform / Alpha-5How Alpha-5
Architecture / local AI platform
How Alpha-5
works.
Alpha-5 is not a chatbot and it is not an RTX 5090 waiting for prompts. It is a set of boundaries that turn local inference into a system people and agents can depend on.
32 GB VRAMPrivate inferenceDurable contextInspectable tools.
01 / System map
System map
A request crosses four deliberate boundaries. Interfaces do not own inference, models do not own memory, and tools do not get hidden inside prompts.
02 / Context
◆Session state
◆Retrieved memory
◆Search results
03 / Reason + act
◆Local model
◆Agent loop
◆MCP tools
04 / Platform
◆vLLM
◆Data services
◆Telemetry
02 / Memory
Memory
Memory is durable application state, not a longer chat transcript. Different kinds of context have different lifetimes and should be retrieved intentionally.
01
Working
The active request, recent turns, and tool results needed for the current loop.
02
Retrieved
Relevant facts and documents selected for this request rather than loaded by default.
03
Durable
Facts, preferences, and operational history stored outside the model and available across sessions.
03 / Inference
Inference
vLLM is the serving boundary. It owns model loading, token generation, and GPU utilization; callers should not need to understand CUDA or model process lifecycle.
Request path
01OpenAI-compatible request
02Capacity + context boundary
03vLLM model server
04RTX 5090 / 32 GB VRAM
05Streamed response
04 / Agents
Agents
The agent is a controlled loop around the model: assemble context, ask for a decision, execute an allowed tool, return evidence, and stop under explicit limits.
01
Assemble context
02
Model proposes
03
Policy checks
04
MCP executes
05
Evidence returns
The model proposes. Code authorizes. Tools act. Durable state remembers.
05 / Scheduling
Scheduling
Alpha-5 currently runs as dedicated compute, so scheduling is a capacity policy rather than a cluster scheduler. The architectural job is still the same: protect interactive work from background demand.
Interactive
Chat and direct tool use
Policy / Latency first
Background
Embedding, ingestion, extraction
Policy / Throughput first
Maintenance
Model changes and index work
Policy / Explicit window
06 / RAG
RAG
Retrieval is a pipeline with observable decisions, not a vector database checkbox. Documents become chunks; chunks become candidates; candidates become bounded evidence.
01
Ingest
02
Normalize
03
Chunk
04
Embed + store
05
Retrieve + rank
06
Ground prompt
pgvectorMeilisearchDocument sourcesCitations
07 / Observability
Observability
The system needs to explain both platform behavior and AI behavior. Host health without request quality is incomplete; model output without resource pressure is equally incomplete.
Platform
GPU utilization · memory · process health
Inference
Queue time · tokens · latency · failures
Retrieval
Candidates · scores · sources · empty results
Agents
Tool calls · loop depth · policy denials
08 / Failure handling
Failure handling
The objective is not to pretend local infrastructure never fails. It is to keep failures bounded, visible, and recoverable without corrupting the context around them.
| Failure | Visible effect | Design response |
|---|---|---|
| Model server unavailable | Requests cannot complete | Keep interfaces and state independent; restart serving without losing durable context. |
| Retrieval degraded | Answers lose grounded context | Fail visibly and prefer an explicit ungrounded state over fabricated confidence. |
| Tool call fails | Agent cannot complete an action | Return typed failure to the loop; preserve the conversation and tool evidence. |
| GPU pressure | Latency rises or allocation fails | Bound concurrency, expose saturation, and protect the host from cascading work. |
| Telemetry unavailable | The system becomes harder to explain | Do not block inference, but mark the observability gap as an operating incident. |
09 / Future roadmap
Future roadmap
The roadmap graduates Alpha-5 from a capable local stack into a more explicit personal AI operating system. These are directions, not claims about what is already shipped.
Now
- ◆Harden serving boundaries
- ◆Document recovery paths
- ◆Publish safe operational signals
Next
- ◆Automated workload admission
- ◆Retrieval evaluation
- ◆Versioned memory policies
Later
- ◆Multi-model routing
- ◆Scheduled agent workflows
- ◆Portable inference capacity
Architecture is the product behind the product.