feat(local-llm): B0.5 — llama-swap + lokale Modellauswahl in Diagnostic
Mehrere lokale Modelle, on-demand geladen/geswappt, in Diagnostic waehlbar. Design: das Brain schickt den Modellnamen (aus local_llm.json) im llm_request mit -> Adapter -> llama-swap laedt/swappt. Keine separate Gamebox-Config noetig. - xtts: `llama`-Container -> `llama-swap` (unified-cuda), config.yaml mit qwen3-8b (Standard) + qwen3-4b; Auto-Download via -hf, Cache /models geteilt (qwen3-8b schon da). Adapter -> llama-swap:8080, Timeout 600s (Erst-Download). - adapter: `model` aus dem Request an llama-swap durchreichen (Fallback env). - brain: router.load_config liest localLlmModel; local_llm_chat(model=...); agent gibt cfg-Modell mit; bridge reicht model durch (_local_llm + Route). - diagnostic: /api/local-models-list (aus /shared/config/local_models.json, seeded), local-llm-config um localLlmModel erweitert; Dropdown "Lokales Modell" im Settings-Block + Erst-Download-Hinweis. BLIND gebaut (Gamebox nicht testbar hier): llama-swap CLI/Config-Pfad beim ersten Start via `docker logs aria-llama-swap` pruefen. Live-Lade-Status (Adapter->Diagnostic) ist B0.5-2 (Folgeschritt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -69,14 +69,16 @@ async def _send(ws, mtype: str, payload: dict) -> None:
|
||||
|
||||
|
||||
async def _call_llama(messages: list, *, max_tokens: int, temperature: float,
|
||||
stop, tools=None) -> dict:
|
||||
"""Ruft llama.cpp /v1/chat/completions (OpenAI-Format). Gibt
|
||||
stop, tools=None, model=None) -> dict:
|
||||
"""Ruft llama.cpp/llama-swap /v1/chat/completions (OpenAI-Format). Gibt
|
||||
{ok, content, tool_calls, error} zurueck — wirft nie.
|
||||
|
||||
tools: optionale OpenAI-Tool-Definitionen (B1b). llama.cpp (--jinja) mit
|
||||
Qwen3 kann natives Tool-Calling und liefert dann message.tool_calls."""
|
||||
model: welches Modell llama-swap laden soll (B0.5). Kommt aus dem Request
|
||||
(Brain -> local_llm.json). Faellt auf LLM_MODEL (env) zurueck.
|
||||
tools: optionale OpenAI-Tool-Definitionen (B1b). Qwen3 (--jinja) kann
|
||||
natives Tool-Calling und liefert dann message.tool_calls."""
|
||||
body = {
|
||||
"model": LLM_MODEL,
|
||||
"model": model or LLM_MODEL,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
@@ -120,13 +122,15 @@ async def _handle_llm_request(ws, payload: dict) -> None:
|
||||
temperature = float(payload.get("temperature", 0.7) or 0.7)
|
||||
stop = payload.get("stop")
|
||||
tools = payload.get("tools") or None
|
||||
model = (payload.get("model") or "").strip() or None
|
||||
t0 = time.time()
|
||||
res = await _call_llama(messages, max_tokens=max_tokens,
|
||||
temperature=temperature, stop=stop, tools=tools)
|
||||
temperature=temperature, stop=stop, tools=tools,
|
||||
model=model)
|
||||
dt = time.time() - t0
|
||||
tc = res.get("tool_calls")
|
||||
logger.info("llm_request id=%s -> ok=%s %.2fs content_len=%d tool_calls=%d",
|
||||
(req_id[:8] if req_id else "?"), res.get("ok"), dt,
|
||||
logger.info("llm_request id=%s model=%s -> ok=%s %.2fs content_len=%d tool_calls=%d",
|
||||
(req_id[:8] if req_id else "?"), model or LLM_MODEL, res.get("ok"), dt,
|
||||
len(res.get("content") or ""), len(tc) if tc else 0)
|
||||
await _send(ws, "llm_response", {
|
||||
"requestId": req_id,
|
||||
@@ -134,7 +138,7 @@ async def _handle_llm_request(ws, payload: dict) -> None:
|
||||
"content": res.get("content", ""),
|
||||
"tool_calls": tc,
|
||||
"error": res.get("error"),
|
||||
"model": LLM_MODEL,
|
||||
"model": model or LLM_MODEL,
|
||||
"elapsedMs": int(dt * 1000),
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user