feat(fleet): Auslastungs-Monitor pro Box — live nvidia-smi + Graphen (Stage E)

Pro Box ein "Auslastung"-Button in der Compute-Flotte → Modal mit live
nvidia-smi (1s), Graphen (GPU-Auslastung + Tokens/Intervall) und Besen-Reset.
Historie liegt auf der Box, Diagnostic holt sie via RVS.

- node_stats.py (identisch in allen 4 Worker-Build-Contexts): Sampler alle 15s
  (nvidia-smi + Token-Delta → Ringpuffer ~500 Punkte, persistent als JSON auf
  der Box), Live-Stream (node_stats, 1s, Auto-Stop 300s), History-Request,
  Reset. nvidia-smi via async subprocess, fail-safe ohne GPU.
- Worker-Wiring (f5tts/whisper/voxtral/llm-adapter): Import, Sampler-Task,
  _stats.handle() nach dem targetInstance-Filter. llm-adapter zaehlt Tokens
  (usage.total_tokens) → Token-Graph nur bei LLM-Boxen. Dockerfiles kopieren
  node_stats.py.
- compose: llm-adapter bekommt runtime:nvidia + NVIDIA_VISIBLE_DEVICES=all +
  DRIVER_CAPABILITIES=utility (nur nvidia-smi, KEIN VRAM/Compute).
- diagnostic/server.js: relay node_stats_* (Browser→Box) + forward (Box→Browser).
- diagnostic/index.html: Auslastung-Button pro Node (Ziel bevorzugt llm-Instanz),
  Modal mit live nvidia-smi + Inline-SVG-Sparklines, Besen-Reset.

Reporter-Wahl bevorzugt die llm-Instanz (sieht alle GPUs + Tokens); GPU-Worker
sehen ihre gepinnte Karte. Gitignored Historie stoert git-Baum der Box nicht.
Deploy: diagnostic + GPU-Boxen neu bauen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-09-19 03:26:17 +02:00
co-authored by Claude Opus 4.8
parent 5c25d6abeb
commit 66781d8d90
15 changed files with 837 additions and 6 deletions
+22 -3
View File
@@ -75,6 +75,13 @@ BASE_CONFIG_PATH = os.getenv("LLAMA_BASE_CONFIG", "/llamaswap/config.yaml")
GEN_CONFIG_PATH = os.getenv("LLAMA_GEN_CONFIG", "/models/llama-swap.config.yaml")
REGISTRY_PATH = os.getenv("LLM_REGISTRY", "/models/aria_models.json")
# ── Auslastungs-Monitor (Stage E) ──────────────────────────
import node_stats
STATS_PATH = os.getenv("STATS_PATH", "/models/aria_stats.json")
_total_tokens = 0 # kumulativ, fuer den Token-Graph
_stats = node_stats.NodeStats(INSTANCE_ID, NODE_NAME, STATS_PATH,
token_getter=lambda: _total_tokens, logger=logger)
def _load_registry() -> list:
try:
@@ -189,11 +196,17 @@ async def _call_llama(messages: list, *, max_tokens: int, temperature: float,
r.raise_for_status()
data = r.json()
msg = (data.get("choices") or [{}])[0].get("message", {}) or {}
usage = data.get("usage") or {}
try:
global _total_tokens
_total_tokens += int(usage.get("total_tokens") or 0)
except Exception:
pass
return {
"ok": True,
"content": msg.get("content") or "",
"tool_calls": msg.get("tool_calls") or None,
"usage": data.get("usage"),
"usage": usage,
}
except Exception as e:
logger.warning("llama.cpp-Call fehlgeschlagen: %s", e)
@@ -387,6 +400,9 @@ async def _run() -> None:
# Reihenfolge ab, falls es kurz vor uns startet).
_generate_config()
# Auslastungs-Sampler (GPU + Tokens) laeuft unabhaengig vom RVS.
asyncio.create_task(_stats.run_sampler())
use_tls = RVS_TLS
retry_s = 2
tls_fallback_tried = False
@@ -410,14 +426,17 @@ async def _run() -> None:
except Exception:
continue
mtype = msg.get("type")
if mtype not in ("llm_request", "llm_provision_model", "llm_remove_model"):
continue
payload = msg.get("payload", {}) or {}
# Redundanz-Routing: gezielt an eine andere Instanz adressiert
# → ignorieren. Ohne targetInstance → wie bisher (jeder nimmt).
tgt = payload.get("targetInstance")
if tgt and tgt != INSTANCE_ID:
continue
# Auslastungs-Monitor (node_stats_*) abfangen.
if await _stats.handle(ws, mtype, payload, _send):
continue
if mtype not in ("llm_request", "llm_provision_model", "llm_remove_model"):
continue
if mtype == "llm_provision_model":
asyncio.create_task(_handle_provision(ws, payload))
elif mtype == "llm_remove_model":