feat(fleet): Auslastungs-Monitor pro Box — live nvidia-smi + Graphen (Stage E)
Pro Box ein "Auslastung"-Button in der Compute-Flotte → Modal mit live nvidia-smi (1s), Graphen (GPU-Auslastung + Tokens/Intervall) und Besen-Reset. Historie liegt auf der Box, Diagnostic holt sie via RVS. - node_stats.py (identisch in allen 4 Worker-Build-Contexts): Sampler alle 15s (nvidia-smi + Token-Delta → Ringpuffer ~500 Punkte, persistent als JSON auf der Box), Live-Stream (node_stats, 1s, Auto-Stop 300s), History-Request, Reset. nvidia-smi via async subprocess, fail-safe ohne GPU. - Worker-Wiring (f5tts/whisper/voxtral/llm-adapter): Import, Sampler-Task, _stats.handle() nach dem targetInstance-Filter. llm-adapter zaehlt Tokens (usage.total_tokens) → Token-Graph nur bei LLM-Boxen. Dockerfiles kopieren node_stats.py. - compose: llm-adapter bekommt runtime:nvidia + NVIDIA_VISIBLE_DEVICES=all + DRIVER_CAPABILITIES=utility (nur nvidia-smi, KEIN VRAM/Compute). - diagnostic/server.js: relay node_stats_* (Browser→Box) + forward (Box→Browser). - diagnostic/index.html: Auslastung-Button pro Node (Ziel bevorzugt llm-Instanz), Modal mit live nvidia-smi + Inline-SVG-Sparklines, Besen-Reset. Reporter-Wahl bevorzugt die llm-Instanz (sieht alle GPUs + Tokens); GPU-Worker sehen ihre gepinnte Karte. Gitignored Historie stoert git-Baum der Box nicht. Deploy: diagnostic + GPU-Boxen neu bauen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -68,6 +68,11 @@ WORKER_SERVICE = "voxtral"
|
||||
INSTANCE_ID = f"{WORKER_SERVICE}@{NODE_NAME}"
|
||||
WORKER_PING_INTERVAL_S = int(os.getenv("WORKER_PING_INTERVAL_S", "10"))
|
||||
|
||||
# ── Auslastungs-Monitor (Stage E) ──────────────────────────
|
||||
import node_stats
|
||||
STATS_PATH = os.getenv("STATS_PATH", f"/root/.cache/huggingface/aria_stats_{WORKER_SERVICE}.json")
|
||||
_stats = node_stats.NodeStats(INSTANCE_ID, NODE_NAME, STATS_PATH, logger=logger)
|
||||
|
||||
STREAM_TRANSCRIBE_INTERVAL_MS = int(os.getenv("STREAM_TRANSCRIBE_INTERVAL_MS", "1000"))
|
||||
STREAM_DEFAULT_ENDPOINT_MS = 2400
|
||||
STREAM_DEFAULT_HARD_CAP_MS = 300000
|
||||
@@ -757,6 +762,9 @@ async def run_loop(sessions: SessionManager) -> None:
|
||||
tgt = payload.get("targetInstance")
|
||||
if tgt and tgt != INSTANCE_ID:
|
||||
continue
|
||||
# Auslastungs-Monitor (node_stats_*) abfangen.
|
||||
if await _stats.handle(ws, mtype, payload, _send):
|
||||
continue
|
||||
if mtype == "stt_stream_start":
|
||||
sessions.start_session(payload)
|
||||
elif mtype == "stt_audio_chunk":
|
||||
@@ -856,6 +864,7 @@ async def main() -> None:
|
||||
await loop.run_in_executor(None, runner.load) # Modell laden (blockierend)
|
||||
sessions = SessionManager(runner)
|
||||
logger.info("Voxtral-Bridge startet — Modell=%s", VOXTRAL_MODEL)
|
||||
asyncio.create_task(_stats.run_sampler()) # Auslastungs-Sampler (Stage E)
|
||||
await asyncio.gather(run_loop(sessions), sessions.run_endpointer())
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user