Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 duffyduck
|
ee6c4f34db | ||
 duffyduckandClaude Opus 4.8
|
75daadf72d | ||
|
duffyduckandClaude Opus 4.8
|
37aaa90239 | ||
|
duffyduckandClaude Opus 4.8
|
03e6d784b6 | ||
|
duffyduckandClaude Opus 4.8
|
762f1a2dd9 | ||
|
duffyduckandClaude Opus 4.8
|
49f6b26ab8 | ||
|
duffyduckandClaude Opus 4.8
|
c340b9d683 | ||
|
duffyduckandClaude Opus 4.8
|
a0429fc91e | ||
|
duffyduckandClaude Opus 4.8
|
174d6d643d | ||
|
duffyduckandClaude Opus 4.8
|
fd6ba73f59 | ||
|
duffyduckandClaude Opus 4.8
|
f5b22253b2 | ||
|
duffyduckandClaude Opus 4.8
|
761f4c8903 |
@@ -0,0 +1,63 @@
|
||||
# ai-box — KI-Box (gpubox) Bootstrap
|
||||
|
||||
Macht aus einem frisch installierten **Debian Trixie** (headless, nur SSH) einen
|
||||
startklaren GPU-Satelliten-Host für den `xtts`-Stack (STT/TTS/LLM).
|
||||
|
||||
## Was das Script tut
|
||||
|
||||
1. Basis-Pakete (curl, gnupg, git …)
|
||||
2. `contrib non-free non-free-firmware` aktivieren (Trixie-**deb822**-Format berücksichtigt)
|
||||
3. **NVIDIA-Treiber** installieren (`nvidia-driver` + firmware)
|
||||
4. **Docker** Engine + Compose-Plugin
|
||||
5. **NVIDIA Container Toolkit** + Docker-Runtime auf NVIDIA konfigurieren
|
||||
6. `xtts/.env` aus `.env.example` anlegen (RVS_TOKEN optional gleich setzen)
|
||||
7. **GPU-im-Container-Test** (`docker run --gpus all … nvidia-smi`)
|
||||
8. optional (`--up`): den `xtts`-Stack hochziehen
|
||||
|
||||
Alles **idempotent** — mehrfach ausführbar.
|
||||
|
||||
## Ablauf
|
||||
|
||||
```bash
|
||||
git clone <repo> ARIA-AGENT
|
||||
cd ARIA-AGENT/ai-box
|
||||
|
||||
sudo ./bootstrap.sh
|
||||
# → Wenn der Treiber frisch installiert wurde: einmal neu starten, dann nochmal:
|
||||
sudo reboot
|
||||
# … nach dem Boot:
|
||||
cd ARIA-AGENT/ai-box
|
||||
sudo ./bootstrap.sh --up --token <DEIN_RVS_TOKEN>
|
||||
```
|
||||
|
||||
Der Treiber-Reboot ist normal (Kernel-Modul wird erst beim Boot geladen). Beim
|
||||
zweiten Lauf überspringt das Script alles Erledigte und macht nur noch den
|
||||
GPU-Test + Stack-Start.
|
||||
|
||||
## Optionen
|
||||
|
||||
| Option | Wirkung |
|
||||
|---|---|
|
||||
| `--up` | am Ende `docker compose up -d --build` (Default-Profil) |
|
||||
| `--token <TOK>` | `RVS_TOKEN` in `xtts/.env` eintragen (auch via `RVS_TOKEN=…` env) |
|
||||
| `--rvs-host <H>` | `RVS_HOST` setzen |
|
||||
|
||||
## Wichtig
|
||||
|
||||
- **Stimm-Daten** (nicht in git): falls von der alten Box noch vorhanden,
|
||||
`xtts/voice-id/` (Speaker-Fingerprint) + `xtts/voices/` (F5-Referenz) herkopieren.
|
||||
Sonst egal — in der App neu anlegen: Stimme neu enrollen (ohne Fingerprint läuft
|
||||
die Speaker-ID fail-open, alles geht durch) + F5-Voice-Referenz neu hochladen.
|
||||
- **Voxtral bleibt aus** auf der 3060 (braucht ≥16 GB VRAM). Das Default-Profil
|
||||
fährt Whisper (mit dem M0.1-Fix) + F5-TTS + lokales LLM. Voxtral erst mit der
|
||||
24-GB-Karte: `docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build`.
|
||||
- **Erster Start lädt Modelle** (mehrere GB via HuggingFace nach `xtts/hf-cache`
|
||||
+ `xtts/models`) — genug Platz (1 TB NVMe ✓) und etwas Geduld.
|
||||
|
||||
## Verifizieren
|
||||
|
||||
```bash
|
||||
nvidia-smi # Host sieht die GPU
|
||||
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi # Container auch
|
||||
docker logs -f aria-whisper-bridge # "RVS verbunden" + service_status ready
|
||||
```
|
||||
Executable
+234
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# ARIA KI-Box (gpubox) Bootstrap — frisches Debian Trixie → startklarer
|
||||
# GPU-Satelliten-Host fuer den xtts-Stack (Voxtral/Whisper STT, F5-TTS, lokales LLM).
|
||||
#
|
||||
# Ablauf nach `git clone`:
|
||||
# cd ARIA-AGENT/ai-box
|
||||
# sudo ./bootstrap.sh # richtet Treiber + Docker + NVIDIA-Toolkit ein
|
||||
# # (falls Treiber frisch installiert: einmal `sudo reboot`, dann Script erneut)
|
||||
# sudo ./bootstrap.sh --up # dazu: xtts-Stack (Whisper+F5+LLM) hochziehen
|
||||
#
|
||||
# Optionen:
|
||||
# --up am Ende den xtts-Stack starten (Default-Profil, OHNE voxtral)
|
||||
# --token <TOK> RVS_TOKEN in xtts/.env eintragen (alternativ: env RVS_TOKEN=...)
|
||||
# --rvs-host <H> RVS_HOST setzen (Default aus .env.example)
|
||||
#
|
||||
# IDEMPOTENT: bereits erledigte Schritte werden uebersprungen. Nach dem
|
||||
# Treiber-Reboot einfach nochmal ausfuehren — der Rest laeuft dann durch.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ── CLI ──
|
||||
DO_UP=0
|
||||
RVS_TOKEN_ARG="${RVS_TOKEN:-}"
|
||||
RVS_HOST_ARG="${RVS_HOST:-}"
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--up) DO_UP=1; shift ;;
|
||||
--token) RVS_TOKEN_ARG="${2:-}"; shift 2 ;;
|
||||
--rvs-host) RVS_HOST_ARG="${2:-}"; shift 2 ;;
|
||||
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "Unbekannte Option: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Log-Helfer ──
|
||||
c_g="\033[1;32m"; c_y="\033[1;33m"; c_r="\033[1;31m"; c_b="\033[1;34m"; c_0="\033[0m"
|
||||
STEP=0
|
||||
step() { STEP=$((STEP+1)); echo -e "\n${c_b}[${STEP}] $*${c_0}"; }
|
||||
ok() { echo -e " ${c_g}✓${c_0} $*"; }
|
||||
warn() { echo -e " ${c_y}!${c_0} $*"; }
|
||||
die() { echo -e "${c_r}✗ $*${c_0}" >&2; exit 1; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
XTTS_DIR="$REPO_ROOT/xtts"
|
||||
|
||||
# ── Preflight ──
|
||||
[[ "$(id -u)" -eq 0 ]] || die "Bitte als root ausfuehren (sudo ./bootstrap.sh)."
|
||||
command -v apt-get >/dev/null || die "Kein apt-get — dieses Script ist fuer Debian/Trixie."
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
echo -e "${c_b}=== ARIA KI-Box Bootstrap ===${c_0}"
|
||||
echo "Repo: $REPO_ROOT"
|
||||
echo "xtts: $XTTS_DIR"
|
||||
|
||||
# ── 1. Basis-Pakete ──
|
||||
step "Basis-Pakete"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq ca-certificates curl gnupg git lsb-release >/dev/null
|
||||
ok "ca-certificates, curl, gnupg, git"
|
||||
|
||||
# ── 2. non-free Repos aktivieren (Trixie deb822 + Legacy) ──
|
||||
step "APT-Komponenten (contrib non-free non-free-firmware)"
|
||||
# Ergaenzt die drei Komponenten in JEDER Components:-Zeile einer deb822-Datei —
|
||||
# pro Zeile nur was fehlt, reihenfolge-robust, idempotent. Die Adressen pruefen
|
||||
# ganze Woerter (non-free-firmware zaehlt NICHT als non-free).
|
||||
add_components() {
|
||||
sed -i -E '/^[Cc]omponents:/{
|
||||
/(^|[[:space:]])contrib([[:space:]]|$)/!s/$/ contrib/
|
||||
/(^|[[:space:]])non-free([[:space:]]|$)/!s/$/ non-free/
|
||||
/(^|[[:space:]])non-free-firmware([[:space:]]|$)/!s/$/ non-free-firmware/
|
||||
}' "$1"
|
||||
}
|
||||
ENABLED_ANY=0
|
||||
shopt -s nullglob
|
||||
for f in /etc/apt/sources.list.d/*.sources; do
|
||||
grep -qE '^[Cc]omponents:' "$f" || continue
|
||||
b="$(md5sum "$f")"; add_components "$f"; a="$(md5sum "$f")"
|
||||
if [[ "$b" != "$a" ]]; then ENABLED_ANY=1; ok "aktualisiert: $(basename "$f")"; fi
|
||||
done
|
||||
shopt -u nullglob
|
||||
# Legacy /etc/apt/sources.list (deb-Zeilen)
|
||||
if [[ -f /etc/apt/sources.list ]] && grep -qE '^deb ' /etc/apt/sources.list; then
|
||||
if ! grep -qE '^deb .*[[:space:]]non-free([[:space:]]|$)' /etc/apt/sources.list; then
|
||||
sed -i -E '/^deb .*debian/ s/$/ contrib non-free non-free-firmware/' /etc/apt/sources.list
|
||||
ENABLED_ANY=1; ok "aktualisiert: sources.list"
|
||||
fi
|
||||
fi
|
||||
[[ $ENABLED_ANY -eq 0 ]] && ok "non-free schon aktiv"
|
||||
apt-get update -qq # immer neu einlesen, damit der Kandidat sicher da ist
|
||||
|
||||
# ── 3. NVIDIA-Treiber ──
|
||||
step "NVIDIA-Treiber"
|
||||
if nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Treiber aktiv: $(nvidia-smi --query-gpu=name --format=csv,noheader | paste -sd', ')"
|
||||
DRIVER_ACTIVE=1
|
||||
else
|
||||
if dpkg -l | grep -q '^ii nvidia-driver '; then
|
||||
warn "Treiber installiert, aber nvidia-smi antwortet nicht → REBOOT noetig."
|
||||
DRIVER_ACTIVE=0
|
||||
else
|
||||
# KEIN separater Kandidaten-Check — die apt-cache-Ausgabe ist locale-/pipe-
|
||||
# fragil (hat faelschlich "kein Kandidat" gemeldet). Der Install IST der Test:
|
||||
# direkt installieren; schlaegt er fehl, einmal volles apt-get update + Retry,
|
||||
# dann erst mit Diagnose abbrechen.
|
||||
# Kernel-Header ZUERST — sonst ueberspringt DKMS den Modulbau ("No kernel
|
||||
# headers were found") und nvidia-smi kann spaeter nicht mit dem Treiber reden.
|
||||
warn "Kernel-Header + nvidia-driver installieren (DKMS-Build, dauert)…"
|
||||
apt-get install -y linux-headers-amd64 || true
|
||||
apt-get install -y "linux-headers-$(uname -r)" || true
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
warn "Install fehlgeschlagen — volles 'apt-get update' + zweiter Versuch…"
|
||||
apt-get update
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
echo
|
||||
warn "nvidia-driver liess sich nicht installieren. Aktive Quellen:"
|
||||
grep -rhE '^[Cc]omponents:' /etc/apt/sources.list.d/*.sources 2>/dev/null | sed 's/^/ /' || true
|
||||
grep -E '^deb ' /etc/apt/sources.list 2>/dev/null | sed 's/^/ /' || true
|
||||
die "Pruefe 'apt-cache policy nvidia-driver' + Netz/non-free."
|
||||
fi
|
||||
fi
|
||||
# Falls nvidia-kernel-dkms schon (ohne Header) installiert war: Modul jetzt bauen.
|
||||
dkms autoinstall >/dev/null 2>&1 || true
|
||||
ok "nvidia-driver + Kernel-Modul installiert"
|
||||
DRIVER_ACTIVE=0
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 4. Docker Engine + Compose-Plugin ──
|
||||
step "Docker"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
ok "Docker vorhanden: $(docker --version)"
|
||||
else
|
||||
warn "Installiere Docker (offizielles get.docker.com)…"
|
||||
curl -fsSL https://get.docker.com | sh >/dev/null
|
||||
systemctl enable --now docker >/dev/null 2>&1 || true
|
||||
ok "Docker installiert: $(docker --version)"
|
||||
fi
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
ok "Compose-Plugin: $(docker compose version | head -1)"
|
||||
else
|
||||
warn "Compose-Plugin fehlt — installiere docker-compose-plugin…"
|
||||
apt-get install -y -qq docker-compose-plugin >/dev/null || \
|
||||
warn "Konnte docker-compose-plugin nicht via apt holen — get.docker.com bringt es normalerweise mit."
|
||||
fi
|
||||
|
||||
# ── 5. NVIDIA Container Toolkit ──
|
||||
step "NVIDIA Container Toolkit"
|
||||
NCT_LIST="/etc/apt/sources.list.d/nvidia-container-toolkit.list"
|
||||
NCT_KEY="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"
|
||||
if ! dpkg -l | grep -q '^ii nvidia-container-toolkit '; then
|
||||
[[ -f "$NCT_KEY" ]] || curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||
| gpg --dearmor -o "$NCT_KEY"
|
||||
if [[ ! -f "$NCT_LIST" ]]; then
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||
| sed "s#deb https://#deb [signed-by=${NCT_KEY}] https://#g" > "$NCT_LIST"
|
||||
fi
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq nvidia-container-toolkit >/dev/null
|
||||
ok "nvidia-container-toolkit installiert"
|
||||
else
|
||||
ok "nvidia-container-toolkit vorhanden"
|
||||
fi
|
||||
# Docker-Runtime auf NVIDIA konfigurieren (idempotent)
|
||||
if ! grep -q '"nvidia"' /etc/docker/daemon.json 2>/dev/null; then
|
||||
nvidia-ctk runtime configure --runtime=docker >/dev/null
|
||||
systemctl restart docker
|
||||
ok "Docker-Runtime auf NVIDIA konfiguriert + Docker neugestartet"
|
||||
else
|
||||
ok "Docker-Runtime bereits NVIDIA-konfiguriert"
|
||||
fi
|
||||
|
||||
# ── 6. xtts/.env vorbereiten ──
|
||||
step "xtts/.env"
|
||||
if [[ -f "$XTTS_DIR/.env" ]]; then
|
||||
ok ".env existiert bereits (unangetastet)"
|
||||
else
|
||||
cp "$XTTS_DIR/.env.example" "$XTTS_DIR/.env"
|
||||
ok ".env aus .env.example erstellt"
|
||||
fi
|
||||
if [[ -n "$RVS_TOKEN_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_TOKEN=.*#RVS_TOKEN=${RVS_TOKEN_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_TOKEN eingetragen"
|
||||
fi
|
||||
if [[ -n "$RVS_HOST_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_HOST=.*#RVS_HOST=${RVS_HOST_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_HOST=${RVS_HOST_ARG} eingetragen"
|
||||
fi
|
||||
if grep -q '^RVS_TOKEN=dein_token_hier' "$XTTS_DIR/.env"; then
|
||||
warn "RVS_TOKEN ist noch der Platzhalter — vor dem Start setzen:"
|
||||
warn " nano $XTTS_DIR/.env (oder: sudo ./bootstrap.sh --token <TOKEN>)"
|
||||
fi
|
||||
warn "Stimm-Daten (nicht in git): falls von der alten Box noch vorhanden, xtts/voice-id/"
|
||||
warn " + xtts/voices/ herkopieren. Sonst egal — in der App neu anlegen:"
|
||||
warn " Sprache neu enrollen (Speaker-ID, sonst fail-open) + F5-Referenz neu hochladen."
|
||||
|
||||
# ── 7. GPU-im-Container verifizieren ──
|
||||
step "GPU-im-Container Test"
|
||||
if [[ "${DRIVER_ACTIVE:-0}" -eq 1 ]]; then
|
||||
if docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Docker sieht die GPU — KI-Box ist einsatzbereit."
|
||||
GPU_READY=1
|
||||
else
|
||||
warn "Host-Treiber ok, aber Container sieht die GPU nicht — Toolkit/Runtime pruefen."
|
||||
GPU_READY=0
|
||||
fi
|
||||
else
|
||||
warn "Treiber noch nicht aktiv → Test uebersprungen."
|
||||
echo
|
||||
echo -e "${c_y}==> REBOOT noetig, dann Script erneut ausfuehren:${c_0}"
|
||||
echo -e "${c_y} sudo reboot && (nach dem Boot) sudo ./bootstrap.sh${c_0}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 8. Optional: xtts-Stack hochziehen ──
|
||||
if [[ $DO_UP -eq 1 && "${GPU_READY:-0}" -eq 1 ]]; then
|
||||
step "xtts-Stack starten (Default-Profil: Whisper + F5 + LLM — OHNE voxtral)"
|
||||
warn "Erster Start laedt Modelle (mehrere GB via HuggingFace) — kann dauern."
|
||||
( cd "$XTTS_DIR" && docker compose up -d --build )
|
||||
ok "Stack laeuft. Logs: docker logs -f aria-whisper-bridge"
|
||||
echo
|
||||
echo " voxtral (STT via Voxtral) braucht >=16 GB VRAM → erst mit der 24-GB-Karte:"
|
||||
echo " cd $XTTS_DIR && docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build"
|
||||
fi
|
||||
|
||||
# ── Abschluss ──
|
||||
echo
|
||||
echo -e "${c_g}=== Fertig. KI-Box startklar. ===${c_0}"
|
||||
if [[ $DO_UP -eq 0 ]]; then
|
||||
echo "Naechster Schritt — Stack starten:"
|
||||
echo " cd $XTTS_DIR && docker compose up -d --build"
|
||||
echo "oder direkt: sudo ./bootstrap.sh --up"
|
||||
fi
|
||||
@@ -79,8 +79,8 @@ android {
|
||||
applicationId "com.ariacockpit"
|
||||
minSdkVersion rootProject.ext.minSdkVersion
|
||||
targetSdkVersion rootProject.ext.targetSdkVersion
|
||||
versionCode 20300
|
||||
versionName "0.2.3.0"
|
||||
versionCode 20301
|
||||
versionName "0.2.3.1"
|
||||
// Fallback fuer Libraries mit Product Flavors
|
||||
missingDimensionStrategy 'react-native-camera', 'general'
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "aria-cockpit",
|
||||
"version": "0.2.3.0",
|
||||
"version": "0.2.3.1",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"android": "react-native run-android",
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* ariaView — Empfaenger der von ARIA komponierten RAEUMLICHEN Ansichten (M1).
|
||||
*
|
||||
* Fluss: ARIA ruft im Brain `present_view` → Brain-Event `aria_view` → Bridge →
|
||||
* RVS `aria_view` → hier gepuffert → WorkspaceCanvas rendert Orb + Karten, die
|
||||
* auf der Flaeche materialisieren.
|
||||
*
|
||||
* Der Service haelt pro Projekt die AKTUELLE View-Spec, damit eine spaet
|
||||
* gemountete Canvas-Kachel sofort den Ist-Stand bekommt. Muster wie
|
||||
* services/codeFile.ts (Singleton, rvs.onMessage).
|
||||
*
|
||||
* Die Karten-Typen sind bewusst offen (string), damit spaetere Renderer (vnc,
|
||||
* chart, file …) ohne Service-Aenderung dazukommen. Der jeweilige Client-Renderer
|
||||
* entscheidet, was er mit einem unbekannten Typ macht (i.d.R. ignorieren).
|
||||
*/
|
||||
|
||||
import rvs, { RVSMessage } from './rvs';
|
||||
|
||||
export type OrbState = 'idle' | 'listening' | 'thinking' | 'speaking' | 'working';
|
||||
|
||||
export interface ViewMarker {
|
||||
lat: number;
|
||||
lon: number;
|
||||
label?: string;
|
||||
}
|
||||
|
||||
export interface ViewCard {
|
||||
type: 'text' | 'image' | 'map' | 'code' | 'list' | string;
|
||||
title?: string;
|
||||
md?: string; // text/list
|
||||
src?: string; // image
|
||||
markers?: ViewMarker[]; // map
|
||||
path?: string; // code
|
||||
lang?: string; // code
|
||||
// Zukuenftige Kartenfelder ohne Service-Aenderung:
|
||||
[k: string]: any;
|
||||
}
|
||||
|
||||
export interface ViewSpec {
|
||||
cards: ViewCard[];
|
||||
orb?: OrbState;
|
||||
title?: string;
|
||||
}
|
||||
|
||||
export interface AriaView {
|
||||
projectId: string;
|
||||
view: ViewSpec;
|
||||
clientMsgId?: string;
|
||||
ts: number;
|
||||
}
|
||||
|
||||
type ViewSub = (v: AriaView) => void;
|
||||
|
||||
class AriaViewService {
|
||||
private views = new Map<string, AriaView>();
|
||||
private subs: ViewSub[] = [];
|
||||
|
||||
constructor() {
|
||||
rvs.onMessage((m) => this.onMessage(m));
|
||||
}
|
||||
|
||||
private onMessage(m: RVSMessage): void {
|
||||
if (m.type !== 'aria_view') return;
|
||||
const p = (m.payload || {}) as any;
|
||||
const raw = (p.view || {}) as any;
|
||||
const cards: ViewCard[] = Array.isArray(raw.cards) ? raw.cards : [];
|
||||
if (cards.length === 0) return; // leere Ansicht ignorieren
|
||||
const view: ViewSpec = {
|
||||
cards,
|
||||
orb: raw.orb || 'speaking',
|
||||
title: raw.title || '',
|
||||
};
|
||||
const projectId: string = p.projectId || '';
|
||||
const entry: AriaView = {
|
||||
projectId,
|
||||
view,
|
||||
clientMsgId: p.clientMsgId || '',
|
||||
ts: Date.now(),
|
||||
};
|
||||
this.views.set(projectId, entry);
|
||||
this.subs.forEach((cb) => {
|
||||
try { cb(entry); } catch {}
|
||||
});
|
||||
}
|
||||
|
||||
/** Aktuelle Ansicht eines Projekts (leer = Hauptchat). */
|
||||
getView(projectId: string): AriaView | undefined {
|
||||
return this.views.get(projectId || '');
|
||||
}
|
||||
|
||||
/** Registriert einen Listener fuer neue Ansichten. */
|
||||
subscribe(cb: ViewSub): () => void {
|
||||
this.subs.push(cb);
|
||||
return () => { this.subs = this.subs.filter((s) => s !== cb); };
|
||||
}
|
||||
|
||||
/** Ansicht eines Projekts verwerfen (z.B. wenn der User sie wegwischt). */
|
||||
clear(projectId: string): void {
|
||||
this.views.delete(projectId || '');
|
||||
}
|
||||
}
|
||||
|
||||
const ariaView = new AriaViewService();
|
||||
export default ariaView;
|
||||
@@ -1145,7 +1145,7 @@ class AudioService {
|
||||
speed: typeof opts.speed === 'number' ? opts.speed : 1.0,
|
||||
interrupted: !!opts.interrupted,
|
||||
location: opts.location || null,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : 1500,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : STT_ENDPOINT_DEFAULT_MS,
|
||||
hardCapMs: typeof opts.hardCapMs === 'number' ? opts.hardCapMs : 60000,
|
||||
sampleRate: 16000,
|
||||
projectId: opts.projectId || '',
|
||||
|
||||
@@ -145,6 +145,17 @@ class GpsTrackingService {
|
||||
// liefert im Hintergrund keine Updates (nur Heartbeat sendet alte Werte).
|
||||
const bgEnabled = await isBackgroundGpsEnabled();
|
||||
if (bgEnabled) {
|
||||
// Ohne ACCESS_BACKGROUND_LOCATION liefert watchPosition im Hintergrund
|
||||
// NICHTS (Android 10+) → der Foreground-Service allein bringt nichts, und
|
||||
// genau der Fall "Ankunft waehrend der Fahrt, Screen aus" faellt durch.
|
||||
// Deshalb erst die Permission sicherstellen (oeffnet ggf. die Android-
|
||||
// Settings fuer "Immer erlauben"), DANN den Location-Foreground-Service
|
||||
// hochziehen — der haelt den Prozess wach, sodass watchPosition + der
|
||||
// 60s-Heartbeat auch unter Doze weiterlaufen.
|
||||
const bgOk = await ensureBackgroundLocationPermission();
|
||||
if (!bgOk) {
|
||||
console.warn('[gps-track] Background-Permission fehlt — Tracking nur im Vordergrund zuverlaessig');
|
||||
}
|
||||
try { await acquireBackgroundAudio('location'); } catch {}
|
||||
}
|
||||
try {
|
||||
|
||||
+104
-6
@@ -627,10 +627,11 @@ META_TOOLS = [
|
||||
"name": "request_location_tracking",
|
||||
"description": (
|
||||
"Bittet die App, das kontinuierliche GPS-Tracking zu aktivieren oder zu "
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). Nutze das wenn du einen "
|
||||
"GPS-basierten Watcher anlegst (z.B. `near(...)`), sonst hat die App "
|
||||
"veraltete Position und der Watcher feuert nie. Auch wieder ausschalten "
|
||||
"wenn der letzte GPS-Watcher geloescht wurde."
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). HINWEIS: Beim Anlegen/Loeschen "
|
||||
"eines Standort-Watchers (`near/entered_near/left_near`) schaltet das System "
|
||||
"das Tracking bereits AUTOMATISCH mit an bzw. aus — dieses Tool brauchst du "
|
||||
"dafuer nicht mehr. Nutze es nur fuer manuelle/explizite Faelle (z.B. Tracking "
|
||||
"kurz einschalten ohne Watcher)."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -642,6 +643,48 @@ META_TOOLS = [
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "present_view",
|
||||
"description": (
|
||||
"Komponiere eine RAEUMLICHE Ansicht, die auf ARIAs Flaeche materialisiert "
|
||||
"(Orb + Karten). Nutze das, wenn eine visuelle Anordnung besser ist als reiner "
|
||||
"Text — z.B. eine Akte/Datei zeigen (Bild links, Text rechts, Karte unten), "
|
||||
"einen Vergleich, eine Liste, eine Karte mit Orten, oder Code. NICHT fuer "
|
||||
"normale Gespraechsantworten. Die Karten erscheinen ZUSAETZLICH zu deiner "
|
||||
"(kurzen) Sprachantwort — halte die Antwort dann knapp, das Visuelle traegt."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cards": {
|
||||
"type": "array",
|
||||
"description": "Die Karten der Ansicht, in sinnvoller Anordnung.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {"type": "string", "enum": ["text", "image", "map", "code", "list"],
|
||||
"description": "Kartentyp"},
|
||||
"title": {"type": "string", "description": "Optionaler Kartentitel"},
|
||||
"md": {"type": "string", "description": "text/list: Markdown-Inhalt"},
|
||||
"src": {"type": "string", "description": "image: Bild-URL oder /shared-Pfad"},
|
||||
"markers": {"type": "array", "items": {"type": "object"},
|
||||
"description": "map: Liste [{lat, lon, label}]"},
|
||||
"path": {"type": "string", "description": "code: Datei-Pfad im Projekt"},
|
||||
"lang": {"type": "string", "description": "code: Sprache fuers Highlighting"},
|
||||
},
|
||||
"required": ["type"],
|
||||
},
|
||||
},
|
||||
"orb": {"type": "string", "enum": ["idle", "speaking", "working"],
|
||||
"description": "Orb-Zustand nach dem Rendern (default: speaking)"},
|
||||
"title": {"type": "string", "description": "Optionaler Titel der ganzen Ansicht"},
|
||||
},
|
||||
"required": ["cards"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
@@ -1355,6 +1398,17 @@ def _skill_to_tool(s: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# Standort-Funktionen in Watcher-Conditions — brauchen aktives GPS-Tracking.
|
||||
_LOCATION_FUNCS = ("near(", "entered_near(", "left_near(")
|
||||
|
||||
|
||||
def _condition_needs_location(condition: str) -> bool:
|
||||
"""True, wenn die Watcher-Condition eine Standort-Funktion nutzt und damit
|
||||
laufende GPS-Updates der App braucht."""
|
||||
c = condition or ""
|
||||
return any(fn in c for fn in _LOCATION_FUNCS)
|
||||
|
||||
|
||||
class Agent:
|
||||
# Mindest-Score den ein Cold-Memory-Treffer haben muss um in den
|
||||
# System-Prompt aufgenommen zu werden. Unter dieser Schwelle ist's
|
||||
@@ -2372,13 +2426,41 @@ class Agent:
|
||||
"trigger": {"name": t["name"], "type": "watcher",
|
||||
"condition": t["condition"], "message": t["message"]},
|
||||
})
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'."
|
||||
# GPS-Kopplung: ein Standort-Watcher (near/entered_near/left_near)
|
||||
# ist tot, wenn die App kein Tracking sendet. Frueher musste ARIA
|
||||
# separat request_location_tracking aufrufen — wurde oft vergessen,
|
||||
# dann feuerte der Trigger nie. Jetzt automatisch mit-anschalten.
|
||||
extra = ""
|
||||
if _condition_needs_location(t["condition"]):
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": True,
|
||||
"reason": f"watcher:{t['name']}",
|
||||
})
|
||||
extra = " GPS-Tracking automatisch aktiviert."
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'.{extra}"
|
||||
if name == "trigger_cancel":
|
||||
try:
|
||||
triggers_mod.delete(arguments["name"])
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht."
|
||||
except ValueError as e:
|
||||
return f"FEHLER: {e}"
|
||||
# GPS-Kopplung (Gegenstueck): existiert kein Standort-Watcher mehr,
|
||||
# Tracking wieder ausschalten (Akku schonen).
|
||||
extra = ""
|
||||
remaining = triggers_mod.list_triggers(active_only=False)
|
||||
still_needs_gps = any(
|
||||
r.get("type") == "watcher"
|
||||
and _condition_needs_location(r.get("condition") or "")
|
||||
for r in remaining
|
||||
)
|
||||
if not still_needs_gps:
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": False,
|
||||
"reason": "no-location-watchers",
|
||||
})
|
||||
extra = " GPS-Tracking deaktiviert (kein Standort-Watcher mehr)."
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht.{extra}"
|
||||
if name == "request_location_tracking":
|
||||
on = bool(arguments.get("on", False))
|
||||
reason = (arguments.get("reason") or "").strip()
|
||||
@@ -2388,6 +2470,22 @@ class Agent:
|
||||
"reason": reason,
|
||||
})
|
||||
return f"OK — Tracking-Request gesendet (on={on}). App wird in Kuerze umschalten."
|
||||
if name == "present_view":
|
||||
cards = arguments.get("cards") or []
|
||||
if not isinstance(cards, list) or not cards:
|
||||
return "FEHLER: present_view braucht mindestens eine Karte in 'cards'."
|
||||
view = {
|
||||
"cards": cards,
|
||||
"orb": arguments.get("orb") or "speaking",
|
||||
"title": arguments.get("title") or "",
|
||||
}
|
||||
self._pending_events.append({
|
||||
"type": "aria_view",
|
||||
"view": view,
|
||||
"project_id": project_id or "",
|
||||
})
|
||||
return (f"OK — Ansicht mit {len(cards)} Karte(n) an die Flaeche geschickt "
|
||||
f"(Kontext: {project_id or 'Hauptchat'}).")
|
||||
if name == "trigger_list":
|
||||
items = triggers_mod.list_triggers(active_only=False)
|
||||
if not items:
|
||||
|
||||
@@ -2080,6 +2080,22 @@ class ARIABridge:
|
||||
proj = event.get("project") or {}
|
||||
logger.info("[brain] Projekt %s: %s (id=%s)",
|
||||
event.get("action") or "?", proj.get("name"), proj.get("id"))
|
||||
elif etype == "aria_view":
|
||||
# M1: ARIA hat via present_view eine raeumliche Ansicht komponiert.
|
||||
# View-Spec (Orb + Karten) + Projekt-Kontext an App/Web/Diagnostic;
|
||||
# deren Renderer materialisieren die Karten auf der Flaeche.
|
||||
view = event.get("view") or {}
|
||||
await self._send_to_rvs({
|
||||
"type": "aria_view",
|
||||
"payload": {
|
||||
"view": view,
|
||||
"projectId": event.get("project_id") or "",
|
||||
"clientMsgId": client_msg_id or "",
|
||||
},
|
||||
"timestamp": int(asyncio.get_event_loop().time() * 1000),
|
||||
})
|
||||
logger.info("[brain] ARIA hat eine Ansicht geschickt: %d Karte(n), orb=%s",
|
||||
len(view.get("cards") or []), view.get("orb"))
|
||||
|
||||
# _process_core_response uebernimmt alles weitere:
|
||||
# File-Marker extrahieren + broadcasten, NO_REPLY-Check, Chat-
|
||||
|
||||
+20
-8
@@ -1767,14 +1767,21 @@
|
||||
|
||||
if (msg.type === 'disk_cleanup') {
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
const dtext = document.getElementById('disk-banner-text');
|
||||
if (msg.status === 'running') {
|
||||
if (btn) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
} else if (msg.status === 'done') {
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Sicher aufraeumen'; }
|
||||
alert('Aufgeraeumt: ' + (msg.freed || '0 MB') + ' frei\n(' + (msg.steps || []).join(', ') + ')');
|
||||
// Feedback NICHT nur per alert() (koennte unterdrueckt sein) —
|
||||
// direkt am Button + im Banner sichtbar machen.
|
||||
if (btn) {
|
||||
btn.disabled = false;
|
||||
btn.textContent = '✓ ' + (msg.freed || '0 MB') + ' frei';
|
||||
setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000);
|
||||
}
|
||||
if (dtext) dtext.textContent = 'Aufgeraeumt: ' + (msg.freed || '0 MB') + ' frei (' + (msg.steps || []).join(', ') + '). Disk-Status aktualisiert sich gleich.';
|
||||
} else if (msg.status === 'error') {
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Sicher aufraeumen'; }
|
||||
alert('Aufraeumen fehlgeschlagen: ' + (msg.error || ''));
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Fehler — nochmal'; setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000); }
|
||||
if (dtext) dtext.textContent = 'Aufraeumen fehlgeschlagen: ' + (msg.error || '');
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -6776,12 +6783,17 @@
|
||||
|
||||
// Fuehrt das Aufraeumen WIRKLICH aus (Server-seitig via Docker-API), statt
|
||||
// nur den Befehl zu kopieren.
|
||||
// WICHTIG: "safe" laeuft OHNE confirm() — Build-Cache + ungenutzte Images
|
||||
// sind ungefaehrlich (keine Volumes/Daten). Frueher blockte ein confirm()
|
||||
// den Klick, wenn der Browser Dialoge unterdrueckt hatte ("Verhindern,
|
||||
// dass diese Seite weitere Dialoge erstellt") → confirm()===false → es ging
|
||||
// NICHTS raus. Nur "aggressive" (Volumes!) fragt noch nach.
|
||||
function runDiskCleanup(variant) {
|
||||
const aggressive = variant === 'aggressive';
|
||||
const q = aggressive
|
||||
? 'AGGRESSIV aufraeumen? Loescht zusaetzlich ungenutzte Volumes — nur wenn ALLE ARIA-Container laufen, sonst Datenverlust!'
|
||||
: 'Sicher aufraeumen? Loescht Build-Cache + ungenutzte Images (keine Volumes, keine Daten gehen verloren).';
|
||||
if (!confirm(q)) return;
|
||||
if (aggressive) {
|
||||
const ok = confirm('AGGRESSIV aufraeumen? Loescht zusaetzlich ungenutzte Volumes — nur wenn ALLE ARIA-Container laufen, sonst Datenverlust!');
|
||||
if (!ok) return;
|
||||
}
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
if (btn && !aggressive) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
send({ action: 'disk_cleanup', variant });
|
||||
|
||||
@@ -70,6 +70,10 @@ const ALLOWED_TYPES = new Set([
|
||||
// spiegelt ARIAs Datei-Writes, und QEMU-VNC wird als RFB-Bytes durch RVS
|
||||
// getunnelt (Base64-in-JSON wie audio_pcm — kein Binaer-Handling noetig).
|
||||
"code_file", "code_file_edit",
|
||||
// M1 Generatives Cockpit: ARIA komponiert via present_view eine View-Spec
|
||||
// (Orb + Karten), die App/Web/Diagnostic mit ihrem jeweiligen Renderer
|
||||
// materialisieren. Brain → Bridge → RVS → Clients.
|
||||
"aria_view",
|
||||
"check_desktop", "desktop_status",
|
||||
"vnc_open", "vnc_close", "vnc_data", "vnc_input",
|
||||
// Satelliten (Info-/Gateway-Aussenposten in fremden Netzen): melden sich mit
|
||||
|
||||
+56
-3
@@ -30,7 +30,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # TTS → GPU 0 (8 GB; F5 ist klein)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./voices:/voices # WAV + TXT Referenz
|
||||
@@ -68,7 +68,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["1"] # STT/groesstes Modell → GPU 1 (12 GB; spaeter Voxtral-STT-3B ~9 GB)
|
||||
capabilities: [gpu]
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
@@ -108,7 +108,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # LLM → GPU 0 (8 GB, teilt sich mit TTS)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./models:/models # HF-Download-Cache (persistent)
|
||||
@@ -137,3 +137,56 @@ services:
|
||||
# Erster Load eines Modells kann ein GGUF ziehen (mehrere GB) — grosszuegig.
|
||||
- LLM_TIMEOUT_SEC=${LLM_TIMEOUT_SEC:-600}
|
||||
restart: unless-stopped
|
||||
|
||||
# ─── Voxtral STT (GPU, Realtime) — PROFIL "voxtral" ───────────
|
||||
# Ersetzt whisper als STT sobald die 24-GB-Karte da ist. Startet NUR mit
|
||||
# docker compose --profile voxtral up -d
|
||||
# (sonst kollidiert es mit whisper — beide wuerden stt_* beantworten).
|
||||
#
|
||||
# ⚠️ VRAM: Voxtral-Mini-4B-Realtime-2602 braucht >=16 GB (BF16, laut vLLM-
|
||||
# Rezept keine Quant). Laeuft NICHT auf der 3060 (12 GB) — erst 24-GB-Karte.
|
||||
# ⚠️ vLLM: Version >=0.20.0 noetig. Entrypoint/Serve-Form beim ersten Lauf
|
||||
# gegen das offizielle Rezept pruefen (siehe voxtral/README.md).
|
||||
voxtral-vllm:
|
||||
image: vllm/vllm-openai:latest
|
||||
container_name: aria-voxtral-vllm
|
||||
profiles: ["voxtral"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./hf-cache:/root/.cache/huggingface # gleicher Modell-Cache wie whisper/f5
|
||||
environment:
|
||||
- VLLM_DISABLE_COMPILE_CACHE=1
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
# Serve-Command aus dem vLLM-Rezept (Voxtral-Mini-4B-Realtime-2602).
|
||||
command:
|
||||
- --model
|
||||
- mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- --tokenizer-mode
|
||||
- mistral
|
||||
- --compilation_config
|
||||
- '{"cudagraph_mode":"PIECEWISE"}'
|
||||
restart: unless-stopped
|
||||
|
||||
# ─── Voxtral-Bridge — RVS <-> vLLM-Realtime-WS (CPU-Glue) ─────
|
||||
voxtral-bridge:
|
||||
build: ./voxtral
|
||||
container_name: aria-voxtral-bridge
|
||||
profiles: ["voxtral"]
|
||||
depends_on:
|
||||
- voxtral-vllm
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
- RVS_PORT=${RVS_PORT:-443}
|
||||
- RVS_TLS=${RVS_TLS:-true}
|
||||
- RVS_TLS_FALLBACK=${RVS_TLS_FALLBACK:-true}
|
||||
- RVS_TOKEN=${RVS_TOKEN}
|
||||
- VOXTRAL_VLLM_URL=ws://voxtral-vllm:8000/v1/realtime
|
||||
- VOXTRAL_MODEL=mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- VOXTRAL_LANGUAGE=${WHISPER_LANGUAGE:-de}
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Voxtral-BRIDGE (nicht das Modell!) — leichte CPU-Glue zwischen RVS und dem
|
||||
# vLLM-Realtime-Server. Das eigentliche Voxtral-Modell laeuft im Container
|
||||
# `voxtral-vllm` (GPU, vllm/vllm-openai). Deshalb hier kein CUDA-Base noetig.
|
||||
FROM python:3.11-slim
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY bridge.py ./
|
||||
|
||||
CMD ["python3", "bridge.py"]
|
||||
@@ -0,0 +1,70 @@
|
||||
# Voxtral-STT-Satellit (M0.3)
|
||||
|
||||
Streaming-STT via **Voxtral-Mini-4B-Realtime-2602** (Mistral, Apache 2.0) auf
|
||||
vLLM. Ersetzt whisper als STT — genauer (~5,9 % WER vs 7,4 % FLEURS) und mit
|
||||
echtem Realtime-Streaming. Deutsch ist in den 13 Sprachen abgedeckt.
|
||||
|
||||
Zwei Container:
|
||||
- **`voxtral-vllm`** — das Modell auf vLLM (GPU). Exponiert die Realtime-WS-API.
|
||||
- **`voxtral-bridge`** — CPU-Glue: RVS ⇄ vLLM-Realtime-WS. Macht das Endpointing
|
||||
selbst (adaptiver Rausch-Boden-VAD, identisch zur whisper-Bridge / M0.1).
|
||||
|
||||
## ⚠️ Hardware-Realität — läuft NICHT auf der 3060
|
||||
|
||||
Das Realtime-Modell braucht laut [vLLM-Rezept](https://recipes.vllm.ai/mistralai/Voxtral-Mini-4B-Realtime-2602)
|
||||
**≥ 16 GB VRAM (BF16, keine Quant)**. Die RTX 3060 hat 12 GB → passt nicht.
|
||||
|
||||
- **Interim-gpubox (nur 3060):** whisper (mit M0.1-Fix) + F5-TTS bleiben aktiv.
|
||||
Voxtral NICHT starten.
|
||||
- **Ab der 24-GB-Karte:** Voxtral-Profil hochziehen, whisper wird Fallback.
|
||||
|
||||
Deshalb liegen beide Services hinter dem Compose-**Profil `voxtral`** und starten
|
||||
NUR explizit — sonst würden whisper *und* voxtral dieselben `stt_*`-Messages
|
||||
beantworten (Kollision).
|
||||
|
||||
## Starten (erst wenn die 24-GB-Karte drin ist)
|
||||
|
||||
```bash
|
||||
cd xtts
|
||||
docker compose --profile voxtral up -d --build
|
||||
docker logs -f aria-voxtral-vllm # laedt Modell (mehrere GB, dauert)
|
||||
docker logs -f aria-voxtral-bridge # "RVS verbunden" + service_status ready
|
||||
```
|
||||
Whisper vorher stoppen, damit nur eine STT-Engine antwortet:
|
||||
```bash
|
||||
docker compose stop whisper-bridge
|
||||
```
|
||||
|
||||
## ⚠️ Auf echter Hardware verifizieren (blind gebaut, kein Test hier)
|
||||
|
||||
1. **vLLM-Version ≥ 0.20.0** und die **Serve-Form**. Das Rezept nutzt
|
||||
`vllm serve <model> …`. Falls das `vllm/vllm-openai`-Image einen anderen
|
||||
Entrypoint hat, das `command:` in `docker-compose.yml` anpassen
|
||||
(Rezept-Command steht dort als Kommentar).
|
||||
2. **Realtime-WS-Frames.** Die exakten Event-Namen sind in `bridge.py` ganz oben
|
||||
als Konstanten gebündelt (`VLLM_SEND_APPEND`, `VLLM_DELTA_SUFFIXES`, …),
|
||||
modelliert nach dem OpenAI-Realtime-Schema. Gegen das offizielle
|
||||
**vLLM-Realtime-Client-Beispiel** prüfen und dort anpassen — nur an dieser
|
||||
einen Stelle. Das Response-Handling ist bereits defensiv (mehrere Feldnamen).
|
||||
3. **Endpoint-URL/Port.** Default `ws://voxtral-vllm:8000/v1/realtime` — prüfen ob
|
||||
vLLM auf 8000 lauscht und `/v1/realtime` registriert (Log-Zeile
|
||||
`Route: /v1/realtime`).
|
||||
4. **Endpointing.** Voxtral liefert keine eigene VAD → unser adaptiver Endpointer
|
||||
entscheidet (akustisch + semantisch am Delta-Wachstum). `endpointMs` kommt wie
|
||||
bei whisper aus der App; `voiceFactor` per Session tunebar.
|
||||
|
||||
## Protokoll (RVS, identisch zu whisper — drop-in)
|
||||
|
||||
Rein: `stt_stream_start`, `stt_audio_chunk` (16 kHz mono s16le, base64), `stt_stream_end`.
|
||||
Raus: `stt_partial`, `stt_endpoint` (das Event, auf das aria-bridge horcht), `stt_stream_done`.
|
||||
|
||||
## TTS-Hinweis
|
||||
|
||||
Voxtral-**TTS** (Voice-Cloning) ist hier NICHT enthalten — braucht ebenfalls
|
||||
16 GB VRAM und ist ein eigener Bau. Bis zur 24-GB-Karte bleibt **F5-TTS** aktiv.
|
||||
Danach: eigener `voxtral-tts`-Satellit (separates Ticket).
|
||||
|
||||
## Quellen
|
||||
- Rezept: https://recipes.vllm.ai/mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- vLLM Speech-to-Text: https://docs.vllm.ai/en/latest/serving/online_serving/speech_to_text/
|
||||
- Modell: https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
@@ -0,0 +1,494 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ARIA Voxtral Bridge — Streaming-STT via Voxtral-Mini-4B-Realtime-2602 (vLLM).
|
||||
|
||||
Zwilling der whisper-Bridge, aber das Transkribieren macht NICHT faster-whisper
|
||||
im selben Prozess, sondern der separate vLLM-Realtime-Server (Container
|
||||
`voxtral-vllm`) ueber dessen WebSocket-API `/v1/realtime`. Diese Bridge ist
|
||||
reine Glue:
|
||||
|
||||
App ──(RVS: stt_stream_start / stt_audio_chunk / stt_stream_end)──▶ diese Bridge
|
||||
diese Bridge ──(WS /v1/realtime: PCM16-b64 append)──▶ voxtral-vllm
|
||||
voxtral-vllm ──(transcription.delta / transcription.done)──▶ diese Bridge
|
||||
diese Bridge ──(RVS: stt_partial / stt_endpoint / stt_stream_done)──▶ App/aria-bridge
|
||||
|
||||
Das RVS-Wire-Protokoll ist IDENTISCH zur whisper-Bridge (drop-in). Das
|
||||
Endpointing (wann hat der User aufgehoert zu sprechen) macht diese Bridge
|
||||
selbst — mit demselben ADAPTIVEN Rausch-Boden-Endpointer wie whisper (Voxtral
|
||||
Realtime liefert laut vLLM-Doku keine eigene VAD/„speaker done"-Semantik, nur
|
||||
transcription.delta/.done). Die akustische Energie messen wir auf unserer
|
||||
eigenen PCM-Kopie, die semantische Stagnation am Delta-Textwachstum.
|
||||
|
||||
⚠️ HARDWARE: Voxtral-Mini-4B-Realtime-2602 braucht >=16 GB VRAM (BF16). Auf der
|
||||
RTX 3060 (12 GB) laeuft es NICHT — erst auf der 24-GB-Karte. Bis dahin
|
||||
bleibt die whisper-Bridge aktiv (Profil-gesteuert im docker-compose).
|
||||
|
||||
⚠️ VERIFY-ON-FIRST-RUN: Die exakten vLLM-Realtime-FRAME-Namen (Audio-Append,
|
||||
Delta/Done-Event-Typen) sind unten als Konstanten gebuendelt und nach dem
|
||||
OpenAI-Realtime-Schema modelliert. Gegen das offizielle vLLM-Realtime-
|
||||
Client-Beispiel pruefen und ggf. anpassen — sie stehen bewusst an EINER
|
||||
Stelle. Response-Handling ist defensiv (mehrere moegliche Feldnamen).
|
||||
|
||||
Env:
|
||||
RVS_HOST, RVS_PORT, RVS_TLS, RVS_TLS_FALLBACK, RVS_TOKEN
|
||||
VOXTRAL_VLLM_URL Default: ws://voxtral-vllm:8000/v1/realtime
|
||||
VOXTRAL_MODEL Default: mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
VOXTRAL_LANGUAGE Default: de
|
||||
"""
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import websockets
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
datefmt="%H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("voxtral-bridge")
|
||||
|
||||
RVS_HOST = os.getenv("RVS_HOST", "").strip()
|
||||
RVS_PORT = int(os.getenv("RVS_PORT", "443"))
|
||||
RVS_TLS = os.getenv("RVS_TLS", "true").lower() == "true"
|
||||
RVS_TLS_FALLBACK = os.getenv("RVS_TLS_FALLBACK", "true").lower() == "true"
|
||||
RVS_TOKEN = os.getenv("RVS_TOKEN", "").strip()
|
||||
|
||||
VOXTRAL_VLLM_URL = os.getenv("VOXTRAL_VLLM_URL", "ws://voxtral-vllm:8000/v1/realtime")
|
||||
VOXTRAL_MODEL = os.getenv("VOXTRAL_MODEL", "mistralai/Voxtral-Mini-4B-Realtime-2602")
|
||||
VOXTRAL_LANGUAGE = os.getenv("VOXTRAL_LANGUAGE", "de")
|
||||
|
||||
# ── vLLM-Realtime-Frames — HIER anpassen falls das Client-Beispiel abweicht ──
|
||||
# Senderichtung (wir → vLLM): PCM16-16kHz-mono base64 anhaengen + committen.
|
||||
VLLM_SEND_APPEND = "input_audio_buffer.append" # {"type":..., "audio": "<b64>"}
|
||||
VLLM_SEND_COMMIT = "input_audio_buffer.commit" # Buffer abschliessen
|
||||
VLLM_AUDIO_FIELD = "audio"
|
||||
# Empfangsrichtung (vLLM → wir): inkrementeller Text + final. Defensiv geprueft.
|
||||
VLLM_DELTA_SUFFIXES = ("transcription.delta",) # msg["type"] endet hierauf
|
||||
VLLM_DONE_SUFFIXES = ("transcription.done", "transcription.completed")
|
||||
VLLM_DELTA_FIELDS = ("delta", "text", "transcription") # eins davon traegt den Text
|
||||
|
||||
# ── Streaming-/Endpointing-Parameter (analog whisper-Bridge) ──
|
||||
STREAM_DEFAULT_ENDPOINT_MS = 2400
|
||||
STREAM_DEFAULT_HARD_CAP_MS = 60000
|
||||
STREAM_MIN_AUDIO_MS = 600
|
||||
STREAM_SESSION_TTL_S = 120
|
||||
STREAM_ENERGY_WINDOW_MS = 300
|
||||
STREAM_SEMANTIC_BACKUP_FACTOR = 2.0
|
||||
# Adaptiver Voice-Schwellwert (siehe whisper-Bridge M0.1): Grenze relativ zum
|
||||
# gemessenen Rausch-Boden statt fix — schneidet leises Sprechen nicht ab.
|
||||
STREAM_VOICE_FACTOR = 2.5
|
||||
STREAM_VOICE_RMS_MIN = 0.005
|
||||
STREAM_VOICE_RMS_MAX = 0.020
|
||||
|
||||
|
||||
def pcm_s16le_to_float32(data: bytes) -> np.ndarray:
|
||||
if not data:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
arr = np.frombuffer(data, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
return arr
|
||||
|
||||
|
||||
async def _send(ws, mtype: str, payload: dict) -> None:
|
||||
try:
|
||||
await ws.send(json.dumps({
|
||||
"type": mtype,
|
||||
"payload": payload,
|
||||
"timestamp": int(time.time() * 1000),
|
||||
}))
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Send fehlgeschlagen (%s): %s", mtype, e)
|
||||
|
||||
|
||||
@dataclass
|
||||
class StreamSession:
|
||||
request_id: str
|
||||
audio_request_id: str
|
||||
language: str
|
||||
endpoint_ms: int
|
||||
hard_cap_ms: int
|
||||
voice: str = ""
|
||||
speed: float = 1.0
|
||||
interrupted: bool = False
|
||||
location: Optional[dict] = None
|
||||
sample_rate: int = 16000
|
||||
voice_factor: float = STREAM_VOICE_FACTOR
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
pcm_buffer: bytearray = field(default_factory=bytearray)
|
||||
started_at: float = field(default_factory=time.time)
|
||||
last_chunk_at: float = field(default_factory=time.time)
|
||||
last_partial: str = ""
|
||||
last_growth_at: float = 0.0
|
||||
last_voice_at: float = 0.0
|
||||
noise_floor: float = 0.0
|
||||
closed: bool = False
|
||||
endpoint_sent: bool = False
|
||||
# vLLM-Realtime-Session
|
||||
vllm_ws: object = None
|
||||
vllm_reader: object = None
|
||||
|
||||
|
||||
class SessionManager:
|
||||
def __init__(self) -> None:
|
||||
self._sessions: dict[str, StreamSession] = {}
|
||||
self._ws = None # RVS
|
||||
|
||||
def attach_ws(self, ws) -> None:
|
||||
self._ws = ws
|
||||
|
||||
async def start_session(self, payload: dict) -> Optional[StreamSession]:
|
||||
request_id = (payload.get("requestId") or "").strip()
|
||||
if not request_id:
|
||||
logger.warning("stt_stream_start ohne requestId — ignoriert")
|
||||
return None
|
||||
try:
|
||||
endpoint_ms = int(payload.get("endpointMs") or STREAM_DEFAULT_ENDPOINT_MS)
|
||||
except (TypeError, ValueError):
|
||||
endpoint_ms = STREAM_DEFAULT_ENDPOINT_MS
|
||||
try:
|
||||
hard_cap_ms = int(payload.get("hardCapMs") or STREAM_DEFAULT_HARD_CAP_MS)
|
||||
except (TypeError, ValueError):
|
||||
hard_cap_ms = STREAM_DEFAULT_HARD_CAP_MS
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
sess = StreamSession(
|
||||
request_id=request_id,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
language=payload.get("language") or VOXTRAL_LANGUAGE,
|
||||
endpoint_ms=endpoint_ms,
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=float(payload.get("speed") or 1.0),
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
)
|
||||
# vLLM-Realtime-Session oeffnen + Reader starten.
|
||||
try:
|
||||
sess.vllm_ws = await websockets.connect(VOXTRAL_VLLM_URL, max_size=8 * 1024 * 1024)
|
||||
await self._vllm_configure(sess)
|
||||
sess.vllm_reader = asyncio.create_task(self._vllm_read_loop(sess))
|
||||
except Exception as e:
|
||||
logger.exception("Stream %s: vLLM-Realtime-Connect fehlgeschlagen: %s",
|
||||
request_id[:8], e)
|
||||
# ohne Backend keine Transkription → sofort leeres Endpoint melden
|
||||
self._sessions[request_id] = sess
|
||||
await self._finalize(sess, reason="vllm_unavailable")
|
||||
return None
|
||||
self._sessions[request_id] = sess
|
||||
logger.info("Voxtral-Session offen: id=%s lang=%s endpointMs=%d",
|
||||
request_id[:8], sess.language, sess.endpoint_ms)
|
||||
return sess
|
||||
|
||||
async def _vllm_configure(self, sess: StreamSession) -> None:
|
||||
"""Optionale Session-Konfig an vLLM (Modell/Sprache/temperature=0).
|
||||
VERIFY: exaktes session.update-Schema gegen vLLM-Realtime-Beispiel.
|
||||
Best-effort — Fehler hier sind nicht fatal."""
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({
|
||||
"type": "session.update",
|
||||
"session": {
|
||||
"model": VOXTRAL_MODEL,
|
||||
"language": sess.language,
|
||||
"temperature": 0.0,
|
||||
"input_audio_format": "pcm16",
|
||||
},
|
||||
}))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _vllm_read_loop(self, sess: StreamSession) -> None:
|
||||
"""Liest transcription.delta/.done vom vLLM-Realtime-Server."""
|
||||
ws = sess.vllm_ws
|
||||
try:
|
||||
async for raw in ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
mtype = str(msg.get("type", ""))
|
||||
if any(mtype.endswith(s) for s in VLLM_DELTA_SUFFIXES):
|
||||
text = self._extract_text(msg)
|
||||
if text:
|
||||
await self._on_delta(sess, text)
|
||||
elif any(mtype.endswith(s) for s in VLLM_DONE_SUFFIXES):
|
||||
text = self._extract_text(msg)
|
||||
if text:
|
||||
await self._on_delta(sess, text, final=True)
|
||||
except Exception:
|
||||
logger.debug("Stream %s: vLLM-Reader beendet", sess.request_id[:8])
|
||||
|
||||
@staticmethod
|
||||
def _extract_text(msg: dict) -> str:
|
||||
for f in VLLM_DELTA_FIELDS:
|
||||
v = msg.get(f)
|
||||
if isinstance(v, str) and v:
|
||||
return v
|
||||
return ""
|
||||
|
||||
async def _on_delta(self, sess: StreamSession, text: str, final: bool = False) -> None:
|
||||
"""Neuer/finaler Transkript-Text vom vLLM. delta = inkrementell; wir
|
||||
haengen an, wenn er den bisherigen Partial verlaengert, sonst ersetzen
|
||||
wir (Voxtral kann korrigieren)."""
|
||||
if final or text.startswith(sess.last_partial):
|
||||
new_full = text if final else text
|
||||
else:
|
||||
new_full = (sess.last_partial + text).strip()
|
||||
new_full = new_full.strip()
|
||||
if new_full and new_full != sess.last_partial:
|
||||
sess.last_partial = new_full
|
||||
sess.last_growth_at = time.time()
|
||||
if self._ws is not None:
|
||||
await _send(self._ws, "stt_partial", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": new_full,
|
||||
})
|
||||
|
||||
def feed_chunk(self, payload: dict) -> bool:
|
||||
request_id = payload.get("requestId", "")
|
||||
sess = self._sessions.get(request_id)
|
||||
if sess is None or sess.closed:
|
||||
return False
|
||||
pcm_b64 = payload.get("pcm", "")
|
||||
if not pcm_b64:
|
||||
return True
|
||||
try:
|
||||
pcm = base64.b64decode(pcm_b64)
|
||||
except Exception:
|
||||
return True
|
||||
sess.pcm_buffer.extend(pcm)
|
||||
sess.last_chunk_at = time.time()
|
||||
# An vLLM weiterreichen (fire-and-forget).
|
||||
if sess.vllm_ws is not None:
|
||||
asyncio.create_task(self._vllm_append(sess, pcm_b64))
|
||||
return True
|
||||
|
||||
async def _vllm_append(self, sess: StreamSession, pcm_b64: str) -> None:
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({
|
||||
"type": VLLM_SEND_APPEND,
|
||||
VLLM_AUDIO_FIELD: pcm_b64,
|
||||
}))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def end_session(self, request_id: str) -> None:
|
||||
sess = self._sessions.get(request_id)
|
||||
if sess is not None:
|
||||
sess.closed = True
|
||||
|
||||
def drop(self, request_id: str) -> None:
|
||||
sess = self._sessions.pop(request_id, None)
|
||||
if sess is not None:
|
||||
self._teardown_vllm(sess)
|
||||
|
||||
def _teardown_vllm(self, sess: StreamSession) -> None:
|
||||
try:
|
||||
if sess.vllm_reader is not None:
|
||||
sess.vllm_reader.cancel()
|
||||
except Exception:
|
||||
pass
|
||||
if sess.vllm_ws is not None:
|
||||
asyncio.create_task(self._close_ws(sess.vllm_ws))
|
||||
sess.vllm_ws = None
|
||||
|
||||
@staticmethod
|
||||
async def _close_ws(ws) -> None:
|
||||
try:
|
||||
await ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── Endpointer (adaptiv, wie whisper-Bridge M0.1) ──
|
||||
def _buffer_duration_ms(self, sess: StreamSession) -> float:
|
||||
samples = len(sess.pcm_buffer) // 2
|
||||
return (samples / sess.sample_rate) * 1000.0 if samples else 0.0
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
win_bytes = int(sess.sample_rate * STREAM_ENERGY_WINDOW_MS / 1000) * 2
|
||||
if win_bytes <= 0:
|
||||
return 0.0
|
||||
tail = sess.pcm_buffer[-win_bytes:]
|
||||
if len(tail) < 2:
|
||||
return 0.0
|
||||
arr = pcm_s16le_to_float32(bytes(tail))
|
||||
if arr.size == 0:
|
||||
return 0.0
|
||||
return float(np.sqrt(np.mean(arr * arr)))
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
async def run_endpointer(self) -> None:
|
||||
logger.info("Voxtral-Endpointer gestartet (adaptiver VAD)")
|
||||
while True:
|
||||
await asyncio.sleep(0.2)
|
||||
now = time.time()
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
try:
|
||||
await self._tick(sess, now)
|
||||
except Exception:
|
||||
logger.exception("Endpointer-Tick crashed (session=%s)", sid[:8])
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
if now - sess.last_chunk_at > STREAM_SESSION_TTL_S:
|
||||
logger.info("Stream %s: TTL — drop", sid[:8])
|
||||
self.drop(sid)
|
||||
|
||||
async def _tick(self, sess: StreamSession, now: float) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
elapsed_ms = (now - sess.started_at) * 1000.0
|
||||
if elapsed_ms > sess.hard_cap_ms and not sess.closed:
|
||||
await self._finalize(sess, reason="hardcap")
|
||||
return
|
||||
if sess.closed:
|
||||
await self._finalize(sess, reason="stream_end")
|
||||
return
|
||||
if self._buffer_duration_ms(sess) < STREAM_MIN_AUDIO_MS:
|
||||
return
|
||||
# adaptive akustische Sprach-Aktivitaet
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
else:
|
||||
self._update_noise_floor(sess, rms)
|
||||
# Endpoint: akustisch (Primaer) oder semantisch (Backstop), sobald Text da
|
||||
if sess.last_growth_at > 0.0:
|
||||
acoustic_silence_ms = (now - sess.last_voice_at) * 1000.0 if sess.last_voice_at > 0 else 0.0
|
||||
semantic_silence_ms = (now - sess.last_growth_at) * 1000.0
|
||||
acoustic_done = sess.last_voice_at > 0 and acoustic_silence_ms >= sess.endpoint_ms
|
||||
semantic_done = semantic_silence_ms >= sess.endpoint_ms * STREAM_SEMANTIC_BACKUP_FACTOR
|
||||
if acoustic_done or semantic_done:
|
||||
await self._finalize(sess, reason="endpoint" if acoustic_done else "endpoint_semantic")
|
||||
|
||||
async def _finalize(self, sess: StreamSession, reason: str) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
sess.endpoint_sent = True
|
||||
# vLLM ggf. committen, damit ein letztes transcription.done kommt.
|
||||
if sess.vllm_ws is not None:
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({"type": VLLM_SEND_COMMIT}))
|
||||
await asyncio.sleep(0.15) # kurz auf finalen Delta warten
|
||||
except Exception:
|
||||
pass
|
||||
final_text = sess.last_partial.strip()
|
||||
duration_s = self._buffer_duration_ms(sess) / 1000.0
|
||||
logger.info("Stream %s: FINAL (reason=%s, %.1fs): %r",
|
||||
sess.request_id[:8], reason, duration_s, final_text[:120])
|
||||
if self._ws is not None:
|
||||
endpoint_payload = {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
"durationS": duration_s,
|
||||
"sttMs": 0,
|
||||
"voice": sess.voice,
|
||||
"speed": sess.speed,
|
||||
"interrupted": sess.interrupted,
|
||||
}
|
||||
if sess.location:
|
||||
endpoint_payload["location"] = sess.location
|
||||
await _send(self._ws, "stt_endpoint", endpoint_payload)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
})
|
||||
self.drop(sess.request_id)
|
||||
|
||||
|
||||
async def _broadcast_status(ws, state: str, **extra) -> None:
|
||||
payload = {"service": "voxtral", "state": state}
|
||||
payload.update(extra)
|
||||
await _send(ws, "service_status", payload)
|
||||
|
||||
|
||||
async def run_loop(sessions: SessionManager) -> None:
|
||||
use_tls = RVS_TLS
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
while True:
|
||||
scheme = "wss" if use_tls else "ws"
|
||||
url = f"{scheme}://{RVS_HOST}:{RVS_PORT}/ws?token={RVS_TOKEN}"
|
||||
masked = url.replace(RVS_TOKEN, "***") if RVS_TOKEN else url
|
||||
try:
|
||||
logger.info("Verbinde zu RVS: %s", masked)
|
||||
async with websockets.connect(url, ping_interval=20, ping_timeout=10,
|
||||
max_size=50 * 1024 * 1024) as ws:
|
||||
logger.info("RVS verbunden")
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
sessions.attach_ws(ws)
|
||||
await _broadcast_status(ws, "ready", model=VOXTRAL_MODEL)
|
||||
await _send(ws, "config_request", {"service": "voxtral"})
|
||||
|
||||
async for raw in ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
mtype = msg.get("type", "")
|
||||
payload = msg.get("payload", {}) or {}
|
||||
if mtype == "stt_stream_start":
|
||||
asyncio.create_task(sessions.start_session(payload))
|
||||
elif mtype == "stt_audio_chunk":
|
||||
sessions.feed_chunk(payload)
|
||||
elif mtype == "stt_stream_end":
|
||||
sessions.end_session(payload.get("requestId", ""))
|
||||
# stt_request (Legacy One-Shot) macht Voxtral hier NICHT —
|
||||
# dafuer bleibt die whisper-Bridge (Fallback).
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Verbindung verloren: %s — retry in %ds", e, retry_s)
|
||||
if use_tls and RVS_TLS_FALLBACK and not tls_fallback_tried:
|
||||
use_tls = False
|
||||
tls_fallback_tried = True
|
||||
logger.info("TLS-Fallback: versuche ws:// (kein TLS)")
|
||||
continue
|
||||
await asyncio.sleep(retry_s)
|
||||
retry_s = min(retry_s * 2, 30)
|
||||
use_tls = RVS_TLS # fuer den naechsten Zyklus zuruecksetzen
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
if not RVS_HOST or not RVS_TOKEN:
|
||||
logger.error("RVS_HOST/RVS_TOKEN fehlen — .env pruefen. Abbruch.")
|
||||
return
|
||||
sessions = SessionManager()
|
||||
logger.info("Voxtral-Bridge startet — vLLM=%s Modell=%s", VOXTRAL_VLLM_URL, VOXTRAL_MODEL)
|
||||
await asyncio.gather(
|
||||
run_loop(sessions),
|
||||
sessions.run_endpointer(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
@@ -0,0 +1,5 @@
|
||||
# Voxtral-Bridge ist reine CPU-Glue (RVS-WS <-> vLLM-Realtime-WS). Das Modell
|
||||
# selbst laeuft im separaten voxtral-vllm-Container (GPU). Deshalb hier KEIN
|
||||
# torch/vllm — nur der WebSocket-Client + numpy fuer die RMS-Energiemessung.
|
||||
websockets>=12.0
|
||||
numpy>=1.24
|
||||
+49
-2
@@ -75,7 +75,17 @@ STREAM_SESSION_TTL_S = 120 # tote Sessions nach 2 min aufraeumen
|
||||
# (Whisper oszilliert/halluziniert). Echte akustische Stille ist das robuste
|
||||
# „User hat aufgehoert"-Signal.
|
||||
STREAM_ENERGY_WINDOW_MS = 300 # RMS ueber die letzten 300ms Audio messen
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # RMS darueber = Sprache (haelt Session am Leben)
|
||||
# --- Adaptiver Voice-Schwellwert (ersetzt die fixe 0.012-Grenze) ---
|
||||
# Problem (Repro dokumentiert in audio.ts): eine FESTE RMS-Grenze schneidet
|
||||
# leises/entferntes Sprechen faelschlich als „Stille" (Handy weiter weg vom Mund,
|
||||
# ruhig im Auto, kurze Sprech-Pause) → Cut mitten im Satz. Loesung: die Grenze
|
||||
# relativ zum gemessenen Rausch-Boden der Session fuehren. Sprache =
|
||||
# noise_floor * Faktor, geklammert auf [MIN, MAX]. Bei noch ungelerntem Boden
|
||||
# gilt MIN → sensibel, lieber nicht abschneiden.
|
||||
STREAM_VOICE_FACTOR = 2.5 # Sprache = noise_floor * Faktor
|
||||
STREAM_VOICE_RMS_MIN = 0.005 # Untergrenze (stiller Raum: nicht auf 0 kollabieren)
|
||||
STREAM_VOICE_RMS_MAX = 0.020 # Obergrenze (lautes Auto: Sprache nie ganz aussperren)
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # Legacy-Konstante (nicht mehr im Cut-Pfad genutzt)
|
||||
# Rein-semantischer Backstop: wenn die Energie NIE faellt (laute Umgebung,
|
||||
# z.B. Auto), endpointen wir trotzdem — aber erst nach diesem Faktor x
|
||||
# endpoint_ms, damit normales Sprechen mit Pausen nicht abgeschnitten wird.
|
||||
@@ -323,6 +333,10 @@ class StreamSession:
|
||||
last_growth_at: float = 0.0
|
||||
last_transcribe_at: float = 0.0
|
||||
last_voice_at: float = 0.0 # letzter Tick mit akustischer Sprach-Energie
|
||||
noise_floor: float = 0.0 # adaptiver Rausch-Boden (0.0 = noch ungelernt)
|
||||
voice_factor: float = STREAM_VOICE_FACTOR # per-Session konfigurierbar (Payload voiceFactor)
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
closed: bool = False # nach stream_end gesetzt
|
||||
endpoint_sent: bool = False # Endpoint nur einmal feuern
|
||||
# Speaker-ID Gating: bei aktiviertem Fingerprint pruefen wir die ersten
|
||||
@@ -372,6 +386,10 @@ class SessionManager:
|
||||
speed = float(payload.get("speed") or 1.0)
|
||||
except (TypeError, ValueError):
|
||||
speed = 1.0
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
session = StreamSession(
|
||||
request_id=request_id,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
@@ -381,6 +399,7 @@ class SessionManager:
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=speed,
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
@@ -549,8 +568,16 @@ class SessionManager:
|
||||
# Akustische Sprach-Aktivitaet JEDEN Tick (~200ms) messen — unabhaengig
|
||||
# vom Transcribe-Throttle. Solange wirklich gesprochen wird, bleibt die
|
||||
# Session am Leben, auch wenn Whisper gerade keinen neuen Text liefert.
|
||||
if self._tail_rms(sess) >= STREAM_VOICE_RMS_THRESHOLD:
|
||||
# ADAPTIV: der Schwellwert richtet sich nach dem gemessenen Rausch-Boden
|
||||
# (fast-down/slow-up), damit leises/entferntes Sprechen nicht faelschlich
|
||||
# als Stille gilt und der Satz mitten drin abgeschnitten wird.
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
else:
|
||||
# Rausch-Boden NUR aus Nicht-Sprache lernen — waehrend Sprache
|
||||
# einfrieren, sonst wandert die Grenze hoch und sperrt Sprache aus.
|
||||
self._update_noise_floor(sess, rms)
|
||||
|
||||
# Endpoint-Entscheidung JEDEN Tick, sobald ueberhaupt Text erkannt wurde:
|
||||
# (a) akustisch: seit endpoint_ms keine Sprach-Energie mehr → User ist
|
||||
@@ -620,6 +647,26 @@ class SessionManager:
|
||||
return 0.0
|
||||
return (samples / sess.sample_rate) * 1000.0
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
"""Adaptiver Voice-Schwellwert = Rausch-Boden * Faktor, geklammert auf
|
||||
[min, max]. Bei noch ungelerntem Boden (0.0) → Untergrenze: sensibel,
|
||||
lieber nicht abschneiden (das war der eigentliche Cutoff-Bug)."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
"""Rausch-Boden nachfuehren: schnell runter (neue, leisere Stille),
|
||||
langsam rauf (Umgebung wird lauter). NUR mit Nicht-Sprache aufrufen."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
"""RMS-Energie der letzten STREAM_ENERGY_WINDOW_MS des Audio-Buffers.
|
||||
Dient als akustisches „redet noch / ist still"-Signal."""
|
||||
|
||||
Reference in New Issue
Block a user