Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b226e1da11 | ||
|
|
0b7ed241b4 | ||
|
|
a2b7e3a48d | ||
|
|
7b72149671 | ||
|
|
e9439dbccb | ||
|
|
219de091d2 | ||
|
|
5992a7e441 | ||
|
|
07ccf05429 | ||
|
|
4ab0e68245 | ||
|
|
9636a702d3 | ||
|
|
5f09e2bca3 | ||
|
|
ebe0e8065f | ||
|
|
9d2c07d8d1 | ||
|
|
9aae5af6a9 | ||
|
|
a8ff73f93d | ||
|
|
0e9adeee5c | ||
|
|
6addb2f8fe | ||
|
|
9bdfb7193e | ||
|
|
9e78d75149 | ||
|
|
517c993ac8 | ||
|
|
c1bd13687d | ||
|
|
d9bb7239c6 | ||
|
|
0265aabb5e | ||
|
|
17bc50b847 | ||
|
|
1f2be4299d | ||
|
|
0ca8a82013 | ||
|
|
7bc3f827d0 | ||
|
|
e7da9cf9c4 | ||
|
|
353fd98d3f | ||
|
|
0350dd33c8 | ||
|
|
7b956c6606 | ||
|
|
36f04f83ff | ||
|
|
01df26e6df | ||
|
|
f226c91973 | ||
|
|
3324d39d50 | ||
|
|
fff2e7df34 | ||
|
|
0aac114142 | ||
|
|
ba60f793fb | ||
|
|
a7c2f07361 | ||
|
|
ccf6dd84fb | ||
|
|
75675ed3aa | ||
|
|
73fee27e90 | ||
|
|
03021a6787 | ||
|
|
b6a5d7029f | ||
|
|
3a8202d2de | ||
|
|
7bbb75481c | ||
|
|
ee6c4f34db | ||
|
|
75daadf72d | ||
|
|
37aaa90239 | ||
|
|
03e6d784b6 | ||
|
|
762f1a2dd9 | ||
|
|
49f6b26ab8 | ||
|
|
c340b9d683 | ||
|
|
a0429fc91e | ||
|
|
174d6d643d | ||
|
|
fd6ba73f59 | ||
|
|
f5b22253b2 | ||
|
|
761f4c8903 |
@@ -0,0 +1,63 @@
|
||||
# ai-box — KI-Box (gpubox) Bootstrap
|
||||
|
||||
Macht aus einem frisch installierten **Debian Trixie** (headless, nur SSH) einen
|
||||
startklaren GPU-Satelliten-Host für den `xtts`-Stack (STT/TTS/LLM).
|
||||
|
||||
## Was das Script tut
|
||||
|
||||
1. Basis-Pakete (curl, gnupg, git …)
|
||||
2. `contrib non-free non-free-firmware` aktivieren (Trixie-**deb822**-Format berücksichtigt)
|
||||
3. **NVIDIA-Treiber** installieren (`nvidia-driver` + firmware)
|
||||
4. **Docker** Engine + Compose-Plugin
|
||||
5. **NVIDIA Container Toolkit** + Docker-Runtime auf NVIDIA konfigurieren
|
||||
6. `xtts/.env` aus `.env.example` anlegen (RVS_TOKEN optional gleich setzen)
|
||||
7. **GPU-im-Container-Test** (`docker run --gpus all … nvidia-smi`)
|
||||
8. optional (`--up`): den `xtts`-Stack hochziehen
|
||||
|
||||
Alles **idempotent** — mehrfach ausführbar.
|
||||
|
||||
## Ablauf
|
||||
|
||||
```bash
|
||||
git clone <repo> ARIA-AGENT
|
||||
cd ARIA-AGENT/ai-box
|
||||
|
||||
sudo ./bootstrap.sh
|
||||
# → Wenn der Treiber frisch installiert wurde: einmal neu starten, dann nochmal:
|
||||
sudo reboot
|
||||
# … nach dem Boot:
|
||||
cd ARIA-AGENT/ai-box
|
||||
sudo ./bootstrap.sh --up --token <DEIN_RVS_TOKEN>
|
||||
```
|
||||
|
||||
Der Treiber-Reboot ist normal (Kernel-Modul wird erst beim Boot geladen). Beim
|
||||
zweiten Lauf überspringt das Script alles Erledigte und macht nur noch den
|
||||
GPU-Test + Stack-Start.
|
||||
|
||||
## Optionen
|
||||
|
||||
| Option | Wirkung |
|
||||
|---|---|
|
||||
| `--up` | am Ende `docker compose up -d --build` (Default-Profil) |
|
||||
| `--token <TOK>` | `RVS_TOKEN` in `xtts/.env` eintragen (auch via `RVS_TOKEN=…` env) |
|
||||
| `--rvs-host <H>` | `RVS_HOST` setzen |
|
||||
|
||||
## Wichtig
|
||||
|
||||
- **Stimm-Daten** (nicht in git): falls von der alten Box noch vorhanden,
|
||||
`xtts/voice-id/` (Speaker-Fingerprint) + `xtts/voices/` (F5-Referenz) herkopieren.
|
||||
Sonst egal — in der App neu anlegen: Stimme neu enrollen (ohne Fingerprint läuft
|
||||
die Speaker-ID fail-open, alles geht durch) + F5-Voice-Referenz neu hochladen.
|
||||
- **Voxtral bleibt aus** auf der 3060 (braucht ≥16 GB VRAM). Das Default-Profil
|
||||
fährt Whisper (mit dem M0.1-Fix) + F5-TTS + lokales LLM. Voxtral erst mit der
|
||||
24-GB-Karte: `docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build`.
|
||||
- **Erster Start lädt Modelle** (mehrere GB via HuggingFace nach `xtts/hf-cache`
|
||||
+ `xtts/models`) — genug Platz (1 TB NVMe ✓) und etwas Geduld.
|
||||
|
||||
## Verifizieren
|
||||
|
||||
```bash
|
||||
nvidia-smi # Host sieht die GPU
|
||||
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi # Container auch
|
||||
docker logs -f aria-whisper-bridge # "RVS verbunden" + service_status ready
|
||||
```
|
||||
Executable
+277
@@ -0,0 +1,277 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# ARIA KI-Box (gpubox) Bootstrap — frisches Debian Trixie → startklarer
|
||||
# GPU-Satelliten-Host fuer den xtts-Stack (Voxtral/Whisper STT, F5-TTS, lokales LLM).
|
||||
#
|
||||
# Ablauf nach `git clone`:
|
||||
# cd ARIA-AGENT/ai-box
|
||||
# sudo ./bootstrap.sh # richtet Treiber + Docker + NVIDIA-Toolkit ein
|
||||
# # (falls Treiber frisch installiert: einmal `sudo reboot`, dann Script erneut)
|
||||
# sudo ./bootstrap.sh --up # dazu: xtts-Stack (Whisper+F5+LLM) hochziehen
|
||||
#
|
||||
# Optionen:
|
||||
# --up am Ende den xtts-Stack starten (Default-Profil, OHNE voxtral)
|
||||
# --upgrade-driver NVIDIA-Treiber aus trixie-backports (modernes CUDA fuer Voxtral).
|
||||
# Danach REBOOT noetig. Default-Bootstrap laesst 550 unangetastet.
|
||||
# --token <TOK> RVS_TOKEN in xtts/.env eintragen (alternativ: env RVS_TOKEN=...)
|
||||
# --rvs-host <H> RVS_HOST setzen (Default aus .env.example)
|
||||
#
|
||||
# IDEMPOTENT: bereits erledigte Schritte werden uebersprungen. Nach dem
|
||||
# Treiber-Reboot einfach nochmal ausfuehren — der Rest laeuft dann durch.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ── CLI ──
|
||||
DO_UP=0
|
||||
DO_UPGRADE_DRIVER=0
|
||||
RVS_TOKEN_ARG="${RVS_TOKEN:-}"
|
||||
RVS_HOST_ARG="${RVS_HOST:-}"
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--up) DO_UP=1; shift ;;
|
||||
--upgrade-driver) DO_UPGRADE_DRIVER=1; shift ;;
|
||||
--token) RVS_TOKEN_ARG="${2:-}"; shift 2 ;;
|
||||
--rvs-host) RVS_HOST_ARG="${2:-}"; shift 2 ;;
|
||||
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "Unbekannte Option: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Log-Helfer ──
|
||||
c_g="\033[1;32m"; c_y="\033[1;33m"; c_r="\033[1;31m"; c_b="\033[1;34m"; c_0="\033[0m"
|
||||
STEP=0
|
||||
step() { STEP=$((STEP+1)); echo -e "\n${c_b}[${STEP}] $*${c_0}"; }
|
||||
ok() { echo -e " ${c_g}✓${c_0} $*"; }
|
||||
warn() { echo -e " ${c_y}!${c_0} $*"; }
|
||||
die() { echo -e "${c_r}✗ $*${c_0}" >&2; exit 1; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
XTTS_DIR="$REPO_ROOT/xtts"
|
||||
|
||||
# ── Preflight ──
|
||||
[[ "$(id -u)" -eq 0 ]] || die "Bitte als root ausfuehren (sudo ./bootstrap.sh)."
|
||||
command -v apt-get >/dev/null || die "Kein apt-get — dieses Script ist fuer Debian/Trixie."
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
echo -e "${c_b}=== ARIA KI-Box Bootstrap ===${c_0}"
|
||||
echo "Repo: $REPO_ROOT"
|
||||
echo "xtts: $XTTS_DIR"
|
||||
|
||||
# ── 1. Basis-Pakete ──
|
||||
step "Basis-Pakete"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq ca-certificates curl gnupg git lsb-release >/dev/null
|
||||
ok "ca-certificates, curl, gnupg, git"
|
||||
|
||||
# ── 2. non-free Repos aktivieren (Trixie deb822 + Legacy) ──
|
||||
step "APT-Komponenten (contrib non-free non-free-firmware)"
|
||||
# Ergaenzt die drei Komponenten in JEDER Components:-Zeile einer deb822-Datei —
|
||||
# pro Zeile nur was fehlt, reihenfolge-robust, idempotent. Die Adressen pruefen
|
||||
# ganze Woerter (non-free-firmware zaehlt NICHT als non-free).
|
||||
add_components() {
|
||||
sed -i -E '/^[Cc]omponents:/{
|
||||
/(^|[[:space:]])contrib([[:space:]]|$)/!s/$/ contrib/
|
||||
/(^|[[:space:]])non-free([[:space:]]|$)/!s/$/ non-free/
|
||||
/(^|[[:space:]])non-free-firmware([[:space:]]|$)/!s/$/ non-free-firmware/
|
||||
}' "$1"
|
||||
}
|
||||
ENABLED_ANY=0
|
||||
shopt -s nullglob
|
||||
for f in /etc/apt/sources.list.d/*.sources; do
|
||||
grep -qE '^[Cc]omponents:' "$f" || continue
|
||||
b="$(md5sum "$f")"; add_components "$f"; a="$(md5sum "$f")"
|
||||
if [[ "$b" != "$a" ]]; then ENABLED_ANY=1; ok "aktualisiert: $(basename "$f")"; fi
|
||||
done
|
||||
shopt -u nullglob
|
||||
# Legacy /etc/apt/sources.list (deb-Zeilen)
|
||||
if [[ -f /etc/apt/sources.list ]] && grep -qE '^deb ' /etc/apt/sources.list; then
|
||||
if ! grep -qE '^deb .*[[:space:]]non-free([[:space:]]|$)' /etc/apt/sources.list; then
|
||||
sed -i -E '/^deb .*debian/ s/$/ contrib non-free non-free-firmware/' /etc/apt/sources.list
|
||||
ENABLED_ANY=1; ok "aktualisiert: sources.list"
|
||||
fi
|
||||
fi
|
||||
[[ $ENABLED_ANY -eq 0 ]] && ok "non-free schon aktiv"
|
||||
apt-get update -qq # immer neu einlesen, damit der Kandidat sicher da ist
|
||||
|
||||
# ── 2b. (opt-in) Treiber-Upgrade via trixie-backports — fuer Voxtral/modernes CUDA ──
|
||||
# Der Trixie-Standardtreiber (550, CUDA 12.4) ist zu alt fuer den modernen Stack
|
||||
# (torch 2.13, vLLM). Backports bringt einen neueren, apt-verwalteten Treiber.
|
||||
# NUR mit --upgrade-driver, damit ein laufendes Setup nicht ungewollt angefasst wird.
|
||||
if [[ "$DO_UPGRADE_DRIVER" -eq 1 ]]; then
|
||||
step "NVIDIA-Treiber-Upgrade (trixie-backports)"
|
||||
BP_FILE="/etc/apt/sources.list.d/backports.sources"
|
||||
if ! grep -rqs "trixie-backports" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null; then
|
||||
cat > "$BP_FILE" <<'EOF'
|
||||
Types: deb
|
||||
URIs: http://deb.debian.org/debian
|
||||
Suites: trixie-backports
|
||||
Components: main contrib non-free non-free-firmware
|
||||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||
EOF
|
||||
ok "trixie-backports hinzugefuegt"
|
||||
else
|
||||
ok "trixie-backports bereits aktiv"
|
||||
fi
|
||||
apt-get update
|
||||
apt-get install -y linux-headers-amd64 || true
|
||||
apt-get install -y "linux-headers-$(uname -r)" || true
|
||||
if apt-get install -y -t trixie-backports nvidia-driver; then
|
||||
dkms autoinstall >/dev/null 2>&1 || true
|
||||
NEWV="$(dpkg-query -W -f='${Version}' nvidia-driver 2>/dev/null || echo '?')"
|
||||
ok "nvidia-driver aus backports installiert: ${NEWV}"
|
||||
echo
|
||||
echo -e "${c_y}==> REBOOT noetig, damit der neue Treiber laedt:${c_0}"
|
||||
echo -e "${c_y} sudo reboot && danach: cd ai-box && sudo ./bootstrap.sh --up --token <TOKEN>${c_0}"
|
||||
echo -e "${c_y} (nvidia-smi zeigt dann die neue Version + CUDA-Level)${c_0}"
|
||||
exit 0
|
||||
else
|
||||
warn "backports-Install fehlgeschlagen — 550er bleibt aktiv."
|
||||
warn "Alternative fuer den neuesten Treiber: NVIDIAs CUDA-Repo fuer debian13"
|
||||
warn " (developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64) → Paket 'cuda-drivers'."
|
||||
die "Treiber-Upgrade nicht moeglich — siehe oben."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 3. NVIDIA-Treiber ──
|
||||
step "NVIDIA-Treiber"
|
||||
if nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Treiber aktiv: $(nvidia-smi --query-gpu=name --format=csv,noheader | paste -sd', ')"
|
||||
DRIVER_ACTIVE=1
|
||||
else
|
||||
if dpkg -l | grep -q '^ii nvidia-driver '; then
|
||||
warn "Treiber installiert, aber nvidia-smi antwortet nicht → REBOOT noetig."
|
||||
DRIVER_ACTIVE=0
|
||||
else
|
||||
# KEIN separater Kandidaten-Check — die apt-cache-Ausgabe ist locale-/pipe-
|
||||
# fragil (hat faelschlich "kein Kandidat" gemeldet). Der Install IST der Test:
|
||||
# direkt installieren; schlaegt er fehl, einmal volles apt-get update + Retry,
|
||||
# dann erst mit Diagnose abbrechen.
|
||||
# Kernel-Header ZUERST — sonst ueberspringt DKMS den Modulbau ("No kernel
|
||||
# headers were found") und nvidia-smi kann spaeter nicht mit dem Treiber reden.
|
||||
warn "Kernel-Header + nvidia-driver installieren (DKMS-Build, dauert)…"
|
||||
apt-get install -y linux-headers-amd64 || true
|
||||
apt-get install -y "linux-headers-$(uname -r)" || true
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
warn "Install fehlgeschlagen — volles 'apt-get update' + zweiter Versuch…"
|
||||
apt-get update
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
echo
|
||||
warn "nvidia-driver liess sich nicht installieren. Aktive Quellen:"
|
||||
grep -rhE '^[Cc]omponents:' /etc/apt/sources.list.d/*.sources 2>/dev/null | sed 's/^/ /' || true
|
||||
grep -E '^deb ' /etc/apt/sources.list 2>/dev/null | sed 's/^/ /' || true
|
||||
die "Pruefe 'apt-cache policy nvidia-driver' + Netz/non-free."
|
||||
fi
|
||||
fi
|
||||
# Falls nvidia-kernel-dkms schon (ohne Header) installiert war: Modul jetzt bauen.
|
||||
dkms autoinstall >/dev/null 2>&1 || true
|
||||
ok "nvidia-driver + Kernel-Modul installiert"
|
||||
DRIVER_ACTIVE=0
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 4. Docker Engine + Compose-Plugin ──
|
||||
step "Docker"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
ok "Docker vorhanden: $(docker --version)"
|
||||
else
|
||||
warn "Installiere Docker (offizielles get.docker.com)…"
|
||||
curl -fsSL https://get.docker.com | sh >/dev/null
|
||||
systemctl enable --now docker >/dev/null 2>&1 || true
|
||||
ok "Docker installiert: $(docker --version)"
|
||||
fi
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
ok "Compose-Plugin: $(docker compose version | head -1)"
|
||||
else
|
||||
warn "Compose-Plugin fehlt — installiere docker-compose-plugin…"
|
||||
apt-get install -y -qq docker-compose-plugin >/dev/null || \
|
||||
warn "Konnte docker-compose-plugin nicht via apt holen — get.docker.com bringt es normalerweise mit."
|
||||
fi
|
||||
|
||||
# ── 5. NVIDIA Container Toolkit ──
|
||||
step "NVIDIA Container Toolkit"
|
||||
NCT_LIST="/etc/apt/sources.list.d/nvidia-container-toolkit.list"
|
||||
NCT_KEY="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"
|
||||
if ! dpkg -l | grep -q '^ii nvidia-container-toolkit '; then
|
||||
[[ -f "$NCT_KEY" ]] || curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||
| gpg --dearmor -o "$NCT_KEY"
|
||||
if [[ ! -f "$NCT_LIST" ]]; then
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||
| sed "s#deb https://#deb [signed-by=${NCT_KEY}] https://#g" > "$NCT_LIST"
|
||||
fi
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq nvidia-container-toolkit >/dev/null
|
||||
ok "nvidia-container-toolkit installiert"
|
||||
else
|
||||
ok "nvidia-container-toolkit vorhanden"
|
||||
fi
|
||||
# Docker-Runtime auf NVIDIA konfigurieren (idempotent)
|
||||
if ! grep -q '"nvidia"' /etc/docker/daemon.json 2>/dev/null; then
|
||||
nvidia-ctk runtime configure --runtime=docker >/dev/null
|
||||
systemctl restart docker
|
||||
ok "Docker-Runtime auf NVIDIA konfiguriert + Docker neugestartet"
|
||||
else
|
||||
ok "Docker-Runtime bereits NVIDIA-konfiguriert"
|
||||
fi
|
||||
|
||||
# ── 6. xtts/.env vorbereiten ──
|
||||
step "xtts/.env"
|
||||
if [[ -f "$XTTS_DIR/.env" ]]; then
|
||||
ok ".env existiert bereits (unangetastet)"
|
||||
else
|
||||
cp "$XTTS_DIR/.env.example" "$XTTS_DIR/.env"
|
||||
ok ".env aus .env.example erstellt"
|
||||
fi
|
||||
if [[ -n "$RVS_TOKEN_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_TOKEN=.*#RVS_TOKEN=${RVS_TOKEN_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_TOKEN eingetragen"
|
||||
fi
|
||||
if [[ -n "$RVS_HOST_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_HOST=.*#RVS_HOST=${RVS_HOST_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_HOST=${RVS_HOST_ARG} eingetragen"
|
||||
fi
|
||||
if grep -q '^RVS_TOKEN=dein_token_hier' "$XTTS_DIR/.env"; then
|
||||
warn "RVS_TOKEN ist noch der Platzhalter — vor dem Start setzen:"
|
||||
warn " nano $XTTS_DIR/.env (oder: sudo ./bootstrap.sh --token <TOKEN>)"
|
||||
fi
|
||||
warn "Stimm-Daten (nicht in git): falls von der alten Box noch vorhanden, xtts/voice-id/"
|
||||
warn " + xtts/voices/ herkopieren. Sonst egal — in der App neu anlegen:"
|
||||
warn " Sprache neu enrollen (Speaker-ID, sonst fail-open) + F5-Referenz neu hochladen."
|
||||
|
||||
# ── 7. GPU-im-Container verifizieren ──
|
||||
step "GPU-im-Container Test"
|
||||
if [[ "${DRIVER_ACTIVE:-0}" -eq 1 ]]; then
|
||||
if docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Docker sieht die GPU — KI-Box ist einsatzbereit."
|
||||
GPU_READY=1
|
||||
else
|
||||
warn "Host-Treiber ok, aber Container sieht die GPU nicht — Toolkit/Runtime pruefen."
|
||||
GPU_READY=0
|
||||
fi
|
||||
else
|
||||
warn "Treiber noch nicht aktiv → Test uebersprungen."
|
||||
echo
|
||||
echo -e "${c_y}==> REBOOT noetig, dann Script erneut ausfuehren:${c_0}"
|
||||
echo -e "${c_y} sudo reboot && (nach dem Boot) sudo ./bootstrap.sh${c_0}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 8. Optional: xtts-Stack hochziehen ──
|
||||
if [[ $DO_UP -eq 1 && "${GPU_READY:-0}" -eq 1 ]]; then
|
||||
step "xtts-Stack starten (Default-Profil: Whisper + F5 + LLM — OHNE voxtral)"
|
||||
warn "Erster Start laedt Modelle (mehrere GB via HuggingFace) — kann dauern."
|
||||
( cd "$XTTS_DIR" && docker compose up -d --build )
|
||||
ok "Stack laeuft. Logs: docker logs -f aria-whisper-bridge"
|
||||
echo
|
||||
echo " voxtral (STT via Voxtral) braucht >=16 GB VRAM → erst mit der 24-GB-Karte:"
|
||||
echo " cd $XTTS_DIR && docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build"
|
||||
fi
|
||||
|
||||
# ── Abschluss ──
|
||||
echo
|
||||
echo -e "${c_g}=== Fertig. KI-Box startklar. ===${c_0}"
|
||||
if [[ $DO_UP -eq 0 ]]; then
|
||||
echo "Naechster Schritt — Stack starten:"
|
||||
echo " cd $XTTS_DIR && docker compose up -d --build"
|
||||
echo "oder direkt: sudo ./bootstrap.sh --up"
|
||||
fi
|
||||
@@ -79,8 +79,8 @@ android {
|
||||
applicationId "com.ariacockpit"
|
||||
minSdkVersion rootProject.ext.minSdkVersion
|
||||
targetSdkVersion rootProject.ext.targetSdkVersion
|
||||
versionCode 20300
|
||||
versionName "0.2.3.0"
|
||||
versionCode 20403
|
||||
versionName "0.2.4.3"
|
||||
// Fallback fuer Libraries mit Product Flavors
|
||||
missingDimensionStrategy 'react-native-camera', 'general'
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
// Trigger eingestuft werden kann. Folge: App pausiert beim Oeffnen die Musik,
|
||||
// weil der False-Positive die AudioFocus-Switch-Logik anwirft (Stefan-Bug 06/2026).
|
||||
// Loesung: in dieser Phase keine Detections an JS weiterleiten.
|
||||
private const val STARTUP_SUPPRESSION_MS = 1500L
|
||||
private const val STARTUP_SUPPRESSION_MS = 600L
|
||||
}
|
||||
|
||||
private val env: OrtEnvironment = OrtEnvironment.getEnvironment()
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "aria-cockpit",
|
||||
"version": "0.2.3.0",
|
||||
"version": "0.2.4.3",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"android": "react-native run-android",
|
||||
|
||||
@@ -35,7 +35,7 @@ import MemoryBrowser from '../components/MemoryBrowser';
|
||||
import ErrorBoundary from '../components/ErrorBoundary';
|
||||
import rvs, { RVSMessage, ConnectionState } from '../services/rvs';
|
||||
import audioService from '../services/audio';
|
||||
import wakeWordService, { loadPassiveListenMs } from '../services/wakeword';
|
||||
import wakeWordService from '../services/wakeword';
|
||||
import ProjectsBrowser from '../components/ProjectsBrowser';
|
||||
import brainApi, { Project as BrainProject } from '../services/brainApi';
|
||||
import projectFocus from '../services/projectFocus';
|
||||
@@ -50,7 +50,7 @@ import VoiceButton from '../components/VoiceButton';
|
||||
import FileUpload, { FileData } from '../components/FileUpload';
|
||||
import CameraUpload, { PhotoData } from '../components/CameraUpload';
|
||||
import MessageText from '../components/MessageText';
|
||||
import { loadConvWindowMs, loadTtsSpeed, TTS_SPEED_DEFAULT, loadSttEndpointMs } from '../services/audio';
|
||||
import { loadTtsSpeed, TTS_SPEED_DEFAULT, loadSttEndpointMs, loadMaxRecordingMs, loadBargeInEnabled } from '../services/audio';
|
||||
import Geolocation from '@react-native-community/geolocation';
|
||||
|
||||
// --- Typen ---
|
||||
@@ -93,6 +93,9 @@ interface ChatMessage {
|
||||
* gespiegelt damit wir die EXAKT richtige Placeholder-Bubble ersetzen,
|
||||
* auch wenn mehrere Aufnahmen parallel offen sind. */
|
||||
audioRequestId?: string;
|
||||
/** Laenge der Sprachaufnahme in Sekunden (aus dem stt_endpoint) — fuer die
|
||||
* Dauer-Anzeige an der Voice-Bubble. */
|
||||
durationS?: number;
|
||||
/** Skill-Created-Bubble: ARIA hat einen neuen Skill angelegt */
|
||||
skillCreated?: {
|
||||
name: string;
|
||||
@@ -184,6 +187,14 @@ function stripSystemHints(text: string): string {
|
||||
}
|
||||
return out;
|
||||
}
|
||||
/** Sekunden → "M:SS" fuer die Sprachnachricht-Dauer. */
|
||||
function formatDur(sec: number): string {
|
||||
const s = Math.max(0, Math.round(sec));
|
||||
const m = Math.floor(s / 60);
|
||||
const r = s % 60;
|
||||
return `${m}:${r.toString().padStart(2, '0')}`;
|
||||
}
|
||||
|
||||
const DEFAULT_ATTACHMENT_DIR = `${RNFS.DocumentDirectoryPath}/chat_attachments`;
|
||||
const STORAGE_PATH_KEY = 'aria_attachment_storage_path';
|
||||
|
||||
@@ -373,6 +384,8 @@ const ChatScreen: React.FC = () => {
|
||||
// stoppen? Kommt als 'converse' in der Chat-Payload; onPlaybackFinished liest
|
||||
// es. Default true (Konversation). false = Einzelaktion/Skill-Antwort.
|
||||
const converseRef = useRef<boolean>(true);
|
||||
// Barge-in erlaubt? Default false = Halb-Duplex (waehrend TTS kein Mikro).
|
||||
const bargeInEnabledRef = useRef<boolean>(false);
|
||||
|
||||
const flatListRef = useRef<FlatList>(null);
|
||||
const messageIdCounter = useRef(0);
|
||||
@@ -651,6 +664,7 @@ const ChatScreen: React.FC = () => {
|
||||
const voice = await AsyncStorage.getItem('aria_xtts_voice');
|
||||
localXttsVoiceRef.current = voice || '';
|
||||
ttsSpeedRef.current = await loadTtsSpeed();
|
||||
bargeInEnabledRef.current = await loadBargeInEnabled();
|
||||
const gps = await AsyncStorage.getItem('aria_gps_enabled');
|
||||
setGpsEnabled(gps === 'true');
|
||||
const hints = await AsyncStorage.getItem('aria_show_hints');
|
||||
@@ -1387,13 +1401,30 @@ const ChatScreen: React.FC = () => {
|
||||
// Fallback mehr: die Bridge schickt speak zuverlaessig mit.
|
||||
// Merken ob nach dem Vorlesen 30s weiterlauschen (Gespraech) oder direkt
|
||||
// stoppen — onPlaybackFinished liest converseRef. Default true.
|
||||
converseRef.current = (message.payload as any).converse !== false;
|
||||
// Passiv-Lauschen (30s) NUR wenn das Brain explizit converse:true schickt.
|
||||
// Vorher default true → jeder Befehl (auch "Spiele Spotify" mit gesproche-
|
||||
// ner Bestaetigung) landete im 30s-Fenster. Jetzt: einzelne Befehle enden
|
||||
// sofort (zurueck aufs Wake-Word), nur echte Gespraeche lauschen weiter.
|
||||
converseRef.current = (message.payload as any).converse === true;
|
||||
const _isSilent = (message.payload as any).speak === false;
|
||||
if (_isSilent && wakeWordService.isConversing()) {
|
||||
// Klarer Steuerbefehl (Liedersteuerung etc.) = KEINE Konversation →
|
||||
// STOP: direkt zurueck aufs Wake-Word. Kein Gong, keine Aufnahme,
|
||||
// kein 30s-Fenster (skipPassive=true).
|
||||
wakeWordService.endConversation(true).catch(() => {});
|
||||
if (_isSilent) {
|
||||
// Steuerbefehl (speak=false) ist ausgefuehrt und wird NICHT vorgelesen.
|
||||
// Ohne TTS feuert onPlaybackFinished nie — der Mikro-/Konversations-
|
||||
// Lifecycle muss hier selbst weitergeschaltet werden, sonst haengt das Ohr.
|
||||
if (converseRef.current) {
|
||||
// Befehlskette laeuft WEITER ([[WEITER]]): Mikro NICHT schliessen,
|
||||
// sondern das passive Lausch-Fenster oeffnen (endConversation(false)),
|
||||
// damit der naechste Kettenbefehl direkt gesprochen werden kann. ARIA
|
||||
// haelt bewusst offen, bis sie [[ENDE]] (converse=false) schickt.
|
||||
if (wakeWordService.isConversing()) {
|
||||
wakeWordService.endConversation(false).catch(() => {});
|
||||
}
|
||||
} else {
|
||||
// Einzelbefehl / [[ENDE]] → ARIA "drueckt selbst Stop": jede offene
|
||||
// Aufnahme schliessen + zurueck aufs Wake-Word, egal in welchem Zustand
|
||||
// (conversing, passives Lauschen ODER offene Streaming-Aufnahme).
|
||||
ariaStopRecording('silent-command').catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1648,7 +1679,11 @@ const ChatScreen: React.FC = () => {
|
||||
rememberMyRequest(audioRequestId);
|
||||
const wasInterrupted = interruptAriaIfBusy();
|
||||
const location = await getCurrentLocation();
|
||||
const windowMs = await loadConvWindowMs();
|
||||
// EIN Wert regiert: die Stille-Toleranz. Sie gilt sowohl als Pause WÄHREND
|
||||
// des Redens (endpointMs) ALS AUCH als "wenn du nicht anfängst zu reden,
|
||||
// ist Schluss" (noSpeechTimeoutMs). Kein separates 30s-Konversationsfenster
|
||||
// mehr — Stefans Modell: sagst du nichts, greift der Stille-Wert.
|
||||
const sttEndpointMs = await loadSttEndpointMs();
|
||||
|
||||
const userMsg: ChatMessage = {
|
||||
id: nextId(),
|
||||
@@ -1666,9 +1701,11 @@ const ChatScreen: React.FC = () => {
|
||||
speed: ttsSpeedRef.current,
|
||||
interrupted: wasInterrupted,
|
||||
location: location || null,
|
||||
noSpeechTimeoutMs: windowMs,
|
||||
endpointMs: await loadSttEndpointMs(),
|
||||
hardCapMs: 60000,
|
||||
noSpeechTimeoutMs: sttEndpointMs,
|
||||
endpointMs: sttEndpointMs,
|
||||
// Notbremse 5 min (nicht 1 min) — der Stille-Endpoint beendet normale
|
||||
// Turns eh sofort; der Cap darf lange Diktate nicht mitten drin kappen.
|
||||
hardCapMs: await loadMaxRecordingMs(),
|
||||
projectId: focusedProjectIdRef.current,
|
||||
});
|
||||
import('../services/logger').then(m => m.reportAppDebug('wake.cb', `startStreamingRecording returned ok=${ok}`)).catch(()=>{});
|
||||
@@ -1696,6 +1733,13 @@ const ChatScreen: React.FC = () => {
|
||||
if (ev.text && ev.text.trim()) {
|
||||
console.log('[Chat] STT-Endpoint: %r (reason=%s, %dms, %.1fs Audio)',
|
||||
ev.text.slice(0, 80), ev.reason, ev.sttMs, ev.durationS);
|
||||
// Aufnahme-Dauer an die passende Voice-Bubble haengen (Anzeige). Der
|
||||
// spaetere STT-Text-Update spreadet die Message, die Dauer bleibt.
|
||||
if (ev.audioRequestId && typeof ev.durationS === 'number' && ev.durationS > 0) {
|
||||
const dur = ev.durationS;
|
||||
setMessages(prev => prev.map(m =>
|
||||
m.audioRequestId === ev.audioRequestId ? { ...m, durationS: dur } : m));
|
||||
}
|
||||
// Wenn passive lauschend: User hat tatsaechlich was gesagt → uebergang
|
||||
// zu 'conversing' damit der normale Flow greift (TTS, resume, etc.)
|
||||
if (wakeWordService.getState() === 'listening') {
|
||||
@@ -1716,12 +1760,12 @@ const ChatScreen: React.FC = () => {
|
||||
!(m.audioRequestId === ev.audioRequestId
|
||||
&& m.text.includes('Spracheingabe wird verarbeitet'))));
|
||||
}
|
||||
// Bei Passive-Listen + speaker_mismatch oder no-speech: erneut passiv
|
||||
// lauschen (Timer im wakeword-service laeuft weiter, regelt das Ende).
|
||||
// Sonst endConversation wie bisher.
|
||||
// Kein Re-Arm mehr: nach ARIAs Antwort gab es EIN Stille-Fenster (=
|
||||
// Stille-Toleranz). Kam nichts, ist Schluss → zurück aufs Wake-Word.
|
||||
// Kein 30s-Nachlauschen. (speaker_mismatch/no-speech landen beide hier.)
|
||||
if (wakeWordService.getState() === 'listening') {
|
||||
console.log('[Chat] Passive-Listen: leeres Endpoint — naechste passive Aufnahme');
|
||||
startPassiveStreamingRecording();
|
||||
console.log('[Chat] Passive-Listen: leeres Endpoint — Ende, zurueck aufs Wake-Word');
|
||||
wakeWordService.exitPassiveListening('timeout').catch(() => {});
|
||||
} else {
|
||||
wakeWordService.endConversation();
|
||||
if (!wakeWordService.isActive()) setWakeWordActive(false);
|
||||
@@ -1751,7 +1795,7 @@ const ChatScreen: React.FC = () => {
|
||||
const audioRequestId = `audio_${Date.now()}_${Math.floor(Math.random() * 100000)}`;
|
||||
rememberMyRequest(audioRequestId);
|
||||
const location = await getCurrentLocation();
|
||||
const windowMs = await loadConvWindowMs();
|
||||
const sttEndpointMs = await loadSttEndpointMs(); // ein Wert für Pause + No-Speech
|
||||
|
||||
const userMsg: ChatMessage = {
|
||||
id: nextId(),
|
||||
@@ -1769,9 +1813,10 @@ const ChatScreen: React.FC = () => {
|
||||
speed: ttsSpeedRef.current,
|
||||
interrupted: true, // Barge-In → Brain weiss "User hat unterbrochen"
|
||||
location: location || null,
|
||||
noSpeechTimeoutMs: windowMs,
|
||||
endpointMs: await loadSttEndpointMs(),
|
||||
hardCapMs: 60000,
|
||||
noSpeechTimeoutMs: sttEndpointMs,
|
||||
endpointMs: sttEndpointMs,
|
||||
// Notbremse 5 min (s.o.) — lange Diktate nicht bei 1 min abschneiden.
|
||||
hardCapMs: await loadMaxRecordingMs(),
|
||||
projectId: focusedProjectIdRef.current,
|
||||
});
|
||||
if (ok) {
|
||||
@@ -1789,7 +1834,9 @@ const ChatScreen: React.FC = () => {
|
||||
// Prozess nicht killt wenn die App im Hintergrund ist.
|
||||
const unsubTtsStart = audioService.onPlaybackStarted(() => {
|
||||
acquireBackgroundAudio('tts').catch(() => {});
|
||||
if (wakeWordService.isConversing() && wakeWordService.hasWakeWord()) {
|
||||
// Barge-Listening (Mikro waehrend TTS) NUR im Barge-in-Modus. Default aus =
|
||||
// Halb-Duplex: ARIA spricht ungestoert zu Ende, dann erst geht das Mikro auf.
|
||||
if (bargeInEnabledRef.current && wakeWordService.isConversing() && wakeWordService.hasWakeWord()) {
|
||||
wakeWordService.startBargeListening().catch(() => {});
|
||||
}
|
||||
});
|
||||
@@ -1828,16 +1875,20 @@ const ChatScreen: React.FC = () => {
|
||||
const audioRequestId = `audio_passive_${Date.now()}_${Math.floor(Math.random() * 100000)}`;
|
||||
rememberMyRequest(audioRequestId);
|
||||
const location = await getCurrentLocation();
|
||||
const passiveMs = await loadPassiveListenMs();
|
||||
// Kein 30s-Passiv-Fenster mehr: nach ARIAs Antwort geht das Mikro auf, und
|
||||
// fängst du nicht innerhalb der Stille-Toleranz an zu reden, ist Schluss →
|
||||
// zurück aufs Wake-Word. Derselbe Wert wie die Pause-Toleranz beim Reden.
|
||||
const sttEndpointMs = await loadSttEndpointMs();
|
||||
const { ok } = await audioService.startStreamingRecording({
|
||||
audioRequestId,
|
||||
voice: localXttsVoiceRef.current,
|
||||
speed: ttsSpeedRef.current,
|
||||
interrupted: false,
|
||||
location: location || null,
|
||||
noSpeechTimeoutMs: Math.min(passiveMs, 30000),
|
||||
endpointMs: await loadSttEndpointMs(),
|
||||
hardCapMs: Math.max(passiveMs + 5000, 35000),
|
||||
noSpeechTimeoutMs: sttEndpointMs,
|
||||
endpointMs: sttEndpointMs,
|
||||
// Lange Antworten nicht kappen (früher 35s → schnitt langes Reden ab).
|
||||
hardCapMs: await loadMaxRecordingMs(),
|
||||
projectId: focusedProjectIdRef.current,
|
||||
});
|
||||
if (!ok) {
|
||||
@@ -2210,15 +2261,16 @@ const ChatScreen: React.FC = () => {
|
||||
advanceQueue(pid);
|
||||
}, [advanceQueue]);
|
||||
|
||||
// Queue-Modus („immer anstellen"): eine neue Sprachnachricht bricht ARIAs
|
||||
// laufende Arbeit NICHT mehr ab. Sie wird — wie Text — angestellt und laeuft
|
||||
// serialisiert (der Brain-Lock pro Projekt reiht /chat-/audio-Turns auf).
|
||||
// Nur das TTS wird akustisch gestoppt, damit das Mikro ARIAs eigene Stimme
|
||||
// nicht mithoert. Explizites Abbrechen laeuft ueber den Stop-Button
|
||||
// (cancelRequest). Rueckgabe = false, weil kein Barge-In/Interrupt mehr.
|
||||
// Nimmt der User das Mikro waehrend ARIA SPRICHT, ist das ein echter Interrupt:
|
||||
// TTS stoppen UND die laufende Brain-Antwort abbrechen (cancel_request). Sonst
|
||||
// produziert das Brain weiter TTS, die ins offene Mikro laeuft → genau der
|
||||
// "Mischmasch" (ARIA antwortet weiter waehrend ich rede). Fuer bewusstes
|
||||
// Nicht-Abbrechen gibt es weiterhin den separaten Zwischenruf-Button (📣).
|
||||
const interruptAriaIfBusy = useCallback(() => {
|
||||
if (audioService.isPlayingAudio()) {
|
||||
audioService.haltAllPlayback('user startet Aufnahme (Queue-Modus, kein Abbruch)');
|
||||
audioService.haltAllPlayback('user startet Aufnahme — Interrupt');
|
||||
rvs.send('cancel_request' as any, { hard: true, source: 'voice-interrupt' });
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}, []);
|
||||
@@ -2255,7 +2307,7 @@ const ChatScreen: React.FC = () => {
|
||||
// die Session auch app-seitig haben wir +2s Toleranz.
|
||||
noSpeechTimeoutMs: 0,
|
||||
endpointMs: await loadSttEndpointMs(),
|
||||
hardCapMs: 300000,
|
||||
hardCapMs: await loadMaxRecordingMs(),
|
||||
projectId: focusedProjectIdRef.current,
|
||||
});
|
||||
if (!ok) {
|
||||
@@ -2267,11 +2319,50 @@ const ChatScreen: React.FC = () => {
|
||||
return true;
|
||||
}, [getCurrentLocation, interruptAriaIfBusy, scheduleStaleAudioCleanup]);
|
||||
|
||||
// ARIA schliesst die Aufnahme SELBST — das programmatische Gegenstueck zum
|
||||
// Stop-Button. Aufgerufen nach einem stillen Steuerbefehl (speak=false): der
|
||||
// Befehl ist ausgefuehrt, ARIA hat die Rueckinfo (Skill-Ergebnis) und antwortet
|
||||
// NICHT vorgelesen. Weil ohne TTS kein onPlaybackFinished kommt, muss der
|
||||
// Aufnahme-/Konversations-Zustand hier aktiv aufgeraeumt werden, sonst bleibt
|
||||
// das Ohr haengen bzw. das Aufnahme-Fenster laeuft leer weiter (Stefans
|
||||
// Reproduktion: "spotify play" und das Mikro wartet trotzdem 30s).
|
||||
// Unterschied zum manuellen Stop: der verwirft NICHT, sondern finalisiert die
|
||||
// Aufnahme (User will seinen Satz verarbeitet haben) — hier ist der Befehl
|
||||
// schon durch, ein evtl. offenes Folge-Fenster wird verworfen.
|
||||
const ariaStopRecording = useCallback(async (reason: string): Promise<void> => {
|
||||
converseRef.current = false;
|
||||
// 1) Passiv-Lauschen: sauber beenden (cancelt den Stream selbst, startet
|
||||
// KEINE neue passive Aufnahme).
|
||||
if (wakeWordService.getState() === 'listening') {
|
||||
await wakeWordService.exitPassiveListening('manual').catch(() => {});
|
||||
return;
|
||||
}
|
||||
// 2) Noch offene Streaming-Aufnahme (aktiv / Barge-In) verwerfen.
|
||||
if (audioService.isStreamingRecording()) {
|
||||
await audioService.cancelStreamingRecording(reason).catch(() => {});
|
||||
}
|
||||
// 3) Konversation beenden → zurueck aufs Wake-Word (skipPassive: kein 30s-Fenster).
|
||||
if (wakeWordService.isConversing()) {
|
||||
await wakeWordService.endConversation(true).catch(() => {});
|
||||
} else if (!wakeWordService.isActive()) {
|
||||
setWakeWordActive(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
// Manueller Aufnahme-Knopf — Stop. Sendet stt_stream_end an Whisper, die
|
||||
// dann ihrerseits den finalen Text als stt_endpoint emittiert. aria-bridge
|
||||
// forwarded direkt an Brain. Im wake-word-conversing-Fall zusaetzlich
|
||||
// endConversation: User hat explizit gestoppt → kein Multi-Turn-Resume.
|
||||
const handleVoiceButtonStop = useCallback(async (): Promise<void> => {
|
||||
// Manueller Stop = endgueltig: auch die NACH der Antwort kommende
|
||||
// onPlaybackFinished darf kein 30s-Passiv-Fenster mehr oeffnen.
|
||||
converseRef.current = false;
|
||||
// Stop = ALLES beenden, vorhersehbar. Spricht ARIA gerade, hart stoppen +
|
||||
// laufende Brain-Antwort abbrechen (sonst "sagt sie ihren letzten Satz").
|
||||
if (audioService.isPlayingAudio()) {
|
||||
audioService.haltAllPlayback('user stop');
|
||||
rvs.send('cancel_request' as any, { hard: true, source: 'voice-stop' });
|
||||
}
|
||||
// Stop WAEHREND des passiven 30s-Lauschens ('listening'): sauber beenden
|
||||
// (zurueck aufs Wake-Word), NICHT den passiven Stream neu starten.
|
||||
// exitPassiveListening cancelt den Stream selbst (via _freeMic) → es feuert
|
||||
@@ -2647,6 +2738,16 @@ const ChatScreen: React.FC = () => {
|
||||
{att.serverPath ? '(tippen zum Laden)' : '(nicht verfuegbar)'}
|
||||
</Text>
|
||||
</TouchableOpacity>
|
||||
) : att.type === 'audio' ? (
|
||||
<View style={styles.attachmentFile}>
|
||||
<Text style={styles.attachmentFileIcon}>{'🎙'}</Text>
|
||||
<Text style={styles.attachmentFileName} numberOfLines={1}>
|
||||
{att.name || 'Sprachaufnahme'}
|
||||
</Text>
|
||||
{typeof item.durationS === 'number' && item.durationS > 0 ? (
|
||||
<Text style={styles.attachmentFileSize}>{formatDur(item.durationS)}</Text>
|
||||
) : null}
|
||||
</View>
|
||||
) : (
|
||||
<TouchableOpacity
|
||||
style={styles.attachmentFile}
|
||||
|
||||
@@ -63,14 +63,16 @@ import {
|
||||
VAD_SILENCE_MIN_SEC,
|
||||
VAD_SILENCE_MAX_SEC,
|
||||
VAD_SILENCE_STORAGE_KEY,
|
||||
CONV_WINDOW_DEFAULT_SEC,
|
||||
CONV_WINDOW_MIN_SEC,
|
||||
CONV_WINDOW_MAX_SEC,
|
||||
CONV_WINDOW_STORAGE_KEY,
|
||||
STT_ENDPOINT_DEFAULT_MS,
|
||||
STT_ENDPOINT_MIN_MS,
|
||||
STT_ENDPOINT_MAX_MS,
|
||||
STT_ENDPOINT_STORAGE_KEY,
|
||||
MAX_RECORDING_DEFAULT_SEC,
|
||||
MAX_RECORDING_MIN_SEC,
|
||||
MAX_RECORDING_MAX_SEC,
|
||||
MAX_RECORDING_STORAGE_KEY,
|
||||
loadBargeInEnabled,
|
||||
saveBargeInEnabled,
|
||||
VAD_SILENCE_DB_DEFAULT,
|
||||
VAD_SILENCE_DB_MIN,
|
||||
VAD_SILENCE_DB_MAX,
|
||||
@@ -110,9 +112,8 @@ import wakeWordService, {
|
||||
WAKE_THRESHOLD_MAX,
|
||||
loadWakeThreshold,
|
||||
saveWakeThreshold,
|
||||
PASSIVE_LISTEN_DEFAULT_MS,
|
||||
loadPassiveListenMs,
|
||||
savePassiveListenMs,
|
||||
loadBgWakeEnabled,
|
||||
saveBgWakeEnabled,
|
||||
} from '../services/wakeword';
|
||||
import ModeSelector from '../components/ModeSelector';
|
||||
import QRScanner from '../components/QRScanner';
|
||||
@@ -195,8 +196,12 @@ const SettingsScreen: React.FC = () => {
|
||||
const [ttsEnabled, setTtsEnabled] = useState(true);
|
||||
const [ttsPrerollSec, setTtsPrerollSec] = useState<number>(TTS_PREROLL_DEFAULT_SEC);
|
||||
const [vadSilenceSec, setVadSilenceSec] = useState<number>(VAD_SILENCE_DEFAULT_SEC);
|
||||
const [convWindowSec, setConvWindowSec] = useState<number>(CONV_WINDOW_DEFAULT_SEC);
|
||||
// Aktive Streaming-Pausen-Toleranz (STT_ENDPOINT) — der "Stille-Toleranz"-Regler
|
||||
// steuert jetzt DIESEN Wert (der alte vadSilenceSec war der tote Legacy-dB-Pfad).
|
||||
const [sttEndpointSec, setSttEndpointSec] = useState<number>(STT_ENDPOINT_DEFAULT_MS / 1000);
|
||||
const [maxRecordingSec, setMaxRecordingSec] = useState<number>(MAX_RECORDING_DEFAULT_SEC);
|
||||
// Barge-in: ARIA waehrend ihrer Antwort unterbrechen duerfen. Default aus (Halb-Duplex).
|
||||
const [bargeIn, setBargeIn] = useState<boolean>(false);
|
||||
// null = automatisch (adaptive Baseline), sonst manueller dB-Override
|
||||
const [vadSilenceDb, setVadSilenceDb] = useState<number | null>(null);
|
||||
const [showVadInfo, setShowVadInfo] = useState(false);
|
||||
@@ -209,7 +214,8 @@ const SettingsScreen: React.FC = () => {
|
||||
const [wakeStatus, setWakeStatus] = useState<string>('');
|
||||
const [wakeReadySound, setWakeReadySound] = useState<boolean>(true);
|
||||
const [wakeThreshold, setWakeThreshold] = useState<number>(WAKE_THRESHOLD_DEFAULT);
|
||||
const [passiveSec, setPassiveSec] = useState<number>(Math.round(PASSIVE_LISTEN_DEFAULT_MS / 1000));
|
||||
// Hintergrund-Wake: auch bei gesperrtem Bildschirm auf das Wake-Wort hoeren. Default aus.
|
||||
const [bgWake, setBgWake] = useState<boolean>(false);
|
||||
const [editingPath, setEditingPath] = useState(false);
|
||||
const [xttsVoice, setXttsVoice] = useState('');
|
||||
const [loadingVoice, setLoadingVoice] = useState<string | null>(null);
|
||||
@@ -298,11 +304,11 @@ const SettingsScreen: React.FC = () => {
|
||||
}
|
||||
}
|
||||
});
|
||||
AsyncStorage.getItem(CONV_WINDOW_STORAGE_KEY).then(saved => {
|
||||
AsyncStorage.getItem(STT_ENDPOINT_STORAGE_KEY).then(saved => {
|
||||
if (saved != null) {
|
||||
const n = parseFloat(saved);
|
||||
if (isFinite(n) && n >= CONV_WINDOW_MIN_SEC && n <= CONV_WINDOW_MAX_SEC) {
|
||||
setConvWindowSec(n);
|
||||
const n = parseInt(saved, 10);
|
||||
if (isFinite(n) && n >= STT_ENDPOINT_MIN_MS && n <= STT_ENDPOINT_MAX_MS) {
|
||||
setSttEndpointSec(n / 1000);
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -314,6 +320,7 @@ const SettingsScreen: React.FC = () => {
|
||||
}
|
||||
}
|
||||
});
|
||||
loadBargeInEnabled().then(setBargeIn).catch(() => {});
|
||||
AsyncStorage.getItem(VAD_SILENCE_DB_OVERRIDE_KEY).then(saved => {
|
||||
if (saved != null && saved !== '') {
|
||||
const n = parseFloat(saved);
|
||||
@@ -333,7 +340,7 @@ const SettingsScreen: React.FC = () => {
|
||||
});
|
||||
isWakeReadySoundEnabled().then(setWakeReadySound);
|
||||
loadWakeThreshold().then(setWakeThreshold).catch(() => {});
|
||||
loadPassiveListenMs().then(ms => setPassiveSec(Math.round(ms / 1000))).catch(() => {});
|
||||
loadBgWakeEnabled().then(setBgWake).catch(() => {});
|
||||
updateService.getApkCacheSize().then(setApkCacheInfo).catch(() => {});
|
||||
audioService.getTtsCacheSize().then(setTtsCacheInfo).catch(() => {});
|
||||
AsyncStorage.getItem('aria_xtts_voice').then(saved => {
|
||||
@@ -1651,72 +1658,56 @@ const SettingsScreen: React.FC = () => {
|
||||
{currentSection === 'voice_input' && (<>
|
||||
<Text style={styles.sectionTitle}>Spracheingabe</Text>
|
||||
<View style={styles.card}>
|
||||
<Text style={styles.toggleLabel}>Stille-Toleranz</Text>
|
||||
<View style={styles.toggleRow}>
|
||||
<View style={styles.toggleInfo}>
|
||||
<Text style={styles.toggleLabel}>Barge-in (unterbrechen)</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
AUS (empfohlen): ARIA spricht ihre Antwort ZU ENDE, dann geht das
|
||||
Mikro auf — sauber, kein Selbst-Echo, du hoerst sie ganz. AN: du
|
||||
kannst sie waehrend des Sprechens per Wake-Wort unterbrechen.
|
||||
</Text>
|
||||
</View>
|
||||
<Switch
|
||||
value={bargeIn}
|
||||
onValueChange={(v) => { setBargeIn(v); saveBargeInEnabled(v).catch(() => {}); }}
|
||||
trackColor={{ false: '#2A2A3E', true: '#0096FF' }}
|
||||
thumbColor={bargeIn ? '#FFFFFF' : '#666680'}
|
||||
/>
|
||||
</View>
|
||||
|
||||
<Text style={[styles.toggleLabel, {marginTop: 20}]}>Stille-Toleranz</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Wie lange du eine Sprechpause machen darfst, bevor die Aufnahme
|
||||
automatisch beendet und gesendet wird. Hoeher = mehr Zeit zum
|
||||
Nachdenken; niedriger = schnelleres Senden.
|
||||
Default: {VAD_SILENCE_DEFAULT_SEC.toFixed(1)}s.
|
||||
Nachdenken (z.B. im Auto); niedriger = schnelleres Senden.
|
||||
Default: {(STT_ENDPOINT_DEFAULT_MS / 1000).toFixed(1)}s.
|
||||
</Text>
|
||||
<View style={styles.prerollRow}>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.max(VAD_SILENCE_MIN_SEC, Math.round((vadSilenceSec - 0.5) * 10) / 10);
|
||||
setVadSilenceSec(next);
|
||||
AsyncStorage.setItem(VAD_SILENCE_STORAGE_KEY, String(next));
|
||||
const next = Math.max(STT_ENDPOINT_MIN_MS / 1000, Math.round((sttEndpointSec - 0.5) * 10) / 10);
|
||||
setSttEndpointSec(next);
|
||||
AsyncStorage.setItem(STT_ENDPOINT_STORAGE_KEY, String(Math.round(next * 1000)));
|
||||
}}
|
||||
disabled={vadSilenceSec <= VAD_SILENCE_MIN_SEC}
|
||||
disabled={sttEndpointSec <= STT_ENDPOINT_MIN_MS / 1000}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>−0.5</Text>
|
||||
</TouchableOpacity>
|
||||
<Text style={styles.prerollValue}>{vadSilenceSec.toFixed(1)} s</Text>
|
||||
<Text style={styles.prerollValue}>{sttEndpointSec.toFixed(1)} s</Text>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.min(VAD_SILENCE_MAX_SEC, Math.round((vadSilenceSec + 0.5) * 10) / 10);
|
||||
setVadSilenceSec(next);
|
||||
AsyncStorage.setItem(VAD_SILENCE_STORAGE_KEY, String(next));
|
||||
const next = Math.min(STT_ENDPOINT_MAX_MS / 1000, Math.round((sttEndpointSec + 0.5) * 10) / 10);
|
||||
setSttEndpointSec(next);
|
||||
AsyncStorage.setItem(STT_ENDPOINT_STORAGE_KEY, String(Math.round(next * 1000)));
|
||||
}}
|
||||
disabled={vadSilenceSec >= VAD_SILENCE_MAX_SEC}
|
||||
disabled={sttEndpointSec >= STT_ENDPOINT_MAX_MS / 1000}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>+0.5</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
|
||||
<Text style={[styles.toggleLabel, {marginTop: 24}]}>Konversations-Fenster</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Im Gespraechsmodus (Ohr-Button): nach ARIA's Antwort hast du so lange
|
||||
Zeit, weiter zu sprechen, bevor die Konversation automatisch beendet wird.
|
||||
Sprichst du nichts → Mikrofon zu.
|
||||
Default: {CONV_WINDOW_DEFAULT_SEC.toFixed(1)}s.
|
||||
</Text>
|
||||
<View style={styles.prerollRow}>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.max(CONV_WINDOW_MIN_SEC, Math.round((convWindowSec - 1) * 10) / 10);
|
||||
setConvWindowSec(next);
|
||||
AsyncStorage.setItem(CONV_WINDOW_STORAGE_KEY, String(next));
|
||||
}}
|
||||
disabled={convWindowSec <= CONV_WINDOW_MIN_SEC}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>−1</Text>
|
||||
</TouchableOpacity>
|
||||
<Text style={styles.prerollValue}>{convWindowSec.toFixed(0)} s</Text>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.min(CONV_WINDOW_MAX_SEC, Math.round((convWindowSec + 1) * 10) / 10);
|
||||
setConvWindowSec(next);
|
||||
AsyncStorage.setItem(CONV_WINDOW_STORAGE_KEY, String(next));
|
||||
}}
|
||||
disabled={convWindowSec >= CONV_WINDOW_MAX_SEC}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>+1</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
|
||||
<Text style={[styles.toggleLabel, {marginTop: 24}]}>Maximale Aufnahmedauer</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Notbremse: nach so vielen Minuten wird die Aufnahme automatisch beendet,
|
||||
@@ -1749,56 +1740,10 @@ const SettingsScreen: React.FC = () => {
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
|
||||
<View style={{flexDirection: 'row', alignItems: 'center', marginTop: 24, gap: 8}}>
|
||||
<Text style={styles.toggleLabel}>Stille-Pegel (dB)</Text>
|
||||
<TouchableOpacity onPress={() => setShowVadInfo(true)} style={styles.infoBtn}>
|
||||
<Text style={styles.infoBtnText}>i</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
<Text style={styles.toggleHint}>
|
||||
Welcher Mikro-Pegel als "Stille" gilt. Standard: automatisch (Baseline aus
|
||||
den ersten 500ms). Manuell setzen wenn Auto nicht zuverlaessig greift.
|
||||
</Text>
|
||||
<View style={styles.prerollRow}>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = vadSilenceDb == null
|
||||
? VAD_SILENCE_DB_DEFAULT - 1
|
||||
: Math.max(VAD_SILENCE_DB_MIN, vadSilenceDb - 1);
|
||||
setVadSilenceDb(next);
|
||||
AsyncStorage.setItem(VAD_SILENCE_DB_OVERRIDE_KEY, String(next));
|
||||
}}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>−1</Text>
|
||||
</TouchableOpacity>
|
||||
<Text style={styles.prerollValue}>
|
||||
{vadSilenceDb == null ? 'auto' : `${vadSilenceDb} dB`}
|
||||
</Text>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = vadSilenceDb == null
|
||||
? VAD_SILENCE_DB_DEFAULT + 1
|
||||
: Math.min(VAD_SILENCE_DB_MAX, vadSilenceDb + 1);
|
||||
setVadSilenceDb(next);
|
||||
AsyncStorage.setItem(VAD_SILENCE_DB_OVERRIDE_KEY, String(next));
|
||||
}}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>+1</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
{vadSilenceDb != null && (
|
||||
<TouchableOpacity
|
||||
onPress={() => {
|
||||
setVadSilenceDb(null);
|
||||
AsyncStorage.removeItem(VAD_SILENCE_DB_OVERRIDE_KEY);
|
||||
}}
|
||||
style={{alignSelf: 'center', marginTop: 8, paddingVertical: 6, paddingHorizontal: 12}}
|
||||
>
|
||||
<Text style={{color: '#0096FF', fontSize: 13}}>↻ Auf automatisch zuruecksetzen</Text>
|
||||
</TouchableOpacity>
|
||||
)}
|
||||
{/* "Stille-Pegel (dB)"-Regler entfernt: der aktive Streaming-STT nutzt
|
||||
einen adaptiven Rausch-Boden (automatisch), ein manueller dB-Wert war
|
||||
wirkungslos. Rauschen-als-Wort verhindert das STT-Modell selbst
|
||||
(no_speech_prob-Filter), nicht die dB-Schwelle. */}
|
||||
</View>
|
||||
|
||||
<Modal
|
||||
@@ -1954,38 +1899,36 @@ const SettingsScreen: React.FC = () => {
|
||||
/>
|
||||
</View>
|
||||
|
||||
<Text style={[styles.toggleLabel, {marginTop: 20}]}>Weiterreden-Fenster (Gespraech)</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Nach einer gesprochenen ARIA-Antwort kannst du so lange einfach
|
||||
weiterreden — ohne Wake-Word — bevor zurueck aufs Wake-Word geschaltet
|
||||
wird. Reine Steuerbefehle (z.B. „nächster Titel") beenden sofort.
|
||||
Default: {Math.round(PASSIVE_LISTEN_DEFAULT_MS / 1000)}s.
|
||||
</Text>
|
||||
<View style={styles.prerollRow}>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.max(10, passiveSec - 5);
|
||||
setPassiveSec(next);
|
||||
savePassiveListenMs(next * 1000);
|
||||
<View style={[styles.toggleRow, {marginTop: 20, borderTopWidth: 1, borderTopColor: '#1E1E2E', paddingTop: 16}]}>
|
||||
<View style={styles.toggleInfo}>
|
||||
<Text style={styles.toggleLabel}>Auch bei gesperrtem Bildschirm zuhören</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
AUS (empfohlen): das Wake-Wort greift nur, wenn die App offen ist —
|
||||
im Hintergrund sind die meisten „Trigger" Fehlalarme (TV, Husten).
|
||||
AN: ARIA hört auch bei gesperrtem Bildschirm / im Hintergrund auf
|
||||
„{KEYWORD_LABELS[wakeKeyword as keyof typeof KEYWORD_LABELS] || wakeKeyword}" — mehr Fehlauslöser möglich.
|
||||
</Text>
|
||||
</View>
|
||||
<Switch
|
||||
value={bgWake}
|
||||
onValueChange={(val) => {
|
||||
setBgWake(val);
|
||||
saveBgWakeEnabled(val).catch(() => {});
|
||||
wakeWordService.setBgWakeEnabled(val);
|
||||
}}
|
||||
disabled={passiveSec <= 10}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>−5</Text>
|
||||
</TouchableOpacity>
|
||||
<Text style={styles.prerollValue}>{passiveSec} s</Text>
|
||||
<TouchableOpacity
|
||||
style={styles.prerollButton}
|
||||
onPress={() => {
|
||||
const next = Math.min(60, passiveSec + 5);
|
||||
setPassiveSec(next);
|
||||
savePassiveListenMs(next * 1000);
|
||||
}}
|
||||
disabled={passiveSec >= 60}
|
||||
>
|
||||
<Text style={styles.prerollButtonText}>+5</Text>
|
||||
</TouchableOpacity>
|
||||
trackColor={{ false: '#2A2A3E', true: '#0096FF' }}
|
||||
thumbColor={bgWake ? '#FFFFFF' : '#666680'}
|
||||
/>
|
||||
</View>
|
||||
|
||||
<Text style={[styles.toggleLabel, {marginTop: 20}]}>Weiterreden nach der Antwort</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Nach einer gesprochenen ARIA-Antwort geht das Mikro auf — du kannst ohne
|
||||
Wake-Word weiterreden. Fängst du nicht innerhalb der „Stille-Toleranz"
|
||||
(Sektion Spracheingabe) an, geht's zurück aufs Wake-Word. Reine
|
||||
Steuerbefehle beenden sofort. Ein separates Zeitfenster gibt es nicht
|
||||
mehr — es zählt überall derselbe Stille-Wert.
|
||||
</Text>
|
||||
</View>
|
||||
</>)}
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* ariaView — Empfaenger der von ARIA komponierten RAEUMLICHEN Ansichten (M1).
|
||||
*
|
||||
* Fluss: ARIA ruft im Brain `present_view` → Brain-Event `aria_view` → Bridge →
|
||||
* RVS `aria_view` → hier gepuffert → WorkspaceCanvas rendert Orb + Karten, die
|
||||
* auf der Flaeche materialisieren.
|
||||
*
|
||||
* Der Service haelt pro Projekt die AKTUELLE View-Spec, damit eine spaet
|
||||
* gemountete Canvas-Kachel sofort den Ist-Stand bekommt. Muster wie
|
||||
* services/codeFile.ts (Singleton, rvs.onMessage).
|
||||
*
|
||||
* Die Karten-Typen sind bewusst offen (string), damit spaetere Renderer (vnc,
|
||||
* chart, file …) ohne Service-Aenderung dazukommen. Der jeweilige Client-Renderer
|
||||
* entscheidet, was er mit einem unbekannten Typ macht (i.d.R. ignorieren).
|
||||
*/
|
||||
|
||||
import rvs, { RVSMessage } from './rvs';
|
||||
|
||||
export type OrbState = 'idle' | 'listening' | 'thinking' | 'speaking' | 'working';
|
||||
|
||||
export interface ViewMarker {
|
||||
lat: number;
|
||||
lon: number;
|
||||
label?: string;
|
||||
}
|
||||
|
||||
export interface ViewCard {
|
||||
type: 'text' | 'image' | 'map' | 'code' | 'list' | string;
|
||||
title?: string;
|
||||
md?: string; // text/list
|
||||
src?: string; // image
|
||||
markers?: ViewMarker[]; // map
|
||||
path?: string; // code
|
||||
lang?: string; // code
|
||||
// Zukuenftige Kartenfelder ohne Service-Aenderung:
|
||||
[k: string]: any;
|
||||
}
|
||||
|
||||
export interface ViewSpec {
|
||||
cards: ViewCard[];
|
||||
orb?: OrbState;
|
||||
title?: string;
|
||||
}
|
||||
|
||||
export interface AriaView {
|
||||
projectId: string;
|
||||
view: ViewSpec;
|
||||
clientMsgId?: string;
|
||||
ts: number;
|
||||
}
|
||||
|
||||
type ViewSub = (v: AriaView) => void;
|
||||
|
||||
class AriaViewService {
|
||||
private views = new Map<string, AriaView>();
|
||||
private subs: ViewSub[] = [];
|
||||
|
||||
constructor() {
|
||||
rvs.onMessage((m) => this.onMessage(m));
|
||||
}
|
||||
|
||||
private onMessage(m: RVSMessage): void {
|
||||
if (m.type !== 'aria_view') return;
|
||||
const p = (m.payload || {}) as any;
|
||||
const raw = (p.view || {}) as any;
|
||||
const cards: ViewCard[] = Array.isArray(raw.cards) ? raw.cards : [];
|
||||
if (cards.length === 0) return; // leere Ansicht ignorieren
|
||||
const view: ViewSpec = {
|
||||
cards,
|
||||
orb: raw.orb || 'speaking',
|
||||
title: raw.title || '',
|
||||
};
|
||||
const projectId: string = p.projectId || '';
|
||||
const entry: AriaView = {
|
||||
projectId,
|
||||
view,
|
||||
clientMsgId: p.clientMsgId || '',
|
||||
ts: Date.now(),
|
||||
};
|
||||
this.views.set(projectId, entry);
|
||||
this.subs.forEach((cb) => {
|
||||
try { cb(entry); } catch {}
|
||||
});
|
||||
}
|
||||
|
||||
/** Aktuelle Ansicht eines Projekts (leer = Hauptchat). */
|
||||
getView(projectId: string): AriaView | undefined {
|
||||
return this.views.get(projectId || '');
|
||||
}
|
||||
|
||||
/** Registriert einen Listener fuer neue Ansichten. */
|
||||
subscribe(cb: ViewSub): () => void {
|
||||
this.subs.push(cb);
|
||||
return () => { this.subs = this.subs.filter((s) => s !== cb); };
|
||||
}
|
||||
|
||||
/** Ansicht eines Projekts verwerfen (z.B. wenn der User sie wegwischt). */
|
||||
clear(projectId: string): void {
|
||||
this.views.delete(projectId || '');
|
||||
}
|
||||
}
|
||||
|
||||
const ariaView = new AriaViewService();
|
||||
export default ariaView;
|
||||
@@ -143,23 +143,34 @@ export const VAD_SILENCE_MIN_SEC = 1.0;
|
||||
export const VAD_SILENCE_MAX_SEC = 8.0;
|
||||
export const VAD_SILENCE_STORAGE_KEY = 'aria_vad_silence_sec';
|
||||
|
||||
// Konversations-Fenster (in Sekunden) — nach ARIA's Antwort hat der User so
|
||||
// lange Zeit, im Gespraechsmodus weiter zu sprechen, ohne dass die Konversation
|
||||
// beendet wird. Sprichst du im Fenster nichts → Konversation aus.
|
||||
export const CONV_WINDOW_DEFAULT_SEC = 8.0;
|
||||
export const CONV_WINDOW_MIN_SEC = 3.0;
|
||||
export const CONV_WINDOW_MAX_SEC = 20.0;
|
||||
export const CONV_WINDOW_STORAGE_KEY = 'aria_conv_window_sec';
|
||||
|
||||
// STT-Endpoint (ms Stille bis "fertig gesprochen"). Zu kurz = schneidet mitten
|
||||
// im Satz ab, besonders im Auto wo man mit Pausen spricht (Reproduktion: die
|
||||
// 11.8s-Frage wurde bei "…ohne dass ein" gekappt). 1500 war zu aggressiv;
|
||||
// 2400 default, im Auto ggf. hoeher. Konfigurierbar in den Settings.
|
||||
// im Satz ab, besonders im Auto oder wenn man zum Nachdenken pausiert. 1500 war
|
||||
// zu aggressiv; 2400 default, bis 8s hoch stellbar (Denkpausen). In den Settings
|
||||
// unter "Stille-Toleranz" konfigurierbar.
|
||||
export const STT_ENDPOINT_DEFAULT_MS = 2400;
|
||||
export const STT_ENDPOINT_MIN_MS = 1000;
|
||||
export const STT_ENDPOINT_MAX_MS = 4000;
|
||||
export const STT_ENDPOINT_MAX_MS = 8000; // bis 8s: genug Zeit zum Ueberlegen
|
||||
export const STT_ENDPOINT_STORAGE_KEY = 'aria_stt_endpoint_ms';
|
||||
|
||||
// Barge-in-Modus: darf man ARIA waehrend ihrer TTS-Antwort unterbrechen (reden)?
|
||||
// Default AUS = sauberes Halb-Duplex (ARIA spricht aus, DANN oeffnet das Mikro —
|
||||
// kein Selbst-Echo, kein Mischmasch). AN = waehrend TTS auf Wake-Wort lauschen.
|
||||
export const BARGE_IN_STORAGE_KEY = 'aria_barge_in_enabled';
|
||||
|
||||
export async function loadBargeInEnabled(): Promise<boolean> {
|
||||
try {
|
||||
return (await AsyncStorage.getItem(BARGE_IN_STORAGE_KEY)) === 'true';
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function saveBargeInEnabled(enabled: boolean): Promise<void> {
|
||||
try {
|
||||
await AsyncStorage.setItem(BARGE_IN_STORAGE_KEY, String(enabled));
|
||||
} catch {}
|
||||
}
|
||||
|
||||
export async function loadSttEndpointMs(): Promise<number> {
|
||||
try {
|
||||
const raw = await AsyncStorage.getItem(STT_ENDPOINT_STORAGE_KEY);
|
||||
@@ -189,18 +200,6 @@ export async function loadTtsSpeed(): Promise<number> {
|
||||
return TTS_SPEED_DEFAULT;
|
||||
}
|
||||
|
||||
export async function loadConvWindowMs(): Promise<number> {
|
||||
try {
|
||||
const raw = await AsyncStorage.getItem(CONV_WINDOW_STORAGE_KEY);
|
||||
if (raw != null) {
|
||||
const n = parseFloat(raw);
|
||||
if (isFinite(n) && n >= CONV_WINDOW_MIN_SEC && n <= CONV_WINDOW_MAX_SEC) {
|
||||
return Math.round(n * 1000);
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
return Math.round(CONV_WINDOW_DEFAULT_SEC * 1000);
|
||||
}
|
||||
|
||||
async function loadVadSilenceMs(): Promise<number> {
|
||||
try {
|
||||
@@ -1145,7 +1144,7 @@ class AudioService {
|
||||
speed: typeof opts.speed === 'number' ? opts.speed : 1.0,
|
||||
interrupted: !!opts.interrupted,
|
||||
location: opts.location || null,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : 1500,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : STT_ENDPOINT_DEFAULT_MS,
|
||||
hardCapMs: typeof opts.hardCapMs === 'number' ? opts.hardCapMs : 60000,
|
||||
sampleRate: 16000,
|
||||
projectId: opts.projectId || '',
|
||||
|
||||
@@ -145,6 +145,17 @@ class GpsTrackingService {
|
||||
// liefert im Hintergrund keine Updates (nur Heartbeat sendet alte Werte).
|
||||
const bgEnabled = await isBackgroundGpsEnabled();
|
||||
if (bgEnabled) {
|
||||
// Ohne ACCESS_BACKGROUND_LOCATION liefert watchPosition im Hintergrund
|
||||
// NICHTS (Android 10+) → der Foreground-Service allein bringt nichts, und
|
||||
// genau der Fall "Ankunft waehrend der Fahrt, Screen aus" faellt durch.
|
||||
// Deshalb erst die Permission sicherstellen (oeffnet ggf. die Android-
|
||||
// Settings fuer "Immer erlauben"), DANN den Location-Foreground-Service
|
||||
// hochziehen — der haelt den Prozess wach, sodass watchPosition + der
|
||||
// 60s-Heartbeat auch unter Doze weiterlaufen.
|
||||
const bgOk = await ensureBackgroundLocationPermission();
|
||||
if (!bgOk) {
|
||||
console.warn('[gps-track] Background-Permission fehlt — Tracking nur im Vordergrund zuverlaessig');
|
||||
}
|
||||
try { await acquireBackgroundAudio('location'); } catch {}
|
||||
}
|
||||
try {
|
||||
|
||||
@@ -30,34 +30,22 @@ type PassiveListenCallback = () => void;
|
||||
|
||||
export type WakeWordState = 'off' | 'armed' | 'conversing' | 'listening';
|
||||
|
||||
/** Default-Dauer fuer den Passive-Listen-Modus nach einer Konversation —
|
||||
* in dem Fenster braucht's kein Wake-Word, Speaker-ID-Filter haelt
|
||||
* fremde Stimmen raus (TV, Familie). 30s default; konfigurierbar. */
|
||||
export const PASSIVE_LISTEN_DEFAULT_MS = 30_000;
|
||||
export const PASSIVE_LISTEN_STORAGE_KEY = 'aria_passive_listen_ms';
|
||||
|
||||
export async function loadPassiveListenMs(): Promise<number> {
|
||||
try {
|
||||
const raw = await AsyncStorage.getItem(PASSIVE_LISTEN_STORAGE_KEY);
|
||||
if (raw) {
|
||||
const n = parseInt(raw, 10);
|
||||
if (isFinite(n) && n >= 0 && n <= 120_000) return n;
|
||||
}
|
||||
} catch {}
|
||||
return PASSIVE_LISTEN_DEFAULT_MS;
|
||||
}
|
||||
|
||||
export async function savePassiveListenMs(ms: number): Promise<void> {
|
||||
await AsyncStorage.setItem(PASSIVE_LISTEN_STORAGE_KEY, String(ms));
|
||||
}
|
||||
/** Reine HANG-Notbremse fuer den Passive-Listen-Modus. Das echte Ende regelt IMMER
|
||||
* die passive Aufnahme selbst: Stille-Toleranz (User pausiert), No-Speech (User
|
||||
* sagt gar nichts) oder Hard-Cap (max. Aufnahmedauer, ~5min) → ChatScreen ruft
|
||||
* dann exitPassiveListening. Dieser Timer darf aktives Reden NIE abschneiden —
|
||||
* deshalb LÄNGER als der Hard-Cap (nur falls ein Endpoint-Event mal verloren geht
|
||||
* und der State sonst ewig 'listening' bliebe). Das alte 30s-Fenster, das lange
|
||||
* Antworten mitten im Satz kappte, ist damit raus. */
|
||||
const PASSIVE_BACKSTOP_MS = 10 * 60_000;
|
||||
|
||||
export const WAKE_KEYWORD_STORAGE = 'aria_wake_keyword';
|
||||
|
||||
// Wake-Word-Empfindlichkeit (openWakeWord-Threshold). Hoeher = strenger =
|
||||
// weniger Fehlauslösung (z.B. durch Musik/Radio ueber die Auto-Lautsprecher,
|
||||
// die das Mikro mithoert — der App-Echo-Canceler kann nur ARIAs eigenes TTS
|
||||
// rausrechnen, NICHT Spotify). Default 0.6 (war 0.5). 0..1.
|
||||
export const WAKE_THRESHOLD_DEFAULT = 0.6;
|
||||
// weniger Fehlauslösung, aber man muss deutlicher/lauter sprechen (fuehlt sich
|
||||
// "traege" an). Fehlausloeser werden ueber Speaker-ID (E3) ohnehin verworfen,
|
||||
// deshalb darf der Default empfindlicher sein. 0.45 (war 0.6/0.5). 0..1.
|
||||
export const WAKE_THRESHOLD_DEFAULT = 0.45;
|
||||
export const WAKE_THRESHOLD_MIN = 0.3;
|
||||
export const WAKE_THRESHOLD_MAX = 0.9;
|
||||
export const WAKE_THRESHOLD_STORAGE_KEY = 'aria_wake_threshold';
|
||||
@@ -77,6 +65,28 @@ export async function saveWakeThreshold(v: number): Promise<void> {
|
||||
await AsyncStorage.setItem(WAKE_THRESHOLD_STORAGE_KEY, String(v));
|
||||
}
|
||||
|
||||
// Hintergrund-Wake: darf das Wake-Wort auch triggern, wenn die App im
|
||||
// Hintergrund / der Bildschirm gesperrt ist? Default AUS — im Hintergrund
|
||||
// sind die meisten „Trigger" Fehlalarme (TV, Husten, AudioFocus-Spikes).
|
||||
// AN = auch bei gesperrtem Bildschirm zuhoeren. Die native Erkennung laeuft
|
||||
// ohnehin durch (Foreground-Service + Wake-Locks) — dieser Schalter oeffnet
|
||||
// nur das JS-Gate in onWakeDetected.
|
||||
export const BG_WAKE_STORAGE_KEY = 'aria_bg_wake_enabled';
|
||||
|
||||
export async function loadBgWakeEnabled(): Promise<boolean> {
|
||||
try {
|
||||
return (await AsyncStorage.getItem(BG_WAKE_STORAGE_KEY)) === 'true';
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function saveBgWakeEnabled(enabled: boolean): Promise<void> {
|
||||
try {
|
||||
await AsyncStorage.setItem(BG_WAKE_STORAGE_KEY, String(enabled));
|
||||
} catch {}
|
||||
}
|
||||
|
||||
/** Verfuegbare Wake-Words — entsprechen den .onnx Dateien in
|
||||
* android/app/src/main/assets/openwakeword/. Custom-Keywords (eigenes
|
||||
* Training via openwakeword Notebook) muessen aktuell als Asset eingebaut
|
||||
@@ -103,7 +113,9 @@ export const KEYWORD_LABELS: Record<WakeKeyword, string> = {
|
||||
// Detection-Tuning. Threshold ist ueber die Settings konfigurierbar
|
||||
// (loadWakeThreshold) — der Wert hier ist nur der Fallback.
|
||||
const DEFAULT_THRESHOLD = WAKE_THRESHOLD_DEFAULT;
|
||||
const DEFAULT_PATIENCE = 2;
|
||||
// patience=1 statt 2: nur EIN Frame ueber Threshold noetig → deutlich schneller.
|
||||
// Speaker-ID filtert Fehlausloeser, also ist das vertretbar.
|
||||
const DEFAULT_PATIENCE = 1;
|
||||
const DEFAULT_DEBOUNCE_MS = 1500;
|
||||
|
||||
interface OpenWakeWordModule {
|
||||
@@ -143,6 +155,10 @@ class WakeWordService {
|
||||
* Hintergrund-Detections sind quasi immer false-positives (TV, Husten,
|
||||
* AudioFocus-Switch beim Wechsel zu Musik etc.). */
|
||||
private inBackground: boolean = false;
|
||||
/** Wenn true: Wake-Wort triggert auch im Hintergrund / bei gesperrtem
|
||||
* Bildschirm. Default false. Wird beim Arm aus AsyncStorage geladen und
|
||||
* bei Aenderung in den Einstellungen via setBgWakeEnabled() aktualisiert. */
|
||||
private bgWakeEnabled: boolean = false;
|
||||
/** Re-Entry-Guard fuer onWakeDetected: native kann mehrere
|
||||
* WakeWordDetected-Events emitten BEVOR OpenWakeWord.stop() in JS
|
||||
* resolved (Bridge-Queue + Doze-Backlog). Mit dem Flag wird das zweite
|
||||
@@ -150,8 +166,9 @@ class WakeWordService {
|
||||
* Ausnahme: bargeListening → Barge-In ist ein legitimer neuer Trigger
|
||||
* waehrend ARIA noch redet, NICHT vom Guard blockieren. */
|
||||
private detectionInProgress: boolean = false;
|
||||
/** Passive-Listen-Timer: feuert nach PASSIVE_LISTEN_MS ohne Stefan-Speech,
|
||||
* beendet den listening-State und geht zurueck zu armed. */
|
||||
/** Passive-Listen-Backstop-Timer: Notbremse (PASSIVE_BACKSTOP_MS). Normal endet
|
||||
* das Fenster ueber die Stille-Toleranz der Aufnahme; feuert dieser Timer
|
||||
* trotzdem, zurueck zu armed. */
|
||||
private passiveListenTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
/** Callbacks fuer den Eintritt in Passive-Listen — ChatScreen startet
|
||||
* hier eine streaming-Aufnahme OHNE User-Bubble (passiv lauschen). */
|
||||
@@ -223,7 +240,8 @@ class WakeWordService {
|
||||
this.initInProgress = (async () => {
|
||||
try {
|
||||
const threshold = await loadWakeThreshold();
|
||||
console.log('[WakeWord] init mit threshold=%s', threshold);
|
||||
this.bgWakeEnabled = await loadBgWakeEnabled();
|
||||
console.log('[WakeWord] init mit threshold=%s, bgWake=%s', threshold, this.bgWakeEnabled);
|
||||
await OpenWakeWord.init(this.keyword, threshold, DEFAULT_PATIENCE, DEFAULT_DEBOUNCE_MS);
|
||||
// Subscribe nur einmal
|
||||
if (!this.eventSub) {
|
||||
@@ -310,7 +328,7 @@ class WakeWordService {
|
||||
/** Cooldown setzen — alle Wake-Word-Detections in den naechsten ms ignorieren.
|
||||
* Wird beim App-Resume gerufen weil AppState-Wechsel Audio-Spikes erzeugen
|
||||
* die openWakeWord faelschlich als Trigger interpretiert. */
|
||||
setResumeCooldown(ms: number = 1500): void {
|
||||
setResumeCooldown(ms: number = 500): void {
|
||||
this.cooldownUntilMs = Date.now() + ms;
|
||||
console.log('[WakeWord] Cooldown aktiv fuer %dms', ms);
|
||||
}
|
||||
@@ -320,23 +338,31 @@ class WakeWordService {
|
||||
* was als „Wake-Word" reinkommt ist Husten/TV/AudioFocus-Switch. */
|
||||
setBackground(): void {
|
||||
this.inBackground = true;
|
||||
console.log('[WakeWord] App im Hintergrund — Detections gesperrt');
|
||||
console.log('[WakeWord] App im Hintergrund — Detections %s',
|
||||
this.bgWakeEnabled ? 'AKTIV (Hintergrund-Wake an)' : 'gesperrt');
|
||||
}
|
||||
|
||||
/** App im Vordergrund: Detections wieder freigeben, plus 3s Cooldown
|
||||
* als Schutz gegen den AudioFocus-/AudioTrack-Spike der direkt nach
|
||||
* dem Resume kommt. Ersetzt das alte setResumeCooldown(3000)-Pattern. */
|
||||
/** Hintergrund-Wake ein/aus schalten (aus den Einstellungen). */
|
||||
setBgWakeEnabled(enabled: boolean): void {
|
||||
this.bgWakeEnabled = enabled;
|
||||
console.log('[WakeWord] Hintergrund-Wake = %s', enabled);
|
||||
}
|
||||
|
||||
/** App im Vordergrund: Detections wieder freigeben, plus kurzer Cooldown
|
||||
* als Schutz gegen den AudioFocus-/AudioTrack-Spike direkt nach dem Resume.
|
||||
* 1s statt 3s — 3s hat sich "traege" angefuehlt (Trigger direkt nach dem
|
||||
* App-Oeffnen wurden verschluckt). */
|
||||
setForeground(): void {
|
||||
this.inBackground = false;
|
||||
this.cooldownUntilMs = Date.now() + 3000;
|
||||
console.log('[WakeWord] App im Vordergrund — Cooldown 3s aktiv');
|
||||
this.cooldownUntilMs = Date.now() + 1000;
|
||||
console.log('[WakeWord] App im Vordergrund — Cooldown 1s aktiv');
|
||||
}
|
||||
|
||||
/** Wake-Word getriggert: Native-Modul pausieren, Konversation starten. */
|
||||
private async onWakeDetected(): Promise<void> {
|
||||
if (this.inBackground) {
|
||||
console.log('[WakeWord] Trigger ignoriert (App im Hintergrund)');
|
||||
import('./logger').then(m => m.reportAppDebug('wake.detect', 'ignored: app in background')).catch(()=>{});
|
||||
if (this.inBackground && !this.bgWakeEnabled) {
|
||||
console.log('[WakeWord] Trigger ignoriert (App im Hintergrund, Hintergrund-Wake aus)');
|
||||
import('./logger').then(m => m.reportAppDebug('wake.detect', 'ignored: app in background (bg-wake off)')).catch(()=>{});
|
||||
return;
|
||||
}
|
||||
// Re-Entry-Guard: blocken wenn ein Detection-Zyklus schon laeuft.
|
||||
@@ -486,13 +512,12 @@ class WakeWordService {
|
||||
import('./logger').then(m => m.reportAppDebug('wake.end',
|
||||
`endConversation called, wasBarge=${wasBarge}, nativeReady=${this.nativeReady}`)).catch(()=>{});
|
||||
|
||||
// Passive-Listen aktiv? Dann nicht direkt zu armed — passive lauschen
|
||||
// fuer N Sekunden, dann erst Wake-Word wieder aktivieren. Speaker-ID
|
||||
// (Phase 3) filtert fremde Stimmen weg, der User kann ohne erneute
|
||||
// Anrede weitersprechen.
|
||||
const passiveMs = await loadPassiveListenMs();
|
||||
if (!skipPassive && passiveMs > 0 && this.nativeReady) {
|
||||
this.enterPassiveListening(passiveMs);
|
||||
// Kein skipPassive? Dann EIN Stille-Fenster zum Weiterreden (kein Wake-Word
|
||||
// noetig). Das echte Ende regelt die Stille-Toleranz der passiven Aufnahme;
|
||||
// der Backstop-Timer ist nur die Notbremse. Der User kann ohne erneute
|
||||
// Anrede weitersprechen; sagt er nichts → zurueck aufs Wake-Word.
|
||||
if (!skipPassive && this.nativeReady) {
|
||||
this.enterPassiveListening(PASSIVE_BACKSTOP_MS);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -534,10 +559,10 @@ class WakeWordService {
|
||||
this.cancelPassiveListenTimer();
|
||||
this.setState('listening');
|
||||
const seconds = Math.round(durationMs / 1000);
|
||||
console.log('[WakeWord] Passive-Listen aktiv (%ds) — Speaker-ID gefiltert', seconds);
|
||||
console.log('[WakeWord] Passive-Listen aktiv (Backstop %ds) — Speaker-ID gefiltert', seconds);
|
||||
import('./logger').then(m => m.reportAppDebug('wake.passive',
|
||||
`entered listening for ${seconds}s, cb-count=${this.passiveListenCallbacks.length}`)).catch(()=>{});
|
||||
ToastAndroid.show(`🎧 ${seconds}s lauscht — sprich einfach weiter`, ToastAndroid.SHORT);
|
||||
`entered listening (backstop ${seconds}s), cb-count=${this.passiveListenCallbacks.length}`)).catch(()=>{});
|
||||
ToastAndroid.show('🎧 sprich einfach weiter', ToastAndroid.SHORT);
|
||||
this.passiveListenTimer = setTimeout(() => {
|
||||
this.passiveListenTimer = null;
|
||||
this.exitPassiveListening('timeout').catch(() => {});
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
/**
|
||||
* AriaViewCanvas — die pannbare Flaeche, auf der ARIAs komponierte Ansicht
|
||||
* (aria_view) MATERIALISIERT: Orb oben, darunter die Karten. Erscheint als
|
||||
* Overlay ueber dem Chat, sobald ARIA present_view aufruft ("sag was → Orb denkt
|
||||
* → Karte fliegt rein"). Der erste, greifbare Vorgeschmack aufs generative
|
||||
* Cockpit (M1).
|
||||
*
|
||||
* Bedienung (NoMachine-Prinzip): 2-Finger halten + schieben bewegt die Welt,
|
||||
* Pinch zoomt. Ein-Finger-Touch geht an die Karten durch (Scrollen). Die Welt
|
||||
* traegt gerenderte/gestreamte Inhalte — interaktive native Panels rasten
|
||||
* spaeter bei Scale 1 ein (Chat bleibt separat darunter).
|
||||
*
|
||||
* Geraete-agnostisch gehalten: liest nur die ViewSpec, damit ein spaeterer Web-/
|
||||
* AR-Renderer dieselbe Spec konsumieren kann.
|
||||
*/
|
||||
|
||||
import React from 'react';
|
||||
import { StyleSheet, Text, TouchableOpacity, View } from 'react-native';
|
||||
import Animated, {
|
||||
FadeInDown,
|
||||
useAnimatedStyle,
|
||||
useSharedValue,
|
||||
withTiming,
|
||||
} from 'react-native-reanimated';
|
||||
import { Gesture, GestureDetector } from 'react-native-gesture-handler';
|
||||
import { ViewSpec } from '../services/ariaView';
|
||||
import Orb from './Orb';
|
||||
import CardView from './CardView';
|
||||
|
||||
const MIN_SCALE = 0.5;
|
||||
const MAX_SCALE = 3;
|
||||
|
||||
interface Props {
|
||||
view: ViewSpec;
|
||||
onClose: () => void;
|
||||
}
|
||||
|
||||
const AriaViewCanvas: React.FC<Props> = ({ view, onClose }) => {
|
||||
const tx = useSharedValue(0);
|
||||
const ty = useSharedValue(0);
|
||||
const scale = useSharedValue(1);
|
||||
const savedTx = useSharedValue(0);
|
||||
const savedTy = useSharedValue(0);
|
||||
const savedScale = useSharedValue(1);
|
||||
|
||||
const pan = Gesture.Pan()
|
||||
.minPointers(2)
|
||||
.maxPointers(2)
|
||||
.onUpdate((e) => {
|
||||
tx.value = savedTx.value + e.translationX;
|
||||
ty.value = savedTy.value + e.translationY;
|
||||
})
|
||||
.onEnd(() => {
|
||||
savedTx.value = tx.value;
|
||||
savedTy.value = ty.value;
|
||||
});
|
||||
|
||||
const pinch = Gesture.Pinch()
|
||||
.onUpdate((e) => {
|
||||
const next = savedScale.value * e.scale;
|
||||
scale.value = Math.max(MIN_SCALE, Math.min(MAX_SCALE, next));
|
||||
})
|
||||
.onEnd(() => {
|
||||
savedScale.value = scale.value;
|
||||
});
|
||||
|
||||
const composed = Gesture.Simultaneous(pan, pinch);
|
||||
|
||||
const worldStyle = useAnimatedStyle(() => ({
|
||||
transform: [
|
||||
{ translateX: tx.value },
|
||||
{ translateY: ty.value },
|
||||
{ scale: scale.value },
|
||||
],
|
||||
}));
|
||||
|
||||
const resetCamera = () => {
|
||||
tx.value = withTiming(0);
|
||||
ty.value = withTiming(0);
|
||||
scale.value = withTiming(1);
|
||||
savedTx.value = 0;
|
||||
savedTy.value = 0;
|
||||
savedScale.value = 1;
|
||||
};
|
||||
|
||||
const cards = Array.isArray(view.cards) ? view.cards : [];
|
||||
|
||||
return (
|
||||
<View style={styles.overlay}>
|
||||
<GestureDetector gesture={composed}>
|
||||
<Animated.View style={[styles.world, worldStyle]}>
|
||||
<View style={styles.orbWrap}>
|
||||
<Orb state={view.orb} size={110} />
|
||||
</View>
|
||||
{!!view.title && <Text style={styles.worldTitle}>{view.title}</Text>}
|
||||
<View style={styles.cards}>
|
||||
{cards.map((c, i) => (
|
||||
<Animated.View
|
||||
key={i}
|
||||
entering={FadeInDown.duration(420).delay(120 + i * 90)}
|
||||
>
|
||||
<CardView card={c} />
|
||||
</Animated.View>
|
||||
))}
|
||||
</View>
|
||||
</Animated.View>
|
||||
</GestureDetector>
|
||||
|
||||
{/* Steuerung — ausserhalb des Transforms, immer bei Scale 1 bedienbar */}
|
||||
<View style={styles.topBar} pointerEvents="box-none">
|
||||
<TouchableOpacity style={styles.iconBtn} onPress={resetCamera}>
|
||||
<Text style={styles.icon}>⤢</Text>
|
||||
</TouchableOpacity>
|
||||
<TouchableOpacity style={styles.iconBtn} onPress={onClose}>
|
||||
<Text style={styles.icon}>✕</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
<View style={styles.hintWrap} pointerEvents="none">
|
||||
<Text style={styles.hint}>2 Finger: schieben · Pinch: zoomen</Text>
|
||||
</View>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
overlay: {
|
||||
...StyleSheet.absoluteFillObject,
|
||||
backgroundColor: 'rgba(6,6,16,0.94)',
|
||||
zIndex: 50,
|
||||
},
|
||||
world: {
|
||||
...StyleSheet.absoluteFillObject,
|
||||
alignItems: 'center',
|
||||
paddingTop: 48,
|
||||
paddingHorizontal: 18,
|
||||
},
|
||||
orbWrap: { marginTop: 8, marginBottom: 6 },
|
||||
worldTitle: {
|
||||
color: '#C9C9FF',
|
||||
fontSize: 18,
|
||||
fontWeight: '700',
|
||||
marginBottom: 4,
|
||||
textAlign: 'center',
|
||||
},
|
||||
cards: { width: '100%', maxWidth: 560 },
|
||||
topBar: {
|
||||
position: 'absolute',
|
||||
top: 10,
|
||||
right: 12,
|
||||
flexDirection: 'row',
|
||||
},
|
||||
iconBtn: {
|
||||
width: 40,
|
||||
height: 40,
|
||||
borderRadius: 20,
|
||||
marginLeft: 10,
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
backgroundColor: 'rgba(30,30,60,0.9)',
|
||||
borderWidth: 1,
|
||||
borderColor: 'rgba(123,92,255,0.4)',
|
||||
},
|
||||
icon: { color: '#C9C9FF', fontSize: 18 },
|
||||
hintWrap: {
|
||||
position: 'absolute',
|
||||
bottom: 14,
|
||||
alignSelf: 'center',
|
||||
},
|
||||
hint: {
|
||||
color: '#6A6A90',
|
||||
fontSize: 12,
|
||||
},
|
||||
});
|
||||
|
||||
export default AriaViewCanvas;
|
||||
@@ -0,0 +1,114 @@
|
||||
/**
|
||||
* CardView — rendert EINE Karte einer aria_view-Spec (M1). Schaltet nach
|
||||
* card.type auf den passenden Renderer. Unbekannte Typen werden als Text-
|
||||
* Fallback gezeigt (nie crashen).
|
||||
*
|
||||
* Bewusst dependency-leicht (v1): Markdown wird als Klartext dargestellt, Map
|
||||
* als Marker-Liste (kein Karten-Lib), Code als Monospace-Block. Spaeter koennen
|
||||
* einzelne Renderer aufgebohrt werden, ohne die Spec/den Fluss zu aendern.
|
||||
*/
|
||||
|
||||
import React from 'react';
|
||||
import { Image, ScrollView, StyleSheet, Text, View } from 'react-native';
|
||||
import { ViewCard, ViewMarker } from '../services/ariaView';
|
||||
|
||||
const ImageBody: React.FC<{ src?: string }> = ({ src }) => {
|
||||
const isUrl = !!src && /^https?:\/\//i.test(src);
|
||||
if (isUrl) {
|
||||
return <Image source={{ uri: src }} style={styles.image} resizeMode="contain" />;
|
||||
}
|
||||
return <Text style={styles.muted}>🖼️ {src || '(kein Bild)'}</Text>;
|
||||
};
|
||||
|
||||
const ListBody: React.FC<{ md?: string }> = ({ md }) => {
|
||||
const lines = (md || '')
|
||||
.split('\n')
|
||||
.map((l) => l.replace(/^\s*[-*•]\s?/, '').trim())
|
||||
.filter(Boolean);
|
||||
if (lines.length === 0) return <Text style={styles.muted}>(leer)</Text>;
|
||||
return (
|
||||
<View>
|
||||
{lines.map((l, i) => (
|
||||
<View key={i} style={styles.listRow}>
|
||||
<Text style={styles.bullet}>•</Text>
|
||||
<Text style={styles.text}>{l}</Text>
|
||||
</View>
|
||||
))}
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const MapBody: React.FC<{ markers?: ViewMarker[] }> = ({ markers }) => {
|
||||
const ms = Array.isArray(markers) ? markers : [];
|
||||
return (
|
||||
<View style={styles.map}>
|
||||
<Text style={styles.mapHint}>🗺️ Karte ({ms.length} Orte)</Text>
|
||||
{ms.map((m, i) => (
|
||||
<Text key={i} style={styles.text}>
|
||||
📍 {m.label || `${m.lat?.toFixed?.(4)}, ${m.lon?.toFixed?.(4)}`}
|
||||
</Text>
|
||||
))}
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const CodeBody: React.FC<{ md?: string; path?: string; lang?: string }> = ({ md, path, lang }) => (
|
||||
<View>
|
||||
{(path || lang) && (
|
||||
<Text style={styles.codeCaption}>
|
||||
{path || ''}{lang ? ` · ${lang}` : ''}
|
||||
</Text>
|
||||
)}
|
||||
<ScrollView horizontal style={styles.codeScroll}>
|
||||
<Text style={styles.code}>{md || ''}</Text>
|
||||
</ScrollView>
|
||||
</View>
|
||||
);
|
||||
|
||||
const CardView: React.FC<{ card: ViewCard }> = ({ card }) => {
|
||||
return (
|
||||
<View style={styles.card}>
|
||||
{!!card.title && <Text style={styles.cardTitle}>{card.title}</Text>}
|
||||
{card.type === 'image' ? (
|
||||
<ImageBody src={card.src} />
|
||||
) : card.type === 'list' ? (
|
||||
<ListBody md={card.md} />
|
||||
) : card.type === 'map' ? (
|
||||
<MapBody markers={card.markers} />
|
||||
) : card.type === 'code' ? (
|
||||
<CodeBody md={card.md} path={card.path} lang={card.lang} />
|
||||
) : (
|
||||
<Text style={styles.text}>{card.md || ''}</Text>
|
||||
)}
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
card: {
|
||||
backgroundColor: 'rgba(18,18,42,0.92)',
|
||||
borderColor: 'rgba(123,92,255,0.35)',
|
||||
borderWidth: 1,
|
||||
borderRadius: 14,
|
||||
padding: 14,
|
||||
marginVertical: 8,
|
||||
shadowColor: '#7B5CFF',
|
||||
shadowOpacity: 0.25,
|
||||
shadowRadius: 12,
|
||||
shadowOffset: { width: 0, height: 2 },
|
||||
elevation: 6,
|
||||
},
|
||||
cardTitle: { color: '#C9C9FF', fontSize: 15, fontWeight: '700', marginBottom: 8 },
|
||||
text: { color: '#E6E6F0', fontSize: 14, lineHeight: 20, flexShrink: 1 },
|
||||
muted: { color: '#8A8AB0', fontSize: 13, fontStyle: 'italic' },
|
||||
image: { width: '100%', height: 200, borderRadius: 8, backgroundColor: '#0D0D1A' },
|
||||
listRow: { flexDirection: 'row', alignItems: 'flex-start', marginVertical: 2 },
|
||||
bullet: { color: '#7B5CFF', marginRight: 8, fontSize: 14, lineHeight: 20 },
|
||||
map: { backgroundColor: '#0D0D1A', borderRadius: 8, padding: 10 },
|
||||
mapHint: { color: '#00B4D8', fontSize: 13, fontWeight: '600', marginBottom: 6 },
|
||||
codeCaption: { color: '#8A8AB0', fontSize: 12, marginBottom: 6 },
|
||||
codeScroll: { backgroundColor: '#0A0A14', borderRadius: 8, padding: 10 },
|
||||
code: { color: '#B9F5C9', fontFamily: 'monospace', fontSize: 12.5, lineHeight: 18 },
|
||||
});
|
||||
|
||||
export default React.memo(CardView);
|
||||
@@ -0,0 +1,106 @@
|
||||
/**
|
||||
* Orb — ARIAs Praesenz-Avatar (M1). Zeigt ihren Zustand (idle/listening/
|
||||
* thinking/speaking/working) als pulsierender Leucht-Kern und ist das
|
||||
* verbindende Element ueber alle Oberflaechen (App/Web/spaeter Brille).
|
||||
*
|
||||
* Reine Optik, keine Logik — der Zustand kommt von aussen (aria_view.orb bzw.
|
||||
* spaeter direkt von Audio/Wake-Word-Signalen). Dependency-leicht: nur
|
||||
* reanimated (schon installiert), kein SVG/Gradient noetig.
|
||||
*/
|
||||
|
||||
import React, { useEffect } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import Animated, {
|
||||
Easing,
|
||||
cancelAnimation,
|
||||
useAnimatedStyle,
|
||||
useSharedValue,
|
||||
withRepeat,
|
||||
withTiming,
|
||||
} from 'react-native-reanimated';
|
||||
import { OrbState } from '../services/ariaView';
|
||||
|
||||
const COLORS: Record<OrbState, string> = {
|
||||
idle: '#3A6EA5',
|
||||
listening: '#00B4D8',
|
||||
thinking: '#7B5CFF',
|
||||
speaking: '#34C759',
|
||||
working: '#FF9500',
|
||||
};
|
||||
|
||||
interface Props {
|
||||
state?: OrbState;
|
||||
size?: number;
|
||||
}
|
||||
|
||||
const Orb: React.FC<Props> = ({ state = 'idle', size = 120 }) => {
|
||||
const pulse = useSharedValue(1);
|
||||
|
||||
useEffect(() => {
|
||||
const fast = state === 'thinking' || state === 'working';
|
||||
cancelAnimation(pulse);
|
||||
pulse.value = 1;
|
||||
pulse.value = withRepeat(
|
||||
withTiming(fast ? 1.14 : 1.07, {
|
||||
duration: fast ? 620 : 1500,
|
||||
easing: Easing.inOut(Easing.ease),
|
||||
}),
|
||||
-1,
|
||||
true,
|
||||
);
|
||||
return () => cancelAnimation(pulse);
|
||||
}, [state, pulse]);
|
||||
|
||||
const animStyle = useAnimatedStyle(() => ({ transform: [{ scale: pulse.value }] }));
|
||||
const color = COLORS[state] || COLORS.idle;
|
||||
|
||||
return (
|
||||
<View style={[styles.wrap, { width: size, height: size }]}>
|
||||
<Animated.View
|
||||
style={[
|
||||
styles.glow,
|
||||
{ width: size, height: size, borderRadius: size / 2, backgroundColor: color },
|
||||
animStyle,
|
||||
]}
|
||||
/>
|
||||
<Animated.View
|
||||
style={[
|
||||
styles.ring,
|
||||
{
|
||||
width: size * 0.72,
|
||||
height: size * 0.72,
|
||||
borderRadius: size * 0.36,
|
||||
borderColor: color,
|
||||
},
|
||||
animStyle,
|
||||
]}
|
||||
/>
|
||||
<View
|
||||
style={[
|
||||
styles.core,
|
||||
{
|
||||
width: size * 0.44,
|
||||
height: size * 0.44,
|
||||
borderRadius: size * 0.22,
|
||||
backgroundColor: color,
|
||||
shadowColor: color,
|
||||
},
|
||||
]}
|
||||
/>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
wrap: { alignItems: 'center', justifyContent: 'center' },
|
||||
glow: { position: 'absolute', opacity: 0.22 },
|
||||
ring: { position: 'absolute', borderWidth: 2, opacity: 0.55 },
|
||||
core: {
|
||||
shadowOpacity: 0.9,
|
||||
shadowRadius: 16,
|
||||
shadowOffset: { width: 0, height: 0 },
|
||||
elevation: 12,
|
||||
},
|
||||
});
|
||||
|
||||
export default React.memo(Orb);
|
||||
@@ -9,6 +9,7 @@
|
||||
*/
|
||||
|
||||
import React, { useEffect, useMemo, useState } from 'react';
|
||||
import { View } from 'react-native';
|
||||
import projectFocus, { FocusSnapshot } from '../services/projectFocus';
|
||||
import codeFile from '../services/codeFile';
|
||||
import brainApi from '../services/brainApi';
|
||||
@@ -16,6 +17,8 @@ import viewMode, { ViewModeValue } from '../services/viewMode';
|
||||
import ChatScreen from '../screens/ChatScreen';
|
||||
import { TileId } from './layout';
|
||||
import WorkspaceDeck from './WorkspaceDeck';
|
||||
import ariaView, { AriaView } from '../services/ariaView';
|
||||
import AriaViewCanvas from './AriaViewCanvas';
|
||||
|
||||
const COCKPIT_PANELS: TileId[] = ['chat', 'files', 'editor', 'vnc'];
|
||||
|
||||
@@ -24,12 +27,21 @@ const WorkspaceScreen: React.FC = () => {
|
||||
const [focus, setFocus] = useState<FocusSnapshot>(projectFocus.get());
|
||||
const [hasCode, setHasCode] = useState(false);
|
||||
const [hasDesktop, setHasDesktop] = useState(false);
|
||||
const [view, setView] = useState<AriaView | undefined>(undefined);
|
||||
|
||||
useEffect(() => viewMode.subscribe(setMode), []);
|
||||
useEffect(() => projectFocus.subscribe(setFocus), []);
|
||||
|
||||
const pid = focus.focusedProjectId;
|
||||
|
||||
// aria_view: ARIAs komponierte Ansicht fuers fokussierte Projekt spiegeln.
|
||||
useEffect(() => {
|
||||
setView(ariaView.getView(pid));
|
||||
return ariaView.subscribe((v) => {
|
||||
if ((v.projectId || '') === (pid || '')) setView(v);
|
||||
});
|
||||
}, [pid]);
|
||||
|
||||
// Code-Signal: hat der Spiegel schon Dateien fuer dieses Projekt?
|
||||
useEffect(() => {
|
||||
setHasCode(codeFile.getFiles(pid).length > 0);
|
||||
@@ -59,13 +71,32 @@ const WorkspaceScreen: React.FC = () => {
|
||||
vnc: hasDesktop ? '#34C759' : undefined,
|
||||
} as Partial<Record<TileId, string>>), [hasCode, hasDesktop]);
|
||||
|
||||
// Kompakt-Ansicht: klassischer Vollbild-Chat, exakt wie vor dem Umbau.
|
||||
if (mode === 'compact') {
|
||||
return <ChatScreen />;
|
||||
}
|
||||
// Kompakt-Ansicht: klassischer Vollbild-Chat; Cockpit: Workbench mit Dock.
|
||||
const content =
|
||||
mode === 'compact' ? (
|
||||
<ChatScreen />
|
||||
) : (
|
||||
<WorkspaceDeck projectId={pid} panels={COCKPIT_PANELS} badges={badges} />
|
||||
);
|
||||
|
||||
// Cockpit: Workbench mit Dock.
|
||||
return <WorkspaceDeck projectId={pid} panels={COCKPIT_PANELS} badges={badges} />;
|
||||
// Generative Flaeche als Overlay, sobald ARIA fuer dieses Projekt eine Ansicht
|
||||
// komponiert hat (present_view → aria_view). Chat/Cockpit bleiben darunter.
|
||||
const showView = !!view && (view.projectId || '') === (pid || '');
|
||||
|
||||
return (
|
||||
<View style={{ flex: 1 }}>
|
||||
{content}
|
||||
{showView && view && (
|
||||
<AriaViewCanvas
|
||||
view={view.view}
|
||||
onClose={() => {
|
||||
ariaView.clear(pid);
|
||||
setView(undefined);
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
export default WorkspaceScreen;
|
||||
|
||||
+245
-12
@@ -627,10 +627,11 @@ META_TOOLS = [
|
||||
"name": "request_location_tracking",
|
||||
"description": (
|
||||
"Bittet die App, das kontinuierliche GPS-Tracking zu aktivieren oder zu "
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). Nutze das wenn du einen "
|
||||
"GPS-basierten Watcher anlegst (z.B. `near(...)`), sonst hat die App "
|
||||
"veraltete Position und der Watcher feuert nie. Auch wieder ausschalten "
|
||||
"wenn der letzte GPS-Watcher geloescht wurde."
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). HINWEIS: Beim Anlegen/Loeschen "
|
||||
"eines Standort-Watchers (`near/entered_near/left_near`) schaltet das System "
|
||||
"das Tracking bereits AUTOMATISCH mit an bzw. aus — dieses Tool brauchst du "
|
||||
"dafuer nicht mehr. Nutze es nur fuer manuelle/explizite Faelle (z.B. Tracking "
|
||||
"kurz einschalten ohne Watcher)."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -642,6 +643,48 @@ META_TOOLS = [
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "present_view",
|
||||
"description": (
|
||||
"Komponiere eine RAEUMLICHE Ansicht, die auf ARIAs Flaeche materialisiert "
|
||||
"(Orb + Karten). Nutze das, wenn eine visuelle Anordnung besser ist als reiner "
|
||||
"Text — z.B. eine Akte/Datei zeigen (Bild links, Text rechts, Karte unten), "
|
||||
"einen Vergleich, eine Liste, eine Karte mit Orten, oder Code. NICHT fuer "
|
||||
"normale Gespraechsantworten. Die Karten erscheinen ZUSAETZLICH zu deiner "
|
||||
"(kurzen) Sprachantwort — halte die Antwort dann knapp, das Visuelle traegt."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cards": {
|
||||
"type": "array",
|
||||
"description": "Die Karten der Ansicht, in sinnvoller Anordnung.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {"type": "string", "enum": ["text", "image", "map", "code", "list"],
|
||||
"description": "Kartentyp"},
|
||||
"title": {"type": "string", "description": "Optionaler Kartentitel"},
|
||||
"md": {"type": "string", "description": "text/list: Markdown-Inhalt"},
|
||||
"src": {"type": "string", "description": "image: Bild-URL oder /shared-Pfad"},
|
||||
"markers": {"type": "array", "items": {"type": "object"},
|
||||
"description": "map: Liste [{lat, lon, label}]"},
|
||||
"path": {"type": "string", "description": "code: Datei-Pfad im Projekt"},
|
||||
"lang": {"type": "string", "description": "code: Sprache fuers Highlighting"},
|
||||
},
|
||||
"required": ["type"],
|
||||
},
|
||||
},
|
||||
"orb": {"type": "string", "enum": ["idle", "speaking", "working"],
|
||||
"description": "Orb-Zustand nach dem Rendern (default: speaking)"},
|
||||
"title": {"type": "string", "description": "Optionaler Titel der ganzen Ansicht"},
|
||||
},
|
||||
"required": ["cards"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
@@ -1304,6 +1347,101 @@ def _extract_await_marker(text: str) -> tuple:
|
||||
return text, False
|
||||
|
||||
|
||||
# ── Sprach-/Gespraechs-Steuermarker (ARIA deklariert die Phase SELBST) ──
|
||||
#
|
||||
# Voice-First: ARIA erkennt aus dem Text, ob Stefan einen BEFEHL gibt (etwas tun)
|
||||
# oder eine FRAGE stellt (etwas wissen), und ob das Gespraech/eine Befehlskette
|
||||
# weiterlaeuft oder endet. Sie haengt dazu Marker ans Ende ihrer Antwort — genau
|
||||
# wie [[AWAIT]], und sie werden ebenso entfernt (nicht angezeigt/vorgelesen/in
|
||||
# History). Der Marker ist AUTORITATIV — er ueberschreibt das Skill-Manifest-Flag,
|
||||
# denn dasselbe Skill (z.B. VM-/GUI-Steuerung) ist mal Befehl, mal Auskunft; nur
|
||||
# ARIA weiss aus dem Kontext, was gerade gemeint ist.
|
||||
#
|
||||
# [[STUMM]] -> reiner Steuerbefehl: NICHT vorlesen (speak=false). Allein =
|
||||
# Einzelbefehl → danach zurueck aufs Wake-Word (converse=false).
|
||||
# [[WEITER]] -> Konversation/Befehlskette laeuft weiter: Mikro offen halten
|
||||
# (converse=true) — kein erneutes "Computer" noetig.
|
||||
# [[ENDE]] -> Konversation/Kette beenden: zurueck aufs Wake-Word (converse=false).
|
||||
_SILENT_MARKER_RE = re.compile(r"\[\[\s*STUMM\s*\]\]", re.IGNORECASE)
|
||||
_CONT_MARKER_RE = re.compile(r"\[\[\s*WEITER\s*\]\]", re.IGNORECASE)
|
||||
_END_MARKER_RE = re.compile(r"\[\[\s*ENDE\s*\]\]", re.IGNORECASE)
|
||||
|
||||
|
||||
def _extract_flow_markers(text: str) -> tuple:
|
||||
"""Zieht [[STUMM]]/[[WEITER]]/[[ENDE]] aus dem finalen Text.
|
||||
Gibt (clean_text, speak_override, converse_override) zurueck; ein Override ist
|
||||
None, wenn der jeweilige Marker fehlt (dann gilt Default/Skill-Flag).
|
||||
Regeln: [[STUMM]] alleine = Einzelbefehl → auch converse=false (Mikro zu),
|
||||
ausser [[WEITER]] haelt es explizit offen. [[ENDE]] gewinnt gegen [[WEITER]]."""
|
||||
if not text:
|
||||
return text, None, None
|
||||
speak_ov = None
|
||||
conv_ov = None
|
||||
if _SILENT_MARKER_RE.search(text):
|
||||
speak_ov = False
|
||||
text = _SILENT_MARKER_RE.sub("", text)
|
||||
if _END_MARKER_RE.search(text):
|
||||
conv_ov = False
|
||||
text = _END_MARKER_RE.sub("", text)
|
||||
if _CONT_MARKER_RE.search(text):
|
||||
# [[ENDE]] hat Vorrang — widerspruechliche Marker → beenden.
|
||||
if conv_ov is None:
|
||||
conv_ov = True
|
||||
text = _CONT_MARKER_RE.sub("", text)
|
||||
# Stiller Einzelbefehl ohne explizites Weiterlauschen → Mikro zu.
|
||||
if speak_ov is False and conv_ov is None:
|
||||
conv_ov = False
|
||||
return text.strip(), speak_ov, conv_ov
|
||||
|
||||
|
||||
# Explizite "Konversation beenden"-Phrasen vom USER — deterministisch, NICHT auf
|
||||
# ARIAs [[ENDE]]-Marker angewiesen. Stefan will "Konversation Ende" o.ae. als
|
||||
# festen Trigger: danach zurueck aufs Wake-Word, egal was ARIA sonst tut. Eine in
|
||||
# derselben Nachricht enthaltene Frage beantwortet sie normal (wird vorgelesen),
|
||||
# aber converse wird auf false gezwungen. Nomen + Ende-Wort in EINEM Satzteil
|
||||
# ([^.!?]{0,15}) in beliebiger Reihenfolge; "befehls?kette" damit "Lieferkette"
|
||||
# o.ae. nicht faelschlich matcht.
|
||||
_CONV_NOUN = r"(?:konversation|gespr[aä]ch|befehls?kette)"
|
||||
_CONV_END_VERB = r"(?:ende|beend\w*|aus|stop\w*|schluss)"
|
||||
_END_CONVERSATION_RE = re.compile(
|
||||
rf"\b{_CONV_NOUN}\b[^.!?]{{0,15}}\b{_CONV_END_VERB}\b"
|
||||
rf"|\b(?:beend\w*|schlie(?:ß|ss)\w*)\b[^.!?]{{0,15}}\b{_CONV_NOUN}\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _user_wants_conversation_end(text: str) -> bool:
|
||||
"""True, wenn der User in dieser Nachricht explizit die Konversation/Kette
|
||||
beenden will (deterministisch, unabhaengig vom LLM-Marker)."""
|
||||
if not text:
|
||||
return False
|
||||
return bool(_END_CONVERSATION_RE.search(_strip_leading_hint_blocks(text)))
|
||||
|
||||
|
||||
# Gegenstueck zu _END: expliziter "Konversation OFFEN halten / fortfuehren"-Wunsch.
|
||||
# Wichtig fuer BEFEHLE die den Fast-Path treffen: "spiel Spotify ab ABER Konversation
|
||||
# fortfuehren" — der Fast-Path (Regex) versteht den Satz-Rest nicht und wuerde mit
|
||||
# converse=false schliessen. Dieser Detektor erzwingt converse=true, auch am
|
||||
# Fast-Path, egal was das Skill-Manifest sagt. Nomen+Verb in einem Satzteil, plus
|
||||
# "weiter reden/sprechen" ohne Nomen.
|
||||
_CONT_VERB = (r"(?:fortf[uü]hr\w*|fortsetz\w*|weiterf[uü]hr\w*|weiter\s*mach\w*|"
|
||||
r"weiter\b|fort\b|offen\s+(?:halten|lassen)|nicht\s+beenden|weiterlauf\w*)")
|
||||
_CONTINUE_CONVERSATION_RE = re.compile(
|
||||
rf"\b{_CONV_NOUN}\b[^.!?]{{0,20}}\b{_CONT_VERB}"
|
||||
rf"|\b{_CONT_VERB}[^.!?]{{0,20}}\b{_CONV_NOUN}\b"
|
||||
rf"|\bweiter\s*(?:reden|sprechen|quatschen|plaudern|labern)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _user_wants_conversation_continue(text: str) -> bool:
|
||||
"""True, wenn der User explizit weiter im Gespraech bleiben will (converse=true
|
||||
erzwingen — auch bei einem Fast-Path-Befehl). [[ENDE]]/_wants_end hat Vorrang."""
|
||||
if not text:
|
||||
return False
|
||||
return bool(_CONTINUE_CONVERSATION_RE.search(_strip_leading_hint_blocks(text)))
|
||||
|
||||
|
||||
def _normalize_for_fast_match(text: str) -> str:
|
||||
norm = _strip_leading_hint_blocks(text).lower()
|
||||
norm = _fold_umlauts(norm)
|
||||
@@ -1355,6 +1493,17 @@ def _skill_to_tool(s: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# Standort-Funktionen in Watcher-Conditions — brauchen aktives GPS-Tracking.
|
||||
_LOCATION_FUNCS = ("near(", "entered_near(", "left_near(")
|
||||
|
||||
|
||||
def _condition_needs_location(condition: str) -> bool:
|
||||
"""True, wenn die Watcher-Condition eine Standort-Funktion nutzt und damit
|
||||
laufende GPS-Updates der App braucht."""
|
||||
c = condition or ""
|
||||
return any(fn in c for fn in _LOCATION_FUNCS)
|
||||
|
||||
|
||||
class Agent:
|
||||
# Mindest-Score den ein Cold-Memory-Treffer haben muss um in den
|
||||
# System-Prompt aufgenommen zu werden. Unter dieser Schwelle ist's
|
||||
@@ -1701,6 +1850,14 @@ class Agent:
|
||||
if not user_message:
|
||||
raise ValueError("Leere Nachricht")
|
||||
|
||||
# Explizite Gespraechs-Steuerung vom USER (deterministisch, an JEDEM Return
|
||||
# angewendet — auch am Fast-Path, den die LLM-Marker nicht erreichen):
|
||||
# _wants_end → converse=false ("Konversation Ende")
|
||||
# _wants_continue → converse=true ("... aber Konversation fortfuehren")
|
||||
# End hat Vorrang bei Widerspruch.
|
||||
_wants_end = _user_wants_conversation_end(user_message)
|
||||
_wants_continue = (not _wants_end) and _user_wants_conversation_continue(user_message)
|
||||
|
||||
# Events vom letzten Turn weglassen
|
||||
self._pending_events = []
|
||||
|
||||
@@ -1728,6 +1885,10 @@ class Agent:
|
||||
speak = bool(getattr(self, "_fast_path_speak", False))
|
||||
# converse folgt dem Skill (Manifest/Output) — nicht mehr generell False.
|
||||
converse = bool(getattr(self, "_fast_path_converse", False))
|
||||
if _wants_end:
|
||||
converse = False
|
||||
elif _wants_continue:
|
||||
converse = True
|
||||
# Fast-Path = reiner Steuerbefehl, nie eine Rueckfrage → awaiting=False.
|
||||
return fast_reply, "fast-path", speak, converse, False
|
||||
|
||||
@@ -1747,6 +1908,10 @@ class Agent:
|
||||
# dem Skill (bzw. Default: Info/Gespraech = vorlesen + 30s).
|
||||
speak = getattr(self, "_local_turn_speak", True)
|
||||
converse = getattr(self, "_local_turn_converse", True)
|
||||
if _wants_end:
|
||||
converse = False
|
||||
elif _wants_continue:
|
||||
converse = True
|
||||
# Local ist tool-loses Reden; blockierende Rueckfragen macht Claude.
|
||||
return local_reply, "local", speak, converse, False
|
||||
|
||||
@@ -1764,6 +1929,15 @@ class Agent:
|
||||
logger.warning("Cold-Search fehlgeschlagen: %s", exc)
|
||||
cold = []
|
||||
|
||||
# 3b. Titel-Index des kalten Gedaechtnisses — ARIA sieht WAS sie an
|
||||
# Nachschlage-Wissen hat (Zugangsdaten, Infra, Projekte) und holt es via
|
||||
# memory_search, statt Stefan danach zu fragen. Nur Titel = billig.
|
||||
try:
|
||||
memory_index = self.store.list_index_titles()
|
||||
except Exception as exc:
|
||||
logger.warning("Titel-Index laden fehlgeschlagen: %s", exc)
|
||||
memory_index = []
|
||||
|
||||
# 4. Aktive Skills holen + Tool-Liste bauen
|
||||
all_skills = skills_mod.list_skills(active_only=False)
|
||||
active_skills = [s for s in all_skills if s.get("active", True)]
|
||||
@@ -1786,7 +1960,8 @@ class Agent:
|
||||
oauth_port = os.environ.get("RVS_PORT_PUBLIC", os.environ.get("RVS_PORT", "443")).strip()
|
||||
oauth_tls = os.environ.get("RVS_TLS", "true").strip().lower() != "false"
|
||||
|
||||
system_prompt = build_system_prompt(hot, cold, skills=all_skills,
|
||||
system_prompt = build_system_prompt(hot, cold, memory_index=memory_index,
|
||||
skills=all_skills,
|
||||
triggers=all_triggers,
|
||||
condition_vars=condition_vars,
|
||||
condition_funcs=condition_funcs,
|
||||
@@ -1979,16 +2154,30 @@ class Agent:
|
||||
# Rueckfrage-Marker aus dem finalen Text ziehen (vor History/Return, damit
|
||||
# er nicht angezeigt/vorgelesen wird und nicht die Conversation vergiftet).
|
||||
final_reply, awaiting_reply = _extract_await_marker(final_reply)
|
||||
# ARIAs Phasen-Marker ([[STUMM]]/[[WEITER]]/[[ENDE]]) ziehen — VOR History,
|
||||
# damit sie nicht angezeigt/vorgelesen/gespeichert werden.
|
||||
final_reply, _speak_ov, _conv_ov = _extract_flow_markers(final_reply)
|
||||
|
||||
# 7. Assistant-Turn (final reply) in die Conversation
|
||||
self.conversation.add("assistant", final_reply,
|
||||
project_id=active_project_id)
|
||||
# speak/converse folgen dem ausgefuehrten Skill (sonst Default: Gespraech);
|
||||
# speak/converse folgen dem ausgefuehrten Skill (sonst Default: Gespraech).
|
||||
# ARIAs Phasen-Marker sind AUTORITATIV: sie kennt aus dem Text den Unter-
|
||||
# schied Befehl/Frage und Kette/Ende, den das Skill-Manifest nicht kennt.
|
||||
speak = bool(getattr(self, "_claude_turn_speak", True))
|
||||
converse = bool(getattr(self, "_claude_turn_converse", True))
|
||||
if _speak_ov is not None:
|
||||
speak = _speak_ov
|
||||
if _conv_ov is not None:
|
||||
converse = _conv_ov
|
||||
# Explizite User-Woerter gewinnen ueber Marker/Manifest: "Konversation
|
||||
# beenden" → zu; "... fortfuehren" → offen halten. End hat Vorrang.
|
||||
if _wants_end:
|
||||
converse = False
|
||||
elif _wants_continue:
|
||||
converse = True
|
||||
# awaiting_reply = ARIA stellt eine blockierende Rueckfrage (Queue pausiert).
|
||||
return (final_reply, "claude",
|
||||
bool(getattr(self, "_claude_turn_speak", True)),
|
||||
bool(getattr(self, "_claude_turn_converse", True)),
|
||||
awaiting_reply)
|
||||
return (final_reply, "claude", speak, converse, awaiting_reply)
|
||||
|
||||
# ── Tool-Dispatcher ───────────────────────────────────────
|
||||
|
||||
@@ -2372,13 +2561,41 @@ class Agent:
|
||||
"trigger": {"name": t["name"], "type": "watcher",
|
||||
"condition": t["condition"], "message": t["message"]},
|
||||
})
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'."
|
||||
# GPS-Kopplung: ein Standort-Watcher (near/entered_near/left_near)
|
||||
# ist tot, wenn die App kein Tracking sendet. Frueher musste ARIA
|
||||
# separat request_location_tracking aufrufen — wurde oft vergessen,
|
||||
# dann feuerte der Trigger nie. Jetzt automatisch mit-anschalten.
|
||||
extra = ""
|
||||
if _condition_needs_location(t["condition"]):
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": True,
|
||||
"reason": f"watcher:{t['name']}",
|
||||
})
|
||||
extra = " GPS-Tracking automatisch aktiviert."
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'.{extra}"
|
||||
if name == "trigger_cancel":
|
||||
try:
|
||||
triggers_mod.delete(arguments["name"])
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht."
|
||||
except ValueError as e:
|
||||
return f"FEHLER: {e}"
|
||||
# GPS-Kopplung (Gegenstueck): existiert kein Standort-Watcher mehr,
|
||||
# Tracking wieder ausschalten (Akku schonen).
|
||||
extra = ""
|
||||
remaining = triggers_mod.list_triggers(active_only=False)
|
||||
still_needs_gps = any(
|
||||
r.get("type") == "watcher"
|
||||
and _condition_needs_location(r.get("condition") or "")
|
||||
for r in remaining
|
||||
)
|
||||
if not still_needs_gps:
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": False,
|
||||
"reason": "no-location-watchers",
|
||||
})
|
||||
extra = " GPS-Tracking deaktiviert (kein Standort-Watcher mehr)."
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht.{extra}"
|
||||
if name == "request_location_tracking":
|
||||
on = bool(arguments.get("on", False))
|
||||
reason = (arguments.get("reason") or "").strip()
|
||||
@@ -2388,6 +2605,22 @@ class Agent:
|
||||
"reason": reason,
|
||||
})
|
||||
return f"OK — Tracking-Request gesendet (on={on}). App wird in Kuerze umschalten."
|
||||
if name == "present_view":
|
||||
cards = arguments.get("cards") or []
|
||||
if not isinstance(cards, list) or not cards:
|
||||
return "FEHLER: present_view braucht mindestens eine Karte in 'cards'."
|
||||
view = {
|
||||
"cards": cards,
|
||||
"orb": arguments.get("orb") or "speaking",
|
||||
"title": arguments.get("title") or "",
|
||||
}
|
||||
self._pending_events.append({
|
||||
"type": "aria_view",
|
||||
"view": view,
|
||||
"project_id": project_id or "",
|
||||
})
|
||||
return (f"OK — Ansicht mit {len(cards)} Karte(n) an die Flaeche geschickt "
|
||||
f"(Kontext: {project_id or 'Hauptchat'}).")
|
||||
if name == "trigger_list":
|
||||
items = triggers_mod.list_triggers(active_only=False)
|
||||
if not items:
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Einmaliger Backfill: weist bestehenden Memory-Punkten ein `scope`
|
||||
(system | personal) zu. Sicher & reversibel — Stefan kann pro Eintrag in der
|
||||
Diagnostic-UI umschalten. Idempotent: laeuft mehrfach ohne Schaden.
|
||||
|
||||
Heuristik (datengetrieben aus dem realen Bestand):
|
||||
- type=preference / fact / conversation / reminder -> personal
|
||||
- source in (seed, auto-feedback) -> system
|
||||
- type=identity -> system
|
||||
- type in (rule, tool, skill) und category in SYSTEM_CATS -> system
|
||||
- sonst -> personal (sicher: nichts leakt)
|
||||
|
||||
Aufruf im Brain-Container:
|
||||
docker exec aria-brain python3 /app/backfill_scope.py # dry-run
|
||||
docker exec aria-brain python3 /app/backfill_scope.py --apply # schreibt
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models as qm
|
||||
|
||||
COLLECTION = "aria_memory"
|
||||
SYSTEM_CATS = {
|
||||
"sicherheit", "arbeitsweise", "architektur", "ehrlichkeit", "verhalten",
|
||||
"voice", "skills", "freigaben", "infrastruktur", "persoenlichkeit",
|
||||
"pentest", "ausgabe",
|
||||
}
|
||||
|
||||
|
||||
def compute_scope(pl: dict) -> str:
|
||||
typ = pl.get("type")
|
||||
src = pl.get("source")
|
||||
cat = (pl.get("category") or "").lower()
|
||||
if typ == "preference":
|
||||
return "personal"
|
||||
if typ in ("fact", "conversation", "reminder"):
|
||||
return "personal"
|
||||
if src in ("seed", "auto-feedback"):
|
||||
return "system"
|
||||
if typ == "identity":
|
||||
return "system"
|
||||
if typ in ("rule", "tool", "skill") and cat in SYSTEM_CATS:
|
||||
return "system"
|
||||
return "personal"
|
||||
|
||||
|
||||
def main():
|
||||
apply = "--apply" in sys.argv
|
||||
force = "--force" in sys.argv # auch schon gesetzte scopes ueberschreiben
|
||||
c = QdrantClient(
|
||||
host=os.environ.get("QDRANT_HOST", "aria-qdrant"),
|
||||
port=int(os.environ.get("QDRANT_PORT", "6333")),
|
||||
)
|
||||
pts, _ = c.scroll(collection_name=COLLECTION, limit=5000,
|
||||
with_payload=True, with_vectors=False)
|
||||
|
||||
per_scope: dict[str, list] = {"system": [], "personal": []}
|
||||
pinned_examples = Counter()
|
||||
skipped = 0
|
||||
for p in pts:
|
||||
pl = p.payload or {}
|
||||
if pl.get("scope") in ("system", "personal") and not force:
|
||||
skipped += 1
|
||||
continue
|
||||
scope = compute_scope(pl)
|
||||
per_scope[scope].append(p.id)
|
||||
if pl.get("pinned"):
|
||||
pinned_examples[(scope, pl.get("source"), pl.get("type"),
|
||||
pl.get("category"))] += 1
|
||||
|
||||
print(f"total={len(pts)} skipped(already set)={skipped}")
|
||||
print(f"-> system={len(per_scope['system'])} personal={len(per_scope['personal'])}")
|
||||
print("pinned split (scope, source, type, category):")
|
||||
for k, v in sorted(pinned_examples.items()):
|
||||
print(" ", k, v)
|
||||
|
||||
if not apply:
|
||||
print("\nDRY-RUN — nichts geschrieben. Mit --apply ausfuehren.")
|
||||
return
|
||||
|
||||
for scope, ids in per_scope.items():
|
||||
if not ids:
|
||||
continue
|
||||
c.set_payload(collection_name=COLLECTION, payload={"scope": scope}, points=ids)
|
||||
print(f"\nAPPLIED: system={len(per_scope['system'])} personal={len(per_scope['personal'])}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+45
-13
@@ -190,6 +190,7 @@ class MemoryIn(BaseModel):
|
||||
pinned: bool = False
|
||||
category: str = ""
|
||||
source: str = "manual"
|
||||
scope: str = "personal" # system | personal — steuert Bootstrap-Export
|
||||
tags: List[str] = Field(default_factory=list)
|
||||
conversation_id: Optional[str] = None
|
||||
# Vorhandene Anhang-Metadaten beim Save mitgeben (i.d.R. werden Anhaenge
|
||||
@@ -203,6 +204,7 @@ class MemoryUpdate(BaseModel):
|
||||
content: Optional[str] = None
|
||||
pinned: Optional[bool] = None
|
||||
category: Optional[str] = None
|
||||
scope: Optional[str] = None # system | personal
|
||||
tags: Optional[List[str]] = None
|
||||
|
||||
|
||||
@@ -214,6 +216,7 @@ class MemoryOut(BaseModel):
|
||||
pinned: bool
|
||||
category: str
|
||||
source: str
|
||||
scope: str = "personal"
|
||||
tags: List[str]
|
||||
created_at: str
|
||||
updated_at: str
|
||||
@@ -328,6 +331,7 @@ def memory_save(body: MemoryIn):
|
||||
pinned=body.pinned,
|
||||
category=body.category,
|
||||
source=body.source,
|
||||
scope=body.scope,
|
||||
tags=body.tags,
|
||||
conversation_id=body.conversation_id,
|
||||
attachments=body.attachments or [],
|
||||
@@ -353,6 +357,8 @@ def memory_update(point_id: str, body: MemoryUpdate):
|
||||
existing.pinned = body.pinned
|
||||
if body.category is not None:
|
||||
existing.category = body.category
|
||||
if body.scope is not None:
|
||||
existing.scope = body.scope
|
||||
if body.tags is not None:
|
||||
existing.tags = body.tags
|
||||
|
||||
@@ -537,12 +543,23 @@ def memory_import_files():
|
||||
# Wiederherstellen einer schlanken ARIA nach Wipe.
|
||||
|
||||
@app.get("/memory/export-bootstrap")
|
||||
def memory_export_bootstrap():
|
||||
"""Gibt alle pinned Memories als JSON zurueck — fuer Browser-Download."""
|
||||
def memory_export_bootstrap(scope: str = "system"):
|
||||
"""Gibt pinned Memories als JSON zurueck — fuer Browser-Download.
|
||||
|
||||
scope='system' → nur generische Regeln (fuer ein frisches System),
|
||||
scope='personal' → nur Stefan-spezifisches (Name, Zugangsdaten, Projekte),
|
||||
scope='all' → alles pinned (Vollbackup).
|
||||
Default 'system', damit man nicht versehentlich Persoenliches teilt."""
|
||||
s = store()
|
||||
pinned = s.list_pinned()
|
||||
if scope == "all":
|
||||
pinned = s.list_pinned()
|
||||
elif scope in ("system", "personal"):
|
||||
pinned = s.list_pinned_by_scope(scope)
|
||||
else:
|
||||
raise HTTPException(400, f"Ungueltiger scope: {scope}")
|
||||
return {
|
||||
"version": 1,
|
||||
"version": 2,
|
||||
"scope": scope,
|
||||
"exported_at": __import__("datetime").datetime.now(
|
||||
__import__("datetime").timezone.utc
|
||||
).isoformat(),
|
||||
@@ -555,6 +572,7 @@ def memory_export_bootstrap():
|
||||
"pinned": True,
|
||||
"category": p.category,
|
||||
"source": p.source,
|
||||
"scope": p.scope,
|
||||
"tags": p.tags,
|
||||
}
|
||||
for p in pinned
|
||||
@@ -564,13 +582,18 @@ def memory_export_bootstrap():
|
||||
|
||||
class BootstrapBundle(BaseModel):
|
||||
version: int = 1
|
||||
scope: Optional[str] = None # system | personal | all (aus dem Export)
|
||||
memories: List[dict]
|
||||
|
||||
|
||||
@app.post("/memory/import-bootstrap")
|
||||
def memory_import_bootstrap(body: BootstrapBundle):
|
||||
"""Loescht alle pinned Memories und importiert die im Bundle.
|
||||
Cold Memory (unpinned) bleibt unangetastet.
|
||||
"""Importiert ein Bootstrap-Bundle scope-sicher.
|
||||
|
||||
Es werden NUR die aktuell pinned Punkte geloescht, deren scope zum Import
|
||||
gehoert — ein System-Import laesst also die persoenlichen pinned Memories
|
||||
(Name, Zugangsdaten) unangetastet und umgekehrt. Bei einem 'all'-Bundle
|
||||
(Vollbackup) werden alle pinned ersetzt.
|
||||
|
||||
Wenn keine Memories im Bundle: nur loeschen ist NICHT erlaubt — der
|
||||
Caller soll erst exportieren und dann importieren.
|
||||
@@ -580,23 +603,31 @@ def memory_import_bootstrap(body: BootstrapBundle):
|
||||
|
||||
s = store()
|
||||
e = embedder()
|
||||
|
||||
# Alle aktuell pinned Punkte loeschen
|
||||
from qdrant_client.http import models as qm
|
||||
from memory.vector_store import COLLECTION
|
||||
|
||||
# Scope bestimmen: explizit aus dem Bundle, sonst aus den memories ableiten.
|
||||
bundle_scope = body.scope
|
||||
if bundle_scope not in ("system", "personal", "all"):
|
||||
scopes_in_mems = {m.get("scope", "personal") for m in body.memories}
|
||||
bundle_scope = scopes_in_mems.pop() if len(scopes_in_mems) == 1 else "all"
|
||||
|
||||
# Nur die pinned Punkte des betroffenen scope loeschen.
|
||||
del_must = [qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True))]
|
||||
if bundle_scope in ("system", "personal"):
|
||||
del_must.append(qm.FieldCondition(key="scope", match=qm.MatchValue(value=bundle_scope)))
|
||||
s.client.delete(
|
||||
collection_name=COLLECTION,
|
||||
points_selector=qm.FilterSelector(filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True))
|
||||
])),
|
||||
points_selector=qm.FilterSelector(filter=qm.Filter(must=del_must)),
|
||||
)
|
||||
|
||||
# Neue Punkte einspeisen
|
||||
# Neue Punkte einspeisen — scope pro memory (Fallback: bundle_scope bzw. personal).
|
||||
created = 0
|
||||
for m in body.memories:
|
||||
content = (m.get("content") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
mscope = m.get("scope") or (bundle_scope if bundle_scope != "all" else "personal")
|
||||
point = MemoryPoint(
|
||||
id="",
|
||||
type=m.get("type", "fact"),
|
||||
@@ -605,13 +636,14 @@ def memory_import_bootstrap(body: BootstrapBundle):
|
||||
pinned=True,
|
||||
category=m.get("category", ""),
|
||||
source=m.get("source", "bootstrap-import"),
|
||||
scope=mscope,
|
||||
tags=list(m.get("tags", [])),
|
||||
)
|
||||
vec = e.embed(content)
|
||||
s.upsert(point, vec)
|
||||
created += 1
|
||||
|
||||
return {"created": created, "deleted_previous_pinned": True}
|
||||
return {"created": created, "scope": bundle_scope, "deleted_previous_pinned": True}
|
||||
|
||||
|
||||
# ─── Conversation-Loop ──────────────────────────────────────────────
|
||||
|
||||
@@ -11,6 +11,10 @@ Punkt-Schema (Payload):
|
||||
content — eigentlicher Text (wird embedded)
|
||||
pinned — bool, True = Hot Memory (immer in Prompt)
|
||||
source — import | conversation | manual
|
||||
scope — system | personal. system = generische Regeln, die JEDER
|
||||
braucht, der das System aufsetzt (Sicherheit, Ehrlichkeit,
|
||||
Skill-Regeln). personal = Stefan-spezifisch (Name, Zugangs-
|
||||
daten, Projekte). Steuert den getrennten Bootstrap-Export.
|
||||
tags — Liste von Strings
|
||||
created_at, updated_at — ISO-Strings
|
||||
conversation_id — optional, nur fuer type=conversation
|
||||
@@ -55,6 +59,7 @@ class MemoryPoint:
|
||||
pinned: bool = False
|
||||
category: str = ""
|
||||
source: str = "manual"
|
||||
scope: str = "personal" # system | personal — steuert Bootstrap-Export
|
||||
tags: List[str] = field(default_factory=list)
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
@@ -74,6 +79,7 @@ class MemoryPoint:
|
||||
"pinned": self.pinned,
|
||||
"category": self.category,
|
||||
"source": self.source,
|
||||
"scope": self.scope,
|
||||
"tags": self.tags,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
@@ -94,6 +100,7 @@ class MemoryPoint:
|
||||
pinned=payload.get("pinned", False),
|
||||
category=payload.get("category", ""),
|
||||
source=payload.get("source", "manual"),
|
||||
scope=payload.get("scope", "personal"),
|
||||
tags=payload.get("tags", []),
|
||||
created_at=payload.get("created_at", ""),
|
||||
updated_at=payload.get("updated_at", ""),
|
||||
@@ -120,14 +127,23 @@ class VectorStore:
|
||||
collection_name=COLLECTION,
|
||||
vectors_config=qm.VectorParams(size=VECTOR_DIM, distance=qm.Distance.COSINE),
|
||||
)
|
||||
# Indexe fuer typische Filter-Felder
|
||||
for field_name in ("type", "pinned", "category", "source", "migration_key"):
|
||||
# Indexe fuer typische Filter-Felder — idempotent, laeuft auch auf
|
||||
# einer bestehenden Collection (fuer neu hinzugekommene Felder wie scope).
|
||||
self._ensure_indexes()
|
||||
|
||||
def _ensure_indexes(self):
|
||||
for field_name in ("type", "pinned", "category", "source", "scope", "migration_key"):
|
||||
schema = (qm.PayloadSchemaType.BOOL if field_name == "pinned"
|
||||
else qm.PayloadSchemaType.KEYWORD)
|
||||
try:
|
||||
self.client.create_payload_index(
|
||||
collection_name=COLLECTION,
|
||||
field_name=field_name,
|
||||
field_schema=qm.PayloadSchemaType.KEYWORD if field_name != "pinned"
|
||||
else qm.PayloadSchemaType.BOOL,
|
||||
field_schema=schema,
|
||||
)
|
||||
except Exception:
|
||||
# Index existiert bereits — kein Problem.
|
||||
pass
|
||||
|
||||
# ─── Schreib-Operationen ─────────────────────────────────────────
|
||||
|
||||
@@ -164,6 +180,38 @@ class VectorStore:
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True))
|
||||
]))
|
||||
|
||||
def list_pinned_by_scope(self, scope: str) -> List[MemoryPoint]:
|
||||
"""Alle pinned Punkte eines scope (system | personal). Fuer den
|
||||
getrennten Bootstrap-Export."""
|
||||
return self._scroll(filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True)),
|
||||
qm.FieldCondition(key="scope", match=qm.MatchValue(value=scope)),
|
||||
]))
|
||||
|
||||
def list_index_titles(self, limit: int = 500) -> List[MemoryPoint]:
|
||||
"""Leichtgewichtiger Titel-Index des kalten Gedaechtnisses fuer den
|
||||
System-Prompt: ARIA sieht WAS sie an Nachschlage-Wissen hat (Zugangs-
|
||||
daten, Infrastruktur, Projekte) und holt den Inhalt bei Bedarf via
|
||||
memory_search — statt Stefan nach etwas zu fragen, das schon da ist.
|
||||
|
||||
Bewusst NUR die deliberat gespeicherten Punkte:
|
||||
- nicht pinned (die sind eh schon voll im Prompt),
|
||||
- kein type=conversation (Chat-Mitschnitte),
|
||||
- kein source=distilled (die 100e auto-destillierten Gespraechs-
|
||||
Fakten — die traegt das semantische Auto-Retrieval, sie hier
|
||||
als Titel zu listen wuerde nur Kontext fressen).
|
||||
So bleibt der Index klein (Dutzende statt Hunderte Zeilen)."""
|
||||
return self._scroll(
|
||||
filter=qm.Filter(
|
||||
must_not=[
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True)),
|
||||
qm.FieldCondition(key="type", match=qm.MatchValue(value="conversation")),
|
||||
qm.FieldCondition(key="source", match=qm.MatchValue(value="distilled")),
|
||||
]
|
||||
),
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
def list_by_type(self, type_: str, limit: int = 100) -> List[MemoryPoint]:
|
||||
return self._scroll(
|
||||
filter=qm.Filter(must=[
|
||||
|
||||
@@ -252,6 +252,7 @@ def _parse_user_md(md: str, source_file: str) -> List[MemoryPoint]:
|
||||
type_="preference", title=f"User: {btitle}",
|
||||
content=btext, category="allgemein",
|
||||
migration_key=f"{source_file}/general-{idx}",
|
||||
scope="personal",
|
||||
))
|
||||
else:
|
||||
cat_key = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") or "allgemein"
|
||||
@@ -259,6 +260,7 @@ def _parse_user_md(md: str, source_file: str) -> List[MemoryPoint]:
|
||||
type_="preference", title=title,
|
||||
content=content, category=cat_key,
|
||||
migration_key=f"{source_file}/{cat_key}",
|
||||
scope="personal",
|
||||
))
|
||||
return points
|
||||
|
||||
@@ -283,7 +285,11 @@ def _mk(
|
||||
migration_key: str,
|
||||
pinned: bool = True,
|
||||
category: str = "",
|
||||
scope: str = "system",
|
||||
) -> MemoryPoint:
|
||||
# scope-Default 'system': AGENT.md + TOOLING.md beschreiben ARIA selbst
|
||||
# (Identitaet, Sicherheit, Architektur) — das braucht jedes System.
|
||||
# USER.md-Praeferenzen sind personal und uebergeben scope='personal'.
|
||||
p = MemoryPoint(
|
||||
id="",
|
||||
type=type_,
|
||||
@@ -292,6 +298,7 @@ def _mk(
|
||||
pinned=pinned,
|
||||
category=category,
|
||||
source="import",
|
||||
scope=scope,
|
||||
tags=[],
|
||||
)
|
||||
# migration_key wird ueber Payload-Index angesprochen — in to_payload manuell anhaengen
|
||||
|
||||
+76
-1
@@ -162,6 +162,46 @@ def build_time_section() -> str:
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
def build_voice_flow_section() -> str:
|
||||
"""Sprach-/Gespraechssteuerung: ARIA erkennt AUS DEM TEXT die Phase (Befehl vs.
|
||||
Frage, Kette vs. Ende) und deklariert sie per Marker — wie [[AWAIT]]. Die
|
||||
Marker werden im Brain entfernt (nie angezeigt/vorgelesen)."""
|
||||
return "\n".join([
|
||||
"## Sprach- & Gespraechssteuerung (Voice-First — du entscheidest die Phase)",
|
||||
"Stefan spricht meist mit dir. DU erkennst aus dem Text, was gerade Phase "
|
||||
"ist — niemand raet das fuer dich. Dazu haengst du EINEN Marker (bei Bedarf "
|
||||
"zwei) ganz ans ENDE deiner Antwort. Sie werden entfernt: nicht angezeigt, "
|
||||
"nicht vorgelesen, nicht gespeichert — genau wie `[[AWAIT]]`.",
|
||||
"",
|
||||
"- `[[STUMM]]` → Deine Antwort ist ein reiner **Steuerbefehl** (du hast etwas "
|
||||
"GETAN: Musik, VNC oeffnen, einen Menuepunkt klicken, Licht …). Sie wird "
|
||||
"NICHT vorgelesen; der kurze Bestaetigungstext steht nur in der Bubble. "
|
||||
"Setz das IMMER, wenn Stefan dir einen Befehl gibt statt eine Frage stellt — "
|
||||
"AUCH wenn du den Befehl ueber ein Skill/Tool ausfuehrst (nicht nur beim "
|
||||
"Fast-Path). `[[STUMM]]` ALLEIN = Einzelbefehl → danach direkt zurueck aufs "
|
||||
"Wake-Word.",
|
||||
"- `[[WEITER]]` → Das Gespraech bzw. eine **Befehlskette** laeuft weiter: das "
|
||||
"Mikro bleibt offen, du wartest auf die naechste Eingabe (kein erneutes "
|
||||
"\"Computer\" noetig). Setz das, wenn Stefan eine Kette ankuendigt ('ich geb "
|
||||
"dir gleich mehrere Befehle', 'wir machen das jetzt Schritt fuer Schritt') "
|
||||
"oder das Gespraech klar weitergeht.",
|
||||
"- `[[ENDE]]` → Konversation/Kette ist zu Ende: zurueck aufs Wake-Word. Setz "
|
||||
"das, wenn Stefan schliesst ('das war's', 'Konversation Ende', 'Befehlskette "
|
||||
"Ende', 'danke, fertig'). Stellt er in DERSELBEN Nachricht noch eine Frage, "
|
||||
"beantworte sie normal (OHNE `[[STUMM]]`, wird also vorgelesen) UND haeng "
|
||||
"`[[ENDE]]` an.",
|
||||
"",
|
||||
"Regeln:",
|
||||
"- Befehl (etwas TUN) → `[[STUMM]]`. Frage (etwas WISSEN / plaudern) → normal, "
|
||||
"ohne Marker (wird vorgelesen).",
|
||||
"- Befehlskette: JEDER Schritt `[[STUMM]] [[WEITER]]` (stumm arbeiten, Mikro "
|
||||
"offen), bis Stefan die Kette beendet → letzter Turn `[[ENDE]]`.",
|
||||
"- Ohne Marker = normales Gespraech: du wirst vorgelesen und ich lausche "
|
||||
"danach kurz weiter (Stefan kann einfach antworten, ohne 'Computer').",
|
||||
"- Nie widerspruechlich: `[[ENDE]]` schlaegt `[[WEITER]]`.",
|
||||
])
|
||||
|
||||
|
||||
TYPE_HEADINGS = {
|
||||
"identity": "## Wer du bist",
|
||||
"rule": "## Sicherheitsregeln & Prinzipien",
|
||||
@@ -260,6 +300,36 @@ def build_cold_memory_section(matches: List[MemoryPoint]) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_memory_index_section(index_titles: List[MemoryPoint]) -> str:
|
||||
"""Titel-Index des kalten Gedaechtnisses: ARIA sieht WELCHES Nachschlage-
|
||||
Wissen sie hat (nur Titel, kein Inhalt = billig), damit sie den Inhalt via
|
||||
memory_search holt statt Stefan nach etwas zu fragen, das schon da ist.
|
||||
Nach Kategorie gruppiert; Conversation-Logs + auto-destillierte Fakten sind
|
||||
bereits ausgefiltert (siehe list_index_titles)."""
|
||||
if not index_titles:
|
||||
return ""
|
||||
grouped: dict[str, List[MemoryPoint]] = {}
|
||||
for p in index_titles:
|
||||
key = (p.category or p.type or "sonstiges").strip() or "sonstiges"
|
||||
grouped.setdefault(key, []).append(p)
|
||||
|
||||
lines = [
|
||||
"## Was in deinem Gedaechtnis liegt (per memory_search abrufbar)",
|
||||
"Diese Eintraege hast DU gespeichert — hier nur die Titel, nicht der "
|
||||
"Inhalt. Wenn einer zur Aufgabe passt, hol den Inhalt mit `memory_search` "
|
||||
"(Titel oder Stichwort). **Frag Stefan NICHT nach etwas, das hier steht** "
|
||||
"(Zugangsdaten, Server/Hosts, Projekt-Stand, Konfig) — erst nachsehen.",
|
||||
"",
|
||||
]
|
||||
for cat in sorted(grouped.keys()):
|
||||
items = grouped[cat]
|
||||
lines.append(f"### {cat}")
|
||||
for p in items:
|
||||
lines.append(f"- {p.title}")
|
||||
lines.append("")
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
|
||||
def build_skills_section(skills: List[dict]) -> str:
|
||||
"""Listet alle Skills (aktiv + deaktiviert) damit ARIA weiss was es gibt
|
||||
und keine doppelt baut. Plus klare Schwelle wann ein Skill sich lohnt."""
|
||||
@@ -450,6 +520,7 @@ def build_flux_section(flux_config: dict) -> str:
|
||||
def build_system_prompt(
|
||||
pinned: List[MemoryPoint],
|
||||
cold: List[MemoryPoint] | None = None,
|
||||
memory_index: List[MemoryPoint] | None = None,
|
||||
skills: List[dict] | None = None,
|
||||
triggers: List[dict] | None = None,
|
||||
condition_vars: List[dict] | None = None,
|
||||
@@ -463,7 +534,8 @@ def build_system_prompt(
|
||||
"""Kompletter System-Prompt: Hot + Cold + Skills + Triggers + FLUX + OAuth."""
|
||||
# Identitaets-Anker IMMER zuerst — vor allen Memories/Sektionen, damit die
|
||||
# ARIA-Rolle auch in Projekten mit injection-artigem Inhalt (Pentest) haelt.
|
||||
parts = [IDENTITY_ANCHOR, "", build_hot_memory_section(pinned), "", build_time_section()]
|
||||
parts = [IDENTITY_ANCHOR, "", build_hot_memory_section(pinned), "", build_time_section(),
|
||||
"", build_voice_flow_section()]
|
||||
if skills:
|
||||
parts.append("")
|
||||
parts.append(build_skills_section(skills))
|
||||
@@ -482,6 +554,9 @@ def build_system_prompt(
|
||||
callback_host=oauth_callback_host,
|
||||
callback_port=oauth_callback_port,
|
||||
callback_tls=oauth_callback_tls))
|
||||
if memory_index:
|
||||
parts.append("")
|
||||
parts.append(build_memory_index_section(memory_index))
|
||||
if cold:
|
||||
parts.append("")
|
||||
parts.append(build_cold_memory_section(cold))
|
||||
|
||||
@@ -915,6 +915,7 @@ def apply(store: VectorStore, embedder: Embedder) -> dict:
|
||||
"pinned": True,
|
||||
"category": rule.get("category", ""),
|
||||
"source": "seed",
|
||||
"scope": "system",
|
||||
"tags": [],
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
|
||||
@@ -2080,6 +2080,22 @@ class ARIABridge:
|
||||
proj = event.get("project") or {}
|
||||
logger.info("[brain] Projekt %s: %s (id=%s)",
|
||||
event.get("action") or "?", proj.get("name"), proj.get("id"))
|
||||
elif etype == "aria_view":
|
||||
# M1: ARIA hat via present_view eine raeumliche Ansicht komponiert.
|
||||
# View-Spec (Orb + Karten) + Projekt-Kontext an App/Web/Diagnostic;
|
||||
# deren Renderer materialisieren die Karten auf der Flaeche.
|
||||
view = event.get("view") or {}
|
||||
await self._send_to_rvs({
|
||||
"type": "aria_view",
|
||||
"payload": {
|
||||
"view": view,
|
||||
"projectId": event.get("project_id") or "",
|
||||
"clientMsgId": client_msg_id or "",
|
||||
},
|
||||
"timestamp": int(asyncio.get_event_loop().time() * 1000),
|
||||
})
|
||||
logger.info("[brain] ARIA hat eine Ansicht geschickt: %d Karte(n), orb=%s",
|
||||
len(view.get("cards") or []), view.get("orb"))
|
||||
|
||||
# _process_core_response uebernimmt alles weitere:
|
||||
# File-Marker extrahieren + broadcasten, NO_REPLY-Check, Chat-
|
||||
|
||||
+73
-21
@@ -788,6 +788,18 @@
|
||||
<div id="voice-id-status" style="font-size:13px;color:#E0E0F0;margin-bottom:10px;">
|
||||
Status wird geladen...
|
||||
</div>
|
||||
<div style="display:flex;align-items:center;gap:12px;margin-bottom:8px;">
|
||||
<label style="color:#8888AA;font-size:12px;min-width:130px;">Nur meine Stimme:</label>
|
||||
<label style="display:flex;align-items:center;gap:8px;cursor:pointer;flex:1;">
|
||||
<input type="checkbox" id="diag-voice-id-enabled" onchange="sendVoiceConfig()">
|
||||
<span style="color:#E0E0F0;font-size:12px;">Speaker-ID-Prüfung aktiv</span>
|
||||
</label>
|
||||
</div>
|
||||
<div style="font-size:10px;color:#555570;margin-bottom:12px;">
|
||||
AUS (Default) = alle Stimmen kommen durch (fail-open). AN = nur der enrollte
|
||||
Sprecher wird ans Brain geleitet, fremde Stimmen werden verworfen. Erst
|
||||
einschalten wenn ein Fingerprint eingelernt ist — sonst hört ARIA niemanden.
|
||||
</div>
|
||||
<div style="display:flex;align-items:center;gap:12px;margin-bottom:8px;">
|
||||
<label style="color:#8888AA;font-size:12px;min-width:130px;">Match-Threshold:</label>
|
||||
<input type="range" id="diag-voice-id-threshold" min="0.30" max="0.70" step="0.05" value="0.50"
|
||||
@@ -1068,11 +1080,13 @@
|
||||
<div style="background:#0D0D1A;border-radius:6px;padding:10px 12px;margin-bottom:8px;">
|
||||
<div style="color:#FFD60A;font-weight:bold;font-size:12px;margin-bottom:4px;">2. Bootstrap-Snapshot (nur pinned)</div>
|
||||
<div style="color:#8888AA;font-size:11px;margin-bottom:8px;">
|
||||
Klein und schnell: <strong>nur</strong> die pinned Memories (Identität, Regeln, Präferenzen, Tools, Skills) als JSON.
|
||||
Use-Case: Wipe → Bootstrap-Import → ARIA hat Persönlichkeit zurück, sonst leer.
|
||||
Cold Memory (Konversations-Fakten) bleibt beim Import unangetastet.
|
||||
Getrennt nach <strong>scope</strong>: <span style="color:#3FFF3F;">System</span> = generische Regeln, die jeder braucht (Sicherheit, Ehrlichkeit, Skill-Regeln) — teilbar für ein frisches System.
|
||||
<span style="color:#FF9F0A;">Persönlich</span> = Stefan-spezifisch (Name, Zugangsdaten, Projekte) — bleibt privat.
|
||||
Import ersetzt nur die pinned Memories des jeweiligen scope; Cold Memory bleibt unangetastet.
|
||||
</div>
|
||||
<button class="btn secondary" onclick="exportBootstrap()" style="color:#FFD60A;border-color:#FFD60A;">⬇ Bootstrap exportieren (JSON)</button>
|
||||
<button class="btn secondary" onclick="exportBootstrap('system')" style="color:#3FFF3F;border-color:#3FFF3F;">⬇ System-Regeln exportieren</button>
|
||||
<button class="btn secondary" onclick="exportBootstrap('personal')" style="color:#FF9F0A;border-color:#FF9F0A;">⬇ Persönliches exportieren</button>
|
||||
<button class="btn secondary" onclick="exportBootstrap('all')" style="color:#FFD60A;border-color:#FFD60A;">⬇ Alles (Vollbackup)</button>
|
||||
<input type="file" id="bootstrap-import-file" accept=".json,application/json" style="display:none" onchange="importBootstrap(event)">
|
||||
<button class="btn secondary" onclick="document.getElementById('bootstrap-import-file').click()" style="color:#FFD60A;border-color:#FFD60A;">⬆ Bootstrap importieren</button>
|
||||
<div id="bootstrap-status" style="margin-top:8px;font-size:11px;color:#8888AA;"></div>
|
||||
@@ -1396,6 +1410,11 @@
|
||||
<input type="checkbox" id="memory-pinned">
|
||||
<span>📌 Pinned (Hot Memory — IMMER im System-Prompt)</span>
|
||||
</label>
|
||||
<label style="display:block;color:#8888AA;font-size:11px;margin-top:10px;margin-bottom:3px;">Scope (steuert Bootstrap-Export):</label>
|
||||
<select id="memory-scope" style="width:100%;background:#0D0D1A;color:#E0E0F0;border:1px solid #1E1E2E;padding:6px;border-radius:4px;font-family:inherit;margin-bottom:10px;">
|
||||
<option value="personal">🟠 Persönlich — Stefan-spezifisch, bleibt privat</option>
|
||||
<option value="system">🟢 System — generische Regel, teilbar für frisches System</option>
|
||||
</select>
|
||||
|
||||
<!-- Anhaenge — nur bei Edit (vorhandene ID) sichtbar -->
|
||||
<div id="memory-attachments-block" style="display:none;margin-top:14px;padding-top:10px;border-top:1px solid #1E1E2E;">
|
||||
@@ -1767,14 +1786,21 @@
|
||||
|
||||
if (msg.type === 'disk_cleanup') {
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
const dtext = document.getElementById('disk-banner-text');
|
||||
if (msg.status === 'running') {
|
||||
if (btn) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
} else if (msg.status === 'done') {
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Sicher aufraeumen'; }
|
||||
alert('Aufgeraeumt: ' + (msg.freed || '0 MB') + ' frei\n(' + (msg.steps || []).join(', ') + ')');
|
||||
// Feedback NICHT nur per alert() (koennte unterdrueckt sein) —
|
||||
// direkt am Button + im Banner sichtbar machen.
|
||||
if (btn) {
|
||||
btn.disabled = false;
|
||||
btn.textContent = '✓ ' + (msg.freed || '0 MB') + ' frei';
|
||||
setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000);
|
||||
}
|
||||
if (dtext) dtext.textContent = 'Aufgeraeumt: ' + (msg.freed || '0 MB') + ' frei (' + (msg.steps || []).join(', ') + '). Disk-Status aktualisiert sich gleich.';
|
||||
} else if (msg.status === 'error') {
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Sicher aufraeumen'; }
|
||||
alert('Aufraeumen fehlgeschlagen: ' + (msg.error || ''));
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Fehler — nochmal'; setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000); }
|
||||
if (dtext) dtext.textContent = 'Aufraeumen fehlgeschlagen: ' + (msg.error || '');
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1892,6 +1918,11 @@
|
||||
if (slider) slider.value = msg.voiceIdThreshold;
|
||||
if (display) display.textContent = Number(msg.voiceIdThreshold).toFixed(2);
|
||||
}
|
||||
// Speaker-ID Gating-Schalter wiederherstellen (Default aus)
|
||||
{
|
||||
const cb = document.getElementById('diag-voice-id-enabled');
|
||||
if (cb) cb.checked = !!msg.voiceIdEnabled;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -3593,13 +3624,14 @@
|
||||
const huggingfaceToken = document.getElementById('diag-flux-hf-token')?.value;
|
||||
const voiceIdThresholdRaw = document.getElementById('diag-voice-id-threshold')?.value;
|
||||
const voiceIdThreshold = voiceIdThresholdRaw ? parseFloat(voiceIdThresholdRaw) : undefined;
|
||||
const voiceIdEnabled = document.getElementById('diag-voice-id-enabled')?.checked;
|
||||
send({
|
||||
action: 'send_voice_config',
|
||||
ttsEnabled, xttsVoice, whisperModel,
|
||||
f5ttsModel, f5ttsCkptFile, f5ttsVocabFile,
|
||||
f5ttsCfgStrength, f5ttsNfeStep,
|
||||
fluxDefaultModel, fluxKeywordRaw, fluxKeywordSwitch, huggingfaceToken,
|
||||
voiceIdThreshold,
|
||||
voiceIdThreshold, voiceIdEnabled,
|
||||
});
|
||||
const statusEl = document.getElementById('voice-status');
|
||||
if (statusEl && xttsVoice) {
|
||||
@@ -5779,9 +5811,15 @@
|
||||
const typeBadge = withScore ? `<span style="color:#0096FF;font-size:10px;margin-right:6px;">${escapeHtml(BRAIN_TYPE_LABELS[m.type] || m.type)}</span>` : '';
|
||||
const attCount = Array.isArray(m.attachments) ? m.attachments.length : 0;
|
||||
const attBadge = attCount > 0 ? `<span style="color:#34C759;font-size:10px;margin-left:6px;" title="${attCount} Anhang${attCount === 1 ? '' : ' / Anhaenge'}">📎${attCount}</span>` : '';
|
||||
// scope-Badge nur bei pinned (nur die werden exportiert — da zaehlt die Trennung).
|
||||
const scopeBadge = m.pinned
|
||||
? (m.scope === 'system'
|
||||
? `<span style="color:#3FFF3F;font-size:9px;margin-left:6px;border:1px solid #3FFF3F;border-radius:3px;padding:0 3px;" title="System-Regel — kommt in den System-Export">SYS</span>`
|
||||
: `<span style="color:#FF9F0A;font-size:9px;margin-left:6px;border:1px solid #FF9F0A;border-radius:3px;padding:0 3px;" title="Persönlich — bleibt privat">PRIV</span>`)
|
||||
: '';
|
||||
return `<div style="padding:6px 0;border-bottom:1px solid #1E1E2E;display:flex;gap:6px;align-items:flex-start;">
|
||||
<div style="flex:1;min-width:0;cursor:pointer;" onclick="openMemoryModal('${m.id}')">
|
||||
<div style="color:#E0E0F0;font-size:12px;">${typeBadge}${pin}<strong>${escapeHtml(m.title || '(ohne Titel)')}</strong>${score}${attBadge}
|
||||
<div style="color:#E0E0F0;font-size:12px;">${typeBadge}${pin}<strong>${escapeHtml(m.title || '(ohne Titel)')}</strong>${score}${attBadge}${scopeBadge}
|
||||
${m.category ? `<span style="color:#555570;font-weight:normal;font-size:10px;margin-left:6px;">[${escapeHtml(m.category)}]</span>` : ''}
|
||||
</div>
|
||||
<div style="color:#888;font-size:11px;line-height:1.4;">${escapeHtml(preview)}${m.content && m.content.length > 140 ? '...' : ''}</div>
|
||||
@@ -5991,6 +6029,7 @@
|
||||
document.getElementById('memory-category').value = m.category || '';
|
||||
document.getElementById('memory-tags').value = (m.tags || []).join(', ');
|
||||
document.getElementById('memory-pinned').checked = !!m.pinned;
|
||||
document.getElementById('memory-scope').value = (m.scope === 'system') ? 'system' : 'personal';
|
||||
// Anhang-Block sichtbar — Liste rendern
|
||||
if (attBlock) attBlock.style.display = 'block';
|
||||
if (attHint) attHint.style.display = 'none';
|
||||
@@ -6004,6 +6043,7 @@
|
||||
document.getElementById('memory-category').value = '';
|
||||
document.getElementById('memory-tags').value = '';
|
||||
document.getElementById('memory-pinned').checked = false;
|
||||
document.getElementById('memory-scope').value = 'personal';
|
||||
// Bei neuem Memory: nur Hinweis, dass Anhaenge nach Save gehen
|
||||
if (attBlock) attBlock.style.display = 'none';
|
||||
if (attHint) attHint.style.display = 'block';
|
||||
@@ -6108,6 +6148,7 @@
|
||||
const category = document.getElementById('memory-category').value.trim();
|
||||
const tags = document.getElementById('memory-tags').value.split(',').map(t => t.trim()).filter(Boolean);
|
||||
const pinned = document.getElementById('memory-pinned').checked;
|
||||
const scope = document.getElementById('memory-scope').value || 'personal';
|
||||
|
||||
if (!title) { errEl.textContent = 'Titel fehlt.'; errEl.style.display = 'block'; return; }
|
||||
if (!content) { errEl.textContent = 'Inhalt fehlt.'; errEl.style.display = 'block'; return; }
|
||||
@@ -6118,13 +6159,13 @@
|
||||
r = await fetch('/api/brain/memory/update/' + encodeURIComponent(id), {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ title, content, pinned, category, tags }),
|
||||
body: JSON.stringify({ title, content, pinned, category, scope, tags }),
|
||||
});
|
||||
} else {
|
||||
r = await fetch('/api/brain/memory/save', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ type, title, content, pinned, category, tags, source: 'manual' }),
|
||||
body: JSON.stringify({ type, title, content, pinned, category, scope, tags, source: 'manual' }),
|
||||
});
|
||||
}
|
||||
if (!r.ok) {
|
||||
@@ -6501,11 +6542,12 @@
|
||||
}
|
||||
|
||||
// ── Bootstrap Export / Import ──────────────────────────
|
||||
async function exportBootstrap() {
|
||||
async function exportBootstrap(scope) {
|
||||
scope = scope || 'system';
|
||||
const status = document.getElementById('bootstrap-status');
|
||||
if (status) status.innerHTML = '⏳ Lade...';
|
||||
try {
|
||||
const r = await fetch('/api/brain/memory/export-bootstrap');
|
||||
const r = await fetch('/api/brain/memory/export-bootstrap?scope=' + encodeURIComponent(scope));
|
||||
if (!r.ok) throw new Error('HTTP ' + r.status);
|
||||
const data = await r.json();
|
||||
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
|
||||
@@ -6513,10 +6555,11 @@
|
||||
const ts = new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = `aria-bootstrap-${ts}.json`;
|
||||
a.download = `aria-bootstrap-${scope}-${ts}.json`;
|
||||
document.body.appendChild(a); a.click();
|
||||
setTimeout(() => { URL.revokeObjectURL(url); a.remove(); }, 100);
|
||||
if (status) status.innerHTML = `<span style="color:#3FFF3F;">✓ ${data.count} pinned Memories exportiert</span>`;
|
||||
const label = scope === 'system' ? 'System-Regeln' : (scope === 'personal' ? 'persönliche Memories' : 'pinned Memories');
|
||||
if (status) status.innerHTML = `<span style="color:#3FFF3F;">✓ ${data.count} ${label} exportiert</span>`;
|
||||
} catch (e) {
|
||||
if (status) status.innerHTML = `<span style="color:#FF6B6B;">✗ ${e.message}</span>`;
|
||||
}
|
||||
@@ -6530,7 +6573,11 @@
|
||||
const text = await file.text();
|
||||
const bundle = JSON.parse(text);
|
||||
if (!Array.isArray(bundle.memories)) throw new Error('Datei hat kein "memories"-Array');
|
||||
if (!confirm(`Bootstrap importieren?\n\n${bundle.memories.length} pinned Memories aus "${file.name}".\n\nALLE aktuell pinned Memories werden überschrieben. Cold Memory bleibt unverändert.`)) {
|
||||
const bScope = bundle.scope || 'all';
|
||||
const scopeInfo = bScope === 'system' ? 'Nur die aktuell pinned SYSTEM-Regeln werden ersetzt — Persönliches bleibt.'
|
||||
: bScope === 'personal' ? 'Nur die aktuell pinned PERSÖNLICHEN Memories werden ersetzt — System-Regeln bleiben.'
|
||||
: 'ALLE aktuell pinned Memories werden überschrieben.';
|
||||
if (!confirm(`Bootstrap importieren? (scope: ${bScope})\n\n${bundle.memories.length} pinned Memories aus "${file.name}".\n\n${scopeInfo} Cold Memory bleibt unverändert.`)) {
|
||||
event.target.value = '';
|
||||
return;
|
||||
}
|
||||
@@ -6776,12 +6823,17 @@
|
||||
|
||||
// Fuehrt das Aufraeumen WIRKLICH aus (Server-seitig via Docker-API), statt
|
||||
// nur den Befehl zu kopieren.
|
||||
// WICHTIG: "safe" laeuft OHNE confirm() — Build-Cache + ungenutzte Images
|
||||
// sind ungefaehrlich (keine Volumes/Daten). Frueher blockte ein confirm()
|
||||
// den Klick, wenn der Browser Dialoge unterdrueckt hatte ("Verhindern,
|
||||
// dass diese Seite weitere Dialoge erstellt") → confirm()===false → es ging
|
||||
// NICHTS raus. Nur "aggressive" (Volumes!) fragt noch nach.
|
||||
function runDiskCleanup(variant) {
|
||||
const aggressive = variant === 'aggressive';
|
||||
const q = aggressive
|
||||
? 'AGGRESSIV aufraeumen? Loescht zusaetzlich ungenutzte Volumes — nur wenn ALLE ARIA-Container laufen, sonst Datenverlust!'
|
||||
: 'Sicher aufraeumen? Loescht Build-Cache + ungenutzte Images (keine Volumes, keine Daten gehen verloren).';
|
||||
if (!confirm(q)) return;
|
||||
if (aggressive) {
|
||||
const ok = confirm('AGGRESSIV aufraeumen? Loescht zusaetzlich ungenutzte Volumes — nur wenn ALLE ARIA-Container laufen, sonst Datenverlust!');
|
||||
if (!ok) return;
|
||||
}
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
if (btn && !aggressive) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
send({ action: 'disk_cleanup', variant });
|
||||
|
||||
@@ -2681,6 +2681,12 @@ wss.on("connection", (ws) => {
|
||||
const t = parseFloat(msg.voiceIdThreshold);
|
||||
if (t >= 0.0 && t <= 1.0) voiceConfig.voiceIdThreshold = t;
|
||||
}
|
||||
// Speaker-ID Gating an/aus ("nur meine Stimme"). Default aus (fail-open) —
|
||||
// bewusster Schalter. voxtral/whisper-bridge lesen voiceIdEnabled aus dem
|
||||
// config-Broadcast; aus = gar keine Pruefung.
|
||||
if (msg.voiceIdEnabled !== undefined) {
|
||||
voiceConfig.voiceIdEnabled = !!msg.voiceIdEnabled;
|
||||
}
|
||||
try {
|
||||
fs.mkdirSync("/shared/config", { recursive: true });
|
||||
fs.writeFileSync("/shared/config/voice_config.json", JSON.stringify(voiceConfig, null, 2));
|
||||
|
||||
@@ -70,6 +70,10 @@ const ALLOWED_TYPES = new Set([
|
||||
// spiegelt ARIAs Datei-Writes, und QEMU-VNC wird als RFB-Bytes durch RVS
|
||||
// getunnelt (Base64-in-JSON wie audio_pcm — kein Binaer-Handling noetig).
|
||||
"code_file", "code_file_edit",
|
||||
// M1 Generatives Cockpit: ARIA komponiert via present_view eine View-Spec
|
||||
// (Orb + Karten), die App/Web/Diagnostic mit ihrem jeweiligen Renderer
|
||||
// materialisieren. Brain → Bridge → RVS → Clients.
|
||||
"aria_view",
|
||||
"check_desktop", "desktop_status",
|
||||
"vnc_open", "vnc_close", "vnc_data", "vnc_input",
|
||||
// Satelliten (Info-/Gateway-Aussenposten in fremden Netzen): melden sich mit
|
||||
|
||||
+35
-3
@@ -30,7 +30,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # TTS → GPU 0 (8 GB; F5 ist klein)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./voices:/voices # WAV + TXT Referenz
|
||||
@@ -63,12 +63,13 @@ services:
|
||||
whisper-bridge:
|
||||
build: ./whisper
|
||||
container_name: aria-whisper-bridge
|
||||
profiles: ["whisper"] # Fallback-STT — startet nur mit --profile whisper
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["1"] # STT/groesstes Modell → GPU 1 (12 GB; spaeter Voxtral-STT-3B ~9 GB)
|
||||
capabilities: [gpu]
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
@@ -108,7 +109,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # LLM → GPU 0 (8 GB, teilt sich mit TTS)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./models:/models # HF-Download-Cache (persistent)
|
||||
@@ -137,3 +138,34 @@ services:
|
||||
# Erster Load eines Modells kann ein GGUF ziehen (mehrere GB) — grosszuegig.
|
||||
- LLM_TIMEOUT_SEC=${LLM_TIMEOUT_SEC:-600}
|
||||
restart: unless-stopped
|
||||
|
||||
# ─── Voxtral STT-3B (Transformers, GPU) — DEFAULT-STT ─────────
|
||||
# Ersetzt whisper als STT. Laeuft auf Treiber 550/CUDA 12.4 (torch cu124, KEIN
|
||||
# Treiber-Upgrade noetig). Modell Voxtral-Mini-3B-2507 (~9 GB bf16) → GPU 1.
|
||||
# Startet bei jedem `docker compose up -d`. Whisper ist der opt-in Fallback
|
||||
# (Profil "whisper") — beide zusammen wuerden stt_* doppelt beantworten, also
|
||||
# immer nur EINEN STT laufen lassen.
|
||||
voxtral-bridge:
|
||||
build: ./voxtral
|
||||
container_name: aria-voxtral-bridge
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["1"] # 12-GB-Karte (STT-3B ~9 GB); GPU 0 (8 GB) bleibt fuer F5/LLM
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./hf-cache:/root/.cache/huggingface # gleicher Modell-Cache wie whisper/f5
|
||||
- ./voice-id:/voice-id # Speaker-Fingerprint (wie whisper)
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
- RVS_PORT=${RVS_PORT:-443}
|
||||
- RVS_TLS=${RVS_TLS:-true}
|
||||
- RVS_TLS_FALLBACK=${RVS_TLS_FALLBACK:-true}
|
||||
- RVS_TOKEN=${RVS_TOKEN}
|
||||
- VOXTRAL_MODEL=mistralai/Voxtral-Mini-3B-2507
|
||||
- VOXTRAL_LANGUAGE=${WHISPER_LANGUAGE:-de}
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} # falls das Modell HF-gated ist
|
||||
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True # weniger VRAM-Fragmentierung
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -9,12 +9,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# PyTorch CUDA-Wheels zuerst (f5-tts zieht sonst CPU-only Torch rein)
|
||||
RUN pip3 install --no-cache-dir torch==2.3.1 torchaudio==2.3.1 \
|
||||
--index-url https://download.pytorch.org/whl/cu121
|
||||
# torch FEST auf cu124 (Treiber 550 = CUDA 12.4). 2.6.0 ist der NEUESTE cu124-
|
||||
# Build — torch 2.7+ gibt es nur noch fuer cu126+, was 550 nicht unterstuetzt
|
||||
# ("NVIDIA driver too old, found 12040"). Der Constraint haelt f5-tts davon ab,
|
||||
# torch beim Dependency-Aufloesen wieder auf eine zu neue CUDA-Version zu ziehen.
|
||||
RUN pip3 install --no-cache-dir torch==2.6.0 torchaudio==2.6.0 \
|
||||
--index-url https://download.pytorch.org/whl/cu124
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip3 install --no-cache-dir -r requirements.txt
|
||||
RUN printf 'torch==2.6.0\ntorchaudio==2.6.0\n' > /tmp/torch-constraint.txt && \
|
||||
pip3 install --no-cache-dir -c /tmp/torch-constraint.txt -r requirements.txt
|
||||
|
||||
COPY bridge.py .
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# Voxtral-STT-3B Bridge (Transformers). Laeuft auf Treiber 550/CUDA 12.4 via
|
||||
# torch cu124 — KEIN Treiber-Upgrade noetig (gleicher Trick wie f5tts).
|
||||
# Modell: Voxtral-Mini-3B-2507 (~9 GB in bf16) → passt auf die 12-GB-Karte (GPU 1).
|
||||
FROM nvidia/cuda:12.2.2-cudnn8-runtime-ubuntu22.04
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3 python3-pip ffmpeg git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# torch FEST auf cu124 (Treiber 550 = CUDA 12.4). 2.6.0 ist der neueste cu124-Build;
|
||||
# torch 2.7+ gibt es nur fuer cu126+ und braeuchte einen neueren Treiber.
|
||||
RUN pip3 install --no-cache-dir torch==2.6.0 torchaudio==2.6.0 \
|
||||
--index-url https://download.pytorch.org/whl/cu124
|
||||
|
||||
COPY requirements.txt .
|
||||
# Constraint haelt transformers/mistral-common davon ab, torch wieder hochzuziehen.
|
||||
RUN printf 'torch==2.6.0\ntorchaudio==2.6.0\n' > /tmp/torch-constraint.txt && \
|
||||
pip3 install --no-cache-dir -c /tmp/torch-constraint.txt -r requirements.txt
|
||||
|
||||
COPY bridge.py speaker_id.py ./
|
||||
|
||||
CMD ["python3", "bridge.py"]
|
||||
@@ -0,0 +1,47 @@
|
||||
# Voxtral-STT-3B-Satellit (Transformers)
|
||||
|
||||
Streaming-STT via **Voxtral-Mini-3B-2507** (Mistral, Apache 2.0) über
|
||||
🤗 Transformers. Ersetzt whisper als STT — genauer, und **ohne Treiber-Upgrade**:
|
||||
läuft auf dem Trixie-Standardtreiber (550/CUDA 12.4) via **torch 2.6.0+cu124**
|
||||
(derselbe Trick wie bei f5tts).
|
||||
|
||||
- Modell ~9 GB (bf16) → **GPU 1** (die 12-GB-Karte; per Compose gepinnt).
|
||||
- **F5-TTS + LLM** bleiben auf GPU 0 (8 GB).
|
||||
- Arbeitsweise = chunked wie whisper: live PCM → alle ~1 s transkribieren
|
||||
(Partials) → **adaptiver Endpointer** (Rausch-Boden-VAD + semantische
|
||||
Stagnation, aus M0.1) feuert `stt_endpoint`. RVS-Protokoll identisch → drop-in.
|
||||
|
||||
## Starten (Profil `voxtral`)
|
||||
|
||||
```bash
|
||||
cd xtts
|
||||
docker compose stop whisper-bridge # sonst beantworten beide stt_*
|
||||
docker compose --profile voxtral up -d --build
|
||||
docker logs -f aria-voxtral-bridge # Modell laedt (mehrere GB), dann "RVS verbunden"
|
||||
```
|
||||
Zurück zu whisper: `docker compose --profile voxtral down && docker compose up -d whisper-bridge`.
|
||||
|
||||
## ⚠️ Auf echter Hardware verifizieren (blind gebaut)
|
||||
|
||||
1. **Transformers-API.** Die exakte Voxtral-Transkriptions-API ist in `bridge.py`
|
||||
in **einer** Methode gekapselt (`VoxtralRunner._transcribe_blocking`), modelliert
|
||||
nach der HF-Modelcard (`apply_transcription_request` → `generate` → `batch_decode`).
|
||||
Beim ersten Lauf gegen die Modelcard prüfen und dort anpassen.
|
||||
2. **HF-Gating.** Ist `Voxtral-Mini-3B-2507` gated, `HF_TOKEN` in `xtts/.env` setzen
|
||||
(wird als `HUGGING_FACE_HUB_TOKEN` durchgereicht).
|
||||
3. **VRAM/Tempo.** 3B in bf16 ~9 GB auf der 12-GB-Karte — Rest fürs KV-Cache. Ist die
|
||||
Partial-Transkription (alle 1 s) zu schwer, `STREAM_TRANSCRIBE_INTERVAL_MS` hochsetzen.
|
||||
4. **torch-Konflikt.** Falls `transformers`/`mistral-common` beim Build torch>2.6
|
||||
erzwingen, meldet der Constraint einen Konflikt → dann brauchen wir doch das
|
||||
Treiber-Upgrade (`bootstrap.sh --upgrade-driver` via NVIDIA-CUDA-Repo) + cu126-torch.
|
||||
|
||||
## Protokoll (RVS, identisch zu whisper — drop-in)
|
||||
Rein: `stt_stream_start`, `stt_audio_chunk` (16 kHz mono s16le, base64), `stt_stream_end`.
|
||||
Raus: `stt_partial`, `stt_endpoint`, `stt_stream_done`.
|
||||
|
||||
## TTS
|
||||
Bleibt **F5-TTS** (klingt gut, passt auf GPU 0). Voxtral-TTS bräuchte ~24 GB — separates Thema.
|
||||
|
||||
## Quellen
|
||||
- Modell: https://huggingface.co/mistralai/Voxtral-Mini-3B-2507
|
||||
- Transformers-Nutzung: HF-Modelcard (Voxtral) + `mistral-common`
|
||||
@@ -0,0 +1,641 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ARIA Voxtral-STT-3B Bridge (Transformers) — Ersatz fuer whisper.
|
||||
|
||||
Laeuft auf Treiber 550/CUDA 12.4 via torch cu124 (kein Treiber-Upgrade noetig).
|
||||
Modell: Voxtral-Mini-3B-2507 (bf16, ~9 GB) → GPU 1 (12 GB, per Compose gepinnt).
|
||||
|
||||
Arbeitsweise = Zwilling der whisper-Bridge: App schickt live PCM-Chunks; wir
|
||||
transkribieren alle ~STREAM_TRANSCRIBE_INTERVAL_MS auf dem Ringbuffer (Partials)
|
||||
und feuern stt_endpoint, sobald der ADAPTIVE Endpointer (Rausch-Boden-VAD +
|
||||
semantische Stagnation, aus M0.1) "fertig" sagt. RVS-Wire-Protokoll identisch zu
|
||||
whisper → drop-in (die App merkt nur bessere Genauigkeit).
|
||||
|
||||
⚠️ VERIFY-ON-FIRST-RUN: Die exakte Transformers-Transkriptions-API von Voxtral
|
||||
(apply_transcription_request / generate / decode) ist unten in EINER Methode
|
||||
(VoxtralRunner._transcribe_blocking) gekapselt und nach dem HF-Modelcard-Muster
|
||||
modelliert. Beim ersten echten Lauf gegen die Voxtral-Modelcard pruefen und dort
|
||||
anpassen. Alles andere (RVS, Endpointer) ist bewaehrt.
|
||||
|
||||
Env:
|
||||
RVS_HOST, RVS_PORT, RVS_TLS, RVS_TLS_FALLBACK, RVS_TOKEN
|
||||
VOXTRAL_MODEL Default: mistralai/Voxtral-Mini-3B-2507
|
||||
VOXTRAL_LANGUAGE Default: de
|
||||
VOXTRAL_DEVICE Default: cuda
|
||||
STREAM_TRANSCRIBE_INTERVAL_MS Default 1000 (3B ist schwerer als whisper-small)
|
||||
"""
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import websockets
|
||||
|
||||
import speaker_id # Speaker-ID (nur Stefans Stimme) — portiert aus der whisper-Bridge
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
datefmt="%H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("voxtral-bridge")
|
||||
|
||||
RVS_HOST = os.getenv("RVS_HOST", "").strip()
|
||||
RVS_PORT = int(os.getenv("RVS_PORT", "443"))
|
||||
RVS_TLS = os.getenv("RVS_TLS", "true").lower() == "true"
|
||||
RVS_TLS_FALLBACK = os.getenv("RVS_TLS_FALLBACK", "true").lower() == "true"
|
||||
RVS_TOKEN = os.getenv("RVS_TOKEN", "").strip()
|
||||
|
||||
VOXTRAL_MODEL = os.getenv("VOXTRAL_MODEL", "mistralai/Voxtral-Mini-3B-2507")
|
||||
VOXTRAL_LANGUAGE = os.getenv("VOXTRAL_LANGUAGE", "de")
|
||||
VOXTRAL_DEVICE = os.getenv("VOXTRAL_DEVICE", "cuda")
|
||||
|
||||
STREAM_TRANSCRIBE_INTERVAL_MS = int(os.getenv("STREAM_TRANSCRIBE_INTERVAL_MS", "1000"))
|
||||
STREAM_DEFAULT_ENDPOINT_MS = 2400
|
||||
STREAM_DEFAULT_HARD_CAP_MS = 300000
|
||||
STREAM_MIN_AUDIO_MS = 600
|
||||
STREAM_SPEAKER_CHECK_MS = 1500 # ab so viel Audio einmalig Speaker-ID pruefen
|
||||
STREAM_SESSION_TTL_S = 120
|
||||
STREAM_ENERGY_WINDOW_MS = 300
|
||||
STREAM_SEMANTIC_BACKUP_FACTOR = 2.0
|
||||
# Adaptiver Voice-Schwellwert (M0.1): relativ zum gemessenen Rausch-Boden.
|
||||
STREAM_VOICE_FACTOR = 2.5
|
||||
STREAM_VOICE_RMS_MIN = 0.005
|
||||
STREAM_VOICE_RMS_MAX = 0.020
|
||||
# Mindest-Stimme (in ~200ms-Endpointer-Frames), ab der eine Aufnahme ueberhaupt
|
||||
# als Sprache gilt. Darunter = Stille / kurzer Geraeusch-Blip → KEIN Transkript
|
||||
# (Voxtral halluziniert aus Fast-Nichts sonst einen Fuellsatz). 2 ≈ 400ms.
|
||||
STREAM_MIN_VOICED_FRAMES = int(os.getenv("STREAM_MIN_VOICED_FRAMES", "2"))
|
||||
|
||||
# Halluzinations-Filter (2. Netz NACH der Transkription). Der voiced_frames-Guard
|
||||
# oben faengt die reine Stille; hier kommt das "borderline"-Band dazu: wenn wenig
|
||||
# echte Stimme da war UND das Transkript ein bekanntes Voxtral-Silence-Artefakt
|
||||
# ist (Untertitel-Credits, Staedte-/Geo-Fakten "Flaeche von X km2"), ist es fast
|
||||
# sicher ein Phantom aus Fast-Nichts → verwerfen. Gegated auf wenig voiced_frames,
|
||||
# damit eine ECHTE Geografie-Frage (die hat normale Stimm-Energie) durchgeht.
|
||||
STREAM_HALLUC_GUARD_FRAMES = int(os.getenv("STREAM_HALLUC_GUARD_FRAMES",
|
||||
str(STREAM_MIN_VOICED_FRAMES * 4))) # ~1.6s
|
||||
_HALLUCINATION_RE = re.compile(
|
||||
r"untertitel"
|
||||
r"|amara\.org"
|
||||
r"|vielen\s+dank\s+f[uü]r'?s?\s+(zuschauen|zusehen|zuh[oö]ren)"
|
||||
r"|bis\s+zum\s+n[aä]chsten\s+mal"
|
||||
r"|abonnier"
|
||||
r"|fl[aä]che\s+von\s+[\d.,]+\s*(km|quadratkilometer)"
|
||||
r"|[\d.,]+\s*(km²|quadratkilometern?|einwohnern?)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Speaker-ID Gating global an/aus. DEFAULT AUS (fail-open) — die "nur meine Stimme"-
|
||||
# Pruefung ist ein BEWUSSTER Schalter, kein Automatismus: ein einziger schlechter
|
||||
# Enroll darf nie die ganze STT lahmlegen (genau das ist passiert). Wird per config-
|
||||
# Broadcast (voiceIdEnabled, aus dem Diagnostic) zur Laufzeit gesetzt. Kann per ENV
|
||||
# vorbelegt werden.
|
||||
SPEAKER_ID_ENABLED = os.getenv("VOICE_ID_ENABLED", "false").lower() in ("1", "true", "yes")
|
||||
|
||||
|
||||
def _set_speaker_id_enabled(val: bool) -> None:
|
||||
global SPEAKER_ID_ENABLED
|
||||
SPEAKER_ID_ENABLED = bool(val)
|
||||
|
||||
|
||||
def pcm_s16le_to_float32(data: bytes) -> np.ndarray:
|
||||
if not data:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
return np.frombuffer(data, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
async def _send(ws, mtype: str, payload: dict) -> None:
|
||||
try:
|
||||
await ws.send(json.dumps({
|
||||
"type": mtype, "payload": payload, "timestamp": int(time.time() * 1000),
|
||||
}))
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Send fehlgeschlagen (%s): %s", mtype, e)
|
||||
|
||||
|
||||
class VoxtralRunner:
|
||||
"""Haelt das Voxtral-Modell (Transformers). transcribe() blockiert → aus dem
|
||||
Event-Loop via run_in_executor aufrufen. Ein Lock serialisiert GPU-Zugriffe."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def load(self) -> None:
|
||||
import torch
|
||||
from transformers import AutoProcessor, VoxtralForConditionalGeneration
|
||||
t0 = time.time()
|
||||
logger.info("Lade Voxtral '%s' (device=%s, bf16)…", VOXTRAL_MODEL, VOXTRAL_DEVICE)
|
||||
self.processor = AutoProcessor.from_pretrained(VOXTRAL_MODEL)
|
||||
self.model = VoxtralForConditionalGeneration.from_pretrained(
|
||||
VOXTRAL_MODEL, torch_dtype=torch.bfloat16, device_map=VOXTRAL_DEVICE,
|
||||
)
|
||||
logger.info("Voxtral geladen in %.1fs", time.time() - t0)
|
||||
|
||||
def _transcribe_blocking(self, audio_f32: np.ndarray, language: str) -> str:
|
||||
import torch
|
||||
proc, model = self.processor, self.model
|
||||
if proc is None or model is None or audio_f32.size == 0:
|
||||
return ""
|
||||
# VoxtralProcessor verlangt bei rohen Arrays ein 'format'. Robuster:
|
||||
# in ein temp-WAV schreiben und den PFAD uebergeben — der Processor liest
|
||||
# Format + Samplerate selbst, kein 'format'-Argument noetig.
|
||||
wav_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tf:
|
||||
wav_path = tf.name
|
||||
sf.write(wav_path, audio_f32, 16000, subtype="PCM_16")
|
||||
inputs = proc.apply_transcription_request(
|
||||
language=language, audio=wav_path, model_id=VOXTRAL_MODEL,
|
||||
)
|
||||
inputs = inputs.to(VOXTRAL_DEVICE, dtype=torch.bfloat16)
|
||||
with torch.no_grad():
|
||||
# hoch genug fuer lange Diktate (stoppt eh am EOS); 512 hat
|
||||
# mehrminutige Aufnahmen abgeschnitten.
|
||||
outputs = model.generate(**inputs, max_new_tokens=4096)
|
||||
trimmed = outputs[:, inputs.input_ids.shape[1]:]
|
||||
text = proc.batch_decode(trimmed, skip_special_tokens=True)
|
||||
return (text[0] if text else "").strip()
|
||||
finally:
|
||||
if wav_path:
|
||||
try:
|
||||
os.unlink(wav_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def transcribe(self, audio_f32: np.ndarray, language: str) -> str:
|
||||
loop = asyncio.get_running_loop()
|
||||
async with self._lock:
|
||||
return await loop.run_in_executor(None, self._transcribe_blocking, audio_f32, language)
|
||||
|
||||
|
||||
@dataclass
|
||||
class StreamSession:
|
||||
request_id: str
|
||||
audio_request_id: str
|
||||
language: str
|
||||
endpoint_ms: int
|
||||
hard_cap_ms: int
|
||||
voice: str = ""
|
||||
speed: float = 1.0
|
||||
interrupted: bool = False
|
||||
location: Optional[dict] = None
|
||||
sample_rate: int = 16000
|
||||
voice_factor: float = STREAM_VOICE_FACTOR
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
pcm_buffer: bytearray = field(default_factory=bytearray)
|
||||
started_at: float = field(default_factory=time.time)
|
||||
last_chunk_at: float = field(default_factory=time.time)
|
||||
last_partial: str = ""
|
||||
last_growth_at: float = 0.0
|
||||
last_transcribe_at: float = 0.0
|
||||
last_voice_at: float = 0.0
|
||||
noise_floor: float = 0.0
|
||||
closed: bool = False
|
||||
endpoint_sent: bool = False
|
||||
# Einmaliges "Sprache erkannt"-Signal an die App gesendet? Voxtral schickt
|
||||
# keine Live-Partials, aber der App-No-Speech-Watchdog wartet auf ein
|
||||
# stt_partial, um "der User redet" zu erkennen — sonst cancelt er mitten im
|
||||
# Satz. Wir feuern EIN leeres stt_partial beim ersten Voice-Frame.
|
||||
speech_signaled: bool = False
|
||||
# Anzahl Endpointer-Frames (~200ms) mit echter Stimme. Gate gegen Halluzination
|
||||
# aus Stille/Blips: unter STREAM_MIN_VOICED_FRAMES wird nicht transkribiert.
|
||||
voiced_frames: int = 0
|
||||
# Speaker-ID Gating (einmalig auf die ersten ~1.5s der Aufnahme)
|
||||
speaker_checked: bool = False
|
||||
speaker_match: Optional[bool] = None
|
||||
speaker_similarity: float = 0.0
|
||||
|
||||
|
||||
class SessionManager:
|
||||
def __init__(self, runner: VoxtralRunner) -> None:
|
||||
self.runner = runner
|
||||
self._sessions: dict[str, StreamSession] = {}
|
||||
self._ws = None
|
||||
|
||||
def attach_ws(self, ws) -> None:
|
||||
self._ws = ws
|
||||
|
||||
def start_session(self, payload: dict) -> None:
|
||||
rid = (payload.get("requestId") or "").strip()
|
||||
if not rid:
|
||||
return
|
||||
try:
|
||||
endpoint_ms = int(payload.get("endpointMs") or STREAM_DEFAULT_ENDPOINT_MS)
|
||||
except (TypeError, ValueError):
|
||||
endpoint_ms = STREAM_DEFAULT_ENDPOINT_MS
|
||||
try:
|
||||
hard_cap_ms = int(payload.get("hardCapMs") or STREAM_DEFAULT_HARD_CAP_MS)
|
||||
except (TypeError, ValueError):
|
||||
hard_cap_ms = STREAM_DEFAULT_HARD_CAP_MS
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
self._sessions[rid] = StreamSession(
|
||||
request_id=rid,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
language=payload.get("language") or VOXTRAL_LANGUAGE,
|
||||
endpoint_ms=endpoint_ms,
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=float(payload.get("speed") or 1.0),
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
)
|
||||
logger.info("Voxtral-Session offen: id=%s lang=%s endpointMs=%d",
|
||||
rid[:8], self._sessions[rid].language, endpoint_ms)
|
||||
|
||||
def feed_chunk(self, payload: dict) -> bool:
|
||||
sess = self._sessions.get(payload.get("requestId", ""))
|
||||
if sess is None or sess.closed:
|
||||
return False
|
||||
pcm_b64 = payload.get("pcm", "")
|
||||
if pcm_b64:
|
||||
try:
|
||||
sess.pcm_buffer.extend(base64.b64decode(pcm_b64))
|
||||
except Exception:
|
||||
pass
|
||||
sess.last_chunk_at = time.time()
|
||||
return True
|
||||
|
||||
def end_session(self, request_id: str) -> None:
|
||||
sess = self._sessions.get(request_id)
|
||||
if sess is not None:
|
||||
sess.closed = True
|
||||
|
||||
def drop(self, request_id: str) -> None:
|
||||
self._sessions.pop(request_id, None)
|
||||
|
||||
# ── Endpointer (adaptiv, M0.1) ──
|
||||
def _buffer_ms(self, sess: StreamSession) -> float:
|
||||
samples = len(sess.pcm_buffer) // 2
|
||||
return (samples / sess.sample_rate) * 1000.0 if samples else 0.0
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
win = int(sess.sample_rate * STREAM_ENERGY_WINDOW_MS / 1000) * 2
|
||||
if win <= 0:
|
||||
return 0.0
|
||||
tail = sess.pcm_buffer[-win:]
|
||||
if len(tail) < 2:
|
||||
return 0.0
|
||||
arr = pcm_s16le_to_float32(bytes(tail))
|
||||
return float(np.sqrt(np.mean(arr * arr))) if arr.size else 0.0
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
async def _check_speaker(self, sess: StreamSession) -> None:
|
||||
"""Einmalig: erste ~1.5s → Embedding → Vergleich mit Fingerprint.
|
||||
Ohne Fingerprint fail-open (match=True). Bei Mismatch: Session beenden."""
|
||||
sess.speaker_checked = True
|
||||
# Schalter aus (Default) → gar keine Pruefung, alles durchlassen.
|
||||
if not SPEAKER_ID_ENABLED:
|
||||
sess.speaker_match = True
|
||||
return
|
||||
head = bytes(sess.pcm_buffer[: STREAM_SPEAKER_CHECK_MS * 32])
|
||||
if len(head) < speaker_id.MIN_SAMPLE_BYTES:
|
||||
sess.speaker_match = True
|
||||
return
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
is_match, sim = await loop.run_in_executor(None, speaker_id.verify, head)
|
||||
except Exception as exc:
|
||||
logger.warning("Stream %s: speaker-check crashed (%s) — fail-open",
|
||||
sess.request_id[:8], exc)
|
||||
sess.speaker_match = True
|
||||
return
|
||||
sess.speaker_match = is_match
|
||||
sess.speaker_similarity = sim
|
||||
logger.info("Stream %s: speaker-check sim=%.2f → %s (thr=%.2f)",
|
||||
sess.request_id[:8], sim, "MATCH" if is_match else "REJECT",
|
||||
speaker_id.DEFAULT_THRESHOLD)
|
||||
if not is_match:
|
||||
await self._finalize_speaker_mismatch(sess, sim)
|
||||
|
||||
async def _finalize_speaker_mismatch(self, sess: StreamSession, similarity: float) -> None:
|
||||
"""Fremde Stimme: synthetisches leeres stt_endpoint (reason=speaker_mismatch),
|
||||
Session droppen — kein Voxtral-Transcribe, kein Brain-Call."""
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
sess.endpoint_sent = True
|
||||
duration_s = self._buffer_ms(sess) / 1000.0
|
||||
logger.info("Stream %s: speaker-mismatch (sim=%.2f) — DROP nach %.1fs",
|
||||
sess.request_id[:8], similarity, duration_s)
|
||||
if self._ws is not None:
|
||||
payload = {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": "speaker_mismatch",
|
||||
"durationS": duration_s, "sttMs": 0,
|
||||
"voice": sess.voice, "speed": sess.speed,
|
||||
"interrupted": sess.interrupted,
|
||||
"speakerSimilarity": float(similarity),
|
||||
}
|
||||
if sess.location:
|
||||
payload["location"] = sess.location
|
||||
await _send(self._ws, "stt_endpoint", payload)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": "speaker_mismatch",
|
||||
})
|
||||
self.drop(sess.request_id)
|
||||
|
||||
async def run_endpointer(self) -> None:
|
||||
logger.info("Voxtral-Endpointer gestartet (adaptiver VAD, interval=%dms)",
|
||||
STREAM_TRANSCRIBE_INTERVAL_MS)
|
||||
while True:
|
||||
await asyncio.sleep(0.2)
|
||||
now = time.time()
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
try:
|
||||
await self._tick(sess, now)
|
||||
except Exception:
|
||||
logger.exception("Tick crashed (session=%s)", sid[:8])
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
if now - sess.last_chunk_at > STREAM_SESSION_TTL_S:
|
||||
logger.info("Stream %s: TTL — drop", sid[:8])
|
||||
self.drop(sid)
|
||||
|
||||
async def _tick(self, sess: StreamSession, now: float) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
if (now - sess.started_at) * 1000.0 > sess.hard_cap_ms and not sess.closed:
|
||||
await self._finalize(sess, "hardcap")
|
||||
return
|
||||
if sess.closed:
|
||||
await self._finalize(sess, "stream_end")
|
||||
return
|
||||
if self._buffer_ms(sess) < STREAM_MIN_AUDIO_MS:
|
||||
return
|
||||
# Speaker-ID einmalig: ist es Stefans Stimme? Fremde → Session verwerfen
|
||||
# (kein Transcribe, kein Brain-Call). Ohne Enrollment fail-open.
|
||||
if not sess.speaker_checked and self._buffer_ms(sess) >= STREAM_SPEAKER_CHECK_MS:
|
||||
await self._check_speaker(sess)
|
||||
if sess.speaker_match is False:
|
||||
return
|
||||
# Adaptive akustische Sprach-Aktivitaet (M0.1). KEINE Live-Partials mehr:
|
||||
# Voxtral-3B transkribiert den ganzen WACHSENDEN Buffer und braucht dafuer
|
||||
# bei langen Aufnahmen 5-6 s — zu langsam fuer Live-Text, UND diese Latenz
|
||||
# hat den semantischen Endpoint faelschlich ausgeloest (Partial-Latenz >
|
||||
# Timeout → willkuerliche Abbrueche nach 20-40 s). Deshalb: Turn-Ende rein
|
||||
# AKUSTISCH, transkribiert wird nur EINMAL im _finalize.
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
sess.voiced_frames += 1
|
||||
# Einmalig der App melden, dass Sprache begonnen hat — aber ERST ab genug
|
||||
# echter Stimme (>= STREAM_MIN_VOICED_FRAMES). Ein einzelner Geraeusch-
|
||||
# Blip darf den No-Speech-Watchdog NICHT loeschen, sonst transkribiert
|
||||
# Voxtral das Fast-Nichts und HALLUZINIERT einen Phantom-Satz. Ohne Live-
|
||||
# Partials wuerde der Watchdog die Aufnahme sonst am Konversationsfenster
|
||||
# canceln, obwohl der User redet ("beendet nach ~4s"-Repro). Leeres
|
||||
# stt_partial: App setzt streamGotPartial=true + loescht den Watchdog.
|
||||
# Nach der Speaker-ID-Pruefung (oben) → fremde Stimmen signalisieren NICHT.
|
||||
if (not sess.speech_signaled and self._ws is not None
|
||||
and sess.voiced_frames >= STREAM_MIN_VOICED_FRAMES):
|
||||
sess.speech_signaled = True
|
||||
await _send(self._ws, "stt_partial", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "",
|
||||
})
|
||||
else:
|
||||
self._update_noise_floor(sess, rms)
|
||||
# Endpoint: hat der User schon gesprochen UND ist es seit endpoint_ms still?
|
||||
if sess.last_voice_at > 0 and (now - sess.last_voice_at) * 1000.0 >= sess.endpoint_ms:
|
||||
await self._finalize(sess, "endpoint")
|
||||
|
||||
async def _finalize(self, sess: StreamSession, reason: str) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
sess.endpoint_sent = True
|
||||
# Halluzinations-Guard: zu wenig echte Stimme (Stille / kurzer Blip im
|
||||
# Passiv-/Wake-Fenster) → NICHT transkribieren. Voxtral (wie Whisper) baut
|
||||
# aus Fast-Nichts gern einen Fuellsatz ("Die Stadt hat eine Flaeche von
|
||||
# 1,5 km2"), der dann als PHANTOM-Nachricht ans Brain geht und das Gespraech
|
||||
# entgleisen laesst (Stefans Repro: "kam Nachricht von mir, obwohl ich
|
||||
# nichts sagte"). Leeres Endpoint = no-speech → App re-armt still.
|
||||
#
|
||||
# WICHTIG (aus dem ai-box-Log gelernt): die Phantome kommen mit
|
||||
# reason=stream_end — Passiv-/Wake-Fenster enden AUCH per stream_end, wenn
|
||||
# sie auf Stille zumachen. stream_end ist also NICHT gleich "manueller Stop".
|
||||
# Deshalb greift der Guard jetzt auch bei stream_end, aber mit niedrigerer
|
||||
# Schwelle (voiced==0 = gar keine Stimme), damit ein kurzes bewusstes Wort
|
||||
# ('ja', 'stopp') am Aufnahme-Button noch durchgeht, echte Stille aber nicht.
|
||||
_min_voiced = STREAM_MIN_VOICED_FRAMES if reason != "stream_end" else 1
|
||||
if sess.voiced_frames < _min_voiced:
|
||||
logger.info("Stream %s: no-speech (voiced_frames=%d<%d, reason=%s) — leeres Endpoint",
|
||||
sess.request_id[:8], sess.voiced_frames, _min_voiced, reason)
|
||||
if self._ws is not None:
|
||||
nospeech = {"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": f"no_speech:{reason}",
|
||||
"durationS": 0.0, "sttMs": 0}
|
||||
await _send(self._ws, "stt_endpoint", nospeech)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": f"no_speech:{reason}"})
|
||||
self.drop(sess.request_id)
|
||||
return
|
||||
audio = pcm_s16le_to_float32(bytes(sess.pcm_buffer))
|
||||
t0 = time.time()
|
||||
try:
|
||||
final_text = (await self.runner.transcribe(audio, sess.language)).strip()
|
||||
except Exception:
|
||||
logger.exception("Stream %s: Final-Transcribe crashed", sess.request_id[:8])
|
||||
final_text = sess.last_partial
|
||||
stt_ms = int((time.time() - t0) * 1000)
|
||||
duration_s = audio.size / 16000.0
|
||||
logger.info("Stream %s: FINAL (reason=%s, %.1fs, %dms): %r",
|
||||
sess.request_id[:8], reason, duration_s, stt_ms, final_text[:120])
|
||||
|
||||
# Halluzinations-Filter (2. Netz): leeres/Artefakt-Transkript im borderline-
|
||||
# Band → als no-speech verwerfen statt ein Phantom ("Die Stadt hat eine
|
||||
# Flaeche von 1,5 km2") ans Brain zu schicken. Gilt fuer ALLE reasons inkl.
|
||||
# stream_end (dort kamen die realen Phantome!) — aber das borderline-Band
|
||||
# (wenig voiced_frames) schuetzt echte, klar gesprochene Eingaben: eine echte
|
||||
# Geografie-FRAGE hat normale Stimm-Energie (voiced_frames >> Schwelle) und
|
||||
# geht durch; das Phantom aus Stille hat ~0 und wird verworfen. Ein leeres
|
||||
# Transkript wird immer verworfen (nichts gesagt = nichts senden).
|
||||
_clean = final_text.strip(" .,!?…-\t\n\r")
|
||||
_borderline = sess.voiced_frames < STREAM_HALLUC_GUARD_FRAMES
|
||||
_is_phantom = (not _clean) or (_borderline and bool(_HALLUCINATION_RE.search(final_text)))
|
||||
if _is_phantom:
|
||||
logger.info("Stream %s: Halluzination verworfen (voiced_frames=%d<%d, %.1fs, text=%r)",
|
||||
sess.request_id[:8], sess.voiced_frames, STREAM_HALLUC_GUARD_FRAMES,
|
||||
duration_s, final_text[:80])
|
||||
if self._ws is not None:
|
||||
nospeech = {"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": f"hallucination:{reason}",
|
||||
"durationS": 0.0, "sttMs": stt_ms}
|
||||
await _send(self._ws, "stt_endpoint", nospeech)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": "", "reason": f"hallucination:{reason}"})
|
||||
self.drop(sess.request_id)
|
||||
return
|
||||
|
||||
if self._ws is not None:
|
||||
payload = {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
"durationS": duration_s,
|
||||
"sttMs": stt_ms,
|
||||
"voice": sess.voice,
|
||||
"speed": sess.speed,
|
||||
"interrupted": sess.interrupted,
|
||||
}
|
||||
if sess.location:
|
||||
payload["location"] = sess.location
|
||||
await _send(self._ws, "stt_endpoint", payload)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
})
|
||||
self.drop(sess.request_id)
|
||||
|
||||
|
||||
async def _broadcast_status(ws, state: str, **extra) -> None:
|
||||
payload = {"service": "voxtral", "state": state}
|
||||
payload.update(extra)
|
||||
await _send(ws, "service_status", payload)
|
||||
|
||||
|
||||
async def run_loop(sessions: SessionManager) -> None:
|
||||
use_tls = RVS_TLS
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
while True:
|
||||
scheme = "wss" if use_tls else "ws"
|
||||
url = f"{scheme}://{RVS_HOST}:{RVS_PORT}/ws?token={RVS_TOKEN}"
|
||||
masked = url.replace(RVS_TOKEN, "***") if RVS_TOKEN else url
|
||||
try:
|
||||
logger.info("Verbinde zu RVS: %s", masked)
|
||||
async with websockets.connect(url, ping_interval=20, ping_timeout=10,
|
||||
max_size=50 * 1024 * 1024) as ws:
|
||||
logger.info("RVS verbunden")
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
sessions.attach_ws(ws)
|
||||
await _broadcast_status(ws, "ready", model=VOXTRAL_MODEL)
|
||||
await _send(ws, "config_request", {"service": "voxtral"})
|
||||
async for raw in ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
mtype = msg.get("type", "")
|
||||
payload = msg.get("payload", {}) or {}
|
||||
if mtype == "stt_stream_start":
|
||||
sessions.start_session(payload)
|
||||
elif mtype == "stt_audio_chunk":
|
||||
sessions.feed_chunk(payload)
|
||||
elif mtype == "stt_stream_end":
|
||||
sessions.end_session(payload.get("requestId", ""))
|
||||
elif mtype == "voice_id_status_request":
|
||||
req_id = payload.get("requestId", "")
|
||||
try:
|
||||
status = speaker_id.status()
|
||||
await _send(ws, "voice_id_status_response",
|
||||
{"requestId": req_id, "ok": True, **status})
|
||||
except Exception as exc:
|
||||
await _send(ws, "voice_id_status_response",
|
||||
{"requestId": req_id, "ok": False, "error": str(exc)[:200]})
|
||||
elif mtype == "voice_id_enroll_request":
|
||||
req_id = payload.get("requestId", "")
|
||||
samples = payload.get("samples") or []
|
||||
logger.info("voice_id_enroll_request: %d Samples (id=%s)", len(samples), req_id[:8])
|
||||
try:
|
||||
result = await asyncio.get_running_loop().run_in_executor(
|
||||
None, speaker_id.enroll_from_samples, samples)
|
||||
await _send(ws, "voice_id_enroll_response", {
|
||||
"requestId": req_id, "ok": True,
|
||||
"sample_count": result.get("sample_count", 0),
|
||||
"rejected": result.get("rejected", []),
|
||||
"updated_at": result.get("updated_at"),
|
||||
"embedding_dim": result.get("embedding_dim"),
|
||||
})
|
||||
except Exception as exc:
|
||||
logger.warning("voice_id_enroll failed: %s", exc)
|
||||
await _send(ws, "voice_id_enroll_response",
|
||||
{"requestId": req_id, "ok": False, "error": str(exc)[:300]})
|
||||
elif mtype == "voice_id_delete_request":
|
||||
req_id = payload.get("requestId", "")
|
||||
removed = speaker_id.delete_fingerprint()
|
||||
await _send(ws, "voice_id_delete_response",
|
||||
{"requestId": req_id, "ok": True, "removed": removed})
|
||||
elif mtype == "config":
|
||||
if "voiceIdThreshold" in payload:
|
||||
try:
|
||||
t = float(payload.get("voiceIdThreshold", 0.5))
|
||||
if 0.0 <= t <= 1.0:
|
||||
speaker_id.DEFAULT_THRESHOLD = t
|
||||
logger.info("[speaker-id] threshold gesetzt: %.2f", t)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if "voiceIdEnabled" in payload:
|
||||
_set_speaker_id_enabled(payload.get("voiceIdEnabled"))
|
||||
logger.info("[speaker-id] Gating %s (voiceIdEnabled)",
|
||||
"AN" if SPEAKER_ID_ENABLED else "AUS")
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Verbindung verloren: %s — retry in %ds", e, retry_s)
|
||||
if use_tls and RVS_TLS_FALLBACK and not tls_fallback_tried:
|
||||
use_tls = False
|
||||
tls_fallback_tried = True
|
||||
continue
|
||||
await asyncio.sleep(retry_s)
|
||||
retry_s = min(retry_s * 2, 30)
|
||||
use_tls = RVS_TLS
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
if not RVS_HOST or not RVS_TOKEN:
|
||||
logger.error("RVS_HOST/RVS_TOKEN fehlen — .env pruefen. Abbruch.")
|
||||
return
|
||||
runner = VoxtralRunner()
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(None, runner.load) # Modell laden (blockierend)
|
||||
sessions = SessionManager(runner)
|
||||
logger.info("Voxtral-Bridge startet — Modell=%s", VOXTRAL_MODEL)
|
||||
await asyncio.gather(run_loop(sessions), sessions.run_endpointer())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
@@ -0,0 +1,10 @@
|
||||
# Voxtral-3B via Transformers. torch/torchaudio kommen cu124-gepinnt aus dem
|
||||
# Dockerfile (nicht hier, sonst zieht pip das Default-CUDA-Wheel).
|
||||
transformers>=4.54
|
||||
mistral-common[audio]>=1.8.1
|
||||
accelerate>=0.30
|
||||
speechbrain>=1.0 # Speaker-ID (ECAPA-TDNN) — nur Stefans Stimme
|
||||
soundfile>=0.12
|
||||
librosa>=0.10 # VoxtralProcessor.load_audio_as nutzt librosa zum WAV-Laden
|
||||
numpy>=1.24
|
||||
websockets>=12.0
|
||||
@@ -0,0 +1,272 @@
|
||||
"""
|
||||
Speaker-ID Backend fuer ARIAs Stimmen-Erkennung.
|
||||
|
||||
Nutzt SpeechBrain ECAPA-TDNN (192-dim Embeddings, auf VoxCeleb-1+2 trainiert).
|
||||
Fingerprint = gemittelter, L2-normalisierter Embedding-Vektor aus N
|
||||
Enrollment-Samples. Verify: cosine_similarity(neue_aufnahme, fingerprint).
|
||||
|
||||
Persistenz: /voice-id/fingerprint.json (Float-Liste + Metadaten).
|
||||
Modell-Cache: /root/.cache/huggingface/ (Bind-Mount mit f5tts geteilt).
|
||||
|
||||
Verhalten OHNE Enrollment (kein Fingerprint vorhanden):
|
||||
verify() → (True, 0.0) — Fail-open, damit Speaker-ID-Gating den
|
||||
ungeenrollten Brain-Pfad nicht versehentlich blockiert.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
VOICE_ID_DIR = Path(os.environ.get("VOICE_ID_DIR", "/voice-id"))
|
||||
FINGERPRINT_FILE = VOICE_ID_DIR / "fingerprint.json"
|
||||
|
||||
# Cosine-Threshold: 0.5 ist konservativ (wenig false-positives), 0.3 ist
|
||||
# locker (mehr Treffer auch bei Nebengeraeuschen). Stefan kann's per
|
||||
# Diagnostic-Setting feintunen.
|
||||
DEFAULT_THRESHOLD = 0.5
|
||||
|
||||
# Minimal-Sample-Laenge fuer ein verlaessliches Embedding (~1s @ 16kHz int16 = 32000 bytes)
|
||||
MIN_SAMPLE_BYTES = 32000
|
||||
|
||||
_model = None
|
||||
|
||||
|
||||
def _ensure_loaded():
|
||||
"""Lazy-Load des ECAPA-TDNN. Holt das Modell beim ersten Aufruf von HF;
|
||||
danach cached im HF-Cache-Volume. Erste Init: ~30s download + load,
|
||||
danach <1s warm. Wirft bei Fehler — Caller muss catchen + fail-open."""
|
||||
global _model
|
||||
if _model is not None:
|
||||
return _model
|
||||
import torch
|
||||
from speechbrain.inference.speaker import EncoderClassifier
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
logger.info("[speaker-id] loading ECAPA-TDNN on %s ...", device)
|
||||
_model = EncoderClassifier.from_hparams(
|
||||
source="speechbrain/spkrec-ecapa-voxceleb",
|
||||
savedir="/root/.cache/huggingface/speechbrain-ecapa",
|
||||
run_opts={"device": device},
|
||||
)
|
||||
logger.info("[speaker-id] model ready (device=%s)", device)
|
||||
return _model
|
||||
|
||||
|
||||
def _decode_compressed_to_pcm(audio_bytes: bytes) -> bytes:
|
||||
"""Dekodiert komprimiertes Audio (MP4/M4A/AAC vom Android-Recorder) via ffmpeg
|
||||
(im Container vorhanden) auf rohes 16kHz mono int16 LE PCM. Input geht ueber
|
||||
eine Temp-Datei (nicht Pipe): Androids MediaRecorder legt das moov-Atom ans
|
||||
ENDE, das braucht seekbaren Input, sonst 'moov atom not found'."""
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
tmp = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tf:
|
||||
tf.write(audio_bytes)
|
||||
tmp = tf.name
|
||||
proc = subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-loglevel", "error", "-i", tmp,
|
||||
"-f", "s16le", "-ac", "1", "-ar", "16000", "pipe:1"],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
)
|
||||
if proc.returncode != 0 or not proc.stdout:
|
||||
raise ValueError(
|
||||
f"ffmpeg decode failed: {proc.stderr.decode('utf-8', 'ignore')[:200]}")
|
||||
return proc.stdout
|
||||
finally:
|
||||
if tmp:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _normalize_audio_bytes(audio_bytes: bytes) -> bytes:
|
||||
"""Akzeptiert rohes 16kHz int16 LE PCM, eine WAV-Datei (RIFF/WAVE) ODER einen
|
||||
komprimierten MP4/M4A/AAC-Container (Android-Recorder). WAV → Header strippen +
|
||||
Format validieren; MP4/AAC → via ffmpeg dekodieren. Ergebnis: rohes PCM."""
|
||||
if (len(audio_bytes) >= 44
|
||||
and audio_bytes[:4] == b"RIFF"
|
||||
and audio_bytes[8:12] == b"WAVE"):
|
||||
import io
|
||||
import wave
|
||||
with wave.open(io.BytesIO(audio_bytes), "rb") as wav:
|
||||
sr = wav.getframerate()
|
||||
ch = wav.getnchannels()
|
||||
sw = wav.getsampwidth()
|
||||
if sr != 16000:
|
||||
raise ValueError(f"WAV-Samplerate {sr} != 16000")
|
||||
if ch != 1:
|
||||
raise ValueError(f"WAV-Kanalzahl {ch} != 1 (mono erwartet)")
|
||||
if sw != 2:
|
||||
raise ValueError(f"WAV-Sampleweite {sw} != 2 (int16 erwartet)")
|
||||
return wav.readframes(wav.getnframes())
|
||||
# MP4/M4A/AAC-Container: Android-AAC-Recorder legt 'ftyp' bei Offset 4 an.
|
||||
if len(audio_bytes) >= 12 and audio_bytes[4:8] == b"ftyp":
|
||||
return _decode_compressed_to_pcm(audio_bytes)
|
||||
return audio_bytes
|
||||
|
||||
|
||||
def _audio_bytes_to_tensor(audio_bytes: bytes):
|
||||
"""int16 LE PCM (16kHz mono) → Torch-Tensor (1, N), normalisiert auf [-1, 1].
|
||||
WAV wird vorher auf rohes PCM reduziert (Header strippen)."""
|
||||
import torch
|
||||
raw = _normalize_audio_bytes(audio_bytes)
|
||||
arr = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
return torch.from_numpy(arr).unsqueeze(0)
|
||||
|
||||
|
||||
def embed(audio_bytes: bytes) -> np.ndarray:
|
||||
"""Berechnet das Speaker-Embedding fuer einen Audio-Chunk.
|
||||
Erwartet 16kHz int16 LE PCM Mono. Returns 192-dim numpy float32."""
|
||||
import torch
|
||||
model = _ensure_loaded()
|
||||
wav = _audio_bytes_to_tensor(audio_bytes)
|
||||
with torch.no_grad():
|
||||
emb = model.encode_batch(wav)
|
||||
return emb.squeeze().cpu().numpy().astype(np.float32)
|
||||
|
||||
|
||||
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
|
||||
"""Kosinus-Aehnlichkeit zwischen zwei 1D-Vektoren, Range [-1, 1].
|
||||
Hoeher = aehnlicher. Bei normalisierten Vektoren ist das gleich dem Skalarprodukt."""
|
||||
na = np.linalg.norm(a)
|
||||
nb = np.linalg.norm(b)
|
||||
if na < 1e-9 or nb < 1e-9:
|
||||
return 0.0
|
||||
return float(np.dot(a, b) / (na * nb))
|
||||
|
||||
|
||||
def save_fingerprint(embeddings: list[np.ndarray], sample_durations_s: list[float]) -> dict:
|
||||
"""Mittelt + L2-normalisiert die Embeddings und schreibt sie nach
|
||||
FINGERPRINT_FILE. Returns das gespeicherte Dict."""
|
||||
if not embeddings:
|
||||
raise ValueError("Keine Embeddings zum Speichern")
|
||||
VOICE_ID_DIR.mkdir(parents=True, exist_ok=True)
|
||||
stacked = np.stack(embeddings)
|
||||
mean = stacked.mean(axis=0)
|
||||
mean = mean / max(np.linalg.norm(mean), 1e-9)
|
||||
data = {
|
||||
"version": 1,
|
||||
"embedding": mean.tolist(),
|
||||
"embedding_dim": int(mean.shape[0]),
|
||||
"sample_count": len(embeddings),
|
||||
"sample_durations_s": [float(s) for s in sample_durations_s],
|
||||
"updated_at": int(time.time()),
|
||||
}
|
||||
FINGERPRINT_FILE.write_text(json.dumps(data, indent=2), encoding="utf-8")
|
||||
logger.info("[speaker-id] fingerprint gespeichert: %d Samples, dim=%d, total_s=%.1f",
|
||||
len(embeddings), mean.shape[0], sum(sample_durations_s))
|
||||
return data
|
||||
|
||||
|
||||
def load_fingerprint() -> Optional[dict]:
|
||||
"""Returns das Fingerprint-Dict oder None wenn noch nicht enrolled."""
|
||||
if not FINGERPRINT_FILE.exists():
|
||||
return None
|
||||
try:
|
||||
return json.loads(FINGERPRINT_FILE.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
logger.warning("[speaker-id] fingerprint laden fehlgeschlagen: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def delete_fingerprint() -> bool:
|
||||
"""Loescht den Fingerprint (z.B. fuer Re-Enrollment). True wenn was weg ist."""
|
||||
if FINGERPRINT_FILE.exists():
|
||||
FINGERPRINT_FILE.unlink()
|
||||
logger.info("[speaker-id] fingerprint geloescht")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def verify(audio_bytes: bytes, threshold: Optional[float] = None) -> tuple[bool, float]:
|
||||
"""Returns (is_match, similarity).
|
||||
|
||||
Wenn threshold=None: nutzt den Modul-Default (DEFAULT_THRESHOLD) — der wird
|
||||
vom config-Broadcast zur Laufzeit auf den Diagnostic-Slider-Wert gesetzt.
|
||||
Default-Arg-Bindung waere zur Def-Zeit, also bewusst None statt direkt.
|
||||
|
||||
Fail-open: wenn kein Fingerprint vorhanden ist oder das Embedding-Modell
|
||||
crasht, returnt (True, 0.0) — kein Filtering. Sonst wuerde ein kaputter
|
||||
Speaker-ID-Service die ganze Aufnahme blockieren."""
|
||||
if threshold is None:
|
||||
threshold = DEFAULT_THRESHOLD
|
||||
fp = load_fingerprint()
|
||||
if fp is None:
|
||||
return True, 0.0
|
||||
if len(audio_bytes) < MIN_SAMPLE_BYTES:
|
||||
# Zu wenig Audio fuer ein verlaessliches Embedding → durchlassen
|
||||
return True, 0.0
|
||||
try:
|
||||
saved_emb = np.array(fp["embedding"], dtype=np.float32)
|
||||
new_emb = embed(audio_bytes)
|
||||
except Exception as exc:
|
||||
logger.warning("[speaker-id] verify embed failed: %s — fail-open", exc)
|
||||
return True, 0.0
|
||||
sim = cosine_similarity(new_emb, saved_emb)
|
||||
return sim >= threshold, sim
|
||||
|
||||
|
||||
def status() -> dict:
|
||||
"""Status-Snapshot fuer die App / Diagnostic."""
|
||||
fp = load_fingerprint()
|
||||
return {
|
||||
"enrolled": fp is not None,
|
||||
"sample_count": fp.get("sample_count", 0) if fp else 0,
|
||||
"sample_durations_s": fp.get("sample_durations_s", []) if fp else [],
|
||||
"updated_at": fp.get("updated_at") if fp else None,
|
||||
"embedding_dim": fp.get("embedding_dim") if fp else None,
|
||||
"default_threshold": DEFAULT_THRESHOLD,
|
||||
}
|
||||
|
||||
|
||||
def enroll_from_samples(samples_b64: list[str]) -> dict:
|
||||
"""Verarbeitet base64-Samples (16kHz int16 LE PCM Mono) zu einem neuen
|
||||
Fingerprint. Returns Status-Dict. Wirft ValueError wenn nichts brauchbar ist."""
|
||||
if not samples_b64:
|
||||
raise ValueError("Keine Samples uebergeben")
|
||||
embeddings: list[np.ndarray] = []
|
||||
durations: list[float] = []
|
||||
rejected: list[dict] = []
|
||||
for idx, s in enumerate(samples_b64):
|
||||
try:
|
||||
raw = base64.b64decode(s)
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"base64: {exc}"})
|
||||
continue
|
||||
# Erst dekodieren (WAV/MP4/AAC → rohes PCM), DANN Laenge pruefen: der
|
||||
# Android-Recorder liefert komprimiertes MP4, dessen Byte-Laenge nichts
|
||||
# ueber die Dauer sagt (4s AAC < 32KB → faelschlich "zu kurz").
|
||||
try:
|
||||
pcm = _normalize_audio_bytes(raw)
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"decode: {exc}"})
|
||||
continue
|
||||
if len(pcm) < MIN_SAMPLE_BYTES:
|
||||
rejected.append({"index": idx, "reason": f"zu kurz ({len(pcm)} bytes PCM)"})
|
||||
continue
|
||||
try:
|
||||
emb = embed(pcm)
|
||||
embeddings.append(emb)
|
||||
durations.append(len(pcm) / 2 / 16000.0)
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"embed: {exc}"})
|
||||
if not embeddings:
|
||||
raise ValueError(
|
||||
f"Keine Samples konnten verarbeitet werden ({len(rejected)} rejected). "
|
||||
f"Details: {rejected[:3]}"
|
||||
)
|
||||
fingerprint = save_fingerprint(embeddings, durations)
|
||||
fingerprint["rejected"] = rejected
|
||||
return fingerprint
|
||||
+68
-2
@@ -75,7 +75,27 @@ STREAM_SESSION_TTL_S = 120 # tote Sessions nach 2 min aufraeumen
|
||||
# (Whisper oszilliert/halluziniert). Echte akustische Stille ist das robuste
|
||||
# „User hat aufgehoert"-Signal.
|
||||
STREAM_ENERGY_WINDOW_MS = 300 # RMS ueber die letzten 300ms Audio messen
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # RMS darueber = Sprache (haelt Session am Leben)
|
||||
# --- Adaptiver Voice-Schwellwert (ersetzt die fixe 0.012-Grenze) ---
|
||||
# Problem (Repro dokumentiert in audio.ts): eine FESTE RMS-Grenze schneidet
|
||||
# leises/entferntes Sprechen faelschlich als „Stille" (Handy weiter weg vom Mund,
|
||||
# ruhig im Auto, kurze Sprech-Pause) → Cut mitten im Satz. Loesung: die Grenze
|
||||
# relativ zum gemessenen Rausch-Boden der Session fuehren. Sprache =
|
||||
# noise_floor * Faktor, geklammert auf [MIN, MAX]. Bei noch ungelerntem Boden
|
||||
# gilt MIN → sensibel, lieber nicht abschneiden.
|
||||
STREAM_VOICE_FACTOR = 2.5 # Sprache = noise_floor * Faktor
|
||||
STREAM_VOICE_RMS_MIN = 0.005 # Untergrenze (stiller Raum: nicht auf 0 kollabieren)
|
||||
STREAM_VOICE_RMS_MAX = 0.020 # Obergrenze (lautes Auto: Sprache nie ganz aussperren)
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # Legacy-Konstante (nicht mehr im Cut-Pfad genutzt)
|
||||
|
||||
# Speaker-ID Gating global an/aus. DEFAULT AUS (fail-open) — bewusster Schalter
|
||||
# ("nur meine Stimme"), kein Automatismus: ein schlechter Enroll darf nie die STT
|
||||
# lahmlegen. Wird per config-Broadcast (voiceIdEnabled) zur Laufzeit gesetzt.
|
||||
SPEAKER_ID_ENABLED = os.getenv("VOICE_ID_ENABLED", "false").lower() in ("1", "true", "yes")
|
||||
|
||||
|
||||
def _set_speaker_id_enabled(val: bool) -> None:
|
||||
global SPEAKER_ID_ENABLED
|
||||
SPEAKER_ID_ENABLED = bool(val)
|
||||
# Rein-semantischer Backstop: wenn die Energie NIE faellt (laute Umgebung,
|
||||
# z.B. Auto), endpointen wir trotzdem — aber erst nach diesem Faktor x
|
||||
# endpoint_ms, damit normales Sprechen mit Pausen nicht abgeschnitten wird.
|
||||
@@ -323,6 +343,10 @@ class StreamSession:
|
||||
last_growth_at: float = 0.0
|
||||
last_transcribe_at: float = 0.0
|
||||
last_voice_at: float = 0.0 # letzter Tick mit akustischer Sprach-Energie
|
||||
noise_floor: float = 0.0 # adaptiver Rausch-Boden (0.0 = noch ungelernt)
|
||||
voice_factor: float = STREAM_VOICE_FACTOR # per-Session konfigurierbar (Payload voiceFactor)
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
closed: bool = False # nach stream_end gesetzt
|
||||
endpoint_sent: bool = False # Endpoint nur einmal feuern
|
||||
# Speaker-ID Gating: bei aktiviertem Fingerprint pruefen wir die ersten
|
||||
@@ -372,6 +396,10 @@ class SessionManager:
|
||||
speed = float(payload.get("speed") or 1.0)
|
||||
except (TypeError, ValueError):
|
||||
speed = 1.0
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
session = StreamSession(
|
||||
request_id=request_id,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
@@ -381,6 +409,7 @@ class SessionManager:
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=speed,
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
@@ -448,6 +477,11 @@ class SessionManager:
|
||||
Ohne Fingerprint → fail-open (match=True). Bei mismatch wird die
|
||||
Session sofort beendet mit synthetischem stt_endpoint."""
|
||||
sess.speaker_checked = True
|
||||
# Schalter aus (Default) → gar keine Pruefung, alles durchlassen.
|
||||
if not SPEAKER_ID_ENABLED:
|
||||
sess.speaker_match = True
|
||||
sess.speaker_similarity = 0.0
|
||||
return
|
||||
# Erste ~1.5s aus dem Buffer entnehmen (16kHz * 2 byte/sample = 32 bytes/ms)
|
||||
head_bytes = bytes(sess.pcm_buffer[: STREAM_SPEAKER_CHECK_MS * 32])
|
||||
if len(head_bytes) < speaker_id.MIN_SAMPLE_BYTES:
|
||||
@@ -549,8 +583,16 @@ class SessionManager:
|
||||
# Akustische Sprach-Aktivitaet JEDEN Tick (~200ms) messen — unabhaengig
|
||||
# vom Transcribe-Throttle. Solange wirklich gesprochen wird, bleibt die
|
||||
# Session am Leben, auch wenn Whisper gerade keinen neuen Text liefert.
|
||||
if self._tail_rms(sess) >= STREAM_VOICE_RMS_THRESHOLD:
|
||||
# ADAPTIV: der Schwellwert richtet sich nach dem gemessenen Rausch-Boden
|
||||
# (fast-down/slow-up), damit leises/entferntes Sprechen nicht faelschlich
|
||||
# als Stille gilt und der Satz mitten drin abgeschnitten wird.
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
else:
|
||||
# Rausch-Boden NUR aus Nicht-Sprache lernen — waehrend Sprache
|
||||
# einfrieren, sonst wandert die Grenze hoch und sperrt Sprache aus.
|
||||
self._update_noise_floor(sess, rms)
|
||||
|
||||
# Endpoint-Entscheidung JEDEN Tick, sobald ueberhaupt Text erkannt wurde:
|
||||
# (a) akustisch: seit endpoint_ms keine Sprach-Energie mehr → User ist
|
||||
@@ -620,6 +662,26 @@ class SessionManager:
|
||||
return 0.0
|
||||
return (samples / sess.sample_rate) * 1000.0
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
"""Adaptiver Voice-Schwellwert = Rausch-Boden * Faktor, geklammert auf
|
||||
[min, max]. Bei noch ungelerntem Boden (0.0) → Untergrenze: sensibel,
|
||||
lieber nicht abschneiden (das war der eigentliche Cutoff-Bug)."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
"""Rausch-Boden nachfuehren: schnell runter (neue, leisere Stille),
|
||||
langsam rauf (Umgebung wird lauter). NUR mit Nicht-Sprache aufrufen."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
"""RMS-Energie der letzten STREAM_ENERGY_WINDOW_MS des Audio-Buffers.
|
||||
Dient als akustisches „redet noch / ist still"-Signal."""
|
||||
@@ -928,6 +990,10 @@ async def run_loop(runner: WhisperRunner, sessions: SessionManager) -> None:
|
||||
logger.info("[speaker-id] threshold gesetzt: %.2f", t)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if "voiceIdEnabled" in payload:
|
||||
_set_speaker_id_enabled(payload.get("voiceIdEnabled"))
|
||||
logger.info("[speaker-id] Gating %s (voiceIdEnabled)",
|
||||
"AN" if SPEAKER_ID_ENABLED else "AUS")
|
||||
if "whisperDebugLog" in payload:
|
||||
global _DEBUG_LOG_TO_BRIDGE
|
||||
old = _DEBUG_LOG_TO_BRIDGE
|
||||
|
||||
@@ -61,10 +61,40 @@ def _ensure_loaded():
|
||||
return _model
|
||||
|
||||
|
||||
def _decode_compressed_to_pcm(audio_bytes: bytes) -> bytes:
|
||||
"""Dekodiert komprimiertes Audio (MP4/M4A/AAC vom Android-Recorder) via ffmpeg
|
||||
(im Container vorhanden) auf rohes 16kHz mono int16 LE PCM. Input geht ueber
|
||||
eine Temp-Datei (nicht Pipe): Androids MediaRecorder legt das moov-Atom ans
|
||||
ENDE, das braucht seekbaren Input, sonst 'moov atom not found'."""
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
tmp = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tf:
|
||||
tf.write(audio_bytes)
|
||||
tmp = tf.name
|
||||
proc = subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-loglevel", "error", "-i", tmp,
|
||||
"-f", "s16le", "-ac", "1", "-ar", "16000", "pipe:1"],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
)
|
||||
if proc.returncode != 0 or not proc.stdout:
|
||||
raise ValueError(
|
||||
f"ffmpeg decode failed: {proc.stderr.decode('utf-8', 'ignore')[:200]}")
|
||||
return proc.stdout
|
||||
finally:
|
||||
if tmp:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _normalize_audio_bytes(audio_bytes: bytes) -> bytes:
|
||||
"""Akzeptiert entweder rohes 16kHz int16 LE PCM ODER eine WAV-Datei (RIFF/WAVE).
|
||||
Bei WAV wird der Header gestrippt + Format validiert (16kHz / mono / int16).
|
||||
Ergebnis: rohes PCM."""
|
||||
"""Akzeptiert rohes 16kHz int16 LE PCM, eine WAV-Datei (RIFF/WAVE) ODER einen
|
||||
komprimierten MP4/M4A/AAC-Container (Android-Recorder). WAV → Header strippen +
|
||||
Format validieren; MP4/AAC → via ffmpeg dekodieren. Ergebnis: rohes PCM."""
|
||||
if (len(audio_bytes) >= 44
|
||||
and audio_bytes[:4] == b"RIFF"
|
||||
and audio_bytes[8:12] == b"WAVE"):
|
||||
@@ -81,6 +111,9 @@ def _normalize_audio_bytes(audio_bytes: bytes) -> bytes:
|
||||
if sw != 2:
|
||||
raise ValueError(f"WAV-Sampleweite {sw} != 2 (int16 erwartet)")
|
||||
return wav.readframes(wav.getnframes())
|
||||
# MP4/M4A/AAC-Container: Android-AAC-Recorder legt 'ftyp' bei Offset 4 an.
|
||||
if len(audio_bytes) >= 12 and audio_bytes[4:8] == b"ftyp":
|
||||
return _decode_compressed_to_pcm(audio_bytes)
|
||||
return audio_bytes
|
||||
|
||||
|
||||
@@ -212,13 +245,21 @@ def enroll_from_samples(samples_b64: list[str]) -> dict:
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"base64: {exc}"})
|
||||
continue
|
||||
if len(raw) < MIN_SAMPLE_BYTES:
|
||||
rejected.append({"index": idx, "reason": f"zu kurz ({len(raw)} bytes)"})
|
||||
# Erst dekodieren (WAV/MP4/AAC → rohes PCM), DANN Laenge pruefen: der
|
||||
# Android-Recorder liefert komprimiertes MP4, dessen Byte-Laenge nichts
|
||||
# ueber die Dauer sagt (4s AAC < 32KB → faelschlich "zu kurz").
|
||||
try:
|
||||
pcm = _normalize_audio_bytes(raw)
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"decode: {exc}"})
|
||||
continue
|
||||
if len(pcm) < MIN_SAMPLE_BYTES:
|
||||
rejected.append({"index": idx, "reason": f"zu kurz ({len(pcm)} bytes PCM)"})
|
||||
continue
|
||||
try:
|
||||
emb = embed(raw)
|
||||
emb = embed(pcm)
|
||||
embeddings.append(emb)
|
||||
durations.append(len(raw) / 2 / 16000.0)
|
||||
durations.append(len(pcm) / 2 / 16000.0)
|
||||
except Exception as exc:
|
||||
rejected.append({"index": idx, "reason": f"embed: {exc}"})
|
||||
if not embeddings:
|
||||
|
||||
Reference in New Issue
Block a user