Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ee6c4f34db | ||
|
|
75daadf72d | ||
|
|
37aaa90239 | ||
|
|
03e6d784b6 | ||
|
|
762f1a2dd9 | ||
|
|
49f6b26ab8 | ||
|
|
c340b9d683 | ||
|
|
a0429fc91e | ||
|
|
174d6d643d | ||
|
|
fd6ba73f59 | ||
|
|
f5b22253b2 | ||
|
|
761f4c8903 | ||
|
|
091a1b7755 | ||
|
|
e2b1eced3c | ||
|
|
fe804fa40e | ||
|
|
18eb94e942 | ||
|
|
74c7ea0a2d | ||
|
|
648e3b04fd | ||
|
|
2ad8c2f245 | ||
|
|
85d190e98c | ||
|
|
70705269fd | ||
|
|
2aef0347ae | ||
|
|
a49c022308 | ||
|
|
2d3ba024a4 | ||
|
|
1c7157327c | ||
|
|
6c27097c96 | ||
|
|
a91325a04f | ||
|
|
ef826d1ed1 | ||
|
|
933836f0a6 | ||
|
|
e04d8f360b | ||
|
|
4685632294 | ||
|
|
769025c41b | ||
|
|
2005e9b85e | ||
|
|
25abd220ad |
@@ -0,0 +1,63 @@
|
||||
# ai-box — KI-Box (gpubox) Bootstrap
|
||||
|
||||
Macht aus einem frisch installierten **Debian Trixie** (headless, nur SSH) einen
|
||||
startklaren GPU-Satelliten-Host für den `xtts`-Stack (STT/TTS/LLM).
|
||||
|
||||
## Was das Script tut
|
||||
|
||||
1. Basis-Pakete (curl, gnupg, git …)
|
||||
2. `contrib non-free non-free-firmware` aktivieren (Trixie-**deb822**-Format berücksichtigt)
|
||||
3. **NVIDIA-Treiber** installieren (`nvidia-driver` + firmware)
|
||||
4. **Docker** Engine + Compose-Plugin
|
||||
5. **NVIDIA Container Toolkit** + Docker-Runtime auf NVIDIA konfigurieren
|
||||
6. `xtts/.env` aus `.env.example` anlegen (RVS_TOKEN optional gleich setzen)
|
||||
7. **GPU-im-Container-Test** (`docker run --gpus all … nvidia-smi`)
|
||||
8. optional (`--up`): den `xtts`-Stack hochziehen
|
||||
|
||||
Alles **idempotent** — mehrfach ausführbar.
|
||||
|
||||
## Ablauf
|
||||
|
||||
```bash
|
||||
git clone <repo> ARIA-AGENT
|
||||
cd ARIA-AGENT/ai-box
|
||||
|
||||
sudo ./bootstrap.sh
|
||||
# → Wenn der Treiber frisch installiert wurde: einmal neu starten, dann nochmal:
|
||||
sudo reboot
|
||||
# … nach dem Boot:
|
||||
cd ARIA-AGENT/ai-box
|
||||
sudo ./bootstrap.sh --up --token <DEIN_RVS_TOKEN>
|
||||
```
|
||||
|
||||
Der Treiber-Reboot ist normal (Kernel-Modul wird erst beim Boot geladen). Beim
|
||||
zweiten Lauf überspringt das Script alles Erledigte und macht nur noch den
|
||||
GPU-Test + Stack-Start.
|
||||
|
||||
## Optionen
|
||||
|
||||
| Option | Wirkung |
|
||||
|---|---|
|
||||
| `--up` | am Ende `docker compose up -d --build` (Default-Profil) |
|
||||
| `--token <TOK>` | `RVS_TOKEN` in `xtts/.env` eintragen (auch via `RVS_TOKEN=…` env) |
|
||||
| `--rvs-host <H>` | `RVS_HOST` setzen |
|
||||
|
||||
## Wichtig
|
||||
|
||||
- **Stimm-Daten** (nicht in git): falls von der alten Box noch vorhanden,
|
||||
`xtts/voice-id/` (Speaker-Fingerprint) + `xtts/voices/` (F5-Referenz) herkopieren.
|
||||
Sonst egal — in der App neu anlegen: Stimme neu enrollen (ohne Fingerprint läuft
|
||||
die Speaker-ID fail-open, alles geht durch) + F5-Voice-Referenz neu hochladen.
|
||||
- **Voxtral bleibt aus** auf der 3060 (braucht ≥16 GB VRAM). Das Default-Profil
|
||||
fährt Whisper (mit dem M0.1-Fix) + F5-TTS + lokales LLM. Voxtral erst mit der
|
||||
24-GB-Karte: `docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build`.
|
||||
- **Erster Start lädt Modelle** (mehrere GB via HuggingFace nach `xtts/hf-cache`
|
||||
+ `xtts/models`) — genug Platz (1 TB NVMe ✓) und etwas Geduld.
|
||||
|
||||
## Verifizieren
|
||||
|
||||
```bash
|
||||
nvidia-smi # Host sieht die GPU
|
||||
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi # Container auch
|
||||
docker logs -f aria-whisper-bridge # "RVS verbunden" + service_status ready
|
||||
```
|
||||
Executable
+234
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# ARIA KI-Box (gpubox) Bootstrap — frisches Debian Trixie → startklarer
|
||||
# GPU-Satelliten-Host fuer den xtts-Stack (Voxtral/Whisper STT, F5-TTS, lokales LLM).
|
||||
#
|
||||
# Ablauf nach `git clone`:
|
||||
# cd ARIA-AGENT/ai-box
|
||||
# sudo ./bootstrap.sh # richtet Treiber + Docker + NVIDIA-Toolkit ein
|
||||
# # (falls Treiber frisch installiert: einmal `sudo reboot`, dann Script erneut)
|
||||
# sudo ./bootstrap.sh --up # dazu: xtts-Stack (Whisper+F5+LLM) hochziehen
|
||||
#
|
||||
# Optionen:
|
||||
# --up am Ende den xtts-Stack starten (Default-Profil, OHNE voxtral)
|
||||
# --token <TOK> RVS_TOKEN in xtts/.env eintragen (alternativ: env RVS_TOKEN=...)
|
||||
# --rvs-host <H> RVS_HOST setzen (Default aus .env.example)
|
||||
#
|
||||
# IDEMPOTENT: bereits erledigte Schritte werden uebersprungen. Nach dem
|
||||
# Treiber-Reboot einfach nochmal ausfuehren — der Rest laeuft dann durch.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ── CLI ──
|
||||
DO_UP=0
|
||||
RVS_TOKEN_ARG="${RVS_TOKEN:-}"
|
||||
RVS_HOST_ARG="${RVS_HOST:-}"
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--up) DO_UP=1; shift ;;
|
||||
--token) RVS_TOKEN_ARG="${2:-}"; shift 2 ;;
|
||||
--rvs-host) RVS_HOST_ARG="${2:-}"; shift 2 ;;
|
||||
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "Unbekannte Option: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Log-Helfer ──
|
||||
c_g="\033[1;32m"; c_y="\033[1;33m"; c_r="\033[1;31m"; c_b="\033[1;34m"; c_0="\033[0m"
|
||||
STEP=0
|
||||
step() { STEP=$((STEP+1)); echo -e "\n${c_b}[${STEP}] $*${c_0}"; }
|
||||
ok() { echo -e " ${c_g}✓${c_0} $*"; }
|
||||
warn() { echo -e " ${c_y}!${c_0} $*"; }
|
||||
die() { echo -e "${c_r}✗ $*${c_0}" >&2; exit 1; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
XTTS_DIR="$REPO_ROOT/xtts"
|
||||
|
||||
# ── Preflight ──
|
||||
[[ "$(id -u)" -eq 0 ]] || die "Bitte als root ausfuehren (sudo ./bootstrap.sh)."
|
||||
command -v apt-get >/dev/null || die "Kein apt-get — dieses Script ist fuer Debian/Trixie."
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
echo -e "${c_b}=== ARIA KI-Box Bootstrap ===${c_0}"
|
||||
echo "Repo: $REPO_ROOT"
|
||||
echo "xtts: $XTTS_DIR"
|
||||
|
||||
# ── 1. Basis-Pakete ──
|
||||
step "Basis-Pakete"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq ca-certificates curl gnupg git lsb-release >/dev/null
|
||||
ok "ca-certificates, curl, gnupg, git"
|
||||
|
||||
# ── 2. non-free Repos aktivieren (Trixie deb822 + Legacy) ──
|
||||
step "APT-Komponenten (contrib non-free non-free-firmware)"
|
||||
# Ergaenzt die drei Komponenten in JEDER Components:-Zeile einer deb822-Datei —
|
||||
# pro Zeile nur was fehlt, reihenfolge-robust, idempotent. Die Adressen pruefen
|
||||
# ganze Woerter (non-free-firmware zaehlt NICHT als non-free).
|
||||
add_components() {
|
||||
sed -i -E '/^[Cc]omponents:/{
|
||||
/(^|[[:space:]])contrib([[:space:]]|$)/!s/$/ contrib/
|
||||
/(^|[[:space:]])non-free([[:space:]]|$)/!s/$/ non-free/
|
||||
/(^|[[:space:]])non-free-firmware([[:space:]]|$)/!s/$/ non-free-firmware/
|
||||
}' "$1"
|
||||
}
|
||||
ENABLED_ANY=0
|
||||
shopt -s nullglob
|
||||
for f in /etc/apt/sources.list.d/*.sources; do
|
||||
grep -qE '^[Cc]omponents:' "$f" || continue
|
||||
b="$(md5sum "$f")"; add_components "$f"; a="$(md5sum "$f")"
|
||||
if [[ "$b" != "$a" ]]; then ENABLED_ANY=1; ok "aktualisiert: $(basename "$f")"; fi
|
||||
done
|
||||
shopt -u nullglob
|
||||
# Legacy /etc/apt/sources.list (deb-Zeilen)
|
||||
if [[ -f /etc/apt/sources.list ]] && grep -qE '^deb ' /etc/apt/sources.list; then
|
||||
if ! grep -qE '^deb .*[[:space:]]non-free([[:space:]]|$)' /etc/apt/sources.list; then
|
||||
sed -i -E '/^deb .*debian/ s/$/ contrib non-free non-free-firmware/' /etc/apt/sources.list
|
||||
ENABLED_ANY=1; ok "aktualisiert: sources.list"
|
||||
fi
|
||||
fi
|
||||
[[ $ENABLED_ANY -eq 0 ]] && ok "non-free schon aktiv"
|
||||
apt-get update -qq # immer neu einlesen, damit der Kandidat sicher da ist
|
||||
|
||||
# ── 3. NVIDIA-Treiber ──
|
||||
step "NVIDIA-Treiber"
|
||||
if nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Treiber aktiv: $(nvidia-smi --query-gpu=name --format=csv,noheader | paste -sd', ')"
|
||||
DRIVER_ACTIVE=1
|
||||
else
|
||||
if dpkg -l | grep -q '^ii nvidia-driver '; then
|
||||
warn "Treiber installiert, aber nvidia-smi antwortet nicht → REBOOT noetig."
|
||||
DRIVER_ACTIVE=0
|
||||
else
|
||||
# KEIN separater Kandidaten-Check — die apt-cache-Ausgabe ist locale-/pipe-
|
||||
# fragil (hat faelschlich "kein Kandidat" gemeldet). Der Install IST der Test:
|
||||
# direkt installieren; schlaegt er fehl, einmal volles apt-get update + Retry,
|
||||
# dann erst mit Diagnose abbrechen.
|
||||
# Kernel-Header ZUERST — sonst ueberspringt DKMS den Modulbau ("No kernel
|
||||
# headers were found") und nvidia-smi kann spaeter nicht mit dem Treiber reden.
|
||||
warn "Kernel-Header + nvidia-driver installieren (DKMS-Build, dauert)…"
|
||||
apt-get install -y linux-headers-amd64 || true
|
||||
apt-get install -y "linux-headers-$(uname -r)" || true
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
warn "Install fehlgeschlagen — volles 'apt-get update' + zweiter Versuch…"
|
||||
apt-get update
|
||||
if ! apt-get install -y nvidia-driver firmware-misc-nonfree; then
|
||||
echo
|
||||
warn "nvidia-driver liess sich nicht installieren. Aktive Quellen:"
|
||||
grep -rhE '^[Cc]omponents:' /etc/apt/sources.list.d/*.sources 2>/dev/null | sed 's/^/ /' || true
|
||||
grep -E '^deb ' /etc/apt/sources.list 2>/dev/null | sed 's/^/ /' || true
|
||||
die "Pruefe 'apt-cache policy nvidia-driver' + Netz/non-free."
|
||||
fi
|
||||
fi
|
||||
# Falls nvidia-kernel-dkms schon (ohne Header) installiert war: Modul jetzt bauen.
|
||||
dkms autoinstall >/dev/null 2>&1 || true
|
||||
ok "nvidia-driver + Kernel-Modul installiert"
|
||||
DRIVER_ACTIVE=0
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 4. Docker Engine + Compose-Plugin ──
|
||||
step "Docker"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
ok "Docker vorhanden: $(docker --version)"
|
||||
else
|
||||
warn "Installiere Docker (offizielles get.docker.com)…"
|
||||
curl -fsSL https://get.docker.com | sh >/dev/null
|
||||
systemctl enable --now docker >/dev/null 2>&1 || true
|
||||
ok "Docker installiert: $(docker --version)"
|
||||
fi
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
ok "Compose-Plugin: $(docker compose version | head -1)"
|
||||
else
|
||||
warn "Compose-Plugin fehlt — installiere docker-compose-plugin…"
|
||||
apt-get install -y -qq docker-compose-plugin >/dev/null || \
|
||||
warn "Konnte docker-compose-plugin nicht via apt holen — get.docker.com bringt es normalerweise mit."
|
||||
fi
|
||||
|
||||
# ── 5. NVIDIA Container Toolkit ──
|
||||
step "NVIDIA Container Toolkit"
|
||||
NCT_LIST="/etc/apt/sources.list.d/nvidia-container-toolkit.list"
|
||||
NCT_KEY="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"
|
||||
if ! dpkg -l | grep -q '^ii nvidia-container-toolkit '; then
|
||||
[[ -f "$NCT_KEY" ]] || curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
||||
| gpg --dearmor -o "$NCT_KEY"
|
||||
if [[ ! -f "$NCT_LIST" ]]; then
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
||||
| sed "s#deb https://#deb [signed-by=${NCT_KEY}] https://#g" > "$NCT_LIST"
|
||||
fi
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq nvidia-container-toolkit >/dev/null
|
||||
ok "nvidia-container-toolkit installiert"
|
||||
else
|
||||
ok "nvidia-container-toolkit vorhanden"
|
||||
fi
|
||||
# Docker-Runtime auf NVIDIA konfigurieren (idempotent)
|
||||
if ! grep -q '"nvidia"' /etc/docker/daemon.json 2>/dev/null; then
|
||||
nvidia-ctk runtime configure --runtime=docker >/dev/null
|
||||
systemctl restart docker
|
||||
ok "Docker-Runtime auf NVIDIA konfiguriert + Docker neugestartet"
|
||||
else
|
||||
ok "Docker-Runtime bereits NVIDIA-konfiguriert"
|
||||
fi
|
||||
|
||||
# ── 6. xtts/.env vorbereiten ──
|
||||
step "xtts/.env"
|
||||
if [[ -f "$XTTS_DIR/.env" ]]; then
|
||||
ok ".env existiert bereits (unangetastet)"
|
||||
else
|
||||
cp "$XTTS_DIR/.env.example" "$XTTS_DIR/.env"
|
||||
ok ".env aus .env.example erstellt"
|
||||
fi
|
||||
if [[ -n "$RVS_TOKEN_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_TOKEN=.*#RVS_TOKEN=${RVS_TOKEN_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_TOKEN eingetragen"
|
||||
fi
|
||||
if [[ -n "$RVS_HOST_ARG" ]]; then
|
||||
sed -i -E "s#^RVS_HOST=.*#RVS_HOST=${RVS_HOST_ARG}#" "$XTTS_DIR/.env"
|
||||
ok "RVS_HOST=${RVS_HOST_ARG} eingetragen"
|
||||
fi
|
||||
if grep -q '^RVS_TOKEN=dein_token_hier' "$XTTS_DIR/.env"; then
|
||||
warn "RVS_TOKEN ist noch der Platzhalter — vor dem Start setzen:"
|
||||
warn " nano $XTTS_DIR/.env (oder: sudo ./bootstrap.sh --token <TOKEN>)"
|
||||
fi
|
||||
warn "Stimm-Daten (nicht in git): falls von der alten Box noch vorhanden, xtts/voice-id/"
|
||||
warn " + xtts/voices/ herkopieren. Sonst egal — in der App neu anlegen:"
|
||||
warn " Sprache neu enrollen (Speaker-ID, sonst fail-open) + F5-Referenz neu hochladen."
|
||||
|
||||
# ── 7. GPU-im-Container verifizieren ──
|
||||
step "GPU-im-Container Test"
|
||||
if [[ "${DRIVER_ACTIVE:-0}" -eq 1 ]]; then
|
||||
if docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi >/dev/null 2>&1; then
|
||||
ok "Docker sieht die GPU — KI-Box ist einsatzbereit."
|
||||
GPU_READY=1
|
||||
else
|
||||
warn "Host-Treiber ok, aber Container sieht die GPU nicht — Toolkit/Runtime pruefen."
|
||||
GPU_READY=0
|
||||
fi
|
||||
else
|
||||
warn "Treiber noch nicht aktiv → Test uebersprungen."
|
||||
echo
|
||||
echo -e "${c_y}==> REBOOT noetig, dann Script erneut ausfuehren:${c_0}"
|
||||
echo -e "${c_y} sudo reboot && (nach dem Boot) sudo ./bootstrap.sh${c_0}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 8. Optional: xtts-Stack hochziehen ──
|
||||
if [[ $DO_UP -eq 1 && "${GPU_READY:-0}" -eq 1 ]]; then
|
||||
step "xtts-Stack starten (Default-Profil: Whisper + F5 + LLM — OHNE voxtral)"
|
||||
warn "Erster Start laedt Modelle (mehrere GB via HuggingFace) — kann dauern."
|
||||
( cd "$XTTS_DIR" && docker compose up -d --build )
|
||||
ok "Stack laeuft. Logs: docker logs -f aria-whisper-bridge"
|
||||
echo
|
||||
echo " voxtral (STT via Voxtral) braucht >=16 GB VRAM → erst mit der 24-GB-Karte:"
|
||||
echo " cd $XTTS_DIR && docker compose stop whisper-bridge && docker compose --profile voxtral up -d --build"
|
||||
fi
|
||||
|
||||
# ── Abschluss ──
|
||||
echo
|
||||
echo -e "${c_g}=== Fertig. KI-Box startklar. ===${c_0}"
|
||||
if [[ $DO_UP -eq 0 ]]; then
|
||||
echo "Naechster Schritt — Stack starten:"
|
||||
echo " cd $XTTS_DIR && docker compose up -d --build"
|
||||
echo "oder direkt: sudo ./bootstrap.sh --up"
|
||||
fi
|
||||
@@ -79,8 +79,8 @@ android {
|
||||
applicationId "com.ariacockpit"
|
||||
minSdkVersion rootProject.ext.minSdkVersion
|
||||
targetSdkVersion rootProject.ext.targetSdkVersion
|
||||
versionCode 20204
|
||||
versionName "0.2.2.4"
|
||||
versionCode 20301
|
||||
versionName "0.2.3.1"
|
||||
// Fallback fuer Libraries mit Product Flavors
|
||||
missingDimensionStrategy 'react-native-camera', 'general'
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "aria-cockpit",
|
||||
"version": "0.2.2.4",
|
||||
"version": "0.2.3.1",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"android": "react-native run-android",
|
||||
|
||||
@@ -2371,6 +2371,10 @@ const ChatScreen: React.FC = () => {
|
||||
size: file.size,
|
||||
base64,
|
||||
projectId: activePid,
|
||||
// Korrelation: dieselbe clientMsgId wie der Text, damit die Bridge
|
||||
// die Datei genau DIESER Nachricht zuordnet — auch wenn Files (fire-
|
||||
// and-forget) und Text (ACK-getrackt, bei Queue verzoegert) desyncen.
|
||||
...(cmid && { clientMsgId: cmid }),
|
||||
...(isPhoto && file.width && { width: file.width, height: file.height }),
|
||||
...(location && { location }),
|
||||
});
|
||||
@@ -2440,6 +2444,24 @@ const ChatScreen: React.FC = () => {
|
||||
}
|
||||
}, [inputText, pendingAttachments, sendPendingAttachments, getCtxState, setCtxState, setCtxQueue, actuallySend]);
|
||||
|
||||
// Zwischenruf: waehrend ARIA arbeitet eine Korrektur MITTEN in den laufenden
|
||||
// Turn schieben — NICHT in die Queue, KEIN Abbruch. Geht als 'interject' ueber
|
||||
// RVS an die Bridge → Proxy → laufender Subprozess (greift es an der naechsten
|
||||
// Tool-Grenze auf). Sichtbar nur, wenn der aktive Kontext gerade arbeitet.
|
||||
const sendInterject = useCallback(() => {
|
||||
const text = inputText.trim();
|
||||
if (!text) return;
|
||||
const activePid = focusedProjectIdRef.current;
|
||||
rvs.send('interject' as any, { projectId: activePid, text });
|
||||
// Lokale Bubble zur Rueckmeldung (laeuft NICHT durch Send/Queue).
|
||||
setMessages(prev => capMessages([...prev, {
|
||||
id: nextId(), sender: 'user', text: `📣 Zwischenruf: ${text}`,
|
||||
timestamp: Date.now(), projectId: activePid,
|
||||
}]));
|
||||
projectDraftsRef.current = { ...projectDraftsRef.current, [activePid]: '' };
|
||||
setInputText('');
|
||||
}, [inputText]);
|
||||
|
||||
// --- Rendering ---
|
||||
|
||||
const renderMessage = ({ item }: { item: ChatMessage }) => {
|
||||
@@ -3214,9 +3236,19 @@ const ChatScreen: React.FC = () => {
|
||||
|
||||
{/* Senden oder Sprache */}
|
||||
{inputText.trim() || pendingAttachments.length > 0 ? (
|
||||
<TouchableOpacity style={styles.sendButton} onPress={sendTextMessage}>
|
||||
<Text style={styles.sendIcon}>{'\u2B06\uFE0F'}</Text>
|
||||
</TouchableOpacity>
|
||||
<>
|
||||
{/* Zwischenruf: nur wenn ARIA im aktiven Kontext gerade arbeitet und
|
||||
Text da ist. Schiebt die Korrektur in den laufenden Turn statt
|
||||
sie anzustellen. */}
|
||||
{inputText.trim() && (agentActivityByCtx[focusedProjectId]?.activity || 'idle') !== 'idle' ? (
|
||||
<TouchableOpacity style={styles.interjectButton} onPress={sendInterject} accessibilityLabel="Zwischenruf">
|
||||
<Text style={styles.interjectIcon}>{'\uD83D\uDCE3'}</Text>
|
||||
</TouchableOpacity>
|
||||
) : null}
|
||||
<TouchableOpacity style={styles.sendButton} onPress={sendTextMessage}>
|
||||
<Text style={styles.sendIcon}>{'\u2B06\uFE0F'}</Text>
|
||||
</TouchableOpacity>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<VoiceButton
|
||||
@@ -3734,6 +3766,18 @@ const styles = StyleSheet.create({
|
||||
sendIcon: {
|
||||
fontSize: 18,
|
||||
},
|
||||
interjectButton: {
|
||||
width: 40,
|
||||
height: 40,
|
||||
borderRadius: 20,
|
||||
backgroundColor: '#FF9500', // orange — Zwischenruf, klar vom blauen Senden getrennt
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
marginRight: 6,
|
||||
},
|
||||
interjectIcon: {
|
||||
fontSize: 18,
|
||||
},
|
||||
wakeWordBtn: {
|
||||
width: 32,
|
||||
height: 32,
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* ariaView — Empfaenger der von ARIA komponierten RAEUMLICHEN Ansichten (M1).
|
||||
*
|
||||
* Fluss: ARIA ruft im Brain `present_view` → Brain-Event `aria_view` → Bridge →
|
||||
* RVS `aria_view` → hier gepuffert → WorkspaceCanvas rendert Orb + Karten, die
|
||||
* auf der Flaeche materialisieren.
|
||||
*
|
||||
* Der Service haelt pro Projekt die AKTUELLE View-Spec, damit eine spaet
|
||||
* gemountete Canvas-Kachel sofort den Ist-Stand bekommt. Muster wie
|
||||
* services/codeFile.ts (Singleton, rvs.onMessage).
|
||||
*
|
||||
* Die Karten-Typen sind bewusst offen (string), damit spaetere Renderer (vnc,
|
||||
* chart, file …) ohne Service-Aenderung dazukommen. Der jeweilige Client-Renderer
|
||||
* entscheidet, was er mit einem unbekannten Typ macht (i.d.R. ignorieren).
|
||||
*/
|
||||
|
||||
import rvs, { RVSMessage } from './rvs';
|
||||
|
||||
export type OrbState = 'idle' | 'listening' | 'thinking' | 'speaking' | 'working';
|
||||
|
||||
export interface ViewMarker {
|
||||
lat: number;
|
||||
lon: number;
|
||||
label?: string;
|
||||
}
|
||||
|
||||
export interface ViewCard {
|
||||
type: 'text' | 'image' | 'map' | 'code' | 'list' | string;
|
||||
title?: string;
|
||||
md?: string; // text/list
|
||||
src?: string; // image
|
||||
markers?: ViewMarker[]; // map
|
||||
path?: string; // code
|
||||
lang?: string; // code
|
||||
// Zukuenftige Kartenfelder ohne Service-Aenderung:
|
||||
[k: string]: any;
|
||||
}
|
||||
|
||||
export interface ViewSpec {
|
||||
cards: ViewCard[];
|
||||
orb?: OrbState;
|
||||
title?: string;
|
||||
}
|
||||
|
||||
export interface AriaView {
|
||||
projectId: string;
|
||||
view: ViewSpec;
|
||||
clientMsgId?: string;
|
||||
ts: number;
|
||||
}
|
||||
|
||||
type ViewSub = (v: AriaView) => void;
|
||||
|
||||
class AriaViewService {
|
||||
private views = new Map<string, AriaView>();
|
||||
private subs: ViewSub[] = [];
|
||||
|
||||
constructor() {
|
||||
rvs.onMessage((m) => this.onMessage(m));
|
||||
}
|
||||
|
||||
private onMessage(m: RVSMessage): void {
|
||||
if (m.type !== 'aria_view') return;
|
||||
const p = (m.payload || {}) as any;
|
||||
const raw = (p.view || {}) as any;
|
||||
const cards: ViewCard[] = Array.isArray(raw.cards) ? raw.cards : [];
|
||||
if (cards.length === 0) return; // leere Ansicht ignorieren
|
||||
const view: ViewSpec = {
|
||||
cards,
|
||||
orb: raw.orb || 'speaking',
|
||||
title: raw.title || '',
|
||||
};
|
||||
const projectId: string = p.projectId || '';
|
||||
const entry: AriaView = {
|
||||
projectId,
|
||||
view,
|
||||
clientMsgId: p.clientMsgId || '',
|
||||
ts: Date.now(),
|
||||
};
|
||||
this.views.set(projectId, entry);
|
||||
this.subs.forEach((cb) => {
|
||||
try { cb(entry); } catch {}
|
||||
});
|
||||
}
|
||||
|
||||
/** Aktuelle Ansicht eines Projekts (leer = Hauptchat). */
|
||||
getView(projectId: string): AriaView | undefined {
|
||||
return this.views.get(projectId || '');
|
||||
}
|
||||
|
||||
/** Registriert einen Listener fuer neue Ansichten. */
|
||||
subscribe(cb: ViewSub): () => void {
|
||||
this.subs.push(cb);
|
||||
return () => { this.subs = this.subs.filter((s) => s !== cb); };
|
||||
}
|
||||
|
||||
/** Ansicht eines Projekts verwerfen (z.B. wenn der User sie wegwischt). */
|
||||
clear(projectId: string): void {
|
||||
this.views.delete(projectId || '');
|
||||
}
|
||||
}
|
||||
|
||||
const ariaView = new AriaViewService();
|
||||
export default ariaView;
|
||||
@@ -1145,7 +1145,7 @@ class AudioService {
|
||||
speed: typeof opts.speed === 'number' ? opts.speed : 1.0,
|
||||
interrupted: !!opts.interrupted,
|
||||
location: opts.location || null,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : 1500,
|
||||
endpointMs: typeof opts.endpointMs === 'number' ? opts.endpointMs : STT_ENDPOINT_DEFAULT_MS,
|
||||
hardCapMs: typeof opts.hardCapMs === 'number' ? opts.hardCapMs : 60000,
|
||||
sampleRate: 16000,
|
||||
projectId: opts.projectId || '',
|
||||
|
||||
@@ -145,6 +145,17 @@ class GpsTrackingService {
|
||||
// liefert im Hintergrund keine Updates (nur Heartbeat sendet alte Werte).
|
||||
const bgEnabled = await isBackgroundGpsEnabled();
|
||||
if (bgEnabled) {
|
||||
// Ohne ACCESS_BACKGROUND_LOCATION liefert watchPosition im Hintergrund
|
||||
// NICHTS (Android 10+) → der Foreground-Service allein bringt nichts, und
|
||||
// genau der Fall "Ankunft waehrend der Fahrt, Screen aus" faellt durch.
|
||||
// Deshalb erst die Permission sicherstellen (oeffnet ggf. die Android-
|
||||
// Settings fuer "Immer erlauben"), DANN den Location-Foreground-Service
|
||||
// hochziehen — der haelt den Prozess wach, sodass watchPosition + der
|
||||
// 60s-Heartbeat auch unter Doze weiterlaufen.
|
||||
const bgOk = await ensureBackgroundLocationPermission();
|
||||
if (!bgOk) {
|
||||
console.warn('[gps-track] Background-Permission fehlt — Tracking nur im Vordergrund zuverlaessig');
|
||||
}
|
||||
try { await acquireBackgroundAudio('location'); } catch {}
|
||||
}
|
||||
try {
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
*/
|
||||
|
||||
import AsyncStorage from '@react-native-async-storage/async-storage';
|
||||
import { Platform, DeviceEventEmitter } from 'react-native';
|
||||
import { Platform, DeviceEventEmitter, AppState } from 'react-native';
|
||||
import rvs from './rvs';
|
||||
|
||||
// Lokales Event damit die SettingsScreen Live Logs / Events Tabs
|
||||
@@ -38,6 +38,23 @@ const noop = () => {};
|
||||
let _verbose = true;
|
||||
let _debugLogsToBridge = false;
|
||||
|
||||
// ─── Crash-Kontext ohne adb ─────────────────────────────────────────
|
||||
// Ein RUN_MARKER bleibt gesetzt, solange die App AKTIV laeuft; bei sauberem
|
||||
// Wechsel in den Hintergrund wird er geloescht. Ist er beim naechsten Start
|
||||
// noch da, ist der vorige Lauf unsauber gestorben (nativer Crash/OOM — der
|
||||
// schreibt KEINEN JS-Fehler, taucht also sonst nirgends auf). Wir melden das
|
||||
// dann via RVS mit dem letzten Breadcrumb (was die App zuletzt tat).
|
||||
const RUN_MARKER_KEY = 'aria_run_marker';
|
||||
const BREADCRUMB_KEY = 'aria_last_breadcrumb';
|
||||
let _breadcrumb: { ts: number; scope: string; message: string } = { ts: 0, scope: '', message: '' };
|
||||
let _breadcrumbDirty = false;
|
||||
|
||||
/** Letzte App-Aktivitaet merken — Crash-Kontext fuer den naechsten Boot. */
|
||||
export function noteBreadcrumb(scope: string, message: string): void {
|
||||
_breadcrumb = { ts: Date.now(), scope: scope || '', message: String(message || '').slice(0, 120) };
|
||||
_breadcrumbDirty = true;
|
||||
}
|
||||
|
||||
function applyState(): void {
|
||||
console.log = _verbose ? originalLog : noop;
|
||||
}
|
||||
@@ -53,6 +70,45 @@ export async function initLogger(): Promise<void> {
|
||||
_debugLogsToBridge = d === 'true'; // default: false
|
||||
} catch {}
|
||||
applyState();
|
||||
await _initCrashDetection();
|
||||
}
|
||||
|
||||
// Native-Crash-Erkennung (ohne adb) — siehe RUN_MARKER-Kommentar oben.
|
||||
async function _initCrashDetection(): Promise<void> {
|
||||
try {
|
||||
const marker = await AsyncStorage.getItem(RUN_MARKER_KEY);
|
||||
if (marker) {
|
||||
let bc: any = {};
|
||||
try { bc = JSON.parse((await AsyncStorage.getItem(BREADCRUMB_KEY)) || '{}'); } catch {}
|
||||
const gap = bc && bc.ts ? Math.round((Date.now() - bc.ts) / 1000) : -1;
|
||||
// Verzoegert melden — RVS ist beim Boot oft noch nicht verbunden.
|
||||
setTimeout(() => {
|
||||
reportAppError({
|
||||
scope: 'app.crash-detected',
|
||||
level: 'warn',
|
||||
message: `Voriger Lauf ohne sauberes Shutdown beendet (nativer Crash/OOM?). `
|
||||
+ `Letzte Aktivitaet: [${(bc && bc.scope) || '?'}] ${(bc && bc.message) || '?'}`
|
||||
+ (gap >= 0 ? ` (vor ~${gap}s)` : ''),
|
||||
});
|
||||
}, 6000);
|
||||
}
|
||||
await AsyncStorage.setItem(RUN_MARKER_KEY, String(Date.now()));
|
||||
} catch {}
|
||||
// Breadcrumb throttled persistieren (alle 5s, nur wenn geaendert).
|
||||
setInterval(() => {
|
||||
if (_breadcrumbDirty) {
|
||||
_breadcrumbDirty = false;
|
||||
AsyncStorage.setItem(BREADCRUMB_KEY, JSON.stringify(_breadcrumb)).catch(() => {});
|
||||
}
|
||||
}, 5000);
|
||||
// Sauberer Hintergrund-Wechsel → Marker weg (kein Crash). Rueckkehr → wieder
|
||||
// scharf. So melden nur echte Aktiv-Crashes, kein normales Backgrounden.
|
||||
try {
|
||||
AppState.addEventListener('change', (s) => {
|
||||
if (s === 'background') AsyncStorage.removeItem(RUN_MARKER_KEY).catch(() => {});
|
||||
else if (s === 'active') AsyncStorage.setItem(RUN_MARKER_KEY, String(Date.now())).catch(() => {});
|
||||
});
|
||||
} catch {}
|
||||
}
|
||||
|
||||
export function isVerboseLogging(): boolean {
|
||||
@@ -94,6 +150,7 @@ let _reportingInstalled = false;
|
||||
/** Schickt einen App-Fehler via RVS an die Bridge. */
|
||||
export function reportAppError(ev: AppErrorEvent): void {
|
||||
const ts = Date.now();
|
||||
noteBreadcrumb(ev.scope, ev.message);
|
||||
try {
|
||||
rvs.send('app_log' as any, {
|
||||
ts,
|
||||
@@ -128,6 +185,9 @@ export function reportAppError(ev: AppErrorEvent): void {
|
||||
* Default aus damit Mama-Modus keine Disk-Schreiblast hat. Error-Reports
|
||||
* (reportAppError) gehen weiterhin IMMER durch. */
|
||||
export function reportAppDebug(scope: string, message: string): void {
|
||||
// Breadcrumb IMMER aktualisieren (auch wenn Debug-Logs-an-Bridge aus ist) —
|
||||
// fuer den Crash-Kontext beim naechsten Boot.
|
||||
noteBreadcrumb(scope, message);
|
||||
if (!_debugLogsToBridge) return;
|
||||
const ts = Date.now();
|
||||
const trimmed = String(message).slice(0, 2000);
|
||||
|
||||
@@ -5,13 +5,13 @@
|
||||
* Cockpit-Modus → Workbench mit Taskleisten-Dock: Chat · Code · Desktop.
|
||||
*
|
||||
* Aktivitaets-Badges am Dock: Editor blau, wenn schon Code-Dateien da sind;
|
||||
* Desktop gruen, wenn eine VM verbunden ist.
|
||||
* Desktop gruen NUR, wenn im aktiven Projekt eine VM laeuft.
|
||||
*/
|
||||
|
||||
import React, { useEffect, useMemo, useState } from 'react';
|
||||
import projectFocus, { FocusSnapshot } from '../services/projectFocus';
|
||||
import codeFile from '../services/codeFile';
|
||||
import desktop from '../services/desktop';
|
||||
import brainApi from '../services/brainApi';
|
||||
import viewMode, { ViewModeValue } from '../services/viewMode';
|
||||
import ChatScreen from '../screens/ChatScreen';
|
||||
import { TileId } from './layout';
|
||||
@@ -29,7 +29,6 @@ const WorkspaceScreen: React.FC = () => {
|
||||
useEffect(() => projectFocus.subscribe(setFocus), []);
|
||||
|
||||
const pid = focus.focusedProjectId;
|
||||
const kind = projectFocus.getProjectKind(pid);
|
||||
|
||||
// Code-Signal: hat der Spiegel schon Dateien fuer dieses Projekt?
|
||||
useEffect(() => {
|
||||
@@ -39,13 +38,21 @@ const WorkspaceScreen: React.FC = () => {
|
||||
});
|
||||
}, [pid]);
|
||||
|
||||
// Desktop-Signal + einmaliger Check beim Betreten eines Code-Projekts.
|
||||
// Desktop-Signal: gruener Punkt NUR, wenn im AKTIVEN Projekt wirklich eine VM
|
||||
// laeuft (nicht generell irgendwo). Quelle ist die projektbezogene VM-Liste;
|
||||
// leichtes Nachfassen, damit Start/Stop sich zeitnah zeigt.
|
||||
useEffect(() => {
|
||||
setHasDesktop(desktop.getStatus().available);
|
||||
const unsub = desktop.subscribeStatus((s) => setHasDesktop(s.available));
|
||||
if (kind === 'code') desktop.requestCheck(pid);
|
||||
return unsub;
|
||||
}, [pid, kind]);
|
||||
if (!pid) { setHasDesktop(false); return; }
|
||||
let alive = true;
|
||||
const check = () => {
|
||||
brainApi.listProjectVms(pid)
|
||||
.then(r => { if (alive) setHasDesktop((r.vms || []).some(v => v.running)); })
|
||||
.catch(() => { if (alive) setHasDesktop(false); });
|
||||
};
|
||||
check();
|
||||
const t = setInterval(check, 6000);
|
||||
return () => { alive = false; clearInterval(t); };
|
||||
}, [pid]);
|
||||
|
||||
const badges = useMemo(() => ({
|
||||
editor: hasCode ? '#0096FF' : undefined,
|
||||
|
||||
@@ -77,14 +77,13 @@ export const NOVNC_HTML = `<!doctype html><html><head><meta charset="utf-8">
|
||||
|
||||
var msg=document.getElementById('msg');
|
||||
|
||||
// Verstecktes Eingabefeld → Software-Tastatur des Handys tippt in die VM.
|
||||
var kbd=document.createElement('input');
|
||||
kbd.setAttribute('autocomplete','off'); kbd.setAttribute('autocorrect','off');
|
||||
kbd.setAttribute('autocapitalize','off'); kbd.spellcheck=false;
|
||||
kbd.style.cssText='position:absolute;left:-1000px;top:0;width:1px;height:1px;opacity:0;';
|
||||
document.body.appendChild(kbd);
|
||||
var SPECIAL={Enter:0xff0d,Backspace:0xff08,Tab:0xff09,Escape:0xff1b,Delete:0xffff,
|
||||
ArrowLeft:0xff51,ArrowUp:0xff52,ArrowRight:0xff53,ArrowDown:0xff54,Home:0xff50,End:0xff57};
|
||||
// Tastatur laeuft NICHT mehr ueber ein verstecktes WebView-Feld (Android
|
||||
// oeffnet die Software-Tastatur dafuer unzuverlaessig). Stattdessen haelt die
|
||||
// App ein echtes RN-<TextInput> und ruft window.ariaVncKey.* per
|
||||
// injectJavaScript auf → wird unten (nach RFB-Init) definiert.
|
||||
// cp<0x100 → Keysym == Codepoint (Latin-1)
|
||||
// sonst → X11-Unicode-Keysym 0x01000000+cp
|
||||
function cpToKeysym(cp){ return cp < 0x100 ? cp : 0x01000000 + cp; }
|
||||
|
||||
import('https://cdn.jsdelivr.net/npm/@novnc/novnc@1.4.0/core/rfb.js').then(function(mod){
|
||||
var RFB = mod.default;
|
||||
@@ -99,19 +98,29 @@ export const NOVNC_HTML = `<!doctype html><html><head><meta charset="utf-8">
|
||||
});
|
||||
window.__rfb = rfb;
|
||||
|
||||
// Tasten aus dem versteckten Feld an die VM schicken.
|
||||
// Down+Up einer Taste an die VM schicken.
|
||||
function tap(keysym, code){ try{ rfb.sendKey(keysym, code||null, true); rfb.sendKey(keysym, code||null, false); }catch(_){} }
|
||||
kbd.addEventListener('keydown', function(e){
|
||||
if(SPECIAL[e.key]!==undefined){ tap(SPECIAL[e.key], e.code); e.preventDefault(); }
|
||||
});
|
||||
kbd.addEventListener('input', function(){
|
||||
var v=kbd.value; for(var i=0;i<v.length;i++){ tap(v.charCodeAt(i)); } kbd.value='';
|
||||
});
|
||||
|
||||
// Empfaenger-API: die App (RN-<TextInput> + Sondertasten-Leiste) ruft das
|
||||
// per injectJavaScript.
|
||||
// char(cp) druckbares Zeichen (Codepoint)
|
||||
// keysym(ks) Sondertaste als fertiges X11-Keysym (Enter/Esc/F1/…)
|
||||
// combo(mods,ks) Modifier(-Keysyms) halten → Taste → wieder loslassen
|
||||
// (Strg+C, Strg+Alt+Entf, …). mods = Array von Keysyms.
|
||||
window.ariaVncKey = {
|
||||
char: function(cp){ tap(cpToKeysym(cp)); },
|
||||
keysym: function(ks){ tap(ks); },
|
||||
combo: function(mods, ks){
|
||||
try{
|
||||
for(var i=0;i<mods.length;i++) rfb.sendKey(mods[i], null, true);
|
||||
rfb.sendKey(ks, null, true); rfb.sendKey(ks, null, false);
|
||||
for(var j=mods.length-1;j>=0;j--) rfb.sendKey(mods[j], null, false);
|
||||
}catch(_){}
|
||||
}
|
||||
};
|
||||
|
||||
// Steuerungs-API fuer die App (per injectJavaScript).
|
||||
window.ariaVncCtl = {
|
||||
focusKeyboard: function(){ try{ kbd.focus(); }catch(_){} },
|
||||
blurKeyboard: function(){ try{ kbd.blur(); }catch(_){} },
|
||||
cad: function(){ try{ rfb.sendCtrlAltDel(); }catch(_){} },
|
||||
toggleFit: function(){ fit=!fit; rfb.scaleViewport=fit; rfb.clipViewport=!fit; post({event:'vnc_fit', fit:fit}); }
|
||||
};
|
||||
|
||||
@@ -26,6 +26,7 @@ const DesktopTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
const [shot, setShot] = useState<{ name: string; b64: string } | null>(null);
|
||||
|
||||
const load = useCallback(() => {
|
||||
if (!projectId) { setVms([]); setErr(''); setLoading(false); return; }
|
||||
setLoading(true); setErr('');
|
||||
brainApi.listProjectVms(projectId)
|
||||
.then(r => setVms(r.vms || []))
|
||||
@@ -71,21 +72,6 @@ const DesktopTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
);
|
||||
}
|
||||
|
||||
// Verbunden → noVNC-Ansicht der VM + Zurück-Leiste.
|
||||
if (connected) {
|
||||
return (
|
||||
<View style={styles.container}>
|
||||
<View style={styles.bar}>
|
||||
<TouchableOpacity onPress={() => setConnected(null)} style={styles.barBtn}>
|
||||
<Text style={styles.barBtnText}>‹ VMs</Text>
|
||||
</TouchableOpacity>
|
||||
<Text style={styles.barTitle} numberOfLines={1}>{connected.name} · :{connected.vnc_display}</Text>
|
||||
</View>
|
||||
<VncTile projectId={projectId} focused port={connected.vnc_port || (5900 + (connected.vnc_display || 1))} />
|
||||
</View>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<View style={styles.container}>
|
||||
<View style={styles.bar}>
|
||||
@@ -97,6 +83,8 @@ const DesktopTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
<ActivityIndicator color="#0096FF" style={{ marginTop: 20 }} />
|
||||
) : err ? (
|
||||
<Text style={styles.err}>{err}</Text>
|
||||
) : !projectId ? (
|
||||
<Text style={styles.empty}>Kein aktives Projekt — wechsle in ein Projekt für dessen VMs.</Text>
|
||||
) : vms.length === 0 ? (
|
||||
<Text style={styles.empty}>
|
||||
Noch keine VM in diesem Projekt.{'\n'}
|
||||
@@ -157,6 +145,18 @@ const DesktopTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
<Text style={styles.shotHint}>Tippen zum Schließen</Text>
|
||||
</TouchableOpacity>
|
||||
</Modal>
|
||||
|
||||
{/* Vollbild-VNC — randlos ueber das ganze Display (Header + Dock weg). */}
|
||||
{connected && (
|
||||
<Modal visible animationType="slide" onRequestClose={() => setConnected(null)} supportedOrientations={['portrait', 'landscape']}>
|
||||
<View style={styles.fs}>
|
||||
<VncTile projectId={projectId} focused port={connected.vnc_port || (5900 + (connected.vnc_display || 1))} />
|
||||
<TouchableOpacity style={styles.fsBack} onPress={() => setConnected(null)} activeOpacity={0.8}>
|
||||
<Text style={styles.fsBackText}>‹ VMs</Text>
|
||||
</TouchableOpacity>
|
||||
</View>
|
||||
</Modal>
|
||||
)}
|
||||
</View>
|
||||
);
|
||||
};
|
||||
@@ -180,6 +180,9 @@ const styles = StyleSheet.create({
|
||||
vmBtns: { flexDirection: 'row', gap: 6, alignItems: 'center' },
|
||||
vmBtn: { borderWidth: 1, borderRadius: 8, paddingHorizontal: 10, paddingVertical: 6, minWidth: 34, alignItems: 'center' },
|
||||
vmBtnText: { fontSize: 12, fontWeight: '700' },
|
||||
fs: { flex: 1, backgroundColor: '#000000' },
|
||||
fsBack: { position: 'absolute', top: 34, left: 10, backgroundColor: 'rgba(18,18,42,0.9)', borderColor: '#2A2A3E', borderWidth: 1, borderRadius: 10, paddingHorizontal: 12, paddingVertical: 7 },
|
||||
fsBackText: { color: '#0096FF', fontSize: 14, fontWeight: '700' },
|
||||
shotOverlay: { flex: 1, backgroundColor: 'rgba(0,0,0,0.92)', alignItems: 'center', justifyContent: 'center', padding: 12 },
|
||||
shotTitle: { color: '#E0E0F0', fontSize: 14, fontWeight: '700', marginBottom: 10 },
|
||||
shotImg: { width: '100%', height: '78%', backgroundColor: '#000' },
|
||||
|
||||
@@ -43,6 +43,7 @@ const FilesTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
const [previewBusy, setPreviewBusy] = useState('');
|
||||
|
||||
const load = useCallback(() => {
|
||||
if (!projectId) { setFiles([]); setErr(''); setLoading(false); return; }
|
||||
setLoading(true); setErr('');
|
||||
brainApi.listProjectFiles(projectId)
|
||||
.then(r => setFiles((r.files || []).slice().sort((a, b) => a.path.localeCompare(b.path))))
|
||||
@@ -88,7 +89,7 @@ const FilesTile: React.FC<Props> = ({ projectId, focused }) => {
|
||||
) : err ? (
|
||||
<Text style={styles.err}>{err}</Text>
|
||||
) : files.length === 0 ? (
|
||||
<Text style={styles.empty}>Noch keine Dateien in diesem Projekt.</Text>
|
||||
<Text style={styles.empty}>{!projectId ? 'Kein aktives Projekt — wechsle in ein Projekt für dessen Dateien.' : 'Noch keine Dateien in diesem Projekt.'}</Text>
|
||||
) : (
|
||||
files.map(f => (
|
||||
<TouchableOpacity key={f.path} onPress={() => open(f)} style={styles.row} disabled={previewBusy === f.path}>
|
||||
|
||||
@@ -2,13 +2,16 @@
|
||||
* VncTile — Live-Desktop der QEMU-VM (noVNC in einer WebView, RFB durch RVS).
|
||||
*
|
||||
* Nur aktiv, wenn das Desktop-Panel offen ist (focused): dann WebView mounten,
|
||||
* bei 'ready' den RVS-VNC-Tunnel oeffnen. Eine kleine Steuerungs-Leiste macht
|
||||
* die VM auf dem Handy bedienbar: Tastatur einblenden (tippt in die VM),
|
||||
* Strg-Alt-Entf, und Fit ↔ 1:1 umschalten.
|
||||
* bei 'ready' den RVS-VNC-Tunnel oeffnen. Zwei Bedien-Leisten machen die VM auf
|
||||
* dem Handy voll bedienbar:
|
||||
* - ctlBar (oben rechts): Fn-Leiste ein/aus, Software-Tastatur, Fit ↔ 1:1.
|
||||
* - keyBar (oben, Fn): echte Steuertasten, die keine Software-Tastatur
|
||||
* liefert — Esc, Tab, Pfeile, Pos1/Ende/Bild, Einfg/Entf, Enter, F1–F12 und
|
||||
* Sticky-Modifier Strg/Alt/Shift (fuer Strg+C, Strg+Alt+Entf, …).
|
||||
*/
|
||||
|
||||
import React, { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { StyleSheet, Text, TouchableOpacity, View } from 'react-native';
|
||||
import { Keyboard, NativeSyntheticEvent, ScrollView, StyleSheet, Text, TextInput, TextInputChangeEventData, TextInputKeyPressEventData, TouchableOpacity, View } from 'react-native';
|
||||
import { WebView, WebViewMessageEvent } from 'react-native-webview';
|
||||
import desktop from '../../services/desktop';
|
||||
import { NOVNC_HTML } from '../assets/novncHtml';
|
||||
@@ -19,9 +22,30 @@ interface Props {
|
||||
port?: number; // VNC-Port der zu verbindenden VM (Default 5901 = Display :1)
|
||||
}
|
||||
|
||||
// X11-Keysyms fuer Sondertasten, die kein druckbares Zeichen liefern.
|
||||
const KEYSYM = { Backspace: 0xff08, Enter: 0xff0d, Tab: 0xff09 };
|
||||
const MOD = { ctrl: 0xffe3, alt: 0xffe9, shift: 0xffe1 };
|
||||
const cpToKeysym = (cp: number) => (cp < 0x100 ? cp : 0x01000000 + cp);
|
||||
|
||||
// Sondertasten fuer die Fn-Leiste (Label → Keysym).
|
||||
const NAV_KEYS: { label: string; ks: number }[] = [
|
||||
{ label: 'Esc', ks: 0xff1b }, { label: 'Tab', ks: 0xff09 },
|
||||
{ label: '←', ks: 0xff51 }, { label: '↑', ks: 0xff52 }, { label: '↓', ks: 0xff54 }, { label: '→', ks: 0xff53 },
|
||||
{ label: 'Pos1', ks: 0xff50 }, { label: 'Ende', ks: 0xff57 },
|
||||
{ label: 'Bild↑', ks: 0xff55 }, { label: 'Bild↓', ks: 0xff56 },
|
||||
{ label: 'Einfg', ks: 0xff63 }, { label: 'Entf', ks: 0xffff }, { label: '⏎', ks: 0xff0d },
|
||||
];
|
||||
const F_KEYS: { label: string; ks: number }[] = Array.from({ length: 12 }, (_, i) => ({ label: 'F' + (i + 1), ks: 0xffbe + i }));
|
||||
|
||||
const VncTile: React.FC<Props> = ({ projectId, focused, port = 5901 }) => {
|
||||
const webRef = useRef<WebView>(null);
|
||||
const kbdRef = useRef<TextInput>(null);
|
||||
const bufRef = useRef(''); // Spiegel des TextInput-Textes
|
||||
const [status, setStatus] = useState<'idle' | 'connecting' | 'connected' | 'disconnected'>('idle');
|
||||
const [kbdOn, setKbdOn] = useState(false);
|
||||
const [keyBar, setKeyBar] = useState(false); // Fn-Leiste sichtbar?
|
||||
const [mods, setMods] = useState({ ctrl: false, alt: false, shift: false });
|
||||
const modRef = useRef({ ctrl: false, alt: false, shift: false }); // Spiegel fuer Closures
|
||||
const unsubDataRef = useRef<null | (() => void)>(null);
|
||||
|
||||
const teardown = useCallback(() => {
|
||||
@@ -34,10 +58,87 @@ const VncTile: React.FC<Props> = ({ projectId, focused, port = 5901 }) => {
|
||||
return () => teardown();
|
||||
}, [focused, teardown]);
|
||||
|
||||
// Button-Zustand an die ECHTE Tastatur-Sichtbarkeit koppeln: Androids
|
||||
// Zurueck-Taste blendet die Tastatur aus, ohne den TextInput zu blurren —
|
||||
// ueber keyboardDidHide setzen wir das ⌨-Symbol trotzdem zurueck.
|
||||
useEffect(() => {
|
||||
if (!focused) return;
|
||||
const show = Keyboard.addListener('keyboardDidShow', () => setKbdOn(true));
|
||||
const hide = Keyboard.addListener('keyboardDidHide', () => setKbdOn(false));
|
||||
return () => { show.remove(); hide.remove(); };
|
||||
}, [focused]);
|
||||
|
||||
const ctl = useCallback((fn: string) => {
|
||||
webRef.current?.injectJavaScript(`window.ariaVncCtl && window.ariaVncCtl.${fn}(); true;`);
|
||||
}, []);
|
||||
|
||||
const sendKeysym = useCallback((ks: number) => {
|
||||
webRef.current?.injectJavaScript(`window.ariaVncKey && window.ariaVncKey.keysym(${ks}); true;`);
|
||||
}, []);
|
||||
const sendCombo = useCallback((modKeysyms: number[], ks: number) => {
|
||||
webRef.current?.injectJavaScript(`window.ariaVncKey && window.ariaVncKey.combo(${JSON.stringify(modKeysyms)}, ${ks}); true;`);
|
||||
}, []);
|
||||
|
||||
// Aktive Sticky-Modifier als Keysym-Liste; nach dem Anwenden one-shot zuruecksetzen.
|
||||
const activeMods = useCallback(() => {
|
||||
const m = modRef.current; const a: number[] = [];
|
||||
if (m.ctrl) a.push(MOD.ctrl); if (m.alt) a.push(MOD.alt); if (m.shift) a.push(MOD.shift);
|
||||
return a;
|
||||
}, []);
|
||||
const clearMods = useCallback(() => {
|
||||
if (modRef.current.ctrl || modRef.current.alt || modRef.current.shift) {
|
||||
modRef.current = { ctrl: false, alt: false, shift: false };
|
||||
setMods(modRef.current);
|
||||
}
|
||||
}, []);
|
||||
const toggleMod = useCallback((k: 'ctrl' | 'alt' | 'shift') => {
|
||||
modRef.current = { ...modRef.current, [k]: !modRef.current[k] };
|
||||
setMods(modRef.current);
|
||||
}, []);
|
||||
|
||||
// Eine Taste (fertiges Keysym) senden — mit ggf. aktiven Modifiern.
|
||||
const pressKey = useCallback((ks: number) => {
|
||||
const m = activeMods();
|
||||
if (m.length) { sendCombo(m, ks); clearMods(); } else sendKeysym(ks);
|
||||
}, [activeMods, sendCombo, clearMods, sendKeysym]);
|
||||
|
||||
// Ein druckbares Zeichen senden — mit ggf. aktiven Modifiern (Strg+C etc.).
|
||||
const pressChar = useCallback((cp: number) => {
|
||||
const m = activeMods();
|
||||
if (m.length) { sendCombo(m, cpToKeysym(cp)); clearMods(); }
|
||||
else webRef.current?.injectJavaScript(`window.ariaVncKey && window.ariaVncKey.char(${cp}); true;`);
|
||||
}, [activeMods, sendCombo, clearMods]);
|
||||
|
||||
// Tastatur ein-/ausblenden. Oeffnen: blur→focus erzwingt das Aufklappen auch
|
||||
// dann, wenn der TextInput noch fokussiert ist (Tastatur per Zurueck-Taste
|
||||
// versteckt). Schliessen: Keyboard.dismiss(); den Button-Zustand setzt der
|
||||
// keyboardDidShow/Hide-Listener — nicht hier —, damit er nie „haengen" bleibt.
|
||||
const toggleKbd = useCallback(() => {
|
||||
if (kbdOn) { Keyboard.dismiss(); }
|
||||
else { kbdRef.current?.blur(); setTimeout(() => kbdRef.current?.focus(), 30); }
|
||||
}, [kbdOn]);
|
||||
|
||||
// Druckbare Zeichen: Prefix-Diff des (wachsenden) Feldes → nur neu Getipptes an
|
||||
// die VM. Loeschungen kommen ueber onKeyPress(Backspace), daher hier nur Inserts.
|
||||
const onKbdChange = useCallback((e: NativeSyntheticEvent<TextInputChangeEventData>) => {
|
||||
const text = e.nativeEvent.text || '';
|
||||
const prev = bufRef.current;
|
||||
let i = 0;
|
||||
const min = Math.min(prev.length, text.length);
|
||||
while (i < min && prev.charCodeAt(i) === text.charCodeAt(i)) i++;
|
||||
for (const ch of text.slice(i)) { const cp = ch.codePointAt(0); if (cp) pressChar(cp); }
|
||||
bufRef.current = text;
|
||||
if (text.length > 200) { bufRef.current = ''; kbdRef.current?.setNativeProps({ text: '' }); }
|
||||
}, [pressChar]);
|
||||
|
||||
// Sondertasten der Software-Tastatur: Backspace feuert auf Android zuverlaessig
|
||||
// als keyPress; die Return-Taste (Haken) kommt als onSubmitEditing (s.u.).
|
||||
const onKbdKeyPress = useCallback((e: NativeSyntheticEvent<TextInputKeyPressEventData>) => {
|
||||
const k = e.nativeEvent.key;
|
||||
if (k === 'Backspace') pressKey(KEYSYM.Backspace);
|
||||
else if (k === 'Enter') pressKey(KEYSYM.Enter);
|
||||
}, [pressKey]);
|
||||
|
||||
const onMessage = useCallback((e: WebViewMessageEvent) => {
|
||||
let m: any;
|
||||
try { m = JSON.parse(e.nativeEvent.data); } catch { return; }
|
||||
@@ -84,14 +185,34 @@ const VncTile: React.FC<Props> = ({ projectId, focused, port = 5901 }) => {
|
||||
keyboardDisplayRequiresUserAction={false}
|
||||
/>
|
||||
|
||||
{/* Verstecktes Eingabefeld: fokussiert → Android-Tastatur tippt in die VM.
|
||||
keyboardType=visible-password schaltet Autokorrektur/Vorschlaege ab und
|
||||
liefert saubere Einzelzeichen. Offscreen, aber fokussierbar. */}
|
||||
<TextInput
|
||||
ref={kbdRef}
|
||||
style={styles.hiddenInput}
|
||||
onChange={onKbdChange}
|
||||
onKeyPress={onKbdKeyPress}
|
||||
onSubmitEditing={() => pressKey(KEYSYM.Enter)}
|
||||
keyboardType="visible-password"
|
||||
returnKeyType="send"
|
||||
autoCapitalize="none"
|
||||
autoCorrect={false}
|
||||
spellCheck={false}
|
||||
blurOnSubmit={false}
|
||||
caretHidden
|
||||
contextMenuHidden
|
||||
multiline={false}
|
||||
/>
|
||||
|
||||
{/* Steuerungs-Leiste — nur wenn verbunden */}
|
||||
{connected && (
|
||||
<View style={styles.ctlBar}>
|
||||
<TouchableOpacity style={styles.ctlBtn} onPress={() => ctl('focusKeyboard')} activeOpacity={0.7}>
|
||||
<Text style={styles.ctlText}>⌨</Text>
|
||||
<TouchableOpacity style={[styles.ctlBtn, keyBar && styles.ctlBtnOn]} onPress={() => setKeyBar(v => !v)} activeOpacity={0.7}>
|
||||
<Text style={styles.ctlText}>Fn</Text>
|
||||
</TouchableOpacity>
|
||||
<TouchableOpacity style={styles.ctlBtn} onPress={() => ctl('cad')} activeOpacity={0.7}>
|
||||
<Text style={styles.ctlTextSmall}>Strg+Alt+Entf</Text>
|
||||
<TouchableOpacity style={[styles.ctlBtn, kbdOn && styles.ctlBtnOn]} onPress={toggleKbd} activeOpacity={0.7}>
|
||||
<Text style={styles.ctlText}>⌨</Text>
|
||||
</TouchableOpacity>
|
||||
<TouchableOpacity style={styles.ctlBtn} onPress={() => ctl('toggleFit')} activeOpacity={0.7}>
|
||||
<Text style={styles.ctlText}>⤢</Text>
|
||||
@@ -99,6 +220,26 @@ const VncTile: React.FC<Props> = ({ projectId, focused, port = 5901 }) => {
|
||||
</View>
|
||||
)}
|
||||
|
||||
{/* Fn-Leiste — echte Steuertasten (oben, ueber der Software-Tastatur). */}
|
||||
{connected && keyBar && (
|
||||
<View style={styles.keyBar} pointerEvents="box-none">
|
||||
<ScrollView horizontal showsHorizontalScrollIndicator={false} keyboardShouldPersistTaps="always" contentContainerStyle={styles.keyRow}>
|
||||
<TouchableOpacity style={[styles.key, mods.ctrl && styles.keyOn]} onPress={() => toggleMod('ctrl')} activeOpacity={0.7}><Text style={styles.keyText}>Strg</Text></TouchableOpacity>
|
||||
<TouchableOpacity style={[styles.key, mods.alt && styles.keyOn]} onPress={() => toggleMod('alt')} activeOpacity={0.7}><Text style={styles.keyText}>Alt</Text></TouchableOpacity>
|
||||
<TouchableOpacity style={[styles.key, mods.shift && styles.keyOn]} onPress={() => toggleMod('shift')} activeOpacity={0.7}><Text style={styles.keyText}>Shift</Text></TouchableOpacity>
|
||||
{NAV_KEYS.map(k => (
|
||||
<TouchableOpacity key={k.label} style={styles.key} onPress={() => pressKey(k.ks)} activeOpacity={0.7}><Text style={styles.keyText}>{k.label}</Text></TouchableOpacity>
|
||||
))}
|
||||
</ScrollView>
|
||||
<ScrollView horizontal showsHorizontalScrollIndicator={false} keyboardShouldPersistTaps="always" contentContainerStyle={styles.keyRow}>
|
||||
{F_KEYS.map(k => (
|
||||
<TouchableOpacity key={k.label} style={styles.key} onPress={() => pressKey(k.ks)} activeOpacity={0.7}><Text style={styles.keyText}>{k.label}</Text></TouchableOpacity>
|
||||
))}
|
||||
<TouchableOpacity style={styles.key} onPress={() => ctl('cad')} activeOpacity={0.7}><Text style={styles.keyTextSm}>Strg+Alt+Entf</Text></TouchableOpacity>
|
||||
</ScrollView>
|
||||
</View>
|
||||
)}
|
||||
|
||||
{!connected && (
|
||||
<View style={styles.overlay} pointerEvents="none">
|
||||
<Text style={styles.overlayText}>
|
||||
@@ -119,7 +260,7 @@ const styles = StyleSheet.create({
|
||||
sub: { color: '#9090B0', fontSize: 14, marginTop: 8 },
|
||||
ctlBar: {
|
||||
position: 'absolute',
|
||||
top: 8,
|
||||
top: 34,
|
||||
right: 8,
|
||||
flexDirection: 'row',
|
||||
gap: 6,
|
||||
@@ -135,8 +276,21 @@ const styles = StyleSheet.create({
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
},
|
||||
ctlBtnOn: { backgroundColor: 'rgba(0,150,255,0.85)', borderColor: '#0096FF' },
|
||||
ctlText: { color: '#E0E0F0', fontSize: 16, fontWeight: '700' },
|
||||
ctlTextSmall: { color: '#E0E0F0', fontSize: 11, fontWeight: '700' },
|
||||
// Fokussierbar (nicht display:none), aber aus dem Sichtfeld geschoben.
|
||||
hiddenInput: { position: 'absolute', width: 1, height: 1, top: -100, left: -100, opacity: 0, padding: 0 },
|
||||
keyBar: { position: 'absolute', top: 74, left: 0, right: 0, gap: 5 },
|
||||
keyRow: { paddingHorizontal: 6, gap: 5, alignItems: 'center' },
|
||||
key: {
|
||||
backgroundColor: 'rgba(18,18,42,0.92)', borderColor: '#2A2A3E', borderWidth: 1,
|
||||
borderRadius: 8, paddingHorizontal: 9, paddingVertical: 7, minWidth: 34,
|
||||
alignItems: 'center', justifyContent: 'center',
|
||||
},
|
||||
keyOn: { backgroundColor: 'rgba(0,150,255,0.85)', borderColor: '#0096FF' },
|
||||
keyText: { color: '#E0E0F0', fontSize: 13, fontWeight: '700' },
|
||||
keyTextSm: { color: '#E0E0F0', fontSize: 10, fontWeight: '700' },
|
||||
overlay: { position: 'absolute', top: 12, left: 0, right: 0, alignItems: 'center' },
|
||||
overlayText: { color: '#9090B0', fontSize: 12, backgroundColor: 'rgba(0,0,0,0.6)', paddingHorizontal: 10, paddingVertical: 4, borderRadius: 10, overflow: 'hidden' },
|
||||
});
|
||||
|
||||
+104
-6
@@ -627,10 +627,11 @@ META_TOOLS = [
|
||||
"name": "request_location_tracking",
|
||||
"description": (
|
||||
"Bittet die App, das kontinuierliche GPS-Tracking zu aktivieren oder zu "
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). Nutze das wenn du einen "
|
||||
"GPS-basierten Watcher anlegst (z.B. `near(...)`), sonst hat die App "
|
||||
"veraltete Position und der Watcher feuert nie. Auch wieder ausschalten "
|
||||
"wenn der letzte GPS-Watcher geloescht wurde."
|
||||
"deaktivieren. Default ist AUS (Akku-Schutz). HINWEIS: Beim Anlegen/Loeschen "
|
||||
"eines Standort-Watchers (`near/entered_near/left_near`) schaltet das System "
|
||||
"das Tracking bereits AUTOMATISCH mit an bzw. aus — dieses Tool brauchst du "
|
||||
"dafuer nicht mehr. Nutze es nur fuer manuelle/explizite Faelle (z.B. Tracking "
|
||||
"kurz einschalten ohne Watcher)."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -642,6 +643,48 @@ META_TOOLS = [
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "present_view",
|
||||
"description": (
|
||||
"Komponiere eine RAEUMLICHE Ansicht, die auf ARIAs Flaeche materialisiert "
|
||||
"(Orb + Karten). Nutze das, wenn eine visuelle Anordnung besser ist als reiner "
|
||||
"Text — z.B. eine Akte/Datei zeigen (Bild links, Text rechts, Karte unten), "
|
||||
"einen Vergleich, eine Liste, eine Karte mit Orten, oder Code. NICHT fuer "
|
||||
"normale Gespraechsantworten. Die Karten erscheinen ZUSAETZLICH zu deiner "
|
||||
"(kurzen) Sprachantwort — halte die Antwort dann knapp, das Visuelle traegt."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cards": {
|
||||
"type": "array",
|
||||
"description": "Die Karten der Ansicht, in sinnvoller Anordnung.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {"type": "string", "enum": ["text", "image", "map", "code", "list"],
|
||||
"description": "Kartentyp"},
|
||||
"title": {"type": "string", "description": "Optionaler Kartentitel"},
|
||||
"md": {"type": "string", "description": "text/list: Markdown-Inhalt"},
|
||||
"src": {"type": "string", "description": "image: Bild-URL oder /shared-Pfad"},
|
||||
"markers": {"type": "array", "items": {"type": "object"},
|
||||
"description": "map: Liste [{lat, lon, label}]"},
|
||||
"path": {"type": "string", "description": "code: Datei-Pfad im Projekt"},
|
||||
"lang": {"type": "string", "description": "code: Sprache fuers Highlighting"},
|
||||
},
|
||||
"required": ["type"],
|
||||
},
|
||||
},
|
||||
"orb": {"type": "string", "enum": ["idle", "speaking", "working"],
|
||||
"description": "Orb-Zustand nach dem Rendern (default: speaking)"},
|
||||
"title": {"type": "string", "description": "Optionaler Titel der ganzen Ansicht"},
|
||||
},
|
||||
"required": ["cards"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
@@ -1355,6 +1398,17 @@ def _skill_to_tool(s: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# Standort-Funktionen in Watcher-Conditions — brauchen aktives GPS-Tracking.
|
||||
_LOCATION_FUNCS = ("near(", "entered_near(", "left_near(")
|
||||
|
||||
|
||||
def _condition_needs_location(condition: str) -> bool:
|
||||
"""True, wenn die Watcher-Condition eine Standort-Funktion nutzt und damit
|
||||
laufende GPS-Updates der App braucht."""
|
||||
c = condition or ""
|
||||
return any(fn in c for fn in _LOCATION_FUNCS)
|
||||
|
||||
|
||||
class Agent:
|
||||
# Mindest-Score den ein Cold-Memory-Treffer haben muss um in den
|
||||
# System-Prompt aufgenommen zu werden. Unter dieser Schwelle ist's
|
||||
@@ -2372,13 +2426,41 @@ class Agent:
|
||||
"trigger": {"name": t["name"], "type": "watcher",
|
||||
"condition": t["condition"], "message": t["message"]},
|
||||
})
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'."
|
||||
# GPS-Kopplung: ein Standort-Watcher (near/entered_near/left_near)
|
||||
# ist tot, wenn die App kein Tracking sendet. Frueher musste ARIA
|
||||
# separat request_location_tracking aufrufen — wurde oft vergessen,
|
||||
# dann feuerte der Trigger nie. Jetzt automatisch mit-anschalten.
|
||||
extra = ""
|
||||
if _condition_needs_location(t["condition"]):
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": True,
|
||||
"reason": f"watcher:{t['name']}",
|
||||
})
|
||||
extra = " GPS-Tracking automatisch aktiviert."
|
||||
return f"OK — Watcher '{t['name']}' angelegt: feuert wenn '{t['condition']}'.{extra}"
|
||||
if name == "trigger_cancel":
|
||||
try:
|
||||
triggers_mod.delete(arguments["name"])
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht."
|
||||
except ValueError as e:
|
||||
return f"FEHLER: {e}"
|
||||
# GPS-Kopplung (Gegenstueck): existiert kein Standort-Watcher mehr,
|
||||
# Tracking wieder ausschalten (Akku schonen).
|
||||
extra = ""
|
||||
remaining = triggers_mod.list_triggers(active_only=False)
|
||||
still_needs_gps = any(
|
||||
r.get("type") == "watcher"
|
||||
and _condition_needs_location(r.get("condition") or "")
|
||||
for r in remaining
|
||||
)
|
||||
if not still_needs_gps:
|
||||
self._pending_events.append({
|
||||
"type": "location_tracking",
|
||||
"on": False,
|
||||
"reason": "no-location-watchers",
|
||||
})
|
||||
extra = " GPS-Tracking deaktiviert (kein Standort-Watcher mehr)."
|
||||
return f"OK — Trigger '{arguments['name']}' geloescht.{extra}"
|
||||
if name == "request_location_tracking":
|
||||
on = bool(arguments.get("on", False))
|
||||
reason = (arguments.get("reason") or "").strip()
|
||||
@@ -2388,6 +2470,22 @@ class Agent:
|
||||
"reason": reason,
|
||||
})
|
||||
return f"OK — Tracking-Request gesendet (on={on}). App wird in Kuerze umschalten."
|
||||
if name == "present_view":
|
||||
cards = arguments.get("cards") or []
|
||||
if not isinstance(cards, list) or not cards:
|
||||
return "FEHLER: present_view braucht mindestens eine Karte in 'cards'."
|
||||
view = {
|
||||
"cards": cards,
|
||||
"orb": arguments.get("orb") or "speaking",
|
||||
"title": arguments.get("title") or "",
|
||||
}
|
||||
self._pending_events.append({
|
||||
"type": "aria_view",
|
||||
"view": view,
|
||||
"project_id": project_id or "",
|
||||
})
|
||||
return (f"OK — Ansicht mit {len(cards)} Karte(n) an die Flaeche geschickt "
|
||||
f"(Kontext: {project_id or 'Hauptchat'}).")
|
||||
if name == "trigger_list":
|
||||
items = triggers_mod.list_triggers(active_only=False)
|
||||
if not items:
|
||||
|
||||
@@ -149,14 +149,23 @@ async def _fire(trigger: dict, agent_factory) -> None:
|
||||
)
|
||||
|
||||
try:
|
||||
agent = agent_factory()
|
||||
reply, _, _, _, _ = agent.chat(prompt, source="trigger")
|
||||
events = agent.pop_events()
|
||||
# WICHTIG: agent.chat() ist ein SYNCHRONER, blockierender Aufruf (Proxy-
|
||||
# HTTP mit bis zu 24h Read-Timeout). NIEMALS direkt im async-Loop —
|
||||
# sonst friert ein einziger getriggerter Turn den GESAMTEN Brain ein
|
||||
# (kein /health, kein weiterer Request). Wie der /chat-Pfad in den
|
||||
# Executor auslagern, damit der Event-Loop frei bleibt.
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _run_turn():
|
||||
a = agent_factory()
|
||||
rep, *_rest = a.chat(prompt, source="trigger")
|
||||
return rep, a.pop_events()
|
||||
|
||||
reply, events = await loop.run_in_executor(None, _run_turn)
|
||||
logger.info("[trigger] %s gefeuert → ARIA-Reply: %s", name, reply[:80])
|
||||
triggers_mod.append_log(name, {"event": "reply", "text": reply[:500]})
|
||||
# Reply an die Bridge pushen, damit App + Diagnostic + TTS sie kriegen.
|
||||
# Ohne diesen Push wuerde die Antwort nur im Brain-Log landen.
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(None, _push_to_bridge, reply, name, ttype, events)
|
||||
except Exception as e:
|
||||
logger.exception("Trigger %s feuern fehlgeschlagen: %s", name, e)
|
||||
|
||||
+29
-23
@@ -963,17 +963,21 @@ def _docker_gateway() -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _ssh_aria_vm(*args: str, timeout: int = 25):
|
||||
def _ssh_host(*cmd: str, timeout: int = 25):
|
||||
import subprocess
|
||||
cmd = ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
|
||||
_ARIA_VM_HOST, "aria-vm", *[str(a) for a in args]]
|
||||
full = ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
|
||||
_ARIA_VM_HOST, *[str(c) for c in cmd]]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
r = subprocess.run(full, capture_output=True, text=True, timeout=timeout)
|
||||
return r.returncode, r.stdout or "", r.stderr or ""
|
||||
except Exception as exc:
|
||||
return 1, "", str(exc)
|
||||
|
||||
|
||||
def _ssh_aria_vm(*args: str, timeout: int = 25):
|
||||
return _ssh_host("aria-vm", *args, timeout=timeout)
|
||||
|
||||
|
||||
def _vm_running_names() -> set:
|
||||
rc, out, _err = _ssh_aria_vm("list", timeout=15)
|
||||
names = set()
|
||||
@@ -1074,11 +1078,13 @@ def project_vm_stop(project_id: str, name: str):
|
||||
|
||||
@app.post("/projects/{project_id}/vms/{name}/screenshot")
|
||||
def project_vm_screenshot(project_id: str, name: str):
|
||||
"""Macht einen Screenshot der laufenden VM (aria-vm screenshot → PNG in
|
||||
/shared/uploads) und liefert ihn als Base64 zurueck. So sieht Stefan den
|
||||
VM-Bildschirm auch ohne Live-VNC (genau wie ARIA es beim Testen macht)."""
|
||||
"""Macht einen Screenshot der laufenden VM und liefert ihn als Base64.
|
||||
|
||||
aria-vm schreibt das PNG ins VM-Verzeichnis (dem aria-User gehoerend — nicht
|
||||
ins /root-Shared-Volume, wo der aria-User keinen Zugriff hat). Der Brain holt
|
||||
die Datei danach per SSH (base64) — funktioniert unabhaengig von Volume-
|
||||
Rechten. Zusaetzlich wird das PNG ins Projekt kopiert (Dateien-Panel)."""
|
||||
import base64
|
||||
import time as _t
|
||||
rc, out, err = _ssh_aria_vm("screenshot", name, timeout=30)
|
||||
if rc != 0:
|
||||
raise HTTPException(status_code=400, detail=f"Screenshot fehlgeschlagen: {(err or out).strip()[:200]}")
|
||||
@@ -1088,28 +1094,28 @@ def project_vm_screenshot(project_id: str, name: str):
|
||||
path = line.split("=", 1)[1].strip()
|
||||
if not path:
|
||||
raise HTTPException(status_code=500, detail=f"Kein Screenshot-Pfad: {out.strip()[:200]}")
|
||||
local = os.path.join("/shared/uploads", os.path.basename(path))
|
||||
for _ in range(10): # kurze Bind-Mount-Latenz abfangen
|
||||
if os.path.isfile(local):
|
||||
break
|
||||
_t.sleep(0.2)
|
||||
if not os.path.isfile(local):
|
||||
raise HTTPException(status_code=500, detail=f"Screenshot-Datei nicht gefunden: {local}")
|
||||
with open(local, "rb") as f:
|
||||
data = f.read()
|
||||
# Auch ins Projekt kopieren → taucht im Dateien-Panel auf.
|
||||
# PNG per SSH als Base64 holen (kein Shared-Volume noetig).
|
||||
rc2, b64, err2 = _ssh_host("base64", "-w0", path, timeout=20)
|
||||
if rc2 != 0 or not b64.strip():
|
||||
raise HTTPException(status_code=500, detail=f"Screenshot konnte nicht gelesen werden: {(err2 or 'leer').strip()[:200]}")
|
||||
b64 = b64.strip()
|
||||
try:
|
||||
data = base64.b64decode(b64)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=f"Base64 ungueltig: {exc}")
|
||||
fname = os.path.basename(path)
|
||||
# Ins Projekt kopieren → taucht im Dateien-Panel auf.
|
||||
proj_rel = ""
|
||||
try:
|
||||
shots_dir = os.path.join(_project_dir(project_id), "screenshots")
|
||||
os.makedirs(shots_dir, exist_ok=True)
|
||||
with open(os.path.join(shots_dir, os.path.basename(local)), "wb") as f:
|
||||
with open(os.path.join(shots_dir, fname), "wb") as f:
|
||||
f.write(data)
|
||||
proj_rel = "screenshots/" + os.path.basename(local)
|
||||
proj_rel = "screenshots/" + fname
|
||||
except Exception:
|
||||
proj_rel = ""
|
||||
return {"ok": True, "name": name, "filename": os.path.basename(local),
|
||||
"projectPath": proj_rel,
|
||||
"base64": base64.b64encode(data).decode("ascii")}
|
||||
return {"ok": True, "name": name, "filename": fname,
|
||||
"projectPath": proj_rel, "base64": b64}
|
||||
|
||||
|
||||
@app.get("/conversation/stats")
|
||||
|
||||
+131
-23
@@ -43,6 +43,25 @@ from openwakeword.model import Model as WakeWordModel
|
||||
|
||||
from modes import Mode, canonical_id, detect_mode_switch, mode_from_id, should_speak
|
||||
|
||||
|
||||
def _docker_gateway() -> str:
|
||||
"""Docker-Gateway-IP (= Host-IP auf DIESEM Container-Netz, aria-net) aus
|
||||
/proc/net/route. Genau die IP, an die der Brain QEMUs VNC bindet — im
|
||||
Gegensatz zu host.docker.internal, das auf die Default-Bridge (docker0)
|
||||
zeigt und daher die VM nicht trifft."""
|
||||
try:
|
||||
import socket as _sock
|
||||
import struct as _struct
|
||||
with open("/proc/net/route") as f:
|
||||
for line in f.readlines()[1:]:
|
||||
fields = line.strip().split()
|
||||
if len(fields) >= 4 and fields[1] == "00000000" and int(fields[3], 16) & 2:
|
||||
return _sock.inet_ntoa(_struct.pack("<L", int(fields[2], 16)))
|
||||
except Exception:
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
# ── Logging ──────────────────────────────────────────────────
|
||||
|
||||
logging.basicConfig(
|
||||
@@ -738,7 +757,10 @@ class ARIABridge:
|
||||
# "task": asyncio.Task}. Wir bruecken rohes RFB-TCP (QEMU-VNC auf dem
|
||||
# Host) <-> RVS (vnc_data/vnc_input, Base64-in-JSON).
|
||||
self._vnc_sessions: dict[str, dict] = {}
|
||||
self._vnc_host: str = os.environ.get("ARIA_VNC_HOST", "host.docker.internal")
|
||||
# VNC-Host: das aria-net-Gateway (dort bindet der Brain QEMUs VNC).
|
||||
# host.docker.internal zeigt faelschlich auf docker0 (172.17.0.1) → refused.
|
||||
self._vnc_host: str = (os.environ.get("ARIA_VNC_HOST")
|
||||
or _docker_gateway() or "host.docker.internal")
|
||||
# Satelliten (Aussenposten in fremden Netzen). id → {location, caps,
|
||||
# control, last_seen}. Registrierung via sat_hello. _pending_sat:
|
||||
# requestId → Future (sat_devices / sat_result), analog _pending_flux.
|
||||
@@ -768,7 +790,8 @@ class ARIABridge:
|
||||
# Anfrage an aria-core. Sonst antwortet ARIA zweimal (einmal "warte auf
|
||||
# Anweisung" beim file, einmal auf den Chat-Text).
|
||||
# Liste von Tuples: (file_path, name, file_type, size_kb, width, height)
|
||||
self._pending_files: list[tuple[str, str, str, int, int, int]] = []
|
||||
# (file_path, name, type, kb, width, height, clientMsgId)
|
||||
self._pending_files: list[tuple[str, str, str, int, int, int, str]] = []
|
||||
self._pending_files_flush_task: Optional[asyncio.Task] = None
|
||||
# Projekt-Kontext der gerade gepufferten Anhaenge (aus dem file-Upload).
|
||||
# Wird beim Flush an send_to_core gegeben, damit Anhaenge im richtigen
|
||||
@@ -1817,16 +1840,16 @@ class ARIABridge:
|
||||
return " ".join(parts) + " " + text
|
||||
return text
|
||||
|
||||
def _build_pending_files_message(self, user_text: str) -> str:
|
||||
"""Baut eine Anweisung an aria-core aus den gepufferten Files + optionalem
|
||||
def _build_pending_files_message(self, user_text: str, files: list) -> str:
|
||||
"""Baut eine Anweisung an aria-core aus den uebergebenen Files + optionalem
|
||||
User-Text. user_text leer → 'warte auf Anweisung'-Variante."""
|
||||
parts: list[str] = []
|
||||
for fp, name, ftype, kb, w, h in self._pending_files:
|
||||
for fp, name, ftype, kb, w, h, _cmid in files:
|
||||
dim = f" {w}x{h}px" if (w and h) else ""
|
||||
kind = "Bild" if ftype.startswith("image/") else "Datei"
|
||||
parts.append(f"- {kind}: {name}{dim} ({ftype}, {kb}KB) liegt unter {fp}")
|
||||
files_summary = "\n".join(parts)
|
||||
n = len(self._pending_files)
|
||||
n = len(files)
|
||||
anhang = "Anhang" if n == 1 else "Anhaenge"
|
||||
if user_text:
|
||||
return (f"Stefan hat dir {n} {anhang} geschickt:\n{files_summary}\n\n"
|
||||
@@ -1835,15 +1858,16 @@ class ARIABridge:
|
||||
f"Warte auf seine Anweisung was du damit tun sollst.")
|
||||
|
||||
async def _flush_pending_files_after(self, delay: float) -> None:
|
||||
"""Wenn nach `delay`s kein chat-Text gekommen ist: Files alleine an
|
||||
aria-core senden ('warte auf Anweisung'-Variante)."""
|
||||
"""Wenn nach `delay`s kein chat-Text gekommen ist: alle noch gepufferten
|
||||
Files alleine an aria-core senden ('warte auf Anweisung'-Variante)."""
|
||||
try:
|
||||
await asyncio.sleep(delay)
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
if not self._pending_files:
|
||||
return
|
||||
text = self._build_pending_files_message("")
|
||||
files = self._pending_files
|
||||
text = self._build_pending_files_message("", files)
|
||||
self._pending_files = []
|
||||
self._pending_files_flush_task = None
|
||||
pid = self._pending_files_project_id
|
||||
@@ -1851,23 +1875,48 @@ class ARIABridge:
|
||||
await self.send_to_core(text, source="app-file", project_id=pid)
|
||||
|
||||
async def _flush_pending_files_with_text(self, user_text: str,
|
||||
project_id: str = "") -> bool:
|
||||
project_id: str = "",
|
||||
client_msg_id: str = "") -> bool:
|
||||
"""Wenn ein chat-Text reinkommt waehrend Files gepuffert sind:
|
||||
Files + Text zu einer einzigen aria-core-Nachricht mergen.
|
||||
Returns True wenn gemerged wurde (Caller soll dann nicht nochmal senden).
|
||||
|
||||
KORRELATION (Fix Queue-Bug): Files tragen dieselbe clientMsgId wie ihr
|
||||
Text. Bei einer Queue gehen Files (fire-and-forget) und Text (ACK-
|
||||
getrackt, ggf. verzoegert) auseinander — ohne Korrelation landeten die
|
||||
Bilder beim falschen Text. Wir mergen darum NUR die Files mit passender
|
||||
cmid; der Rest bleibt gepuffert fuer seine eigene Nachricht. Fallback
|
||||
(Legacy-App ohne cmid an Files, oder cmid ohne Treffer): altes Verhalten
|
||||
(alle Files mit diesem Text), damit nie ein Bild verloren geht.
|
||||
|
||||
project_id: Projekt-Kontext aus dem chat-Payload (der sichtbare Focus
|
||||
beim Absenden). Faellt auf den beim File-Upload gemerkten Kontext
|
||||
zurueck, damit Anhaenge im richtigen Projekt landen statt im Hauptchat."""
|
||||
beim Absenden). Faellt auf den beim File-Upload gemerkten Kontext zurueck.
|
||||
"""
|
||||
if not self._pending_files:
|
||||
return False
|
||||
cmid = (client_msg_id or "").strip()
|
||||
matching = [f for f in self._pending_files if cmid and f[6] == cmid]
|
||||
if not matching:
|
||||
# Kein cmid-Treffer → altes Verhalten: alle gepufferten Files mergen.
|
||||
matching = list(self._pending_files)
|
||||
remaining: list = []
|
||||
else:
|
||||
remaining = [f for f in self._pending_files if f not in matching]
|
||||
|
||||
text = self._build_pending_files_message(user_text, matching)
|
||||
self._pending_files = remaining
|
||||
pid = (project_id or "").strip() or self._pending_files_project_id
|
||||
# Flush-Timer neu setzen wenn noch Files anderer Nachrichten warten,
|
||||
# sonst zuruecksetzen.
|
||||
if self._pending_files_flush_task and not self._pending_files_flush_task.done():
|
||||
self._pending_files_flush_task.cancel()
|
||||
self._pending_files_flush_task = None
|
||||
text = self._build_pending_files_message(user_text)
|
||||
self._pending_files = []
|
||||
pid = (project_id or "").strip() or self._pending_files_project_id
|
||||
self._pending_files_project_id = ""
|
||||
if remaining:
|
||||
self._pending_files_flush_task = asyncio.create_task(
|
||||
self._flush_pending_files_after(self._PENDING_FILES_WINDOW_SEC)
|
||||
)
|
||||
else:
|
||||
self._pending_files_project_id = ""
|
||||
# create_task statt await — sonst blockt der RVS-recv-Loop bis Brain
|
||||
# fertig ist (siehe chat-handler oben).
|
||||
asyncio.create_task(self.send_to_core(text, source="app-file+chat", project_id=pid))
|
||||
@@ -1914,10 +1963,13 @@ class ARIABridge:
|
||||
url, data=payload, method="POST",
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
# 20 Min Timeout — lange Multi-Tool-Workflows (Karten,
|
||||
# PDFs, viele curl-Calls) brauchen das. 5 Min waren chronisch
|
||||
# zu knapp und haben ARIA mitten in der Arbeit gekappt.
|
||||
with urllib.request.urlopen(req, timeout=1200) as resp:
|
||||
# Timeout MUSS zum Proxy passen (der laesst ARIA bis 24h
|
||||
# rechnen). 1200s (20 Min) war zu knapp: lange Software-Dev-
|
||||
# Turns dauern laenger → die Bridge gab auf, die Antwort ging
|
||||
# verloren (kein Bubble), der Kontext blieb auf 'running'
|
||||
# haengen. Jetzt 24h (env BRAIN_CHAT_TIMEOUT_SEC).
|
||||
_chat_timeout = float(os.environ.get("BRAIN_CHAT_TIMEOUT_SEC", "86400"))
|
||||
with urllib.request.urlopen(req, timeout=_chat_timeout) as resp:
|
||||
return resp.status, resp.read().decode("utf-8", errors="ignore")
|
||||
except Exception as exc:
|
||||
return None, str(exc)
|
||||
@@ -2028,6 +2080,22 @@ class ARIABridge:
|
||||
proj = event.get("project") or {}
|
||||
logger.info("[brain] Projekt %s: %s (id=%s)",
|
||||
event.get("action") or "?", proj.get("name"), proj.get("id"))
|
||||
elif etype == "aria_view":
|
||||
# M1: ARIA hat via present_view eine raeumliche Ansicht komponiert.
|
||||
# View-Spec (Orb + Karten) + Projekt-Kontext an App/Web/Diagnostic;
|
||||
# deren Renderer materialisieren die Karten auf der Flaeche.
|
||||
view = event.get("view") or {}
|
||||
await self._send_to_rvs({
|
||||
"type": "aria_view",
|
||||
"payload": {
|
||||
"view": view,
|
||||
"projectId": event.get("project_id") or "",
|
||||
"clientMsgId": client_msg_id or "",
|
||||
},
|
||||
"timestamp": int(asyncio.get_event_loop().time() * 1000),
|
||||
})
|
||||
logger.info("[brain] ARIA hat eine Ansicht geschickt: %d Karte(n), orb=%s",
|
||||
len(view.get("cards") or []), view.get("orb"))
|
||||
|
||||
# _process_core_response uebernimmt alles weitere:
|
||||
# File-Marker extrahieren + broadcasten, NO_REPLY-Check, Chat-
|
||||
@@ -2342,7 +2410,8 @@ class ARIABridge:
|
||||
# gesendet), mergen wir sie zu einer einzigen Anfrage statt
|
||||
# zwei separater send_to_core-Calls.
|
||||
merged = await self._flush_pending_files_with_text(
|
||||
text, project_id=str(payload.get("projectId") or ""))
|
||||
text, project_id=str(payload.get("projectId") or ""),
|
||||
client_msg_id=client_msg_id or "")
|
||||
if merged:
|
||||
logger.info("[rvs] App-Chat (mit Anhaengen) project=%s: '%s'",
|
||||
str(payload.get("projectId") or "") or "(main)", text[:80])
|
||||
@@ -2384,6 +2453,20 @@ class ARIABridge:
|
||||
await self._emit_activity("idle", "", project_id=cancel_pid)
|
||||
return
|
||||
|
||||
if msg_type == "interject":
|
||||
# Zwischenruf: waehrend eines laufenden Turns eine Korrektur
|
||||
# reinschieben — KEIN Abbruch, keine Queue. Geht an den Proxy-
|
||||
# internen /interject, der die Message in den laufenden Subprozess
|
||||
# des Kontexts schreibt (claude greift sie an der naechsten Tool-
|
||||
# Grenze auf).
|
||||
interject_pid = str(payload.get("projectId") or "")
|
||||
interject_text = str(payload.get("text") or "")
|
||||
logger.info("[rvs] Zwischenruf project=%s: '%s'",
|
||||
interject_pid or "(main)", interject_text[:80])
|
||||
if interject_text.strip():
|
||||
await self._interject_proxy_for_project(interject_pid, interject_text)
|
||||
return
|
||||
|
||||
elif msg_type == "audio_pcm":
|
||||
# Audio-PCM geht direkt von XTTS-Bridge an die App.
|
||||
# Die aria-bridge darf es NICHT rebroadcasten — sonst bekommt die App
|
||||
@@ -2636,8 +2719,11 @@ class ARIABridge:
|
||||
logger.warning("[rvs] Bild-Resize fehlgeschlagen (%s) — Original wird genutzt: %s",
|
||||
file_name, e)
|
||||
|
||||
# In Pending-Queue + Flush-Timer (anti-spam Buffering)
|
||||
self._pending_files.append((file_path, file_name, file_type, size_kb, int(width or 0), int(height or 0)))
|
||||
# In Pending-Queue + Flush-Timer (anti-spam Buffering).
|
||||
# clientMsgId mitpuffern → spaeterer Text-Flush ordnet die Datei
|
||||
# genau SEINER Nachricht zu (Queue-Korrelation, s. _flush_*).
|
||||
file_cmid = str(payload.get("clientMsgId") or "")
|
||||
self._pending_files.append((file_path, file_name, file_type, size_kb, int(width or 0), int(height or 0), file_cmid))
|
||||
if self._pending_files_flush_task and not self._pending_files_flush_task.done():
|
||||
self._pending_files_flush_task.cancel()
|
||||
self._pending_files_flush_task = asyncio.create_task(
|
||||
@@ -4079,6 +4165,28 @@ class ARIABridge:
|
||||
logger.info("[cancel] proxy /cancel project=%s: %s %s",
|
||||
project_id or "(main)", status, body)
|
||||
|
||||
async def _interject_proxy_for_project(self, project_id: str, text: str) -> None:
|
||||
"""Zwischenruf: schiebt eine User-Message in den laufenden Turn dieses
|
||||
Kontexts (proxy-internes /interject) — ohne Abbruch. claude greift sie
|
||||
an der naechsten Tool-Grenze auf."""
|
||||
url = os.environ.get("PROXY_INTERNAL_URL", "http://aria-proxy:3457") + "/interject"
|
||||
data = json.dumps({"projectId": project_id or "", "text": text}).encode("utf-8")
|
||||
|
||||
def _do_request():
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
url, method="POST", data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=3) as resp:
|
||||
return resp.status, resp.read().decode("utf-8", "ignore")[:200]
|
||||
except Exception as e:
|
||||
return f"error: {e}", ""
|
||||
|
||||
status, body = await asyncio.get_event_loop().run_in_executor(None, _do_request)
|
||||
logger.info("[interject] proxy /interject project=%s: %s %s",
|
||||
project_id or "(main)", status, body)
|
||||
|
||||
async def _emit_activity(self, activity: str, tool: str = "", force: bool = False,
|
||||
project_id: str = "") -> None:
|
||||
"""Sendet agent_activity an die App — nur wenn sich der State geaendert hat.
|
||||
|
||||
+79
-1
@@ -198,7 +198,7 @@
|
||||
<div style="display:flex;align-items:center;gap:10px;flex-wrap:wrap;">
|
||||
<span id="disk-banner-icon" style="font-size:18px;">⚠️</span>
|
||||
<span id="disk-banner-text" style="flex:1;min-width:200px;font-weight:600;"></span>
|
||||
<button onclick="copyDiskCmd('safe')" class="btn secondary" style="padding:4px 10px;font-size:11px;" title="docker builder prune -a -f && docker image prune -a -f">
|
||||
<button id="disk-clean-btn" onclick="runDiskCleanup('safe')" class="btn secondary" style="padding:4px 10px;font-size:11px;" title="Fuehrt aus: docker builder prune -a && docker image prune -a (ohne Volumes)">
|
||||
Sicher aufraeumen
|
||||
</button>
|
||||
<button onclick="document.getElementById('disk-banner-aggressive').style.display=(document.getElementById('disk-banner-aggressive').style.display==='none'?'flex':'none')"
|
||||
@@ -216,6 +216,7 @@
|
||||
<div style="color:#FFAA55;">
|
||||
<b>Aggressiv</b> — zusaetzlich ungenutzte Volumes. <b>Nur wenn alle ARIA-Container laufen</b>, sonst riskierst du Daten-Verlust (Sessions, SSH-Keys, Shared):<br>
|
||||
<code style="font-family:monospace;">docker system prune -a --volumes -f</code>
|
||||
<button onclick="runDiskCleanup('aggressive')" class="btn secondary" style="padding:2px 8px;font-size:10px;margin-left:6px;border-color:#FF3B30;color:#FF3B30;">Jetzt ausfuehren</button>
|
||||
<button onclick="copyDiskCmd('aggressive')" class="btn secondary" style="padding:2px 8px;font-size:10px;margin-left:6px;">Kopieren</button>
|
||||
</div>
|
||||
</div>
|
||||
@@ -328,6 +329,7 @@
|
||||
<input type="file" id="diag-file-input" multiple accept="image/*,application/pdf,.doc,.docx,.txt" style="display:none;" onchange="handleDiagFileSelect(this.files)">
|
||||
</label>
|
||||
<textarea id="chat-input" placeholder="Nachricht an ARIA... (Enter sendet, Shift+Enter neue Zeile)" rows="2" onpaste="handleDiagPaste(event)" oninput="autoResizeTextarea(this)"></textarea>
|
||||
<button class="btn secondary" onclick="interjectDiag('chat-input')" title="Zwischenruf — Korrektur in den laufenden Turn (kein Abbruch, keine Queue)" style="border-color:#FF9500;color:#FF9500;">📣</button>
|
||||
<button class="btn" id="btn-rvs" onclick="testRVS()">Senden</button>
|
||||
</div>
|
||||
</div>
|
||||
@@ -345,6 +347,7 @@
|
||||
</div>
|
||||
<div class="input-row" style="margin-top:8px;">
|
||||
<textarea id="chat-input-fs" placeholder="Nachricht an ARIA... (Enter sendet, Shift+Enter neue Zeile)" rows="2" oninput="autoResizeTextarea(this)"></textarea>
|
||||
<button class="btn secondary" onclick="interjectDiag('chat-input-fs')" title="Zwischenruf — Korrektur in den laufenden Turn (kein Abbruch, keine Queue)" style="border-color:#FF9500;color:#FF9500;">📣</button>
|
||||
<button class="btn" onclick="testRVSFS()">Senden</button>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1585,9 +1588,34 @@
|
||||
const d = await r.json();
|
||||
diagQueueStatus = d?.contexts || {};
|
||||
renderContextStrip();
|
||||
reconcileDiagStates();
|
||||
} catch {}
|
||||
}
|
||||
|
||||
// Selbstheilung: haengt ein Kontext lokal auf 'running', obwohl der Brain
|
||||
// ihn NICHT als busy meldet, ist der Turn (z.B. durch einen Bridge-Timeout)
|
||||
// ohne Antwort-Bubble abgebrochen — der Automat blieb sonst ewig 'running'
|
||||
// (Folge-Nachrichten wandern in die Queue, obwohl ARIA idle ist; frueher
|
||||
// half nur Ctrl+R, was die Queue-Nachricht verlor). Nach 2 aufeinander-
|
||||
// folgenden not-busy-Polls (~4s Karenz gegen das Sende-Fenster) schalten
|
||||
// wir die Queue weiter: angestellte Nachricht geht automatisch raus, sonst
|
||||
// idle.
|
||||
const diagStuckCounts = {};
|
||||
function reconcileDiagStates() {
|
||||
const ctxs = diagQueueStatus || {};
|
||||
for (const pid of Object.keys(diagCtxStates)) {
|
||||
if (diagCtxStates[pid] !== 'running') { diagStuckCounts[pid] = 0; continue; }
|
||||
const busy = !!(ctxs[pid || '__main__'] && ctxs[pid || '__main__'].busy);
|
||||
if (busy) { diagStuckCounts[pid] = 0; continue; }
|
||||
diagStuckCounts[pid] = (diagStuckCounts[pid] || 0) + 1;
|
||||
if (diagStuckCounts[pid] >= 2) {
|
||||
diagStuckCounts[pid] = 0;
|
||||
console.warn('[diag] Kontext', pid || '(main)', 'haengt auf running, Brain idle → Queue weiterschalten');
|
||||
advanceDiagQueue(pid);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshDiagProjectsCache() {
|
||||
try {
|
||||
const r = await fetch('/api/brain/projects/list?include_archived=false');
|
||||
@@ -1737,6 +1765,27 @@
|
||||
return;
|
||||
}
|
||||
|
||||
if (msg.type === 'disk_cleanup') {
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
const dtext = document.getElementById('disk-banner-text');
|
||||
if (msg.status === 'running') {
|
||||
if (btn) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
} else if (msg.status === 'done') {
|
||||
// Feedback NICHT nur per alert() (koennte unterdrueckt sein) —
|
||||
// direkt am Button + im Banner sichtbar machen.
|
||||
if (btn) {
|
||||
btn.disabled = false;
|
||||
btn.textContent = '✓ ' + (msg.freed || '0 MB') + ' frei';
|
||||
setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000);
|
||||
}
|
||||
if (dtext) dtext.textContent = 'Aufgeraeumt: ' + (msg.freed || '0 MB') + ' frei (' + (msg.steps || []).join(', ') + '). Disk-Status aktualisiert sich gleich.';
|
||||
} else if (msg.status === 'error') {
|
||||
if (btn) { btn.disabled = false; btn.textContent = 'Fehler — nochmal'; setTimeout(() => { btn.textContent = 'Sicher aufraeumen'; }, 6000); }
|
||||
if (dtext) dtext.textContent = 'Aufraeumen fehlgeschlagen: ' + (msg.error || '');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (msg.type === 'mode' && msg.payload) {
|
||||
// Bridge hat den Modus geaendert (evtl. von anderer App/Diagnostic) — UI syncen
|
||||
const mode = (msg.payload.mode || '').toLowerCase();
|
||||
@@ -2244,6 +2293,18 @@
|
||||
diagCtxStates[pid] = 'running';
|
||||
renderDiagQueue();
|
||||
}
|
||||
// Zwischenruf: waehrend ARIA arbeitet eine Korrektur in den laufenden Turn
|
||||
// schieben — NICHT anstellen, NICHT abbrechen.
|
||||
function interjectDiag(inputId) {
|
||||
const input = document.getElementById(inputId || 'chat-input');
|
||||
const text = (input.value || '').trim();
|
||||
if (!text) return;
|
||||
send({ action: 'interject', text, projectId: focusedContextId });
|
||||
addChat('sent', '📣 Zwischenruf: ' + text, 'interject', { projectId: focusedContextId });
|
||||
diagDrafts[focusedContextId] = '';
|
||||
localStorage.setItem('diag_ctx_drafts', JSON.stringify(diagDrafts));
|
||||
input.value = '';
|
||||
}
|
||||
function advanceDiagQueue(pid) {
|
||||
const q = diagCtxQueues[pid] || [];
|
||||
if (q.length === 0) { diagCtxStates[pid] = 'idle'; renderDiagQueue(); return; }
|
||||
@@ -6720,6 +6781,23 @@
|
||||
banner.style.display = 'block';
|
||||
}
|
||||
|
||||
// Fuehrt das Aufraeumen WIRKLICH aus (Server-seitig via Docker-API), statt
|
||||
// nur den Befehl zu kopieren.
|
||||
// WICHTIG: "safe" laeuft OHNE confirm() — Build-Cache + ungenutzte Images
|
||||
// sind ungefaehrlich (keine Volumes/Daten). Frueher blockte ein confirm()
|
||||
// den Klick, wenn der Browser Dialoge unterdrueckt hatte ("Verhindern,
|
||||
// dass diese Seite weitere Dialoge erstellt") → confirm()===false → es ging
|
||||
// NICHTS raus. Nur "aggressive" (Volumes!) fragt noch nach.
|
||||
function runDiskCleanup(variant) {
|
||||
const aggressive = variant === 'aggressive';
|
||||
if (aggressive) {
|
||||
const ok = confirm('AGGRESSIV aufraeumen? Loescht zusaetzlich ungenutzte Volumes — nur wenn ALLE ARIA-Container laufen, sonst Datenverlust!');
|
||||
if (!ok) return;
|
||||
}
|
||||
const btn = document.getElementById('disk-clean-btn');
|
||||
if (btn && !aggressive) { btn.disabled = true; btn.textContent = 'Raeume auf...'; }
|
||||
send({ action: 'disk_cleanup', variant });
|
||||
}
|
||||
function copyDiskCmd(variant) {
|
||||
const cmd = variant === 'aggressive'
|
||||
? 'docker system prune -a --volumes -f'
|
||||
|
||||
+81
-3
@@ -1559,6 +1559,70 @@ function dockerExec(containerName, cmd) {
|
||||
});
|
||||
}
|
||||
|
||||
// POST gegen die Docker-Daemon-API (via gemountetem Socket). Fuer prune-
|
||||
// Endpoints — die geben SpaceReclaimed (Bytes) zurueck.
|
||||
function dockerApiPost(apiPath) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = http.request({
|
||||
socketPath: "/var/run/docker.sock",
|
||||
path: apiPath,
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json", "Content-Length": 0 },
|
||||
}, (res) => {
|
||||
let data = "";
|
||||
res.on("data", (c) => data += c);
|
||||
res.on("end", () => {
|
||||
if (res.statusCode >= 200 && res.statusCode < 300) {
|
||||
try { resolve(JSON.parse(data || "{}")); } catch { resolve({}); }
|
||||
} else {
|
||||
reject(new Error(`Docker API ${apiPath}: HTTP ${res.statusCode} — ${String(data).slice(0, 200)}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
req.on("error", reject);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
// "Sicher aufraeumen": Build-Cache + ungenutzte Images prunen — OHNE Volumes
|
||||
// (keine Daten weg). "aggressive": zusaetzlich gestoppte Container + ungenutzte
|
||||
// Volumes (kann Daten kosten → nur auf ausdrueckliche Wahl). Fuehrt es WIRKLICH
|
||||
// aus (frueher kopierte der Button nur den Befehl in die Zwischenablage).
|
||||
async function handleDiskCleanup(clientWs, variant) {
|
||||
const aggressive = variant === "aggressive";
|
||||
const send = (o) => { try { clientWs.send(JSON.stringify(o)); } catch (_) {} };
|
||||
send({ type: "disk_cleanup", status: "running", variant });
|
||||
log("warn", "server", `Disk-Cleanup gestartet (${aggressive ? "aggressive" : "safe"})`);
|
||||
try {
|
||||
let reclaimed = 0;
|
||||
const steps = [];
|
||||
const bp = await dockerApiPost("/build/prune?all=true");
|
||||
reclaimed += (bp.SpaceReclaimed || 0);
|
||||
steps.push("Build-Cache");
|
||||
// dangling=false → ALLE ungenutzten Images (nicht nur dangling).
|
||||
// Docker-API-Filterformat: map[string][]string.
|
||||
const imgFilter = encodeURIComponent(JSON.stringify({ dangling: ["false"] }));
|
||||
const ip = await dockerApiPost("/images/prune?filters=" + imgFilter);
|
||||
reclaimed += (ip.SpaceReclaimed || 0);
|
||||
steps.push("ungenutzte Images");
|
||||
if (aggressive) {
|
||||
const cp = await dockerApiPost("/containers/prune");
|
||||
reclaimed += (cp.SpaceReclaimed || 0);
|
||||
steps.push("gestoppte Container");
|
||||
const vp = await dockerApiPost("/volumes/prune");
|
||||
reclaimed += (vp.SpaceReclaimed || 0);
|
||||
steps.push("ungenutzte Volumes");
|
||||
}
|
||||
const mb = (reclaimed / (1024 * 1024));
|
||||
const freed = mb >= 1024 ? (mb / 1024).toFixed(2) + " GB" : mb.toFixed(0) + " MB";
|
||||
log("info", "server", `Disk-Cleanup fertig: ${freed} frei (${steps.join(", ")})`);
|
||||
send({ type: "disk_cleanup", status: "done", variant, reclaimedBytes: reclaimed, freed, steps });
|
||||
} catch (err) {
|
||||
log("error", "server", `Disk-Cleanup fehlgeschlagen: ${err.message}`);
|
||||
send({ type: "disk_cleanup", status: "error", variant, error: String(err && err.message || err) });
|
||||
}
|
||||
}
|
||||
|
||||
// ── Hilfsfunktionen ─────────────────────────────────────
|
||||
|
||||
function waitForMessage(ws, timeoutMs) {
|
||||
@@ -2455,6 +2519,16 @@ wss.on("connection", (ws) => {
|
||||
} else if (msg.action === "test_rvs") {
|
||||
traceStart("RVS", msg.text || "aria lebst du noch?");
|
||||
sendToRVS(msg.text || "aria lebst du noch?", true, msg.projectId || "");
|
||||
} else if (msg.action === "interject") {
|
||||
// Zwischenruf: in den laufenden Turn schieben (kein Abbruch, keine
|
||||
// Queue) → RVS interject → Bridge → Proxy /interject.
|
||||
const t = String(msg.text || "");
|
||||
if (t.trim()) {
|
||||
sendToRVS_raw({ type: "interject", payload: { projectId: msg.projectId || "", text: t }, timestamp: Date.now() });
|
||||
log("info", "server", "Zwischenruf an RVS (project=" + (msg.projectId || "(main)") + "): " + t.slice(0, 60));
|
||||
}
|
||||
} else if (msg.action === "disk_cleanup") {
|
||||
handleDiskCleanup(ws, msg.variant === "aggressive" ? "aggressive" : "safe");
|
||||
} else if (msg.action === "reconnect_gateway") {
|
||||
connectGateway();
|
||||
} else if (msg.action === "reconnect_rvs") {
|
||||
@@ -2499,14 +2573,18 @@ wss.on("connection", (ws) => {
|
||||
});
|
||||
log("info", "server", `Datei gesendet: ${msg.name} (${msg.type})`);
|
||||
} else if (msg.action === "cancel_request") {
|
||||
// Laufende Anfrage abbrechen — doctor --fix beendet stuck runs
|
||||
log("warn", "server", "Anfrage abgebrochen — fuehre doctor --fix aus");
|
||||
// Laufende Anfrage abbrechen — ECHTER Cancel: RVS cancel_request (hard)
|
||||
// an die Bridge, die den Proxy-/cancel-all Side-Channel anruft und den
|
||||
// laufenden claude-Subprozess killt. Das alte `openclaw doctor --fix`
|
||||
// zielte auf den Container aria-core, den es nicht mehr gibt — es
|
||||
// beendete den Run nie (ARIA lief munter weiter).
|
||||
log("warn", "server", "Anfrage abgebrochen — cancel_request (hard) an Bridge/Proxy");
|
||||
pendingMessageTime = 0;
|
||||
watchdogWarned = false;
|
||||
watchdogFixAttempted = false;
|
||||
if (traceActive) traceEnd(false, "Vom Benutzer abgebrochen");
|
||||
broadcast({ type: "agent_activity", activity: "idle" });
|
||||
dockerExec("aria-core", "openclaw doctor --fix 2>/dev/null || true").catch(() => {});
|
||||
sendToRVS_raw({ type: "cancel_request", payload: { hard: true, source: "diagnostic-cancel" }, timestamp: Date.now() });
|
||||
} else if (msg.action === "aria_panic_stop") {
|
||||
// NOT-AUS aus ARIA-Live-View: lokales /api/cancel UND Hard-Kill via
|
||||
// Bridge (die wiederum den Proxy-Side-Channel /cancel-all anruft).
|
||||
|
||||
+1
-5
@@ -11,11 +11,7 @@ services:
|
||||
npm install -g @anthropic-ai/claude-code claude-max-api-proxy &&
|
||||
DIST=$$(find /usr/local/lib -path '*/claude-max-api-proxy/dist' -type d | head -1) &&
|
||||
sed -i 's/startServer({ port })/startServer({ port, host: process.env.HOST || \"127.0.0.1\" })/' $$DIST/server/standalone.js &&
|
||||
sed -i 's/\"--no-session-persistence\",/\"--no-session-persistence\",\"--dangerously-skip-permissions\",/' $$DIST/subprocess/manager.js &&
|
||||
sed -i 's/\"--dangerously-skip-permissions\",/\"--dangerously-skip-permissions\",\"--system-prompt\",options.systemPrompt,/' $$DIST/subprocess/manager.js &&
|
||||
sed -i 's/const DEFAULT_TIMEOUT = 300000;/const DEFAULT_TIMEOUT = 86400000;/' $$DIST/subprocess/manager.js &&
|
||||
sed -i '/prompt, \\/\\/ Pass prompt as argument/d' $$DIST/subprocess/manager.js &&
|
||||
sed -i 's|this\\.process\\.stdin?\\.end();|this.process.stdin?.end(prompt);|' $$DIST/subprocess/manager.js &&
|
||||
cp /proxy-patches/manager.js $$DIST/subprocess/manager.js &&
|
||||
cp /proxy-patches/openai-to-cli.js $$DIST/adapter/openai-to-cli.js &&
|
||||
cp /proxy-patches/cli-to-openai.js $$DIST/adapter/cli-to-openai.js &&
|
||||
cp /proxy-patches/routes.js $$DIST/server/routes.js &&
|
||||
|
||||
@@ -147,16 +147,20 @@ cmd_screenshot() {
|
||||
local d; d="$(vm_dir "${name}")"
|
||||
vm_running "${name}" || die "VM '${name}' laeuft nicht"
|
||||
command -v socat >/dev/null || die "socat fehlt (qemu-setup.sh?)"
|
||||
mkdir -p "${SHOT_DIR}"
|
||||
# Standard: ins VM-Verzeichnis schreiben (dem aria-User gehoerend) — NICHT
|
||||
# nach /root/... (da kommt der aria-User nicht hin). Der Brain holt das PNG
|
||||
# danach per SSH (base64). Ueberschreibbar via ARIA_VM_SHOT_DIR.
|
||||
local out_dir="${ARIA_VM_SHOT_DIR:-${d}}"
|
||||
mkdir -p "${out_dir}"
|
||||
local ts; ts="$(date +%s)"
|
||||
local ppm="${d}/shot-${ts}.ppm"
|
||||
printf 'screendump %s\n' "${ppm}" | socat - "unix-connect:${d}/monitor.sock" >/dev/null
|
||||
sleep 0.3
|
||||
local out="${SHOT_DIR}/${name}-${ts}.png"
|
||||
local out="${out_dir}/${name}-${ts}.png"
|
||||
if command -v convert >/dev/null; then
|
||||
convert "${ppm}" "${out}" && rm -f "${ppm}"
|
||||
else
|
||||
out="${SHOT_DIR}/${name}-${ts}.ppm"; mv "${ppm}" "${out}"
|
||||
out="${out_dir}/${name}-${ts}.ppm"; mv "${ppm}" "${out}"
|
||||
fi
|
||||
echo "screenshot=${out}"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
/**
|
||||
* Claude Code CLI Subprocess Manager — ARIA-Patch
|
||||
*
|
||||
* Basis: claude-max-api-proxy dist/subprocess/manager.js, plus die bisher per
|
||||
* sed in docker-compose.yml eingespielten Anpassungen (dangerously-skip-
|
||||
* permissions, system-prompt, 24h-Timeout, Prompt via stdin) — hier fest im
|
||||
* File, damit der groessere Zwischenruf-Umbau nicht per sed gefrickelt werden
|
||||
* muss. Wird per `cp` ueber die npm-Version gelegt (siehe docker-compose.yml).
|
||||
*
|
||||
* ZWISCHENRUF (interject): Statt den Prompt als Text zu schreiben und stdin
|
||||
* sofort zu schliessen (--print/text), laeuft claude jetzt im
|
||||
* `--input-format stream-json`-Modus. Der initiale Prompt geht als
|
||||
* stream-json User-Message rein, stdin bleibt OFFEN — so kann waehrend des
|
||||
* laufenden Turns per sendMessage() eine weitere User-Message reingeschoben
|
||||
* werden, die claude an der naechsten Tool-Grenze aufgreift (kein Abbruch).
|
||||
* Bei 'result' (Turn fertig) wird stdin geschlossen, damit claude sauber
|
||||
* beendet und die HTTP-Response (in routes.js an 'close' gebunden) rausgeht.
|
||||
*/
|
||||
import { spawn } from "child_process";
|
||||
import { EventEmitter } from "events";
|
||||
import { isAssistantMessage, isResultMessage, isContentDelta } from "../types/claude-cli.js";
|
||||
const DEFAULT_TIMEOUT = 86400000; // 24h — lange Agent-Loops (Pentests etc.)
|
||||
export class ClaudeSubprocess extends EventEmitter {
|
||||
process = null;
|
||||
buffer = "";
|
||||
timeoutId = null;
|
||||
isKilled = false;
|
||||
_stdinClosed = false;
|
||||
/**
|
||||
* Start the Claude CLI subprocess with the given prompt
|
||||
*/
|
||||
async start(prompt, options) {
|
||||
const args = this.buildArgs(prompt, options);
|
||||
const timeout = options.timeout || DEFAULT_TIMEOUT;
|
||||
return new Promise((resolve, reject) => {
|
||||
try {
|
||||
// Use spawn() for security - no shell interpretation
|
||||
this.process = spawn("claude", args, {
|
||||
cwd: options.cwd || process.cwd(),
|
||||
env: { ...process.env },
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
// Set timeout
|
||||
this.timeoutId = setTimeout(() => {
|
||||
if (!this.isKilled) {
|
||||
this.isKilled = true;
|
||||
this.process?.kill("SIGTERM");
|
||||
this.emit("error", new Error(`Request timed out after ${timeout}ms`));
|
||||
}
|
||||
}, timeout);
|
||||
// Handle spawn errors (e.g., claude not found)
|
||||
this.process.on("error", (err) => {
|
||||
this.clearTimeout();
|
||||
if (err.message.includes("ENOENT")) {
|
||||
reject(new Error("Claude CLI not found. Install with: npm install -g @anthropic-ai/claude-code"));
|
||||
}
|
||||
else {
|
||||
reject(err);
|
||||
}
|
||||
});
|
||||
// stdin BLEIBT OFFEN: initialen Prompt als stream-json User-
|
||||
// Message schreiben; spaetere Zwischenrufe kommen via
|
||||
// sendMessage(). Geschlossen wird bei 'result' (s. processBuffer).
|
||||
this._writeUserMessage(prompt);
|
||||
// Falls stdin (z.B. EPIPE) frueh stirbt: nicht crashen.
|
||||
this.process.stdin?.on("error", () => {});
|
||||
console.error(`[Subprocess] Process spawned with PID: ${this.process.pid}`);
|
||||
// Parse JSON stream from stdout
|
||||
this.process.stdout?.on("data", (chunk) => {
|
||||
const data = chunk.toString();
|
||||
console.error(`[Subprocess] Received ${data.length} bytes of stdout`);
|
||||
this.buffer += data;
|
||||
this.processBuffer();
|
||||
});
|
||||
// Capture stderr for debugging
|
||||
this.process.stderr?.on("data", (chunk) => {
|
||||
const errorText = chunk.toString().trim();
|
||||
if (errorText) {
|
||||
// Don't emit as error unless it's actually an error
|
||||
// Claude CLI may write debug info to stderr
|
||||
console.error("[Subprocess stderr]:", errorText.slice(0, 200));
|
||||
}
|
||||
});
|
||||
// Handle process close
|
||||
this.process.on("close", (code) => {
|
||||
console.error(`[Subprocess] Process closed with code: ${code}`);
|
||||
this.clearTimeout();
|
||||
// Process any remaining buffer
|
||||
if (this.buffer.trim()) {
|
||||
this.processBuffer();
|
||||
}
|
||||
this.emit("close", code);
|
||||
});
|
||||
// Resolve immediately since we're streaming
|
||||
resolve();
|
||||
}
|
||||
catch (err) {
|
||||
this.clearTimeout();
|
||||
reject(err);
|
||||
}
|
||||
});
|
||||
}
|
||||
/**
|
||||
* Build CLI arguments array
|
||||
*/
|
||||
buildArgs(prompt, options) {
|
||||
const args = [
|
||||
"--print", // Non-interactive mode
|
||||
"--output-format",
|
||||
"stream-json", // JSON streaming output
|
||||
"--verbose", // Required for stream-json
|
||||
"--include-partial-messages", // Enable streaming chunks
|
||||
"--input-format",
|
||||
"stream-json", // ARIA: User-Messages via stdin (Zwischenruf)
|
||||
"--model",
|
||||
options.model, // Model alias (opus/sonnet/haiku)
|
||||
"--no-session-persistence", "--dangerously-skip-permissions", "--system-prompt", options.systemPrompt, "--safe-mode",
|
||||
];
|
||||
if (options.sessionId) {
|
||||
args.push("--session-id", options.sessionId);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
/**
|
||||
* Eine User-Message im stream-json-Input-Format an stdin schreiben.
|
||||
* Genutzt fuer den initialen Prompt UND fuer Zwischenrufe (sendMessage).
|
||||
*/
|
||||
_writeUserMessage(text) {
|
||||
const p = this.process;
|
||||
if (!p || !p.stdin || p.stdin.destroyed || this._stdinClosed)
|
||||
return false;
|
||||
try {
|
||||
p.stdin.write(JSON.stringify({ type: "user", message: { role: "user", content: String(text) } }) + "\n");
|
||||
return true;
|
||||
}
|
||||
catch (_) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Zwischenruf: waehrend eines laufenden Turns eine weitere User-Message
|
||||
* reinschieben. claude greift sie an der naechsten Tool-Grenze auf, ohne
|
||||
* den Turn abzubrechen. Kein Effekt, wenn stdin schon geschlossen ist
|
||||
* (Turn praktisch fertig) — dann ist der Zwischenruf schlicht zu spaet.
|
||||
*/
|
||||
sendMessage(text) {
|
||||
return this._writeUserMessage(text);
|
||||
}
|
||||
/**
|
||||
* stdin schliessen → claude beendet den stream-json-Input und exit't.
|
||||
*/
|
||||
_closeStdin() {
|
||||
if (this._stdinClosed)
|
||||
return;
|
||||
this._stdinClosed = true;
|
||||
try {
|
||||
this.process?.stdin?.end();
|
||||
}
|
||||
catch (_) { }
|
||||
}
|
||||
/**
|
||||
* Process the buffer and emit parsed messages
|
||||
*/
|
||||
processBuffer() {
|
||||
const lines = this.buffer.split("\n");
|
||||
this.buffer = lines.pop() || ""; // Keep incomplete line
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed)
|
||||
continue;
|
||||
try {
|
||||
const message = JSON.parse(trimmed);
|
||||
this.emit("message", message);
|
||||
if (isContentDelta(message)) {
|
||||
// Emit content delta for streaming
|
||||
this.emit("content_delta", message);
|
||||
}
|
||||
else if (isAssistantMessage(message)) {
|
||||
this.emit("assistant", message);
|
||||
}
|
||||
else if (isResultMessage(message)) {
|
||||
this.emit("result", message);
|
||||
// Turn fertig → stdin schliessen, sonst wartet claude im
|
||||
// stream-json-Input auf weitere Messages und der Prozess
|
||||
// (und damit die HTTP-Response) haengt fuer immer.
|
||||
this._closeStdin();
|
||||
}
|
||||
}
|
||||
catch {
|
||||
// Non-JSON output, emit as raw
|
||||
this.emit("raw", trimmed);
|
||||
}
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Clear the timeout timer
|
||||
*/
|
||||
clearTimeout() {
|
||||
if (this.timeoutId) {
|
||||
clearTimeout(this.timeoutId);
|
||||
this.timeoutId = null;
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Kill the subprocess
|
||||
*/
|
||||
kill(signal = "SIGTERM") {
|
||||
if (!this.isKilled && this.process) {
|
||||
this.isKilled = true;
|
||||
this.clearTimeout();
|
||||
this.process.kill(signal);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Check if the process is still running
|
||||
*/
|
||||
isRunning() {
|
||||
return this.process !== null && !this.isKilled && this.process.exitCode === null;
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Verify that Claude CLI is installed and accessible
|
||||
*/
|
||||
export async function verifyClaude() {
|
||||
return new Promise((resolve) => {
|
||||
const proc = spawn("claude", ["--version"], { stdio: "pipe" });
|
||||
let output = "";
|
||||
proc.stdout?.on("data", (chunk) => {
|
||||
output += chunk.toString();
|
||||
});
|
||||
proc.on("error", () => {
|
||||
resolve({
|
||||
ok: false,
|
||||
error: "Claude CLI not found. Install with: npm install -g @anthropic-ai/claude-code",
|
||||
});
|
||||
});
|
||||
proc.on("close", (code) => {
|
||||
if (code === 0) {
|
||||
resolve({ ok: true, version: output.trim() });
|
||||
}
|
||||
else {
|
||||
resolve({
|
||||
ok: false,
|
||||
error: "Claude CLI returned non-zero exit code",
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
/**
|
||||
* Check if Claude CLI is authenticated
|
||||
*/
|
||||
export async function verifyAuth() {
|
||||
return { ok: true };
|
||||
}
|
||||
//# sourceMappingURL=manager.js.map
|
||||
@@ -25,6 +25,13 @@ const MODEL_MAP = {
|
||||
"opus": "opus",
|
||||
"sonnet": "sonnet",
|
||||
"haiku": "haiku",
|
||||
"fable": "fable",
|
||||
"claude-fable-5": "fable",
|
||||
"claude-code-cli/fable": "fable",
|
||||
// Volle aktuelle IDs (falls die App/Diagnostic sie mal direkt setzt)
|
||||
"claude-opus-5": "opus",
|
||||
"claude-sonnet-5": "sonnet",
|
||||
"claude-haiku-4-5": "haiku",
|
||||
};
|
||||
|
||||
export function extractModel(model) {
|
||||
|
||||
+46
-4
@@ -514,12 +514,18 @@ async function handleNonStreamingResponse(res, subprocess, cliInput, requestId)
|
||||
// Datei, greifen die eingebauten Defaults; die Datei wird dann einmalig mit
|
||||
// diesen Defaults angelegt, damit es was zu editieren gibt.
|
||||
const MODELS_FILE = process.env.ARIA_MODELS_FILE || "/shared/config/models.json";
|
||||
// Tier-Aliase als id (opus/sonnet/haiku/fable) — die CLI loest sie automatisch
|
||||
// auf die AKTUELLE Version des Tiers auf (Stand 2026-07: fable→Fable 5,
|
||||
// opus→Opus 5, sonnet→Sonnet 5, haiku→Haiku 4.5). So bleibt die Liste
|
||||
// versions-robust; die display_name-Texte nur bei Tier-Wechsel anpassen.
|
||||
const DEFAULT_MODELS = [
|
||||
{ id: "claude-sonnet-4", tier: "sonnet", display_name: "Sonnet (aktuell: Sonnet 5)",
|
||||
{ id: "fable", tier: "fable", display_name: "Fable (aktuell: Fable 5)",
|
||||
description: "Staerkstes Modell — fuer die haertesten Aufgaben (Software-Entwicklung, lange Agent-Laeufe)." },
|
||||
{ id: "opus", tier: "opus", display_name: "Opus (aktuell: Opus 5)",
|
||||
description: "Sehr schlau, schneller als Fable — fuer schwere/lange Aufgaben." },
|
||||
{ id: "sonnet", tier: "sonnet", display_name: "Sonnet (aktuell: Sonnet 5)",
|
||||
description: "Schnell & gut — Standard fuer den Alltag." },
|
||||
{ id: "claude-opus-4", tier: "opus", display_name: "Opus (aktuell: Opus 4.8)",
|
||||
description: "Langsamer, aber am schlausten — fuer schwere/lange Aufgaben." },
|
||||
{ id: "claude-haiku-4", tier: "haiku", display_name: "Haiku (aktuell: Haiku 4.5)",
|
||||
{ id: "haiku", tier: "haiku", display_name: "Haiku (aktuell: Haiku 4.5)",
|
||||
description: "Sehr schnell & guenstig, kleinerer Kontext — fuer einfache Tasks." },
|
||||
];
|
||||
|
||||
@@ -621,6 +627,28 @@ function _cancelByProject(projectId) {
|
||||
return { killed, requestIds: ids, projectId: pid };
|
||||
}
|
||||
|
||||
// Zwischenruf: schiebt eine User-Message in den/die laufenden Subprozess(e)
|
||||
// eines Kontexts, OHNE sie zu killen. claude greift sie an der naechsten Tool-
|
||||
// Grenze auf (stream-json-Input, s. manager.js). Kein Treffer / stdin schon
|
||||
// zu (Turn quasi fertig) → delivered=0.
|
||||
function _interjectByProject(projectId, text) {
|
||||
const pid = String(projectId || "");
|
||||
const ids = [];
|
||||
let delivered = 0;
|
||||
for (const [id, entry] of Array.from(_activeSubprocesses)) {
|
||||
if (entry.projectId !== pid) continue;
|
||||
try {
|
||||
if (typeof entry.subprocess.sendMessage === "function" && entry.subprocess.sendMessage(text)) {
|
||||
delivered++;
|
||||
ids.push(id);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error("[aria-interject] sendMessage failed for", id, e?.message);
|
||||
}
|
||||
}
|
||||
return { delivered, requestIds: ids, projectId: pid };
|
||||
}
|
||||
|
||||
try {
|
||||
const internalServer = http.createServer((req, res) => {
|
||||
if (req.method === "POST" && req.url === "/cancel-all") {
|
||||
@@ -645,6 +673,20 @@ try {
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (req.method === "POST" && req.url === "/interject") {
|
||||
// Body: {projectId, text}. Zwischenruf in den laufenden Turn.
|
||||
let raw = "";
|
||||
req.on("data", (c) => { raw += c; if (raw.length > 65536) req.destroy(); });
|
||||
req.on("end", () => {
|
||||
let projectId = "", text = "";
|
||||
try { const b = JSON.parse(raw || "{}"); projectId = String(b.projectId || ""); text = String(b.text || ""); } catch (_) {}
|
||||
const result = text ? _interjectByProject(projectId, text) : { delivered: 0, requestIds: [], projectId };
|
||||
console.warn("[aria-interject] /interject project=%s — delivered %d", projectId || "(main)", result.delivered);
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ ok: true, ...result }));
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (req.method === "GET" && req.url === "/health") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ ok: true, active: _activeSubprocesses.size }));
|
||||
|
||||
+5
-1
@@ -17,7 +17,7 @@ const ALLOWED_TYPES = new Set([
|
||||
"file_request", "file_response", "file_saved", "stt_result", "config", "tts_request",
|
||||
"xtts_request", "xtts_response", "xtts_list_voices", "xtts_voices_list", "voice_upload", "xtts_voice_saved",
|
||||
"update_check", "update_available", "update_download", "update_data",
|
||||
"agent_activity", "cancel_request",
|
||||
"agent_activity", "cancel_request", "interject",
|
||||
"audio_pcm",
|
||||
"file_from_aria",
|
||||
"container_restart",
|
||||
@@ -70,6 +70,10 @@ const ALLOWED_TYPES = new Set([
|
||||
// spiegelt ARIAs Datei-Writes, und QEMU-VNC wird als RFB-Bytes durch RVS
|
||||
// getunnelt (Base64-in-JSON wie audio_pcm — kein Binaer-Handling noetig).
|
||||
"code_file", "code_file_edit",
|
||||
// M1 Generatives Cockpit: ARIA komponiert via present_view eine View-Spec
|
||||
// (Orb + Karten), die App/Web/Diagnostic mit ihrem jeweiligen Renderer
|
||||
// materialisieren. Brain → Bridge → RVS → Clients.
|
||||
"aria_view",
|
||||
"check_desktop", "desktop_status",
|
||||
"vnc_open", "vnc_close", "vnc_data", "vnc_input",
|
||||
// Satelliten (Info-/Gateway-Aussenposten in fremden Netzen): melden sich mit
|
||||
|
||||
+56
-3
@@ -30,7 +30,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # TTS → GPU 0 (8 GB; F5 ist klein)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./voices:/voices # WAV + TXT Referenz
|
||||
@@ -68,7 +68,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["1"] # STT/groesstes Modell → GPU 1 (12 GB; spaeter Voxtral-STT-3B ~9 GB)
|
||||
capabilities: [gpu]
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
@@ -108,7 +108,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
device_ids: ["0"] # LLM → GPU 0 (8 GB, teilt sich mit TTS)
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./models:/models # HF-Download-Cache (persistent)
|
||||
@@ -137,3 +137,56 @@ services:
|
||||
# Erster Load eines Modells kann ein GGUF ziehen (mehrere GB) — grosszuegig.
|
||||
- LLM_TIMEOUT_SEC=${LLM_TIMEOUT_SEC:-600}
|
||||
restart: unless-stopped
|
||||
|
||||
# ─── Voxtral STT (GPU, Realtime) — PROFIL "voxtral" ───────────
|
||||
# Ersetzt whisper als STT sobald die 24-GB-Karte da ist. Startet NUR mit
|
||||
# docker compose --profile voxtral up -d
|
||||
# (sonst kollidiert es mit whisper — beide wuerden stt_* beantworten).
|
||||
#
|
||||
# ⚠️ VRAM: Voxtral-Mini-4B-Realtime-2602 braucht >=16 GB (BF16, laut vLLM-
|
||||
# Rezept keine Quant). Laeuft NICHT auf der 3060 (12 GB) — erst 24-GB-Karte.
|
||||
# ⚠️ vLLM: Version >=0.20.0 noetig. Entrypoint/Serve-Form beim ersten Lauf
|
||||
# gegen das offizielle Rezept pruefen (siehe voxtral/README.md).
|
||||
voxtral-vllm:
|
||||
image: vllm/vllm-openai:latest
|
||||
container_name: aria-voxtral-vllm
|
||||
profiles: ["voxtral"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
volumes:
|
||||
- ./hf-cache:/root/.cache/huggingface # gleicher Modell-Cache wie whisper/f5
|
||||
environment:
|
||||
- VLLM_DISABLE_COMPILE_CACHE=1
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
# Serve-Command aus dem vLLM-Rezept (Voxtral-Mini-4B-Realtime-2602).
|
||||
command:
|
||||
- --model
|
||||
- mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- --tokenizer-mode
|
||||
- mistral
|
||||
- --compilation_config
|
||||
- '{"cudagraph_mode":"PIECEWISE"}'
|
||||
restart: unless-stopped
|
||||
|
||||
# ─── Voxtral-Bridge — RVS <-> vLLM-Realtime-WS (CPU-Glue) ─────
|
||||
voxtral-bridge:
|
||||
build: ./voxtral
|
||||
container_name: aria-voxtral-bridge
|
||||
profiles: ["voxtral"]
|
||||
depends_on:
|
||||
- voxtral-vllm
|
||||
environment:
|
||||
- RVS_HOST=${RVS_HOST}
|
||||
- RVS_PORT=${RVS_PORT:-443}
|
||||
- RVS_TLS=${RVS_TLS:-true}
|
||||
- RVS_TLS_FALLBACK=${RVS_TLS_FALLBACK:-true}
|
||||
- RVS_TOKEN=${RVS_TOKEN}
|
||||
- VOXTRAL_VLLM_URL=ws://voxtral-vllm:8000/v1/realtime
|
||||
- VOXTRAL_MODEL=mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- VOXTRAL_LANGUAGE=${WHISPER_LANGUAGE:-de}
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Voxtral-BRIDGE (nicht das Modell!) — leichte CPU-Glue zwischen RVS und dem
|
||||
# vLLM-Realtime-Server. Das eigentliche Voxtral-Modell laeuft im Container
|
||||
# `voxtral-vllm` (GPU, vllm/vllm-openai). Deshalb hier kein CUDA-Base noetig.
|
||||
FROM python:3.11-slim
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY bridge.py ./
|
||||
|
||||
CMD ["python3", "bridge.py"]
|
||||
@@ -0,0 +1,70 @@
|
||||
# Voxtral-STT-Satellit (M0.3)
|
||||
|
||||
Streaming-STT via **Voxtral-Mini-4B-Realtime-2602** (Mistral, Apache 2.0) auf
|
||||
vLLM. Ersetzt whisper als STT — genauer (~5,9 % WER vs 7,4 % FLEURS) und mit
|
||||
echtem Realtime-Streaming. Deutsch ist in den 13 Sprachen abgedeckt.
|
||||
|
||||
Zwei Container:
|
||||
- **`voxtral-vllm`** — das Modell auf vLLM (GPU). Exponiert die Realtime-WS-API.
|
||||
- **`voxtral-bridge`** — CPU-Glue: RVS ⇄ vLLM-Realtime-WS. Macht das Endpointing
|
||||
selbst (adaptiver Rausch-Boden-VAD, identisch zur whisper-Bridge / M0.1).
|
||||
|
||||
## ⚠️ Hardware-Realität — läuft NICHT auf der 3060
|
||||
|
||||
Das Realtime-Modell braucht laut [vLLM-Rezept](https://recipes.vllm.ai/mistralai/Voxtral-Mini-4B-Realtime-2602)
|
||||
**≥ 16 GB VRAM (BF16, keine Quant)**. Die RTX 3060 hat 12 GB → passt nicht.
|
||||
|
||||
- **Interim-gpubox (nur 3060):** whisper (mit M0.1-Fix) + F5-TTS bleiben aktiv.
|
||||
Voxtral NICHT starten.
|
||||
- **Ab der 24-GB-Karte:** Voxtral-Profil hochziehen, whisper wird Fallback.
|
||||
|
||||
Deshalb liegen beide Services hinter dem Compose-**Profil `voxtral`** und starten
|
||||
NUR explizit — sonst würden whisper *und* voxtral dieselben `stt_*`-Messages
|
||||
beantworten (Kollision).
|
||||
|
||||
## Starten (erst wenn die 24-GB-Karte drin ist)
|
||||
|
||||
```bash
|
||||
cd xtts
|
||||
docker compose --profile voxtral up -d --build
|
||||
docker logs -f aria-voxtral-vllm # laedt Modell (mehrere GB, dauert)
|
||||
docker logs -f aria-voxtral-bridge # "RVS verbunden" + service_status ready
|
||||
```
|
||||
Whisper vorher stoppen, damit nur eine STT-Engine antwortet:
|
||||
```bash
|
||||
docker compose stop whisper-bridge
|
||||
```
|
||||
|
||||
## ⚠️ Auf echter Hardware verifizieren (blind gebaut, kein Test hier)
|
||||
|
||||
1. **vLLM-Version ≥ 0.20.0** und die **Serve-Form**. Das Rezept nutzt
|
||||
`vllm serve <model> …`. Falls das `vllm/vllm-openai`-Image einen anderen
|
||||
Entrypoint hat, das `command:` in `docker-compose.yml` anpassen
|
||||
(Rezept-Command steht dort als Kommentar).
|
||||
2. **Realtime-WS-Frames.** Die exakten Event-Namen sind in `bridge.py` ganz oben
|
||||
als Konstanten gebündelt (`VLLM_SEND_APPEND`, `VLLM_DELTA_SUFFIXES`, …),
|
||||
modelliert nach dem OpenAI-Realtime-Schema. Gegen das offizielle
|
||||
**vLLM-Realtime-Client-Beispiel** prüfen und dort anpassen — nur an dieser
|
||||
einen Stelle. Das Response-Handling ist bereits defensiv (mehrere Feldnamen).
|
||||
3. **Endpoint-URL/Port.** Default `ws://voxtral-vllm:8000/v1/realtime` — prüfen ob
|
||||
vLLM auf 8000 lauscht und `/v1/realtime` registriert (Log-Zeile
|
||||
`Route: /v1/realtime`).
|
||||
4. **Endpointing.** Voxtral liefert keine eigene VAD → unser adaptiver Endpointer
|
||||
entscheidet (akustisch + semantisch am Delta-Wachstum). `endpointMs` kommt wie
|
||||
bei whisper aus der App; `voiceFactor` per Session tunebar.
|
||||
|
||||
## Protokoll (RVS, identisch zu whisper — drop-in)
|
||||
|
||||
Rein: `stt_stream_start`, `stt_audio_chunk` (16 kHz mono s16le, base64), `stt_stream_end`.
|
||||
Raus: `stt_partial`, `stt_endpoint` (das Event, auf das aria-bridge horcht), `stt_stream_done`.
|
||||
|
||||
## TTS-Hinweis
|
||||
|
||||
Voxtral-**TTS** (Voice-Cloning) ist hier NICHT enthalten — braucht ebenfalls
|
||||
16 GB VRAM und ist ein eigener Bau. Bis zur 24-GB-Karte bleibt **F5-TTS** aktiv.
|
||||
Danach: eigener `voxtral-tts`-Satellit (separates Ticket).
|
||||
|
||||
## Quellen
|
||||
- Rezept: https://recipes.vllm.ai/mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
- vLLM Speech-to-Text: https://docs.vllm.ai/en/latest/serving/online_serving/speech_to_text/
|
||||
- Modell: https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
@@ -0,0 +1,494 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ARIA Voxtral Bridge — Streaming-STT via Voxtral-Mini-4B-Realtime-2602 (vLLM).
|
||||
|
||||
Zwilling der whisper-Bridge, aber das Transkribieren macht NICHT faster-whisper
|
||||
im selben Prozess, sondern der separate vLLM-Realtime-Server (Container
|
||||
`voxtral-vllm`) ueber dessen WebSocket-API `/v1/realtime`. Diese Bridge ist
|
||||
reine Glue:
|
||||
|
||||
App ──(RVS: stt_stream_start / stt_audio_chunk / stt_stream_end)──▶ diese Bridge
|
||||
diese Bridge ──(WS /v1/realtime: PCM16-b64 append)──▶ voxtral-vllm
|
||||
voxtral-vllm ──(transcription.delta / transcription.done)──▶ diese Bridge
|
||||
diese Bridge ──(RVS: stt_partial / stt_endpoint / stt_stream_done)──▶ App/aria-bridge
|
||||
|
||||
Das RVS-Wire-Protokoll ist IDENTISCH zur whisper-Bridge (drop-in). Das
|
||||
Endpointing (wann hat der User aufgehoert zu sprechen) macht diese Bridge
|
||||
selbst — mit demselben ADAPTIVEN Rausch-Boden-Endpointer wie whisper (Voxtral
|
||||
Realtime liefert laut vLLM-Doku keine eigene VAD/„speaker done"-Semantik, nur
|
||||
transcription.delta/.done). Die akustische Energie messen wir auf unserer
|
||||
eigenen PCM-Kopie, die semantische Stagnation am Delta-Textwachstum.
|
||||
|
||||
⚠️ HARDWARE: Voxtral-Mini-4B-Realtime-2602 braucht >=16 GB VRAM (BF16). Auf der
|
||||
RTX 3060 (12 GB) laeuft es NICHT — erst auf der 24-GB-Karte. Bis dahin
|
||||
bleibt die whisper-Bridge aktiv (Profil-gesteuert im docker-compose).
|
||||
|
||||
⚠️ VERIFY-ON-FIRST-RUN: Die exakten vLLM-Realtime-FRAME-Namen (Audio-Append,
|
||||
Delta/Done-Event-Typen) sind unten als Konstanten gebuendelt und nach dem
|
||||
OpenAI-Realtime-Schema modelliert. Gegen das offizielle vLLM-Realtime-
|
||||
Client-Beispiel pruefen und ggf. anpassen — sie stehen bewusst an EINER
|
||||
Stelle. Response-Handling ist defensiv (mehrere moegliche Feldnamen).
|
||||
|
||||
Env:
|
||||
RVS_HOST, RVS_PORT, RVS_TLS, RVS_TLS_FALLBACK, RVS_TOKEN
|
||||
VOXTRAL_VLLM_URL Default: ws://voxtral-vllm:8000/v1/realtime
|
||||
VOXTRAL_MODEL Default: mistralai/Voxtral-Mini-4B-Realtime-2602
|
||||
VOXTRAL_LANGUAGE Default: de
|
||||
"""
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import websockets
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
datefmt="%H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("voxtral-bridge")
|
||||
|
||||
RVS_HOST = os.getenv("RVS_HOST", "").strip()
|
||||
RVS_PORT = int(os.getenv("RVS_PORT", "443"))
|
||||
RVS_TLS = os.getenv("RVS_TLS", "true").lower() == "true"
|
||||
RVS_TLS_FALLBACK = os.getenv("RVS_TLS_FALLBACK", "true").lower() == "true"
|
||||
RVS_TOKEN = os.getenv("RVS_TOKEN", "").strip()
|
||||
|
||||
VOXTRAL_VLLM_URL = os.getenv("VOXTRAL_VLLM_URL", "ws://voxtral-vllm:8000/v1/realtime")
|
||||
VOXTRAL_MODEL = os.getenv("VOXTRAL_MODEL", "mistralai/Voxtral-Mini-4B-Realtime-2602")
|
||||
VOXTRAL_LANGUAGE = os.getenv("VOXTRAL_LANGUAGE", "de")
|
||||
|
||||
# ── vLLM-Realtime-Frames — HIER anpassen falls das Client-Beispiel abweicht ──
|
||||
# Senderichtung (wir → vLLM): PCM16-16kHz-mono base64 anhaengen + committen.
|
||||
VLLM_SEND_APPEND = "input_audio_buffer.append" # {"type":..., "audio": "<b64>"}
|
||||
VLLM_SEND_COMMIT = "input_audio_buffer.commit" # Buffer abschliessen
|
||||
VLLM_AUDIO_FIELD = "audio"
|
||||
# Empfangsrichtung (vLLM → wir): inkrementeller Text + final. Defensiv geprueft.
|
||||
VLLM_DELTA_SUFFIXES = ("transcription.delta",) # msg["type"] endet hierauf
|
||||
VLLM_DONE_SUFFIXES = ("transcription.done", "transcription.completed")
|
||||
VLLM_DELTA_FIELDS = ("delta", "text", "transcription") # eins davon traegt den Text
|
||||
|
||||
# ── Streaming-/Endpointing-Parameter (analog whisper-Bridge) ──
|
||||
STREAM_DEFAULT_ENDPOINT_MS = 2400
|
||||
STREAM_DEFAULT_HARD_CAP_MS = 60000
|
||||
STREAM_MIN_AUDIO_MS = 600
|
||||
STREAM_SESSION_TTL_S = 120
|
||||
STREAM_ENERGY_WINDOW_MS = 300
|
||||
STREAM_SEMANTIC_BACKUP_FACTOR = 2.0
|
||||
# Adaptiver Voice-Schwellwert (siehe whisper-Bridge M0.1): Grenze relativ zum
|
||||
# gemessenen Rausch-Boden statt fix — schneidet leises Sprechen nicht ab.
|
||||
STREAM_VOICE_FACTOR = 2.5
|
||||
STREAM_VOICE_RMS_MIN = 0.005
|
||||
STREAM_VOICE_RMS_MAX = 0.020
|
||||
|
||||
|
||||
def pcm_s16le_to_float32(data: bytes) -> np.ndarray:
|
||||
if not data:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
arr = np.frombuffer(data, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
return arr
|
||||
|
||||
|
||||
async def _send(ws, mtype: str, payload: dict) -> None:
|
||||
try:
|
||||
await ws.send(json.dumps({
|
||||
"type": mtype,
|
||||
"payload": payload,
|
||||
"timestamp": int(time.time() * 1000),
|
||||
}))
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Send fehlgeschlagen (%s): %s", mtype, e)
|
||||
|
||||
|
||||
@dataclass
|
||||
class StreamSession:
|
||||
request_id: str
|
||||
audio_request_id: str
|
||||
language: str
|
||||
endpoint_ms: int
|
||||
hard_cap_ms: int
|
||||
voice: str = ""
|
||||
speed: float = 1.0
|
||||
interrupted: bool = False
|
||||
location: Optional[dict] = None
|
||||
sample_rate: int = 16000
|
||||
voice_factor: float = STREAM_VOICE_FACTOR
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
pcm_buffer: bytearray = field(default_factory=bytearray)
|
||||
started_at: float = field(default_factory=time.time)
|
||||
last_chunk_at: float = field(default_factory=time.time)
|
||||
last_partial: str = ""
|
||||
last_growth_at: float = 0.0
|
||||
last_voice_at: float = 0.0
|
||||
noise_floor: float = 0.0
|
||||
closed: bool = False
|
||||
endpoint_sent: bool = False
|
||||
# vLLM-Realtime-Session
|
||||
vllm_ws: object = None
|
||||
vllm_reader: object = None
|
||||
|
||||
|
||||
class SessionManager:
|
||||
def __init__(self) -> None:
|
||||
self._sessions: dict[str, StreamSession] = {}
|
||||
self._ws = None # RVS
|
||||
|
||||
def attach_ws(self, ws) -> None:
|
||||
self._ws = ws
|
||||
|
||||
async def start_session(self, payload: dict) -> Optional[StreamSession]:
|
||||
request_id = (payload.get("requestId") or "").strip()
|
||||
if not request_id:
|
||||
logger.warning("stt_stream_start ohne requestId — ignoriert")
|
||||
return None
|
||||
try:
|
||||
endpoint_ms = int(payload.get("endpointMs") or STREAM_DEFAULT_ENDPOINT_MS)
|
||||
except (TypeError, ValueError):
|
||||
endpoint_ms = STREAM_DEFAULT_ENDPOINT_MS
|
||||
try:
|
||||
hard_cap_ms = int(payload.get("hardCapMs") or STREAM_DEFAULT_HARD_CAP_MS)
|
||||
except (TypeError, ValueError):
|
||||
hard_cap_ms = STREAM_DEFAULT_HARD_CAP_MS
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
sess = StreamSession(
|
||||
request_id=request_id,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
language=payload.get("language") or VOXTRAL_LANGUAGE,
|
||||
endpoint_ms=endpoint_ms,
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=float(payload.get("speed") or 1.0),
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
)
|
||||
# vLLM-Realtime-Session oeffnen + Reader starten.
|
||||
try:
|
||||
sess.vllm_ws = await websockets.connect(VOXTRAL_VLLM_URL, max_size=8 * 1024 * 1024)
|
||||
await self._vllm_configure(sess)
|
||||
sess.vllm_reader = asyncio.create_task(self._vllm_read_loop(sess))
|
||||
except Exception as e:
|
||||
logger.exception("Stream %s: vLLM-Realtime-Connect fehlgeschlagen: %s",
|
||||
request_id[:8], e)
|
||||
# ohne Backend keine Transkription → sofort leeres Endpoint melden
|
||||
self._sessions[request_id] = sess
|
||||
await self._finalize(sess, reason="vllm_unavailable")
|
||||
return None
|
||||
self._sessions[request_id] = sess
|
||||
logger.info("Voxtral-Session offen: id=%s lang=%s endpointMs=%d",
|
||||
request_id[:8], sess.language, sess.endpoint_ms)
|
||||
return sess
|
||||
|
||||
async def _vllm_configure(self, sess: StreamSession) -> None:
|
||||
"""Optionale Session-Konfig an vLLM (Modell/Sprache/temperature=0).
|
||||
VERIFY: exaktes session.update-Schema gegen vLLM-Realtime-Beispiel.
|
||||
Best-effort — Fehler hier sind nicht fatal."""
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({
|
||||
"type": "session.update",
|
||||
"session": {
|
||||
"model": VOXTRAL_MODEL,
|
||||
"language": sess.language,
|
||||
"temperature": 0.0,
|
||||
"input_audio_format": "pcm16",
|
||||
},
|
||||
}))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _vllm_read_loop(self, sess: StreamSession) -> None:
|
||||
"""Liest transcription.delta/.done vom vLLM-Realtime-Server."""
|
||||
ws = sess.vllm_ws
|
||||
try:
|
||||
async for raw in ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
mtype = str(msg.get("type", ""))
|
||||
if any(mtype.endswith(s) for s in VLLM_DELTA_SUFFIXES):
|
||||
text = self._extract_text(msg)
|
||||
if text:
|
||||
await self._on_delta(sess, text)
|
||||
elif any(mtype.endswith(s) for s in VLLM_DONE_SUFFIXES):
|
||||
text = self._extract_text(msg)
|
||||
if text:
|
||||
await self._on_delta(sess, text, final=True)
|
||||
except Exception:
|
||||
logger.debug("Stream %s: vLLM-Reader beendet", sess.request_id[:8])
|
||||
|
||||
@staticmethod
|
||||
def _extract_text(msg: dict) -> str:
|
||||
for f in VLLM_DELTA_FIELDS:
|
||||
v = msg.get(f)
|
||||
if isinstance(v, str) and v:
|
||||
return v
|
||||
return ""
|
||||
|
||||
async def _on_delta(self, sess: StreamSession, text: str, final: bool = False) -> None:
|
||||
"""Neuer/finaler Transkript-Text vom vLLM. delta = inkrementell; wir
|
||||
haengen an, wenn er den bisherigen Partial verlaengert, sonst ersetzen
|
||||
wir (Voxtral kann korrigieren)."""
|
||||
if final or text.startswith(sess.last_partial):
|
||||
new_full = text if final else text
|
||||
else:
|
||||
new_full = (sess.last_partial + text).strip()
|
||||
new_full = new_full.strip()
|
||||
if new_full and new_full != sess.last_partial:
|
||||
sess.last_partial = new_full
|
||||
sess.last_growth_at = time.time()
|
||||
if self._ws is not None:
|
||||
await _send(self._ws, "stt_partial", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": new_full,
|
||||
})
|
||||
|
||||
def feed_chunk(self, payload: dict) -> bool:
|
||||
request_id = payload.get("requestId", "")
|
||||
sess = self._sessions.get(request_id)
|
||||
if sess is None or sess.closed:
|
||||
return False
|
||||
pcm_b64 = payload.get("pcm", "")
|
||||
if not pcm_b64:
|
||||
return True
|
||||
try:
|
||||
pcm = base64.b64decode(pcm_b64)
|
||||
except Exception:
|
||||
return True
|
||||
sess.pcm_buffer.extend(pcm)
|
||||
sess.last_chunk_at = time.time()
|
||||
# An vLLM weiterreichen (fire-and-forget).
|
||||
if sess.vllm_ws is not None:
|
||||
asyncio.create_task(self._vllm_append(sess, pcm_b64))
|
||||
return True
|
||||
|
||||
async def _vllm_append(self, sess: StreamSession, pcm_b64: str) -> None:
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({
|
||||
"type": VLLM_SEND_APPEND,
|
||||
VLLM_AUDIO_FIELD: pcm_b64,
|
||||
}))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def end_session(self, request_id: str) -> None:
|
||||
sess = self._sessions.get(request_id)
|
||||
if sess is not None:
|
||||
sess.closed = True
|
||||
|
||||
def drop(self, request_id: str) -> None:
|
||||
sess = self._sessions.pop(request_id, None)
|
||||
if sess is not None:
|
||||
self._teardown_vllm(sess)
|
||||
|
||||
def _teardown_vllm(self, sess: StreamSession) -> None:
|
||||
try:
|
||||
if sess.vllm_reader is not None:
|
||||
sess.vllm_reader.cancel()
|
||||
except Exception:
|
||||
pass
|
||||
if sess.vllm_ws is not None:
|
||||
asyncio.create_task(self._close_ws(sess.vllm_ws))
|
||||
sess.vllm_ws = None
|
||||
|
||||
@staticmethod
|
||||
async def _close_ws(ws) -> None:
|
||||
try:
|
||||
await ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── Endpointer (adaptiv, wie whisper-Bridge M0.1) ──
|
||||
def _buffer_duration_ms(self, sess: StreamSession) -> float:
|
||||
samples = len(sess.pcm_buffer) // 2
|
||||
return (samples / sess.sample_rate) * 1000.0 if samples else 0.0
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
win_bytes = int(sess.sample_rate * STREAM_ENERGY_WINDOW_MS / 1000) * 2
|
||||
if win_bytes <= 0:
|
||||
return 0.0
|
||||
tail = sess.pcm_buffer[-win_bytes:]
|
||||
if len(tail) < 2:
|
||||
return 0.0
|
||||
arr = pcm_s16le_to_float32(bytes(tail))
|
||||
if arr.size == 0:
|
||||
return 0.0
|
||||
return float(np.sqrt(np.mean(arr * arr)))
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
async def run_endpointer(self) -> None:
|
||||
logger.info("Voxtral-Endpointer gestartet (adaptiver VAD)")
|
||||
while True:
|
||||
await asyncio.sleep(0.2)
|
||||
now = time.time()
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
try:
|
||||
await self._tick(sess, now)
|
||||
except Exception:
|
||||
logger.exception("Endpointer-Tick crashed (session=%s)", sid[:8])
|
||||
for sid, sess in list(self._sessions.items()):
|
||||
if now - sess.last_chunk_at > STREAM_SESSION_TTL_S:
|
||||
logger.info("Stream %s: TTL — drop", sid[:8])
|
||||
self.drop(sid)
|
||||
|
||||
async def _tick(self, sess: StreamSession, now: float) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
elapsed_ms = (now - sess.started_at) * 1000.0
|
||||
if elapsed_ms > sess.hard_cap_ms and not sess.closed:
|
||||
await self._finalize(sess, reason="hardcap")
|
||||
return
|
||||
if sess.closed:
|
||||
await self._finalize(sess, reason="stream_end")
|
||||
return
|
||||
if self._buffer_duration_ms(sess) < STREAM_MIN_AUDIO_MS:
|
||||
return
|
||||
# adaptive akustische Sprach-Aktivitaet
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
else:
|
||||
self._update_noise_floor(sess, rms)
|
||||
# Endpoint: akustisch (Primaer) oder semantisch (Backstop), sobald Text da
|
||||
if sess.last_growth_at > 0.0:
|
||||
acoustic_silence_ms = (now - sess.last_voice_at) * 1000.0 if sess.last_voice_at > 0 else 0.0
|
||||
semantic_silence_ms = (now - sess.last_growth_at) * 1000.0
|
||||
acoustic_done = sess.last_voice_at > 0 and acoustic_silence_ms >= sess.endpoint_ms
|
||||
semantic_done = semantic_silence_ms >= sess.endpoint_ms * STREAM_SEMANTIC_BACKUP_FACTOR
|
||||
if acoustic_done or semantic_done:
|
||||
await self._finalize(sess, reason="endpoint" if acoustic_done else "endpoint_semantic")
|
||||
|
||||
async def _finalize(self, sess: StreamSession, reason: str) -> None:
|
||||
if sess.endpoint_sent:
|
||||
return
|
||||
sess.endpoint_sent = True
|
||||
# vLLM ggf. committen, damit ein letztes transcription.done kommt.
|
||||
if sess.vllm_ws is not None:
|
||||
try:
|
||||
await sess.vllm_ws.send(json.dumps({"type": VLLM_SEND_COMMIT}))
|
||||
await asyncio.sleep(0.15) # kurz auf finalen Delta warten
|
||||
except Exception:
|
||||
pass
|
||||
final_text = sess.last_partial.strip()
|
||||
duration_s = self._buffer_duration_ms(sess) / 1000.0
|
||||
logger.info("Stream %s: FINAL (reason=%s, %.1fs): %r",
|
||||
sess.request_id[:8], reason, duration_s, final_text[:120])
|
||||
if self._ws is not None:
|
||||
endpoint_payload = {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
"durationS": duration_s,
|
||||
"sttMs": 0,
|
||||
"voice": sess.voice,
|
||||
"speed": sess.speed,
|
||||
"interrupted": sess.interrupted,
|
||||
}
|
||||
if sess.location:
|
||||
endpoint_payload["location"] = sess.location
|
||||
await _send(self._ws, "stt_endpoint", endpoint_payload)
|
||||
await _send(self._ws, "stt_stream_done", {
|
||||
"requestId": sess.request_id,
|
||||
"audioRequestId": sess.audio_request_id,
|
||||
"text": final_text,
|
||||
"reason": reason,
|
||||
})
|
||||
self.drop(sess.request_id)
|
||||
|
||||
|
||||
async def _broadcast_status(ws, state: str, **extra) -> None:
|
||||
payload = {"service": "voxtral", "state": state}
|
||||
payload.update(extra)
|
||||
await _send(ws, "service_status", payload)
|
||||
|
||||
|
||||
async def run_loop(sessions: SessionManager) -> None:
|
||||
use_tls = RVS_TLS
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
while True:
|
||||
scheme = "wss" if use_tls else "ws"
|
||||
url = f"{scheme}://{RVS_HOST}:{RVS_PORT}/ws?token={RVS_TOKEN}"
|
||||
masked = url.replace(RVS_TOKEN, "***") if RVS_TOKEN else url
|
||||
try:
|
||||
logger.info("Verbinde zu RVS: %s", masked)
|
||||
async with websockets.connect(url, ping_interval=20, ping_timeout=10,
|
||||
max_size=50 * 1024 * 1024) as ws:
|
||||
logger.info("RVS verbunden")
|
||||
retry_s = 2
|
||||
tls_fallback_tried = False
|
||||
sessions.attach_ws(ws)
|
||||
await _broadcast_status(ws, "ready", model=VOXTRAL_MODEL)
|
||||
await _send(ws, "config_request", {"service": "voxtral"})
|
||||
|
||||
async for raw in ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
mtype = msg.get("type", "")
|
||||
payload = msg.get("payload", {}) or {}
|
||||
if mtype == "stt_stream_start":
|
||||
asyncio.create_task(sessions.start_session(payload))
|
||||
elif mtype == "stt_audio_chunk":
|
||||
sessions.feed_chunk(payload)
|
||||
elif mtype == "stt_stream_end":
|
||||
sessions.end_session(payload.get("requestId", ""))
|
||||
# stt_request (Legacy One-Shot) macht Voxtral hier NICHT —
|
||||
# dafuer bleibt die whisper-Bridge (Fallback).
|
||||
except Exception as e:
|
||||
logger.warning("RVS-Verbindung verloren: %s — retry in %ds", e, retry_s)
|
||||
if use_tls and RVS_TLS_FALLBACK and not tls_fallback_tried:
|
||||
use_tls = False
|
||||
tls_fallback_tried = True
|
||||
logger.info("TLS-Fallback: versuche ws:// (kein TLS)")
|
||||
continue
|
||||
await asyncio.sleep(retry_s)
|
||||
retry_s = min(retry_s * 2, 30)
|
||||
use_tls = RVS_TLS # fuer den naechsten Zyklus zuruecksetzen
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
if not RVS_HOST or not RVS_TOKEN:
|
||||
logger.error("RVS_HOST/RVS_TOKEN fehlen — .env pruefen. Abbruch.")
|
||||
return
|
||||
sessions = SessionManager()
|
||||
logger.info("Voxtral-Bridge startet — vLLM=%s Modell=%s", VOXTRAL_VLLM_URL, VOXTRAL_MODEL)
|
||||
await asyncio.gather(
|
||||
run_loop(sessions),
|
||||
sessions.run_endpointer(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
@@ -0,0 +1,5 @@
|
||||
# Voxtral-Bridge ist reine CPU-Glue (RVS-WS <-> vLLM-Realtime-WS). Das Modell
|
||||
# selbst laeuft im separaten voxtral-vllm-Container (GPU). Deshalb hier KEIN
|
||||
# torch/vllm — nur der WebSocket-Client + numpy fuer die RMS-Energiemessung.
|
||||
websockets>=12.0
|
||||
numpy>=1.24
|
||||
+49
-2
@@ -75,7 +75,17 @@ STREAM_SESSION_TTL_S = 120 # tote Sessions nach 2 min aufraeumen
|
||||
# (Whisper oszilliert/halluziniert). Echte akustische Stille ist das robuste
|
||||
# „User hat aufgehoert"-Signal.
|
||||
STREAM_ENERGY_WINDOW_MS = 300 # RMS ueber die letzten 300ms Audio messen
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # RMS darueber = Sprache (haelt Session am Leben)
|
||||
# --- Adaptiver Voice-Schwellwert (ersetzt die fixe 0.012-Grenze) ---
|
||||
# Problem (Repro dokumentiert in audio.ts): eine FESTE RMS-Grenze schneidet
|
||||
# leises/entferntes Sprechen faelschlich als „Stille" (Handy weiter weg vom Mund,
|
||||
# ruhig im Auto, kurze Sprech-Pause) → Cut mitten im Satz. Loesung: die Grenze
|
||||
# relativ zum gemessenen Rausch-Boden der Session fuehren. Sprache =
|
||||
# noise_floor * Faktor, geklammert auf [MIN, MAX]. Bei noch ungelerntem Boden
|
||||
# gilt MIN → sensibel, lieber nicht abschneiden.
|
||||
STREAM_VOICE_FACTOR = 2.5 # Sprache = noise_floor * Faktor
|
||||
STREAM_VOICE_RMS_MIN = 0.005 # Untergrenze (stiller Raum: nicht auf 0 kollabieren)
|
||||
STREAM_VOICE_RMS_MAX = 0.020 # Obergrenze (lautes Auto: Sprache nie ganz aussperren)
|
||||
STREAM_VOICE_RMS_THRESHOLD = 0.012 # Legacy-Konstante (nicht mehr im Cut-Pfad genutzt)
|
||||
# Rein-semantischer Backstop: wenn die Energie NIE faellt (laute Umgebung,
|
||||
# z.B. Auto), endpointen wir trotzdem — aber erst nach diesem Faktor x
|
||||
# endpoint_ms, damit normales Sprechen mit Pausen nicht abgeschnitten wird.
|
||||
@@ -323,6 +333,10 @@ class StreamSession:
|
||||
last_growth_at: float = 0.0
|
||||
last_transcribe_at: float = 0.0
|
||||
last_voice_at: float = 0.0 # letzter Tick mit akustischer Sprach-Energie
|
||||
noise_floor: float = 0.0 # adaptiver Rausch-Boden (0.0 = noch ungelernt)
|
||||
voice_factor: float = STREAM_VOICE_FACTOR # per-Session konfigurierbar (Payload voiceFactor)
|
||||
voice_rms_min: float = STREAM_VOICE_RMS_MIN
|
||||
voice_rms_max: float = STREAM_VOICE_RMS_MAX
|
||||
closed: bool = False # nach stream_end gesetzt
|
||||
endpoint_sent: bool = False # Endpoint nur einmal feuern
|
||||
# Speaker-ID Gating: bei aktiviertem Fingerprint pruefen wir die ersten
|
||||
@@ -372,6 +386,10 @@ class SessionManager:
|
||||
speed = float(payload.get("speed") or 1.0)
|
||||
except (TypeError, ValueError):
|
||||
speed = 1.0
|
||||
try:
|
||||
voice_factor = float(payload.get("voiceFactor") or STREAM_VOICE_FACTOR)
|
||||
except (TypeError, ValueError):
|
||||
voice_factor = STREAM_VOICE_FACTOR
|
||||
session = StreamSession(
|
||||
request_id=request_id,
|
||||
audio_request_id=payload.get("audioRequestId", "") or "",
|
||||
@@ -381,6 +399,7 @@ class SessionManager:
|
||||
hard_cap_ms=hard_cap_ms,
|
||||
voice=payload.get("voice", "") or "",
|
||||
speed=speed,
|
||||
voice_factor=voice_factor,
|
||||
interrupted=bool(payload.get("interrupted", False)),
|
||||
location=payload.get("location") or None,
|
||||
sample_rate=int(payload.get("sampleRate") or 16000),
|
||||
@@ -549,8 +568,16 @@ class SessionManager:
|
||||
# Akustische Sprach-Aktivitaet JEDEN Tick (~200ms) messen — unabhaengig
|
||||
# vom Transcribe-Throttle. Solange wirklich gesprochen wird, bleibt die
|
||||
# Session am Leben, auch wenn Whisper gerade keinen neuen Text liefert.
|
||||
if self._tail_rms(sess) >= STREAM_VOICE_RMS_THRESHOLD:
|
||||
# ADAPTIV: der Schwellwert richtet sich nach dem gemessenen Rausch-Boden
|
||||
# (fast-down/slow-up), damit leises/entferntes Sprechen nicht faelschlich
|
||||
# als Stille gilt und der Satz mitten drin abgeschnitten wird.
|
||||
rms = self._tail_rms(sess)
|
||||
if rms >= self._voice_threshold(sess):
|
||||
sess.last_voice_at = now
|
||||
else:
|
||||
# Rausch-Boden NUR aus Nicht-Sprache lernen — waehrend Sprache
|
||||
# einfrieren, sonst wandert die Grenze hoch und sperrt Sprache aus.
|
||||
self._update_noise_floor(sess, rms)
|
||||
|
||||
# Endpoint-Entscheidung JEDEN Tick, sobald ueberhaupt Text erkannt wurde:
|
||||
# (a) akustisch: seit endpoint_ms keine Sprach-Energie mehr → User ist
|
||||
@@ -620,6 +647,26 @@ class SessionManager:
|
||||
return 0.0
|
||||
return (samples / sess.sample_rate) * 1000.0
|
||||
|
||||
def _voice_threshold(self, sess: StreamSession) -> float:
|
||||
"""Adaptiver Voice-Schwellwert = Rausch-Boden * Faktor, geklammert auf
|
||||
[min, max]. Bei noch ungelerntem Boden (0.0) → Untergrenze: sensibel,
|
||||
lieber nicht abschneiden (das war der eigentliche Cutoff-Bug)."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
return sess.voice_rms_min
|
||||
return min(max(nf * sess.voice_factor, sess.voice_rms_min), sess.voice_rms_max)
|
||||
|
||||
def _update_noise_floor(self, sess: StreamSession, rms: float) -> None:
|
||||
"""Rausch-Boden nachfuehren: schnell runter (neue, leisere Stille),
|
||||
langsam rauf (Umgebung wird lauter). NUR mit Nicht-Sprache aufrufen."""
|
||||
nf = sess.noise_floor
|
||||
if nf <= 0.0:
|
||||
sess.noise_floor = rms
|
||||
elif rms < nf:
|
||||
sess.noise_floor = 0.90 * nf + 0.10 * rms
|
||||
else:
|
||||
sess.noise_floor = 0.98 * nf + 0.02 * rms
|
||||
|
||||
def _tail_rms(self, sess: StreamSession) -> float:
|
||||
"""RMS-Energie der letzten STREAM_ENERGY_WINDOW_MS des Audio-Buffers.
|
||||
Dient als akustisches „redet noch / ist still"-Signal."""
|
||||
|
||||
Reference in New Issue
Block a user