Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 55ef207454 | |||
| 6651f5937d | |||
| e9e7dd804f | |||
| ec9530f17f | |||
| 97cb7be313 | |||
| 77e927ffcd |
@@ -383,6 +383,7 @@ API-Endpoint fuer andere Services: `GET http://localhost:3001/api/session`
|
||||
- **Wake-Word** (on-device, openWakeWord ONNX): "Hey Jarvis", "Alexa", "Hey Mycroft", "Hey Rhasspy" — Mikrofon hoert passiv mit, Konversation startet beim Schluesselwort. Komplett on-device via ONNX Runtime, kein API-Key, kein Cloud-Roundtrip, Audio verlaesst das Geraet nicht.
|
||||
- **VAD (Voice Activity Detection)**: Adaptive Schwelle (Baseline aus ersten 500ms Mic-Pegel + 6dB Offset). Konfigurierbare Stille-Toleranz (1.0–8.0s, Default 2.8s) bevor Auto-Stop greift. Max-Aufnahme einstellbar (1–30 min, Default 5 min)
|
||||
- **Barge-In**: Wenn du waehrend ARIAs Antwort eine neue Sprach-/Text-Nachricht reinschickst, wird sie unterbrochen + bekommt den Hint "das ist eine Korrektur"
|
||||
- **Wake-Word waehrend TTS**: Du kannst "Computer" sagen waehrend ARIA noch redet — AcousticEchoCanceler verhindert dass ARIAs eigene Stimme das Wake-Word triggert
|
||||
- **Anruf-Pause**: TTS verstummt automatisch wenn das Telefon klingelt (READ_PHONE_STATE Permission)
|
||||
- **Speech Gate**: Aufnahme wird verworfen wenn keine Sprache erkannt
|
||||
- **STT (Speech-to-Text)**: 16kHz mono → Bridge → Gamebox-Whisper (CUDA) → Text im Chat. Fast in Echtzeit.
|
||||
@@ -417,7 +418,7 @@ Community-Modelle stammen aus [fwartner/home-assistant-wakewords-collection](htt
|
||||
**Bedienung:**
|
||||
- App → **Einstellungen** → **Wake-Word** → gewuenschtes Keyword waehlen → **Speichern + Aktivieren**
|
||||
- **Ohr-Button (👂)** in der Statusleiste tippen → Wake-Word ist scharf, App hoert passiv mit
|
||||
- Wake-Word sagen → Symbol wechselt auf 🎙️, Konversation laeuft
|
||||
- Wake-Word sagen → Symbol wechselt auf 🎙️, **Bereit-Sound** (Ding-Dong, optional in Settings) + Toast "🎤 sprich jetzt" sobald das Mikro wirklich offen ist
|
||||
- Nach jeder ARIA-Antwort oeffnet sich das Mikro nochmal — Stille → zurueck zu 👂
|
||||
- Erneut tippen → Ohr aus (🔇)
|
||||
|
||||
@@ -847,6 +848,9 @@ docker exec aria-core ssh aria-wohnung hostname
|
||||
- [x] Anruf-Pause: TTS verstummt bei eingehendem Anruf (PhoneStateListener)
|
||||
- [x] Settings-Sub-Screens: 8 Kategorien statt langer Liste
|
||||
- [x] APK ABI-Split arm64-v8a: 35 MB statt 136 MB
|
||||
- [x] Sprachnachrichten-Bubble: audioRequestId statt Substring-Match — keine vertauschten Bubbles mehr bei parallelen Aufnahmen
|
||||
- [x] Bereit-Sound (Airplane Ding-Dong) wenn Mikro nach Wake-Word offen ist — akustische Bestaetigung, in Settings abschaltbar
|
||||
- [x] Wake-Word parallel zu TTS mit AcousticEchoCanceler — "Computer" sagen waehrend ARIA spricht stoppt sie und oeffnet Mikro
|
||||
- [x] Disk-Voll Banner in Diagnostic mit copy-baren Cleanup-Befehlen
|
||||
- [x] Wake-Word on-device via openWakeWord (ONNX Runtime, kein API-Key) + State-Icon
|
||||
|
||||
|
||||
@@ -79,8 +79,8 @@ android {
|
||||
applicationId "com.ariacockpit"
|
||||
minSdkVersion rootProject.ext.minSdkVersion
|
||||
targetSdkVersion rootProject.ext.targetSdkVersion
|
||||
versionCode 705
|
||||
versionName "0.0.7.5"
|
||||
versionCode 707
|
||||
versionName "0.0.7.7"
|
||||
// Fallback fuer Libraries mit Product Flavors
|
||||
missingDimensionStrategy 'react-native-camera', 'general'
|
||||
}
|
||||
|
||||
@@ -8,6 +8,9 @@ import android.content.pm.PackageManager
|
||||
import android.media.AudioFormat
|
||||
import android.media.AudioRecord
|
||||
import android.media.MediaRecorder
|
||||
import android.media.audiofx.AcousticEchoCanceler
|
||||
import android.media.audiofx.AutomaticGainControl
|
||||
import android.media.audiofx.NoiseSuppressor
|
||||
import android.util.Log
|
||||
import androidx.core.content.ContextCompat
|
||||
import com.facebook.react.bridge.Promise
|
||||
@@ -70,6 +73,13 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
private val running = AtomicBoolean(false)
|
||||
private var captureThread: Thread? = null
|
||||
|
||||
// Audio-Effects: Echo-Cancellation (gegen ARIAs eigene TTS-Stimme die sonst
|
||||
// das Wake-Word triggern wuerde) + Noise-Suppression. Per VOICE_COMMUNICATION
|
||||
// Audio-Source schon vorhanden, aber explizites Aktivieren ist robuster.
|
||||
private var aec: AcousticEchoCanceler? = null
|
||||
private var ns: NoiseSuppressor? = null
|
||||
private var agc: AutomaticGainControl? = null
|
||||
|
||||
// Inferenz-State
|
||||
private val melBuffer: ArrayList<FloatArray> = ArrayList(256) // Liste von 32-dim Frames
|
||||
private var melProcessedIdx: Int = 0
|
||||
@@ -146,8 +156,12 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(CHUNK_SAMPLES * 2 * 4)
|
||||
|
||||
// VOICE_COMMUNICATION-Source: aktiviert auf den meisten Android-Geraeten
|
||||
// automatisch Echo-Cancellation + Noise-Suppression. Wichtig damit
|
||||
// ARIAs eigene Stimme nicht das Wake-Word triggert wenn parallel
|
||||
// zur TTS-Wiedergabe gelauscht wird.
|
||||
val record = AudioRecord(
|
||||
MediaRecorder.AudioSource.MIC,
|
||||
MediaRecorder.AudioSource.VOICE_COMMUNICATION,
|
||||
SAMPLE_RATE,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
@@ -159,6 +173,27 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
return
|
||||
}
|
||||
audioRecord = record
|
||||
|
||||
// Audio-Effects ZUSAETZLICH explizit aktivieren — manche Geraete
|
||||
// benoetigen das, obwohl VOICE_COMMUNICATION es eigentlich schon
|
||||
// mitbringt. Failure ist nicht kritisch (continue ohne Effects).
|
||||
try {
|
||||
if (AcousticEchoCanceler.isAvailable()) {
|
||||
aec = AcousticEchoCanceler.create(record.audioSessionId)?.apply { enabled = true }
|
||||
Log.i(TAG, "AEC aktiviert (enabled=${aec?.enabled})")
|
||||
}
|
||||
} catch (e: Exception) { Log.w(TAG, "AEC failed: ${e.message}") }
|
||||
try {
|
||||
if (NoiseSuppressor.isAvailable()) {
|
||||
ns = NoiseSuppressor.create(record.audioSessionId)?.apply { enabled = true }
|
||||
}
|
||||
} catch (e: Exception) { Log.w(TAG, "NS failed: ${e.message}") }
|
||||
try {
|
||||
if (AutomaticGainControl.isAvailable()) {
|
||||
agc = AutomaticGainControl.create(record.audioSessionId)?.apply { enabled = true }
|
||||
}
|
||||
} catch (e: Exception) { Log.w(TAG, "AGC failed: ${e.message}") }
|
||||
|
||||
resetInferenceState()
|
||||
running.set(true)
|
||||
record.startRecording()
|
||||
@@ -179,6 +214,13 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
}
|
||||
}
|
||||
|
||||
private fun releaseAudioEffects() {
|
||||
try { aec?.release() } catch (_: Exception) {}
|
||||
try { ns?.release() } catch (_: Exception) {}
|
||||
try { agc?.release() } catch (_: Exception) {}
|
||||
aec = null; ns = null; agc = null
|
||||
}
|
||||
|
||||
@ReactMethod
|
||||
fun stop(promise: Promise) {
|
||||
running.set(false)
|
||||
@@ -189,6 +231,7 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
try { audioRecord?.stop() } catch (_: Exception) {}
|
||||
try { audioRecord?.release() } catch (_: Exception) {}
|
||||
audioRecord = null
|
||||
releaseAudioEffects()
|
||||
Log.i(TAG, "Lauschen gestoppt")
|
||||
promise.resolve(true)
|
||||
}
|
||||
@@ -201,6 +244,7 @@ class OpenWakeWordModule(reactContext: ReactApplicationContext) : ReactContextBa
|
||||
try { audioRecord?.stop() } catch (_: Exception) {}
|
||||
try { audioRecord?.release() } catch (_: Exception) {}
|
||||
audioRecord = null
|
||||
releaseAudioEffects()
|
||||
disposeSessions()
|
||||
promise.resolve(true)
|
||||
}
|
||||
|
||||
Binary file not shown.
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "aria-cockpit",
|
||||
"version": "0.0.7.5",
|
||||
"version": "0.0.7.7",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"android": "react-native run-android",
|
||||
|
||||
Binary file not shown.
@@ -26,6 +26,7 @@ import rvs, { RVSMessage, ConnectionState } from '../services/rvs';
|
||||
import audioService from '../services/audio';
|
||||
import wakeWordService from '../services/wakeword';
|
||||
import phoneCallService from '../services/phoneCall';
|
||||
import { playWakeReadySound } from '../services/wakeReadySound';
|
||||
import updateService from '../services/updater';
|
||||
import VoiceButton from '../components/VoiceButton';
|
||||
import FileUpload, { FileData } from '../components/FileUpload';
|
||||
@@ -55,6 +56,10 @@ interface ChatMessage {
|
||||
messageId?: string;
|
||||
/** Lokaler Pfad zur gecachten TTS-Audio-Datei (file://...) */
|
||||
audioPath?: string;
|
||||
/** Korrelations-ID fuer Sprachnachrichten — wird mit dem STT-Result zurueck-
|
||||
* gespiegelt damit wir die EXAKT richtige Placeholder-Bubble ersetzen,
|
||||
* auch wenn mehrere Aufnahmen parallel offen sind. */
|
||||
audioRequestId?: string;
|
||||
}
|
||||
|
||||
// --- Konstanten ---
|
||||
@@ -292,46 +297,42 @@ const ChatScreen: React.FC = () => {
|
||||
// den gleichen Text bekommen (Bug: zweite Antwort ueberschreibt erste).
|
||||
if (sender === 'stt') {
|
||||
const sttText = (message.payload.text as string) || '';
|
||||
// Debug-Toast: visualisiert dass das STT-Event in der App angekommen ist.
|
||||
// Wenn dieser Toast NICHT erscheint, kommt das Event nicht durch (Bridge
|
||||
// oder RVS broadcastet es nicht), und der Bug liegt server-side.
|
||||
ToastAndroid.show(`STT empfangen: "${sttText.slice(0, 40)}"`, ToastAndroid.SHORT);
|
||||
if (sttText) {
|
||||
setMessages(prev => {
|
||||
const idx = prev.findIndex(m =>
|
||||
m.sender === 'user' && m.text.includes('Spracheingabe wird verarbeitet')
|
||||
);
|
||||
const placeholderCount = prev.filter(m =>
|
||||
m.sender === 'user' && m.text.includes('Spracheingabe wird verarbeitet')
|
||||
).length;
|
||||
console.log('[Chat] STT-Result: idx=%d text="%s" placeholders=%d',
|
||||
idx, sttText.slice(0, 60), placeholderCount);
|
||||
// Zweiter Toast: zeigt ob die Placeholder gefunden wurde.
|
||||
ToastAndroid.show(
|
||||
idx < 0
|
||||
? `STT: keine Placeholder (${placeholderCount}) \u2192 neue Bubble`
|
||||
: `STT: Bubble #${idx} ersetzt`,
|
||||
ToastAndroid.SHORT,
|
||||
);
|
||||
const newText = `\uD83C\uDFA4 ${sttText}`;
|
||||
if (idx < 0) {
|
||||
// Defensiv: wenn keine Placeholder im State (z.B. weil sie nie
|
||||
// hinzugefuegt wurde oder schon durch ein anderes Update verloren
|
||||
// ging), die Sprachnachricht trotzdem als neue Bubble einfuegen.
|
||||
// Sonst kommt ARIAs Antwort ohne sichtbare User-Nachricht.
|
||||
return capMessages([...prev, {
|
||||
id: nextId(),
|
||||
sender: 'user',
|
||||
text: newText,
|
||||
timestamp: message.timestamp,
|
||||
attachments: [{ type: 'audio', name: 'Sprachaufnahme' }],
|
||||
}]);
|
||||
const sttAudioReqId = (message.payload.audioRequestId as string) || '';
|
||||
if (!sttText) {
|
||||
return;
|
||||
}
|
||||
setMessages(prev => {
|
||||
const newText = `\uD83C\uDFA4 ${sttText}`;
|
||||
// Primaer: matche per audioRequestId (eindeutig pro Aufnahme).
|
||||
// So gibt's keine Verwechslung wenn zwei Audios kurz hintereinander
|
||||
// gesendet wurden und ihre STT-Results ueberlappen.
|
||||
if (sttAudioReqId) {
|
||||
const idxById = prev.findIndex(m => m.audioRequestId === sttAudioReqId);
|
||||
if (idxById >= 0) {
|
||||
const next = prev.slice();
|
||||
next[idxById] = { ...next[idxById], text: newText };
|
||||
return next;
|
||||
}
|
||||
}
|
||||
// Fallback: alte Bridge-Version ohne audioRequestId \u2014 match per Substring,
|
||||
// nimmt die ERSTE noch unaufgeloeste Placeholder.
|
||||
const idx = prev.findIndex(m =>
|
||||
m.sender === 'user' && m.text.includes('Spracheingabe wird verarbeitet')
|
||||
);
|
||||
if (idx >= 0) {
|
||||
const next = prev.slice();
|
||||
next[idx] = { ...next[idx], text: newText };
|
||||
return next;
|
||||
});
|
||||
}
|
||||
}
|
||||
// Letzter Fallback: gar keine Placeholder \u2192 neue Bubble einfuegen
|
||||
return capMessages([...prev, {
|
||||
id: nextId(),
|
||||
sender: 'user',
|
||||
text: newText,
|
||||
timestamp: message.timestamp,
|
||||
attachments: [{ type: 'audio', name: 'Sprachaufnahme' }],
|
||||
}]);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -493,7 +494,14 @@ const ChatScreen: React.FC = () => {
|
||||
// Conversation-Window: User hat X Sekunden um anzufangen, sonst Konversation aus
|
||||
const windowMs = await loadConvWindowMs();
|
||||
const started = await audioService.startRecording(true, windowMs);
|
||||
if (!started) {
|
||||
if (started) {
|
||||
// Erst JETZT signalisieren dass das Mikro wirklich offen ist —
|
||||
// vorher war's noch in der Init-Phase. So weiss der User exakt
|
||||
// ab wann er reden kann. "Bereit"-Sound (Ding-Dong) ist optional
|
||||
// ueber Settings → Wake-Word abschaltbar.
|
||||
ToastAndroid.show('🎤 Mikro offen — sprich jetzt', ToastAndroid.SHORT);
|
||||
playWakeReadySound().catch(() => {});
|
||||
} else {
|
||||
// Mikrofon nicht verfuegbar, naechsten Versuch
|
||||
wakeWordService.resume();
|
||||
}
|
||||
@@ -507,12 +515,14 @@ const ChatScreen: React.FC = () => {
|
||||
// Barge-In: laufende ARIA-Aktivitaet abbrechen wenn welche da ist.
|
||||
const wasInterrupted = interruptAriaIfBusy();
|
||||
const location = await getCurrentLocation();
|
||||
const audioRequestId = `audio_${Date.now()}_${Math.floor(Math.random() * 100000)}`;
|
||||
const userMsg: ChatMessage = {
|
||||
id: nextId(),
|
||||
sender: 'user',
|
||||
text: '🎙 Spracheingabe wird verarbeitet...',
|
||||
timestamp: Date.now(),
|
||||
attachments: [{ type: 'audio', name: 'Sprachaufnahme' }],
|
||||
audioRequestId,
|
||||
};
|
||||
setMessages(prev => capMessages([...prev, userMsg]));
|
||||
rvs.send('audio', {
|
||||
@@ -522,6 +532,7 @@ const ChatScreen: React.FC = () => {
|
||||
voice: localXttsVoiceRef.current,
|
||||
speed: ttsSpeedRef.current,
|
||||
interrupted: wasInterrupted,
|
||||
audioRequestId,
|
||||
...(location && { location }),
|
||||
});
|
||||
// resume() wird durch onPlaybackFinished nach ARIAs Antwort getriggert.
|
||||
@@ -534,9 +545,43 @@ const ChatScreen: React.FC = () => {
|
||||
}
|
||||
});
|
||||
|
||||
// Barge-In via Wake-Word: User sagt "Computer" waehrend ARIA spricht.
|
||||
// Wake-Word-Service hat bei TTS-Start parallel zu lauschen begonnen
|
||||
// (mit AcousticEchoCanceler damit ARIAs eigene Stimme nicht triggert).
|
||||
const unsubBarge = wakeWordService.onBargeIn(async () => {
|
||||
console.log('[Chat] Barge-In via Wake-Word — TTS abbrechen + neue Aufnahme');
|
||||
audioService.haltAllPlayback('barge-in via wake-word');
|
||||
setAgentActivity({ activity: 'idle', tool: '' });
|
||||
rvs.send('cancel_request' as any, {});
|
||||
// Kurze Pause damit halt durchgreift, dann neue Aufnahme starten
|
||||
await new Promise(r => setTimeout(r, 150));
|
||||
const windowMs = await loadConvWindowMs();
|
||||
const started = await audioService.startRecording(true, windowMs);
|
||||
if (started) {
|
||||
ToastAndroid.show('🎤 Mikro offen — sprich jetzt', ToastAndroid.SHORT);
|
||||
playWakeReadySound().catch(() => {});
|
||||
}
|
||||
});
|
||||
|
||||
// TTS-Lifecycle: solange ARIA spricht und Wake-Word verfuegbar ist,
|
||||
// parallel mitlauschen — User kann "Computer" sagen statt manuell tappen.
|
||||
const unsubTtsStart = audioService.onPlaybackStarted(() => {
|
||||
if (wakeWordService.isConversing() && wakeWordService.hasWakeWord()) {
|
||||
wakeWordService.startBargeListening().catch(() => {});
|
||||
}
|
||||
});
|
||||
const unsubTtsEnd = audioService.onPlaybackFinished(() => {
|
||||
// Vor naechster Aufnahme: barge-listening aus damit der AudioRecorder
|
||||
// das Mikro greifen kann.
|
||||
wakeWordService.stopBargeListening().catch(() => {});
|
||||
});
|
||||
|
||||
return () => {
|
||||
unsubWake();
|
||||
unsubSilence();
|
||||
unsubBarge();
|
||||
unsubTtsStart();
|
||||
unsubTtsEnd();
|
||||
};
|
||||
}, [wakeWordActive]);
|
||||
|
||||
@@ -677,12 +722,14 @@ const ChatScreen: React.FC = () => {
|
||||
// Barge-In: laufende ARIA-Aktivitaet abbrechen falls aktiv.
|
||||
const wasInterrupted = interruptAriaIfBusy();
|
||||
const location = await getCurrentLocation();
|
||||
const audioRequestId = `audio_${Date.now()}_${Math.floor(Math.random() * 100000)}`;
|
||||
|
||||
const userMsg: ChatMessage = {
|
||||
id: nextId(),
|
||||
sender: 'user',
|
||||
text: '🎙 Spracheingabe wird verarbeitet...',
|
||||
timestamp: Date.now(),
|
||||
audioRequestId,
|
||||
};
|
||||
setMessages(prev => capMessages([...prev, userMsg]));
|
||||
|
||||
@@ -693,6 +740,7 @@ const ChatScreen: React.FC = () => {
|
||||
voice: localXttsVoiceRef.current,
|
||||
speed: ttsSpeedRef.current,
|
||||
interrupted: wasInterrupted,
|
||||
audioRequestId,
|
||||
...(location && { location }),
|
||||
});
|
||||
}, [getCurrentLocation, interruptAriaIfBusy]);
|
||||
|
||||
@@ -44,6 +44,11 @@ import {
|
||||
TTS_SPEED_MAX,
|
||||
TTS_SPEED_STORAGE_KEY,
|
||||
} from '../services/audio';
|
||||
import {
|
||||
isWakeReadySoundEnabled,
|
||||
setWakeReadySoundEnabled,
|
||||
playWakeReadySound,
|
||||
} from '../services/wakeReadySound';
|
||||
import wakeWordService, {
|
||||
WAKE_KEYWORDS,
|
||||
KEYWORD_LABELS,
|
||||
@@ -122,6 +127,7 @@ const SettingsScreen: React.FC = () => {
|
||||
const [ttsSpeed, setTtsSpeed] = useState<number>(TTS_SPEED_DEFAULT);
|
||||
const [wakeKeyword, setWakeKeyword] = useState<string>(DEFAULT_KEYWORD);
|
||||
const [wakeStatus, setWakeStatus] = useState<string>('');
|
||||
const [wakeReadySound, setWakeReadySound] = useState<boolean>(true);
|
||||
const [editingPath, setEditingPath] = useState(false);
|
||||
const [xttsVoice, setXttsVoice] = useState('');
|
||||
const [loadingVoice, setLoadingVoice] = useState<string | null>(null);
|
||||
@@ -194,6 +200,7 @@ const SettingsScreen: React.FC = () => {
|
||||
AsyncStorage.getItem(WAKE_KEYWORD_STORAGE).then(saved => {
|
||||
if (saved && (WAKE_KEYWORDS as readonly string[]).includes(saved)) setWakeKeyword(saved);
|
||||
});
|
||||
isWakeReadySoundEnabled().then(setWakeReadySound);
|
||||
AsyncStorage.getItem('aria_xtts_voice').then(saved => {
|
||||
if (saved) setXttsVoice(saved);
|
||||
});
|
||||
@@ -828,6 +835,31 @@ const SettingsScreen: React.FC = () => {
|
||||
{!!wakeStatus && (
|
||||
<Text style={{marginTop: 8, fontSize: 12, color: '#8888AA'}}>{wakeStatus}</Text>
|
||||
)}
|
||||
|
||||
<View style={[styles.toggleRow, {marginTop: 20, borderTopWidth: 1, borderTopColor: '#1E1E2E', paddingTop: 16}]}>
|
||||
<View style={styles.toggleInfo}>
|
||||
<Text style={styles.toggleLabel}>Bereit-Sound abspielen</Text>
|
||||
<Text style={styles.toggleHint}>
|
||||
Kurzer Ding-Dong wenn das Mikro nach Wake-Word offen ist —
|
||||
akustische Bestaetigung dass du jetzt sprechen darfst.
|
||||
</Text>
|
||||
</View>
|
||||
<Switch
|
||||
value={wakeReadySound}
|
||||
onValueChange={async (val) => {
|
||||
setWakeReadySound(val);
|
||||
await setWakeReadySoundEnabled(val);
|
||||
if (val) {
|
||||
// Direkt eine Vorschau abspielen damit der User weiss wie's klingt.
|
||||
// playWakeReadySound checked das gerade gesetzte Flag — wenn val=true,
|
||||
// wird abgespielt; bei false bleibt es still.
|
||||
setTimeout(() => playWakeReadySound().catch(() => {}), 150);
|
||||
}
|
||||
}}
|
||||
trackColor={{ false: '#2A2A3E', true: '#0096FF' }}
|
||||
thumbColor={wakeReadySound ? '#FFFFFF' : '#666680'}
|
||||
/>
|
||||
</View>
|
||||
</View>
|
||||
</>)}
|
||||
|
||||
|
||||
@@ -668,6 +668,7 @@ class AudioService {
|
||||
}
|
||||
this._cancelDeferredFocusRelease();
|
||||
AudioFocus?.requestDuck().catch(() => {});
|
||||
this._firePlaybackStarted();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -782,6 +783,7 @@ class AudioService {
|
||||
|
||||
// Callback wenn alle Audio-Teile abgespielt sind
|
||||
private playbackFinishedListeners: (() => void)[] = [];
|
||||
private playbackStartedListeners: (() => void)[] = [];
|
||||
|
||||
onPlaybackFinished(callback: () => void): () => void {
|
||||
this.playbackFinishedListeners.push(callback);
|
||||
@@ -790,6 +792,21 @@ class AudioService {
|
||||
};
|
||||
}
|
||||
|
||||
/** Callback wenn ARIAs TTS-Wiedergabe startet — fuer Wake-Word-parallel-
|
||||
* Listening waehrend ARIA spricht (Barge-In via "Computer" sagen). */
|
||||
onPlaybackStarted(callback: () => void): () => void {
|
||||
this.playbackStartedListeners.push(callback);
|
||||
return () => {
|
||||
this.playbackStartedListeners = this.playbackStartedListeners.filter(cb => cb !== callback);
|
||||
};
|
||||
}
|
||||
|
||||
private _firePlaybackStarted(): void {
|
||||
this.playbackStartedListeners.forEach(cb => {
|
||||
try { cb(); } catch (e) { console.warn('[Audio] playbackStarted listener err:', e); }
|
||||
});
|
||||
}
|
||||
|
||||
/** Naechstes Audio aus der Queue abspielen */
|
||||
private async _playNext(): Promise<void> {
|
||||
if (this.audioQueue.length === 0) {
|
||||
@@ -802,10 +819,11 @@ class AudioService {
|
||||
return;
|
||||
}
|
||||
|
||||
// Beim ersten Playback-Start: andere Apps ducken
|
||||
// Beim ersten Playback-Start: andere Apps ducken + Listener informieren
|
||||
if (!this.isPlaying) {
|
||||
this._cancelDeferredFocusRelease();
|
||||
AudioFocus?.requestDuck().catch(() => {});
|
||||
this._firePlaybackStarted();
|
||||
}
|
||||
this.isPlaying = true;
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* Spielt einen kurzen "Bereit"-Sound (Airplane Ding-Dong) wenn das Mikrofon
|
||||
* nach Wake-Word-Erkennung wirklich offen ist. Datei liegt in
|
||||
* android/app/src/main/res/raw/wake_ready_sound.mp3 — wird ueber Android's
|
||||
* Resource-System per react-native-sound abgespielt.
|
||||
*
|
||||
* Toggle: AsyncStorage-Key 'aria_wake_ready_sound_enabled' (default true).
|
||||
*/
|
||||
|
||||
import Sound from 'react-native-sound';
|
||||
import AsyncStorage from '@react-native-async-storage/async-storage';
|
||||
|
||||
export const WAKE_READY_SOUND_STORAGE_KEY = 'aria_wake_ready_sound_enabled';
|
||||
|
||||
Sound.setCategory('Playback', false);
|
||||
|
||||
let cachedSound: Sound | null = null;
|
||||
let cachedFailed = false;
|
||||
|
||||
function getSound(): Promise<Sound | null> {
|
||||
if (cachedFailed) return Promise.resolve(null);
|
||||
if (cachedSound) return Promise.resolve(cachedSound);
|
||||
return new Promise(resolve => {
|
||||
const s = new Sound('wake_ready_sound', Sound.MAIN_BUNDLE, (err) => {
|
||||
if (err) {
|
||||
console.warn('[WakeReadySound] Konnte nicht geladen werden:', err);
|
||||
cachedFailed = true;
|
||||
resolve(null);
|
||||
return;
|
||||
}
|
||||
cachedSound = s;
|
||||
resolve(s);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** True wenn der User den "Bereit"-Sound aktiviert hat. Default: true. */
|
||||
export async function isWakeReadySoundEnabled(): Promise<boolean> {
|
||||
try {
|
||||
const raw = await AsyncStorage.getItem(WAKE_READY_SOUND_STORAGE_KEY);
|
||||
if (raw === null) return true; // Default an
|
||||
return raw === 'true';
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
export async function setWakeReadySoundEnabled(enabled: boolean): Promise<void> {
|
||||
try {
|
||||
await AsyncStorage.setItem(WAKE_READY_SOUND_STORAGE_KEY, String(enabled));
|
||||
} catch {}
|
||||
}
|
||||
|
||||
/** Spielt den Bereit-Sound einmal ab — non-blocking. Wenn der User ihn
|
||||
* in den Settings deaktiviert hat oder die Datei nicht ladbar ist,
|
||||
* passiert einfach nichts. */
|
||||
export async function playWakeReadySound(): Promise<void> {
|
||||
if (!(await isWakeReadySoundEnabled())) return;
|
||||
const s = await getSound();
|
||||
if (!s) return;
|
||||
try {
|
||||
s.stop(() => {
|
||||
s.setCurrentTime(0);
|
||||
s.play((success) => {
|
||||
if (!success) console.warn('[WakeReadySound] Wiedergabe fehlgeschlagen');
|
||||
});
|
||||
});
|
||||
} catch (e) {
|
||||
console.warn('[WakeReadySound] play() Exception:', e);
|
||||
}
|
||||
}
|
||||
@@ -72,6 +72,11 @@ class WakeWordService {
|
||||
private state: WakeWordState = 'off';
|
||||
private wakeCallbacks: WakeWordCallback[] = [];
|
||||
private stateCallbacks: StateCallback[] = [];
|
||||
/** Barge-In-Callbacks: feuern wenn Wake-Word WAEHREND ARIA spricht erkannt
|
||||
* wird. ChatScreen reagiert mit TTS-stop + neuer Aufnahme. */
|
||||
private bargeCallbacks: WakeWordCallback[] = [];
|
||||
/** True solange Wake-Word parallel zu TTS aktiv ist. */
|
||||
private bargeListening: boolean = false;
|
||||
|
||||
private keyword: WakeKeyword = DEFAULT_KEYWORD;
|
||||
private nativeReady: boolean = false;
|
||||
@@ -191,16 +196,28 @@ class WakeWordService {
|
||||
if (this.nativeReady && OpenWakeWord) {
|
||||
try { await OpenWakeWord.stop(); } catch {}
|
||||
}
|
||||
this.bargeListening = false;
|
||||
this.setState('off');
|
||||
}
|
||||
|
||||
/** Wake-Word getriggert: Native-Modul pausieren, Konversation starten. */
|
||||
private async onWakeDetected(): Promise<void> {
|
||||
console.log('[WakeWord] Wake-Word "%s" erkannt!', this.keyword);
|
||||
ToastAndroid.show(`Wake-Word "${KEYWORD_LABELS[this.keyword]}" erkannt — sprich jetzt`, ToastAndroid.SHORT);
|
||||
console.log('[WakeWord] Wake-Word "%s" erkannt! (state=%s, barge=%s)',
|
||||
this.keyword, this.state, this.bargeListening);
|
||||
if (this.nativeReady && OpenWakeWord) {
|
||||
try { await OpenWakeWord.stop(); } catch {}
|
||||
}
|
||||
this.bargeListening = false;
|
||||
// Wenn wir bereits in 'conversing' sind und der Trigger waehrend ARIAs TTS
|
||||
// kam (Barge-In via Wake-Word), feuern wir einen separaten Callback damit
|
||||
// ChatScreen das TTS abbrechen + neue Aufnahme starten kann. Sonst normal.
|
||||
if (this.state === 'conversing') {
|
||||
this.bargeCallbacks.forEach(cb => {
|
||||
try { cb(); } catch (e) { console.warn('[WakeWord] barge cb err:', e); }
|
||||
});
|
||||
// Kein erneutes setState — wir bleiben in 'conversing'.
|
||||
return;
|
||||
}
|
||||
this.setState('conversing');
|
||||
setTimeout(() => {
|
||||
if (this.state === 'conversing') {
|
||||
@@ -209,6 +226,35 @@ class WakeWordService {
|
||||
}, 200);
|
||||
}
|
||||
|
||||
/** Wake-Word PARALLEL zur TTS-Wiedergabe lauschen lassen — User kann
|
||||
* "Computer" sagen waehrend ARIA noch redet, AcousticEchoCanceler im
|
||||
* Native-Modul verhindert dass ARIAs eigene Stimme triggert.
|
||||
* Voraussetzung: AudioRecorder muss frei sein (Recording aus). Wenn der
|
||||
* AudioRecorder gerade laeuft, hat der Vorrang — Wake-Word geht nicht. */
|
||||
async startBargeListening(): Promise<void> {
|
||||
if (!this.nativeReady || !OpenWakeWord) return;
|
||||
if (this.state !== 'conversing') return;
|
||||
if (this.bargeListening) return;
|
||||
try {
|
||||
await OpenWakeWord.start();
|
||||
this.bargeListening = true;
|
||||
console.log('[WakeWord] Barge-Listening aktiv (parallel zu TTS)');
|
||||
} catch (err) {
|
||||
console.warn('[WakeWord] Barge-Listening start fehlgeschlagen:', err);
|
||||
}
|
||||
}
|
||||
|
||||
/** Barge-Listening wieder aus — z.B. wenn der AudioRecorder fuer die
|
||||
* naechste Aufnahme das Mikro braucht. */
|
||||
async stopBargeListening(): Promise<void> {
|
||||
if (!this.bargeListening) return;
|
||||
if (this.nativeReady && OpenWakeWord) {
|
||||
try { await OpenWakeWord.stop(); } catch {}
|
||||
}
|
||||
this.bargeListening = false;
|
||||
console.log('[WakeWord] Barge-Listening aus');
|
||||
}
|
||||
|
||||
/** Konversation beenden — User hat im Window nichts gesagt.
|
||||
* Mit Wake-Word: zurueck zu 'armed' (Listener wieder an).
|
||||
* Ohne: zurueck zu 'off'.
|
||||
@@ -268,6 +314,19 @@ class WakeWordService {
|
||||
};
|
||||
}
|
||||
|
||||
/** Subscribe auf Barge-In-Events: Wake-Word erkannt waehrend ARIA noch
|
||||
* spricht. ChatScreen sollte dann TTS abbrechen + neue Aufnahme starten. */
|
||||
onBargeIn(callback: WakeWordCallback): () => void {
|
||||
this.bargeCallbacks.push(callback);
|
||||
return () => {
|
||||
this.bargeCallbacks = this.bargeCallbacks.filter(cb => cb !== callback);
|
||||
};
|
||||
}
|
||||
|
||||
isBargeListening(): boolean {
|
||||
return this.bargeListening;
|
||||
}
|
||||
|
||||
onStateChange(callback: StateCallback): () => void {
|
||||
this.stateCallbacks.push(callback);
|
||||
return () => {
|
||||
|
||||
+20
-9
@@ -1510,10 +1510,12 @@ class ARIABridge:
|
||||
except (TypeError, ValueError):
|
||||
self._next_speed_override = None
|
||||
interrupted = bool(payload.get("interrupted", False))
|
||||
logger.info("[rvs] Audio empfangen: %s, %dms, %dKB%s",
|
||||
audio_request_id = payload.get("audioRequestId", "") or ""
|
||||
logger.info("[rvs] Audio empfangen: %s, %dms, %dKB%s%s",
|
||||
mime_type, duration_ms, len(audio_b64) // 1365,
|
||||
" [BARGE-IN]" if interrupted else "")
|
||||
asyncio.create_task(self._process_app_audio(audio_b64, mime_type, interrupted))
|
||||
" [BARGE-IN]" if interrupted else "",
|
||||
f" reqId={audio_request_id[:16]}" if audio_request_id else "")
|
||||
asyncio.create_task(self._process_app_audio(audio_b64, mime_type, interrupted, audio_request_id))
|
||||
|
||||
elif msg_type == "stt_response":
|
||||
# Antwort der whisper-bridge auf unseren stt_request
|
||||
@@ -1569,13 +1571,19 @@ class ARIABridge:
|
||||
_STT_REMOTE_TIMEOUT_READY_S = 45.0
|
||||
_STT_REMOTE_TIMEOUT_LOADING_S = 300.0
|
||||
|
||||
async def _process_app_audio(self, audio_b64: str, mime_type: str, interrupted: bool = False) -> None:
|
||||
async def _process_app_audio(self, audio_b64: str, mime_type: str,
|
||||
interrupted: bool = False,
|
||||
audio_request_id: str = "") -> None:
|
||||
"""App-Audio → STT → aria-core. Primaer via whisper-bridge (RVS), Fallback lokal.
|
||||
|
||||
interrupted=True wenn der User waehrend ARIA noch sprach/dachte aufgenommen hat
|
||||
(Barge-In). Wird als Hinweis-Praefix an aria-core mitgegeben damit ARIA die
|
||||
Korrektur/Unterbrechung in den Kontext einordnen kann statt als reine
|
||||
Folgefrage zu behandeln."""
|
||||
Folgefrage zu behandeln.
|
||||
|
||||
audio_request_id: Korrelations-ID die die App im audio-Event mitschickt — wird
|
||||
unveraendert ans STT-Result zurueckgegeben damit die App die EXAKT richtige
|
||||
'wird verarbeitet'-Bubble ersetzen kann (auch bei mehreren parallelen Aufnahmen)."""
|
||||
# Erst Remote versuchen
|
||||
text = await self._stt_remote(audio_b64, mime_type)
|
||||
if text is None:
|
||||
@@ -1601,12 +1609,15 @@ class ARIABridge:
|
||||
# STT-Text an RVS senden (fuer Anzeige in App + Diagnostic)
|
||||
# sender="stt" damit Bridge es ignoriert (kein Loop)
|
||||
try:
|
||||
stt_payload = {
|
||||
"text": text,
|
||||
"sender": "stt",
|
||||
}
|
||||
if audio_request_id:
|
||||
stt_payload["audioRequestId"] = audio_request_id
|
||||
ok = await self._send_to_rvs({
|
||||
"type": "chat",
|
||||
"payload": {
|
||||
"text": text,
|
||||
"sender": "stt",
|
||||
},
|
||||
"payload": stt_payload,
|
||||
"timestamp": int(asyncio.get_event_loop().time() * 1000),
|
||||
})
|
||||
if ok:
|
||||
|
||||
@@ -104,11 +104,14 @@
|
||||
- [x] Push-to-Talk raus, nur noch Tap-to-Talk (verhinderte Touch-Race-Probleme)
|
||||
- [x] Settings-Sub-Screens: 8 Kategorien (Verbindung, Allgemein, Spracheingabe, Wake-Word, Sprachausgabe, Speicher, Protokoll, Ueber) statt langer Liste
|
||||
- [x] Textauswahl in Bubbles wieder funktional (nested Text+onPress raus, dataDetectorType="all" macht Links automatisch klickbar)
|
||||
- [x] **Placeholder-Race bei parallelen Sprachnachrichten geloest**: jede Aufnahme bekommt eine eindeutige audioRequestId, Bridge gibt sie ans STT-Result zurueck — App matcht jetzt punktgenau die richtige Bubble statt per Substring "Spracheingabe wird verarbeitet"
|
||||
- [x] Mikro-Offen-Toast "🎤 sprich jetzt" erscheint erst wenn audioService.startRecording wirklich erfolgreich war (statt ~400ms vorher beim Wake-Word-Detect)
|
||||
- [x] **Bereit-Sound (Airplane Ding-Dong) wenn Mikro nach Wake-Word offen** — akustische Bestaetigung statt nur Toast. Toggle in Settings → Wake-Word, default aktiv
|
||||
- [x] **Wake-Word parallel zu TTS** mit AcousticEchoCanceler: User sagt "Computer" waehrend ARIA spricht → TTS verstummt sofort, neue Aufnahme startet. Native AEC verhindert dass ARIAs eigene Stimme das Wake-Word triggert. Audio-Source ist VOICE_COMMUNICATION + zusaetzlich AEC/NS/AGC-Effekte aktiviert
|
||||
|
||||
## Offen
|
||||
|
||||
### Bugs
|
||||
- [ ] App: STT-Text ersetzt Placeholder nicht — Toast-Debug + Bridge-Log eingebaut, beim naechsten Test pruefen ob das chat-Event mit sender=stt in der App ankommt
|
||||
|
||||
### App Features
|
||||
- [ ] Chat-History zuverlaessiger laden (AsyncStorage Race Condition)
|
||||
|
||||
Reference in New Issue
Block a user