feat(brain): Phase B — Vector-DB-Memory, Conversation-Loop, Skills, Tool-Use
OpenClaw (aria-core) ist raus, ARIA laeuft jetzt mit eigenem Agent-Framework
im aria-brain Container. Vector-DB-basiertes Gedaechtnis statt Sessions,
eigener Conversation-Loop mit Hot+Cold-Memory + Rolling Window, Tool-Use
fuer Skills, Memory-Destillat-Pipeline.
aria-brain/ (neuer Container)
- main.py FastAPI auf 8080, alle Endpoints
- agent.py Conversation-Loop mit Tool-Use (skill_create + run_<skill>)
- conversation.py Rolling Window, JSONL-Persistenz, Distill-Marker
- proxy_client.py httpx-Wrapper zum Claude-Proxy, OpenAI-Format
- prompts.py System-Prompt aus Hot+Cold+Skills
- migration.py Markdown-Parser fuer brain-import/ → atomare Memories
- skills.py Filesystem-Layer fuer /data/skills/<name>/ (Python-only,
venv pro Skill, tar.gz Export/Import, Run-Logs)
- memory/ Embedder (sentence-transformers, multilingual MiniLM)
+ VectorStore (Qdrant-Wrapper)
docker-compose.yml
- aria-core (OpenClaw) raus, openclaw-config Volume raus
- aria-brain Service (FastAPI + Memory)
- aria-qdrant Service (Vector-DB) mit Bind-Mount aria-data/brain/qdrant/
- Diagnostic teilt jetzt Netzwerk mit Bridge (vorher: aria-core)
- Brain bekommt SSH-Mount fuer aria-wohnung + /import fuer brain-import/
bridge/aria_bridge.py
- send_to_core → HTTP-Call an aria-brain:8080/chat (statt OpenClaw-WS)
- aria-core-spezifische Handler raus: doctor_fix, aria_restart,
aria_session_reset, Auto-Compact-Logik, OpenClaw-Handshake
- Generischer container_restart-Handler (Whitelist Bridge/Brain/Qdrant)
- Side-Channel-Events aus /chat-Response (z.B. skill_created) werden
als RVS-Events forwarded
- file_list_request / file_delete_request → an Diagnostic forwarded
- Tote OpenClaw-Connection-Logik bleibt im Code als Referenz (nicht aktiv)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
# ════════════════════════════════════════════════════════════
|
||||
# ARIA Brain — Agent + Memory Container
|
||||
#
|
||||
# FastAPI-Server mit Vector-DB-Memory (Qdrant).
|
||||
# Spricht via HTTP/WebSocket mit Bridge und Diagnostic.
|
||||
# LLM-Calls gehen ueber den Proxy (claude-max-api-proxy).
|
||||
# ════════════════════════════════════════════════════════════
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
# System-Tools die Skills brauchen koennten (curl, jq, git, ssh-client,
|
||||
# Build-Basics fuer venv-Compiles). Bewusst sparsam — alles weitere
|
||||
# bringt der Skill selbst mit (siehe execution=local-bin).
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
jq \
|
||||
git \
|
||||
openssh-client \
|
||||
ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
# Embedding-Model-Cache und Skills landen unter /data (Volume)
|
||||
ENV SENTENCE_TRANSFORMERS_HOME=/data/_models
|
||||
ENV ARIA_DATA_DIR=/data
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
@@ -0,0 +1,385 @@
|
||||
"""
|
||||
Conversation-Loop. Eine Anfrage von Stefan, eine Antwort von ARIA.
|
||||
|
||||
Pro Turn:
|
||||
1. user-Turn an die laufende Conversation appenden
|
||||
2. Hot Memory holen (alle pinned Punkte)
|
||||
3. Cold Memory holen (Top-K semantisch zur user-Nachricht)
|
||||
4. System-Prompt aus Hot+Cold bauen
|
||||
5. Messages = [system, *window, user]
|
||||
6. Claude via Proxy aufrufen
|
||||
7. Assistant-Reply in Conversation appenden + zurueckgeben
|
||||
|
||||
Memory-Destillat laeuft asynchron NACH dem Reply, gesteuert vom
|
||||
/chat-Endpoint ueber BackgroundTasks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from conversation import Conversation, Turn
|
||||
from memory import Embedder, VectorStore, MemoryPoint
|
||||
from prompts import build_system_prompt
|
||||
from proxy_client import ProxyClient, Message as ProxyMessage
|
||||
import skills as skills_mod
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Meta-Tool: ARIA kann selbst neue Skills bauen
|
||||
META_TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "skill_create",
|
||||
"description": (
|
||||
"Erstelle einen neuen Skill (wiederverwendbare Faehigkeit). "
|
||||
"Skills sind IMMER Python — jeder Skill bekommt seine eigene venv "
|
||||
"mit den pip_packages die er braucht.\n\n"
|
||||
"HARTE REGEL — IMMER Skill anlegen wenn: die Loesung erfordert eine "
|
||||
"pip-Library. Sonst muesste der Install bei jedem Container-Restart "
|
||||
"neu laufen (Brain hat keinen persistenten State ausser /data/skills/).\n\n"
|
||||
"Sonst NUR wenn ALLE Kriterien erfuellt sind:\n"
|
||||
" 1) wiederkehrend (Aufgabe kommt realistisch nochmal),\n"
|
||||
" 2) nicht-trivial (mehrere Schritte),\n"
|
||||
" 3) parametrisierbar (nimmt Eingaben, gibt Ergebnis),\n"
|
||||
" 4) wiederverwendbar als ganzes Paket.\n"
|
||||
"NICHT fuer einzelne Shell-Befehle (date, hostname, ls etc.) und "
|
||||
"nicht fuer Einmal-Faelle. Stefan kann Skill-Erstellung explizit "
|
||||
"triggern (\"bau daraus einen Skill\").\n\n"
|
||||
"Wenn etwas nur via apt-Paket geht — Stefan fragen ob es ins "
|
||||
"Brain-Dockerfile soll, NICHT als Skill bauen."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "description": "kurz, kebab-case, a-z 0-9 - _"},
|
||||
"description": {"type": "string", "description": "Was kann der Skill? 1 Satz."},
|
||||
"entry_code": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Python-Code. Args lesen via os.environ['ARG_NAME']. "
|
||||
"Resultat per print() (stdout) zurueck. Bei Fehler: "
|
||||
"non-zero exit (sys.exit(1) o.ae.)."
|
||||
),
|
||||
},
|
||||
"readme": {"type": "string", "description": "Markdown — was macht der Skill, Beispiel-Aufrufe"},
|
||||
"pip_packages": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "pip-Pakete die in der venv installiert werden (z.B. requests, yt-dlp, pypdf)",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "object"},
|
||||
"description": "Argumente-Schema [{name, type, required, description}]",
|
||||
},
|
||||
},
|
||||
"required": ["name", "description", "entry_code"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "skill_list",
|
||||
"description": "Zeigt alle Skills (inkl. deaktivierte). Sollte selten noetig sein — die Liste steht eh im System-Prompt.",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _skill_to_tool(s: dict) -> dict:
|
||||
"""Mappt einen Skill auf ein OpenAI-Function-Tool."""
|
||||
args = s.get("args") or []
|
||||
props = {}
|
||||
required = []
|
||||
for a in args:
|
||||
if not isinstance(a, dict):
|
||||
continue
|
||||
name = a.get("name") or ""
|
||||
if not name:
|
||||
continue
|
||||
props[name] = {
|
||||
"type": a.get("type", "string"),
|
||||
"description": a.get("description", ""),
|
||||
}
|
||||
if a.get("required"):
|
||||
required.append(name)
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": f"run_{s['name']}",
|
||||
"description": s.get("description", "(ohne Beschreibung)"),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": props,
|
||||
"required": required,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class Agent:
|
||||
def __init__(self, store: VectorStore, embedder: Embedder,
|
||||
conversation: Conversation, proxy: ProxyClient,
|
||||
cold_k: int = 5):
|
||||
self.store = store
|
||||
self.embedder = embedder
|
||||
self.conversation = conversation
|
||||
self.proxy = proxy
|
||||
self.cold_k = cold_k
|
||||
# Side-Channel-Events die im Turn entstehen (z.B. skill_create).
|
||||
# Werden vom /chat-Endpoint in der Response mitgeschickt, damit
|
||||
# Stefan in der App und Diagnostic eine sichtbare Bubble bekommt.
|
||||
self._pending_events: list[dict] = []
|
||||
|
||||
def pop_events(self) -> list[dict]:
|
||||
"""Holt die Events des letzten chat()-Calls und leert die Liste."""
|
||||
events = self._pending_events
|
||||
self._pending_events = []
|
||||
return events
|
||||
|
||||
# ── Hauptpfad: ein User-Turn → Tool-Loop → finaler Reply ──
|
||||
|
||||
MAX_TOOL_ITERATIONS = 8 # Schutz vor Endlos-Loops
|
||||
|
||||
def chat(self, user_message: str, source: str = "") -> str:
|
||||
user_message = (user_message or "").strip()
|
||||
if not user_message:
|
||||
raise ValueError("Leere Nachricht")
|
||||
|
||||
# Events vom letzten Turn weglassen
|
||||
self._pending_events = []
|
||||
|
||||
# 1. User-Turn an die Konversation
|
||||
self.conversation.add("user", user_message, source=source)
|
||||
|
||||
# 2. Hot Memory (alle pinned Punkte)
|
||||
hot = self.store.list_pinned()
|
||||
|
||||
# 3. Cold Memory (Top-K semantic)
|
||||
try:
|
||||
qvec = self.embedder.embed(user_message)
|
||||
cold = self.store.search(qvec, k=self.cold_k, exclude_pinned=True)
|
||||
except Exception as exc:
|
||||
logger.warning("Cold-Search fehlgeschlagen: %s", exc)
|
||||
cold = []
|
||||
|
||||
# 4. Aktive Skills holen + Tool-Liste bauen
|
||||
all_skills = skills_mod.list_skills(active_only=False)
|
||||
active_skills = [s for s in all_skills if s.get("active", True)]
|
||||
tools = list(META_TOOLS) + [_skill_to_tool(s) for s in active_skills]
|
||||
|
||||
# 5. System-Prompt + Window-Messages
|
||||
system_prompt = build_system_prompt(hot, cold, skills=all_skills)
|
||||
messages = [ProxyMessage(role="system", content=system_prompt)]
|
||||
for t in self.conversation.window():
|
||||
messages.append(ProxyMessage(role=t.role, content=t.content))
|
||||
|
||||
logger.info("chat: pinned=%d cold=%d skills=%d/%d window=%d prompt_chars=%d",
|
||||
len(hot), len(cold), len(active_skills), len(all_skills),
|
||||
len(self.conversation.window()), len(system_prompt))
|
||||
|
||||
# 6. Tool-Use-Loop
|
||||
final_reply = ""
|
||||
for iteration in range(self.MAX_TOOL_ITERATIONS):
|
||||
result = self.proxy.chat_full(messages, tools=tools)
|
||||
if result.tool_calls:
|
||||
# Assistant-Turn mit tool_calls in messages anhaengen (nicht in Conversation!)
|
||||
messages.append(ProxyMessage(
|
||||
role="assistant",
|
||||
content=result.content or None,
|
||||
tool_calls=[{
|
||||
"id": tc["id"], "type": "function",
|
||||
"function": {"name": tc["name"], "arguments": json.dumps(tc["arguments"])},
|
||||
} for tc in result.tool_calls],
|
||||
))
|
||||
# Tools ausfuehren + Ergebnis als role=tool zurueck
|
||||
for tc in result.tool_calls:
|
||||
tool_result = self._dispatch_tool(tc["name"], tc["arguments"])
|
||||
messages.append(ProxyMessage(
|
||||
role="tool",
|
||||
tool_call_id=tc["id"],
|
||||
name=tc["name"],
|
||||
content=tool_result[:8000],
|
||||
))
|
||||
continue # next iteration mit Tool-Results
|
||||
# Kein Tool-Call mehr → final reply
|
||||
final_reply = (result.content or "").strip()
|
||||
break
|
||||
else:
|
||||
# Loop-Limit erreicht
|
||||
final_reply = "[Tool-Loop-Limit erreicht — ARIA hat zu viele Tool-Calls gemacht ohne fertig zu werden]"
|
||||
logger.warning("Tool-Loop hit MAX_TOOL_ITERATIONS=%d", self.MAX_TOOL_ITERATIONS)
|
||||
|
||||
if not final_reply:
|
||||
raise RuntimeError("Leerer Reply vom Proxy")
|
||||
|
||||
# 7. Assistant-Turn (final reply) in die Conversation
|
||||
self.conversation.add("assistant", final_reply)
|
||||
return final_reply
|
||||
|
||||
# ── Tool-Dispatcher ───────────────────────────────────────
|
||||
|
||||
def _dispatch_tool(self, name: str, arguments: dict) -> str:
|
||||
"""Fuehrt einen Tool-Call aus und gibt ein kurzes Text-Resultat zurueck.
|
||||
Niemals werfen — Fehler werden als Text-Resultat reportet damit Claude
|
||||
weitermachen kann."""
|
||||
try:
|
||||
if name == "skill_create":
|
||||
# ARIA-Skills sind immer Python — execution ist nicht mehr im Schema
|
||||
manifest = skills_mod.create_skill(
|
||||
name=arguments["name"],
|
||||
description=arguments["description"],
|
||||
execution="local-venv",
|
||||
entry_code=arguments["entry_code"],
|
||||
readme=arguments.get("readme", ""),
|
||||
args=arguments.get("args", []),
|
||||
pip_packages=arguments.get("pip_packages", []),
|
||||
author="aria",
|
||||
)
|
||||
# Side-Channel-Event: Stefan soll sehen wenn ARIA was anlegt
|
||||
self._pending_events.append({
|
||||
"type": "skill_created",
|
||||
"skill": {
|
||||
"name": manifest["name"],
|
||||
"description": manifest.get("description", ""),
|
||||
"execution": manifest.get("execution", ""),
|
||||
"active": manifest.get("active", True),
|
||||
"setup_error": manifest.get("setup_error"),
|
||||
},
|
||||
})
|
||||
return f"OK — Skill '{manifest['name']}' erstellt (active={manifest['active']})."
|
||||
if name == "skill_list":
|
||||
items = skills_mod.list_skills(active_only=False)
|
||||
if not items:
|
||||
return "(keine Skills vorhanden)"
|
||||
return "\n".join(
|
||||
f"- {s['name']} ({s['execution']}) {'aktiv' if s.get('active', True) else 'DEAKTIVIERT'}: {s.get('description', '')}"
|
||||
for s in items
|
||||
)
|
||||
if name.startswith("run_"):
|
||||
skill_name = name[len("run_"):]
|
||||
res = skills_mod.run_skill(skill_name, args=arguments)
|
||||
snippet = (res.get("stdout") or "")[:2000] or "(kein stdout)"
|
||||
err = (res.get("stderr") or "")[:500]
|
||||
marker = "OK" if res["ok"] else f"FEHLER (exit={res['exit_code']})"
|
||||
out = f"{marker} · {res['duration_sec']}s\nstdout:\n{snippet}"
|
||||
if err:
|
||||
out += f"\nstderr:\n{err}"
|
||||
return out
|
||||
return f"Unbekanntes Tool: {name}"
|
||||
except Exception as exc:
|
||||
logger.exception("Tool '%s' fehlgeschlagen", name)
|
||||
return f"FEHLER: {exc}"
|
||||
|
||||
# ── Memory-Destillat (laeuft im Hintergrund) ──────────────
|
||||
|
||||
def distill_old_turns(self) -> dict:
|
||||
"""Nimmt die N aeltesten Turns und destilliert sie zu fact-Memories.
|
||||
|
||||
Pattern: separater Claude-Call, lieferte 3-7 JSON-Facts, die als
|
||||
type=fact, source=distilled gespeichert werden. Erfolgreiches
|
||||
Schreiben → Turns aus dem Window entfernen.
|
||||
"""
|
||||
if not self.conversation.needs_distill():
|
||||
return {"distilled": 0, "reason": "kein Bedarf"}
|
||||
|
||||
old_turns = self.conversation.take_oldest_for_distill()
|
||||
if not old_turns:
|
||||
return {"distilled": 0, "reason": "keine alten Turns"}
|
||||
|
||||
# Konversation als Klartext bauen
|
||||
transcript = "\n".join(
|
||||
f"[{t.role.upper()}] {t.content}" for t in old_turns
|
||||
)[:30000] # Cap auf 30k Zeichen damit der Prompt nicht explodiert
|
||||
|
||||
system = (
|
||||
"Du extrahierst aus einer Konversation zwischen Stefan und ARIA die "
|
||||
"wichtigsten dauerhaft relevanten Fakten — keine Smalltalk-Details, "
|
||||
"keine flüchtigen Zustände. Antworte AUSSCHLIESSLICH mit gültigem JSON "
|
||||
"im Format: {\"facts\": [{\"title\": \"kurz, max 80 Zeichen\", "
|
||||
"\"content\": \"1-3 Sätze, konkret und nützlich\"}]}. "
|
||||
"Mindestens 0, höchstens 7 Facts. Wenn nichts wichtig genug ist: leeres Array."
|
||||
)
|
||||
user = (
|
||||
"Hier ist der Konversations-Abschnitt:\n\n"
|
||||
f"{transcript}\n\n"
|
||||
"Extrahiere die wichtigsten Fakten als JSON."
|
||||
)
|
||||
|
||||
try:
|
||||
raw = self.proxy.chat([
|
||||
ProxyMessage(role="system", content=system),
|
||||
ProxyMessage(role="user", content=user),
|
||||
])
|
||||
except Exception as exc:
|
||||
logger.warning("Destillat-Call fehlgeschlagen: %s — Turns bleiben", exc)
|
||||
return {"distilled": 0, "error": str(exc)}
|
||||
|
||||
facts = self._parse_facts(raw)
|
||||
if facts is None:
|
||||
logger.warning("Destillat lieferte unparsbares JSON: %r", raw[:200])
|
||||
return {"distilled": 0, "error": "JSON parse failed", "raw": raw[:200]}
|
||||
|
||||
# Facts in die DB schreiben
|
||||
created = 0
|
||||
for f in facts:
|
||||
content = (f.get("content") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
title = (f.get("title") or "").strip()[:120] or "Fakt"
|
||||
point = MemoryPoint(
|
||||
id="",
|
||||
type="fact",
|
||||
title=title,
|
||||
content=content,
|
||||
pinned=False,
|
||||
category="konversation",
|
||||
source="distilled",
|
||||
tags=[],
|
||||
)
|
||||
try:
|
||||
vec = self.embedder.embed(content)
|
||||
self.store.upsert(point, vec)
|
||||
created += 1
|
||||
except Exception as exc:
|
||||
logger.warning("Fakt schreiben fehlgeschlagen: %s", exc)
|
||||
|
||||
# Erst nach erfolgreichem Schreiben aus dem Window entfernen
|
||||
last_ts = old_turns[-1].ts
|
||||
self.conversation.commit_distill(last_ts)
|
||||
logger.info("Destillat: %d Facts geschrieben, %d Turns aus Window entfernt",
|
||||
created, len(old_turns))
|
||||
return {"distilled": created, "removed_turns": len(old_turns)}
|
||||
|
||||
@staticmethod
|
||||
def _parse_facts(raw: str) -> Optional[list]:
|
||||
if not raw:
|
||||
return None
|
||||
# JSON robust extrahieren — Claude kann Code-Fences setzen
|
||||
cleaned = raw.strip()
|
||||
if cleaned.startswith("```"):
|
||||
# ```json oder ``` rauswerfen
|
||||
cleaned = cleaned.split("\n", 1)[1] if "\n" in cleaned else cleaned[3:]
|
||||
if cleaned.endswith("```"):
|
||||
cleaned = cleaned[: -3]
|
||||
cleaned = cleaned.strip()
|
||||
# Erstes { bis letztes }
|
||||
start = cleaned.find("{")
|
||||
end = cleaned.rfind("}")
|
||||
if start == -1 or end == -1 or end < start:
|
||||
return None
|
||||
try:
|
||||
obj = json.loads(cleaned[start: end + 1])
|
||||
except Exception:
|
||||
return None
|
||||
facts = obj.get("facts") if isinstance(obj, dict) else None
|
||||
if not isinstance(facts, list):
|
||||
return None
|
||||
return facts
|
||||
@@ -0,0 +1,130 @@
|
||||
"""
|
||||
Conversation-State — ein einziger Rolling-Window-State fuer ARIAs
|
||||
laufendes Gespraech mit Stefan.
|
||||
|
||||
Stefan-Entscheidung: KEINE Sessions, KEIN Multi-Thread. EIN Strang,
|
||||
intern rollend. Was rausfaellt, wird ggf. destilliert und landet
|
||||
als type=fact Memory in der Vector-DB.
|
||||
|
||||
Persistenz: append-only JSONL unter /data/conversation.jsonl.
|
||||
Bei Restart wird die letzte N gelesen (komplett vermeidet Memory-
|
||||
Overhead bei sehr langen Verlaeufen).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CONVERSATION_FILE = Path(os.environ.get("CONVERSATION_FILE", "/data/conversation.jsonl"))
|
||||
|
||||
|
||||
@dataclass
|
||||
class Turn:
|
||||
role: str # "user" | "assistant"
|
||||
content: str
|
||||
ts: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
|
||||
source: str = "" # "app" / "diagnostic" / "stt" — optional
|
||||
|
||||
|
||||
class Conversation:
|
||||
"""In-Memory Rolling Window, mit JSONL-Persistenz."""
|
||||
|
||||
def __init__(self, max_window: int = 50, distill_threshold: int = 60,
|
||||
distill_count: int = 30):
|
||||
self.max_window = max_window
|
||||
self.distill_threshold = distill_threshold
|
||||
self.distill_count = distill_count
|
||||
self.turns: List[Turn] = []
|
||||
self._load()
|
||||
|
||||
def _load(self):
|
||||
if not CONVERSATION_FILE.exists():
|
||||
return
|
||||
try:
|
||||
lines = CONVERSATION_FILE.read_text(encoding="utf-8").splitlines()
|
||||
except Exception as exc:
|
||||
logger.warning("Konversation laden fehlgeschlagen: %s", exc)
|
||||
return
|
||||
loaded: List[Turn] = []
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if obj.get("op") == "distill":
|
||||
# Marker: bis hierhin wurde alles destilliert
|
||||
drop_until_ts = obj.get("ts", "")
|
||||
if drop_until_ts:
|
||||
loaded = [t for t in loaded if t.ts > drop_until_ts]
|
||||
continue
|
||||
role = obj.get("role")
|
||||
content = obj.get("content")
|
||||
if role in ("user", "assistant") and isinstance(content, str):
|
||||
loaded.append(Turn(role=role, content=content,
|
||||
ts=obj.get("ts", ""),
|
||||
source=obj.get("source", "")))
|
||||
self.turns = loaded
|
||||
logger.info("Konversation geladen: %d Turns aus %s", len(self.turns), CONVERSATION_FILE)
|
||||
|
||||
def _append_to_file(self, record: dict):
|
||||
try:
|
||||
CONVERSATION_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with CONVERSATION_FILE.open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
except Exception as exc:
|
||||
logger.warning("Konversation persist fehlgeschlagen: %s", exc)
|
||||
|
||||
def add(self, role: str, content: str, source: str = "") -> Turn:
|
||||
t = Turn(role=role, content=content, source=source)
|
||||
self.turns.append(t)
|
||||
self._append_to_file({
|
||||
"ts": t.ts, "role": t.role, "content": t.content, "source": t.source,
|
||||
})
|
||||
return t
|
||||
|
||||
def window(self) -> List[Turn]:
|
||||
"""Die letzten max_window Turns — gehen in den LLM-Prompt."""
|
||||
return self.turns[-self.max_window:]
|
||||
|
||||
def needs_distill(self) -> bool:
|
||||
return len(self.turns) > self.distill_threshold
|
||||
|
||||
def take_oldest_for_distill(self) -> List[Turn]:
|
||||
"""Gibt die N aeltesten Turns zurueck — fuer den Destillat-Call.
|
||||
Entfernt sie NICHT — das macht commit_distill nach erfolgreichem Call."""
|
||||
return self.turns[: self.distill_count]
|
||||
|
||||
def commit_distill(self, last_distilled_ts: str):
|
||||
"""Schreibt einen Distill-Marker, entfernt aus dem In-Memory-Window."""
|
||||
self._append_to_file({"op": "distill", "ts": last_distilled_ts})
|
||||
self.turns = [t for t in self.turns if t.ts > last_distilled_ts]
|
||||
logger.info("Distill commit bei ts=%s — Window jetzt %d Turns", last_distilled_ts, len(self.turns))
|
||||
|
||||
def reset(self):
|
||||
"""Hardes Reset — verwende vorsichtig (Diagnostic-Button)."""
|
||||
try:
|
||||
if CONVERSATION_FILE.exists():
|
||||
CONVERSATION_FILE.unlink()
|
||||
except Exception:
|
||||
pass
|
||||
self.turns = []
|
||||
logger.warning("Konversation komplett zurueckgesetzt")
|
||||
|
||||
def stats(self) -> dict:
|
||||
return {
|
||||
"turns": len(self.turns),
|
||||
"max_window": self.max_window,
|
||||
"distill_threshold": self.distill_threshold,
|
||||
"needs_distill": self.needs_distill(),
|
||||
}
|
||||
@@ -0,0 +1,518 @@
|
||||
"""
|
||||
ARIA Brain — FastAPI-Einstieg.
|
||||
|
||||
Phase B Punkt 1: nur Skeleton.
|
||||
- /health → Liveness
|
||||
- /memory/list → alle Punkte (gefiltert)
|
||||
- /memory/pinned → Hot Memory
|
||||
- /memory/search?q=...&k=5 → semantische Suche
|
||||
- /memory/save → neuen Punkt anlegen
|
||||
- /memory/update/{id} → Punkt aendern (re-embed wenn content geaendert)
|
||||
- /memory/delete/{id} → Punkt loeschen
|
||||
- /memory/stats → Anzahl Punkte pro Type
|
||||
|
||||
/chat (Conversation-Loop) und /skills/* kommen in spaeteren Phasen.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import FastAPI, HTTPException, BackgroundTasks, Request
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from memory import Embedder, VectorStore, MemoryPoint
|
||||
from conversation import Conversation
|
||||
from proxy_client import ProxyClient
|
||||
from agent import Agent
|
||||
import skills as skills_mod
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
|
||||
logger = logging.getLogger("aria-brain")
|
||||
|
||||
QDRANT_HOST = os.environ.get("QDRANT_HOST", "aria-qdrant")
|
||||
QDRANT_PORT = int(os.environ.get("QDRANT_PORT", "6333"))
|
||||
|
||||
app = FastAPI(title="ARIA Brain", version="0.1.0")
|
||||
|
||||
_embedder: Optional[Embedder] = None
|
||||
_store: Optional[VectorStore] = None
|
||||
_conversation: Optional[Conversation] = None
|
||||
_proxy: Optional[ProxyClient] = None
|
||||
_agent: Optional[Agent] = None
|
||||
|
||||
|
||||
def embedder() -> Embedder:
|
||||
global _embedder
|
||||
if _embedder is None:
|
||||
_embedder = Embedder()
|
||||
return _embedder
|
||||
|
||||
|
||||
def store() -> VectorStore:
|
||||
global _store
|
||||
if _store is None:
|
||||
_store = VectorStore(host=QDRANT_HOST, port=QDRANT_PORT)
|
||||
return _store
|
||||
|
||||
|
||||
def conversation() -> Conversation:
|
||||
global _conversation
|
||||
if _conversation is None:
|
||||
_conversation = Conversation()
|
||||
return _conversation
|
||||
|
||||
|
||||
def proxy_client() -> ProxyClient:
|
||||
global _proxy
|
||||
if _proxy is None:
|
||||
_proxy = ProxyClient()
|
||||
return _proxy
|
||||
|
||||
|
||||
def agent() -> Agent:
|
||||
global _agent
|
||||
if _agent is None:
|
||||
_agent = Agent(store(), embedder(), conversation(), proxy_client())
|
||||
return _agent
|
||||
|
||||
|
||||
# ─── Pydantic-Schemas ─────────────────────────────────────────────────
|
||||
|
||||
class MemoryIn(BaseModel):
|
||||
type: str = Field(..., description="identity|rule|preference|tool|skill|fact|conversation|reminder")
|
||||
title: str
|
||||
content: str
|
||||
pinned: bool = False
|
||||
category: str = ""
|
||||
source: str = "manual"
|
||||
tags: List[str] = Field(default_factory=list)
|
||||
conversation_id: Optional[str] = None
|
||||
|
||||
|
||||
class MemoryUpdate(BaseModel):
|
||||
title: Optional[str] = None
|
||||
content: Optional[str] = None
|
||||
pinned: Optional[bool] = None
|
||||
category: Optional[str] = None
|
||||
tags: Optional[List[str]] = None
|
||||
|
||||
|
||||
class MemoryOut(BaseModel):
|
||||
id: str
|
||||
type: str
|
||||
title: str
|
||||
content: str
|
||||
pinned: bool
|
||||
category: str
|
||||
source: str
|
||||
tags: List[str]
|
||||
created_at: str
|
||||
updated_at: str
|
||||
conversation_id: Optional[str] = None
|
||||
score: Optional[float] = None
|
||||
|
||||
@classmethod
|
||||
def from_point(cls, p: MemoryPoint) -> "MemoryOut":
|
||||
return cls(**p.__dict__)
|
||||
|
||||
|
||||
# ─── Health ───────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
try:
|
||||
n = store().count()
|
||||
return {"status": "ok", "memory_count": n, "qdrant": f"{QDRANT_HOST}:{QDRANT_PORT}"}
|
||||
except Exception as exc:
|
||||
return {"status": "degraded", "error": str(exc), "qdrant": f"{QDRANT_HOST}:{QDRANT_PORT}"}
|
||||
|
||||
|
||||
# ─── Memory-Endpoints ─────────────────────────────────────────────────
|
||||
|
||||
@app.get("/memory/stats")
|
||||
def memory_stats():
|
||||
s = store()
|
||||
points = s.list_all()
|
||||
by_type = {}
|
||||
pinned = 0
|
||||
for p in points:
|
||||
by_type[p.type] = by_type.get(p.type, 0) + 1
|
||||
if p.pinned:
|
||||
pinned += 1
|
||||
return {"total": len(points), "pinned": pinned, "by_type": by_type}
|
||||
|
||||
|
||||
@app.get("/memory/list", response_model=List[MemoryOut])
|
||||
def memory_list(type: Optional[str] = None, limit: int = 200):
|
||||
s = store()
|
||||
points = s.list_by_type(type, limit=limit) if type else s.list_all(limit=limit)
|
||||
return [MemoryOut.from_point(p) for p in points]
|
||||
|
||||
|
||||
@app.get("/memory/pinned", response_model=List[MemoryOut])
|
||||
def memory_pinned():
|
||||
return [MemoryOut.from_point(p) for p in store().list_pinned()]
|
||||
|
||||
|
||||
@app.get("/memory/search", response_model=List[MemoryOut])
|
||||
def memory_search(q: str, k: int = 5, type: Optional[str] = None, include_pinned: bool = False):
|
||||
vec = embedder().embed(q)
|
||||
points = store().search(vec, k=k, type_filter=type, exclude_pinned=not include_pinned)
|
||||
return [MemoryOut.from_point(p) for p in points]
|
||||
|
||||
|
||||
@app.post("/memory/save", response_model=MemoryOut)
|
||||
def memory_save(body: MemoryIn):
|
||||
s = store()
|
||||
vec = embedder().embed(body.content)
|
||||
point = MemoryPoint(
|
||||
id="",
|
||||
type=body.type,
|
||||
title=body.title,
|
||||
content=body.content,
|
||||
pinned=body.pinned,
|
||||
category=body.category,
|
||||
source=body.source,
|
||||
tags=body.tags,
|
||||
conversation_id=body.conversation_id,
|
||||
)
|
||||
pid = s.upsert(point, vec)
|
||||
saved = s.get(pid)
|
||||
return MemoryOut.from_point(saved)
|
||||
|
||||
|
||||
@app.patch("/memory/update/{point_id}", response_model=MemoryOut)
|
||||
def memory_update(point_id: str, body: MemoryUpdate):
|
||||
s = store()
|
||||
existing = s.get(point_id)
|
||||
if not existing:
|
||||
raise HTTPException(404, f"Memory {point_id} nicht gefunden")
|
||||
|
||||
content_changed = body.content is not None and body.content != existing.content
|
||||
if body.title is not None:
|
||||
existing.title = body.title
|
||||
if body.content is not None:
|
||||
existing.content = body.content
|
||||
if body.pinned is not None:
|
||||
existing.pinned = body.pinned
|
||||
if body.category is not None:
|
||||
existing.category = body.category
|
||||
if body.tags is not None:
|
||||
existing.tags = body.tags
|
||||
|
||||
vec = embedder().embed(existing.content) if content_changed else None
|
||||
if vec is None:
|
||||
# Vektor unveraendert lassen — nur Payload neu schreiben
|
||||
from qdrant_client.http import models as qm
|
||||
from memory.vector_store import COLLECTION
|
||||
s.client.set_payload(
|
||||
collection_name=COLLECTION,
|
||||
payload=existing.to_payload() | {"updated_at": __import__("datetime").datetime.now(__import__("datetime").timezone.utc).isoformat()},
|
||||
points=[point_id],
|
||||
)
|
||||
saved = s.get(point_id)
|
||||
else:
|
||||
s.upsert(existing, vec)
|
||||
saved = s.get(point_id)
|
||||
return MemoryOut.from_point(saved)
|
||||
|
||||
|
||||
@app.delete("/memory/delete/{point_id}")
|
||||
def memory_delete(point_id: str):
|
||||
s = store()
|
||||
if not s.get(point_id):
|
||||
raise HTTPException(404, f"Memory {point_id} nicht gefunden")
|
||||
s.delete(point_id)
|
||||
return {"deleted": point_id}
|
||||
|
||||
|
||||
# ─── Migration aus brain-import/ ──────────────────────────────────────
|
||||
|
||||
IMPORT_DIR = os.environ.get("IMPORT_DIR", "/import")
|
||||
|
||||
|
||||
@app.post("/memory/migrate")
|
||||
def memory_migrate():
|
||||
"""Liest /import/*.md und schreibt atomare Memory-Punkte in die DB.
|
||||
Idempotent: bei Re-Run werden Punkte mit gleicher migration_key ersetzt."""
|
||||
from pathlib import Path
|
||||
from migration import run_migration
|
||||
s = store()
|
||||
e = embedder()
|
||||
result = run_migration(Path(IMPORT_DIR), s, e)
|
||||
return result
|
||||
|
||||
|
||||
@app.get("/memory/import-files")
|
||||
def memory_import_files():
|
||||
"""Listet was unter /import/ liegt — fuer die Diagnostic-UI."""
|
||||
from pathlib import Path
|
||||
d = Path(IMPORT_DIR)
|
||||
if not d.exists():
|
||||
return {"import_dir": str(d), "exists": False, "files": []}
|
||||
out = []
|
||||
for p in sorted(d.iterdir()):
|
||||
if p.is_file():
|
||||
try:
|
||||
out.append({"name": p.name, "size": p.stat().st_size})
|
||||
except Exception:
|
||||
pass
|
||||
return {"import_dir": str(d), "exists": True, "files": out}
|
||||
|
||||
|
||||
# ─── Bootstrap-Snapshot ───────────────────────────────────────────────
|
||||
# "Bootstrap" = alle pinned Memories. Export/Import zum schnellen
|
||||
# Wiederherstellen einer schlanken ARIA nach Wipe.
|
||||
|
||||
@app.get("/memory/export-bootstrap")
|
||||
def memory_export_bootstrap():
|
||||
"""Gibt alle pinned Memories als JSON zurueck — fuer Browser-Download."""
|
||||
s = store()
|
||||
pinned = s.list_pinned()
|
||||
return {
|
||||
"version": 1,
|
||||
"exported_at": __import__("datetime").datetime.now(
|
||||
__import__("datetime").timezone.utc
|
||||
).isoformat(),
|
||||
"count": len(pinned),
|
||||
"memories": [
|
||||
{
|
||||
"type": p.type,
|
||||
"title": p.title,
|
||||
"content": p.content,
|
||||
"pinned": True,
|
||||
"category": p.category,
|
||||
"source": p.source,
|
||||
"tags": p.tags,
|
||||
}
|
||||
for p in pinned
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class BootstrapBundle(BaseModel):
|
||||
version: int = 1
|
||||
memories: List[dict]
|
||||
|
||||
|
||||
@app.post("/memory/import-bootstrap")
|
||||
def memory_import_bootstrap(body: BootstrapBundle):
|
||||
"""Loescht alle pinned Memories und importiert die im Bundle.
|
||||
Cold Memory (unpinned) bleibt unangetastet.
|
||||
|
||||
Wenn keine Memories im Bundle: nur loeschen ist NICHT erlaubt — der
|
||||
Caller soll erst exportieren und dann importieren.
|
||||
"""
|
||||
if not body.memories:
|
||||
raise HTTPException(400, "Bundle hat keine memories — Abbruch zur Sicherheit")
|
||||
|
||||
s = store()
|
||||
e = embedder()
|
||||
|
||||
# Alle aktuell pinned Punkte loeschen
|
||||
from qdrant_client.http import models as qm
|
||||
from memory.vector_store import COLLECTION
|
||||
s.client.delete(
|
||||
collection_name=COLLECTION,
|
||||
points_selector=qm.FilterSelector(filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True))
|
||||
])),
|
||||
)
|
||||
|
||||
# Neue Punkte einspeisen
|
||||
created = 0
|
||||
for m in body.memories:
|
||||
content = (m.get("content") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
point = MemoryPoint(
|
||||
id="",
|
||||
type=m.get("type", "fact"),
|
||||
title=m.get("title", "(ohne Titel)"),
|
||||
content=content,
|
||||
pinned=True,
|
||||
category=m.get("category", ""),
|
||||
source=m.get("source", "bootstrap-import"),
|
||||
tags=list(m.get("tags", [])),
|
||||
)
|
||||
vec = e.embed(content)
|
||||
s.upsert(point, vec)
|
||||
created += 1
|
||||
|
||||
return {"created": created, "deleted_previous_pinned": True}
|
||||
|
||||
|
||||
# ─── Conversation-Loop ──────────────────────────────────────────────
|
||||
|
||||
class ChatIn(BaseModel):
|
||||
message: str
|
||||
source: str = "" # "app" / "diagnostic" / "stt" — optional
|
||||
|
||||
|
||||
class ChatOut(BaseModel):
|
||||
reply: str
|
||||
turns: int
|
||||
distilling: bool
|
||||
events: list = Field(default_factory=list)
|
||||
|
||||
|
||||
@app.post("/chat", response_model=ChatOut)
|
||||
def chat(body: ChatIn, background: BackgroundTasks):
|
||||
"""Hauptpfad. Antwort kommt synchron. Memory-Destillat laeuft
|
||||
im Hintergrund nachdem die Response rausging."""
|
||||
a = agent()
|
||||
try:
|
||||
reply = a.chat(body.message, source=body.source)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc))
|
||||
except RuntimeError as exc:
|
||||
logger.error("chat fehlgeschlagen: %s", exc)
|
||||
raise HTTPException(502, str(exc))
|
||||
|
||||
needs_distill = a.conversation.needs_distill()
|
||||
if needs_distill:
|
||||
background.add_task(a.distill_old_turns)
|
||||
return ChatOut(
|
||||
reply=reply,
|
||||
turns=len(a.conversation.turns),
|
||||
distilling=needs_distill,
|
||||
events=a.pop_events(),
|
||||
)
|
||||
|
||||
|
||||
@app.get("/conversation/stats")
|
||||
def conversation_stats():
|
||||
return conversation().stats()
|
||||
|
||||
|
||||
@app.post("/conversation/reset")
|
||||
def conversation_reset():
|
||||
"""Hardes Reset — der Rolling-Window-Verlauf wird komplett geleert.
|
||||
Destillierte facts bleiben in der DB."""
|
||||
conversation().reset()
|
||||
return {"ok": True, "turns": 0}
|
||||
|
||||
|
||||
@app.post("/conversation/distill")
|
||||
def conversation_distill_now():
|
||||
"""Manueller Trigger fuer Destillat — fuer Tests oder vor einem
|
||||
bewussten Reset."""
|
||||
return agent().distill_old_turns()
|
||||
|
||||
|
||||
# ─── Skills ─────────────────────────────────────────────────────────
|
||||
|
||||
class SkillCreate(BaseModel):
|
||||
name: str
|
||||
description: str
|
||||
execution: str # local-venv | local-bin | bash
|
||||
entry_code: str
|
||||
readme: str = ""
|
||||
args: list = Field(default_factory=list)
|
||||
requires: dict = Field(default_factory=dict)
|
||||
pip_packages: list = Field(default_factory=list)
|
||||
author: str = "stefan"
|
||||
|
||||
|
||||
class SkillRun(BaseModel):
|
||||
name: str
|
||||
args: dict = Field(default_factory=dict)
|
||||
timeout_sec: int = 300
|
||||
|
||||
|
||||
class SkillPatch(BaseModel):
|
||||
description: str | None = None
|
||||
active: bool | None = None
|
||||
args: list | None = None
|
||||
|
||||
|
||||
@app.get("/skills/list")
|
||||
def skills_list(active_only: bool = False):
|
||||
return {"skills": skills_mod.list_skills(active_only=active_only)}
|
||||
|
||||
|
||||
@app.get("/skills/{name}")
|
||||
def skills_get(name: str):
|
||||
m = skills_mod.read_manifest(name)
|
||||
if m is None:
|
||||
raise HTTPException(404, f"Skill '{name}' nicht gefunden")
|
||||
readme = skills_mod.read_readme(name)
|
||||
return {"manifest": m, "readme": readme}
|
||||
|
||||
|
||||
@app.post("/skills/create")
|
||||
def skills_create(body: SkillCreate):
|
||||
try:
|
||||
return skills_mod.create_skill(
|
||||
name=body.name,
|
||||
description=body.description,
|
||||
execution=body.execution,
|
||||
entry_code=body.entry_code,
|
||||
readme=body.readme,
|
||||
args=body.args,
|
||||
requires=body.requires,
|
||||
pip_packages=body.pip_packages,
|
||||
author=body.author,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc))
|
||||
|
||||
|
||||
@app.post("/skills/run")
|
||||
def skills_run(body: SkillRun):
|
||||
try:
|
||||
return skills_mod.run_skill(body.name, args=body.args, timeout_sec=body.timeout_sec)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc))
|
||||
|
||||
|
||||
@app.patch("/skills/{name}")
|
||||
def skills_patch(name: str, body: SkillPatch):
|
||||
patch = {k: v for k, v in body.model_dump().items() if v is not None}
|
||||
try:
|
||||
return skills_mod.update_skill(name, patch)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
|
||||
|
||||
@app.delete("/skills/{name}")
|
||||
def skills_delete(name: str):
|
||||
try:
|
||||
skills_mod.delete_skill(name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
return {"deleted": name}
|
||||
|
||||
|
||||
@app.get("/skills/{name}/logs")
|
||||
def skills_logs(name: str, limit: int = 50):
|
||||
return {"logs": skills_mod.list_logs(name, limit=limit)}
|
||||
|
||||
|
||||
@app.get("/skills/{name}/export")
|
||||
def skills_export(name: str):
|
||||
try:
|
||||
data = skills_mod.export_skill(name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
return Response(
|
||||
content=data,
|
||||
media_type="application/gzip",
|
||||
headers={"Content-Disposition": f'attachment; filename="skill-{name}.tar.gz"'},
|
||||
)
|
||||
|
||||
|
||||
@app.post("/skills/import")
|
||||
async def skills_import(request: Request, overwrite: bool = False):
|
||||
data = await request.body()
|
||||
if not data:
|
||||
raise HTTPException(400, "Leerer Body")
|
||||
try:
|
||||
manifest = skills_mod.import_skill(data, overwrite=overwrite)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc))
|
||||
return {"imported": manifest}
|
||||
@@ -0,0 +1,4 @@
|
||||
from .embedder import Embedder
|
||||
from .vector_store import VectorStore, MemoryPoint, MemoryType
|
||||
|
||||
__all__ = ["Embedder", "VectorStore", "MemoryPoint", "MemoryType"]
|
||||
@@ -0,0 +1,42 @@
|
||||
"""
|
||||
Lokaler Embedder fuer Memory-Texte.
|
||||
|
||||
Nutzt sentence-transformers (paraphrase-multilingual-MiniLM-L12-v2):
|
||||
- Deutsch + Englisch
|
||||
- 384-dimensionale Vektoren
|
||||
- Laeuft auf CPU, ~30ms pro kurzer Text
|
||||
- Modell wird beim ersten Aufruf in /data/_models gecached
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODEL_NAME = "paraphrase-multilingual-MiniLM-L12-v2"
|
||||
VECTOR_DIM = 384
|
||||
|
||||
|
||||
class Embedder:
|
||||
def __init__(self, model_name: str = MODEL_NAME):
|
||||
self.model_name = model_name
|
||||
self._model = None
|
||||
|
||||
def _load(self):
|
||||
if self._model is None:
|
||||
logger.info("Lade Embedding-Modell %s ...", self.model_name)
|
||||
from sentence_transformers import SentenceTransformer
|
||||
self._model = SentenceTransformer(self.model_name)
|
||||
logger.info("Embedding-Modell geladen.")
|
||||
|
||||
def embed(self, text: str) -> List[float]:
|
||||
self._load()
|
||||
vec = self._model.encode(text, convert_to_numpy=True, normalize_embeddings=True)
|
||||
return vec.tolist()
|
||||
|
||||
def embed_batch(self, texts: List[str]) -> List[List[float]]:
|
||||
self._load()
|
||||
vecs = self._model.encode(texts, convert_to_numpy=True, normalize_embeddings=True)
|
||||
return vecs.tolist()
|
||||
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
Vector-Store-Wrapper um Qdrant.
|
||||
|
||||
Eine Collection "aria_memory" haelt ALLE Memory-Punkte.
|
||||
Trennung nach Type/Pinned-Status via Payload-Filter.
|
||||
|
||||
Punkt-Schema (Payload):
|
||||
type — identity | rule | preference | tool | skill | fact | conversation | reminder
|
||||
category — frei, fuer UI-Gruppierung
|
||||
title — kurze Ueberschrift
|
||||
content — eigentlicher Text (wird embedded)
|
||||
pinned — bool, True = Hot Memory (immer in Prompt)
|
||||
source — import | conversation | manual
|
||||
tags — Liste von Strings
|
||||
created_at, updated_at — ISO-Strings
|
||||
conversation_id — optional, nur fuer type=conversation
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import List, Optional
|
||||
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models as qm
|
||||
|
||||
from .embedder import VECTOR_DIM
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
COLLECTION = "aria_memory"
|
||||
|
||||
|
||||
class MemoryType(str, Enum):
|
||||
IDENTITY = "identity"
|
||||
RULE = "rule"
|
||||
PREFERENCE = "preference"
|
||||
TOOL = "tool"
|
||||
SKILL = "skill"
|
||||
FACT = "fact"
|
||||
CONVERSATION = "conversation"
|
||||
REMINDER = "reminder"
|
||||
|
||||
|
||||
@dataclass
|
||||
class MemoryPoint:
|
||||
id: str
|
||||
type: str
|
||||
title: str
|
||||
content: str
|
||||
pinned: bool = False
|
||||
category: str = ""
|
||||
source: str = "manual"
|
||||
tags: List[str] = field(default_factory=list)
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
conversation_id: Optional[str] = None
|
||||
score: Optional[float] = None # nur bei Search gesetzt
|
||||
|
||||
def to_payload(self) -> dict:
|
||||
p = {
|
||||
"type": self.type,
|
||||
"title": self.title,
|
||||
"content": self.content,
|
||||
"pinned": self.pinned,
|
||||
"category": self.category,
|
||||
"source": self.source,
|
||||
"tags": self.tags,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
}
|
||||
if self.conversation_id:
|
||||
p["conversation_id"] = self.conversation_id
|
||||
return p
|
||||
|
||||
@classmethod
|
||||
def from_qdrant(cls, point) -> "MemoryPoint":
|
||||
payload = point.payload or {}
|
||||
return cls(
|
||||
id=str(point.id),
|
||||
type=payload.get("type", "fact"),
|
||||
title=payload.get("title", ""),
|
||||
content=payload.get("content", ""),
|
||||
pinned=payload.get("pinned", False),
|
||||
category=payload.get("category", ""),
|
||||
source=payload.get("source", "manual"),
|
||||
tags=payload.get("tags", []),
|
||||
created_at=payload.get("created_at", ""),
|
||||
updated_at=payload.get("updated_at", ""),
|
||||
conversation_id=payload.get("conversation_id"),
|
||||
score=getattr(point, "score", None),
|
||||
)
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
class VectorStore:
|
||||
def __init__(self, host: str, port: int = 6333):
|
||||
self.client = QdrantClient(host=host, port=port)
|
||||
self._ensure_collection()
|
||||
|
||||
def _ensure_collection(self):
|
||||
existing = [c.name for c in self.client.get_collections().collections]
|
||||
if COLLECTION not in existing:
|
||||
logger.info("Erstelle Collection %s ...", COLLECTION)
|
||||
self.client.create_collection(
|
||||
collection_name=COLLECTION,
|
||||
vectors_config=qm.VectorParams(size=VECTOR_DIM, distance=qm.Distance.COSINE),
|
||||
)
|
||||
# Indexe fuer typische Filter-Felder
|
||||
for field_name in ("type", "pinned", "category", "source", "migration_key"):
|
||||
self.client.create_payload_index(
|
||||
collection_name=COLLECTION,
|
||||
field_name=field_name,
|
||||
field_schema=qm.PayloadSchemaType.KEYWORD if field_name != "pinned"
|
||||
else qm.PayloadSchemaType.BOOL,
|
||||
)
|
||||
|
||||
# ─── Schreib-Operationen ─────────────────────────────────────────
|
||||
|
||||
def upsert(self, point: MemoryPoint, vector: List[float]) -> str:
|
||||
if not point.id:
|
||||
point.id = str(uuid.uuid4())
|
||||
if not point.created_at:
|
||||
point.created_at = _now()
|
||||
point.updated_at = _now()
|
||||
|
||||
self.client.upsert(
|
||||
collection_name=COLLECTION,
|
||||
points=[qm.PointStruct(id=point.id, vector=vector, payload=point.to_payload())],
|
||||
)
|
||||
return point.id
|
||||
|
||||
def delete(self, point_id: str):
|
||||
self.client.delete(
|
||||
collection_name=COLLECTION,
|
||||
points_selector=qm.PointIdsList(points=[point_id]),
|
||||
)
|
||||
|
||||
# ─── Lese-Operationen ────────────────────────────────────────────
|
||||
|
||||
def get(self, point_id: str) -> Optional[MemoryPoint]:
|
||||
result = self.client.retrieve(collection_name=COLLECTION, ids=[point_id], with_payload=True)
|
||||
if not result:
|
||||
return None
|
||||
return MemoryPoint.from_qdrant(result[0])
|
||||
|
||||
def list_pinned(self) -> List[MemoryPoint]:
|
||||
"""Alle pinned Punkte — Hot Memory."""
|
||||
return self._scroll(filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True))
|
||||
]))
|
||||
|
||||
def list_by_type(self, type_: str, limit: int = 100) -> List[MemoryPoint]:
|
||||
return self._scroll(
|
||||
filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="type", match=qm.MatchValue(value=type_))
|
||||
]),
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
def list_all(self, limit: int = 1000) -> List[MemoryPoint]:
|
||||
return self._scroll(filter=None, limit=limit)
|
||||
|
||||
def _scroll(self, filter, limit: int = 1000) -> List[MemoryPoint]:
|
||||
points, _ = self.client.scroll(
|
||||
collection_name=COLLECTION,
|
||||
scroll_filter=filter,
|
||||
limit=limit,
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
return [MemoryPoint.from_qdrant(p) for p in points]
|
||||
|
||||
def search(
|
||||
self,
|
||||
query_vector: List[float],
|
||||
k: int = 5,
|
||||
type_filter: Optional[str] = None,
|
||||
exclude_pinned: bool = True,
|
||||
) -> List[MemoryPoint]:
|
||||
"""Semantische Search. Standard: pinned-Punkte ausgeschlossen
|
||||
(die kommen separat via list_pinned in den Prompt)."""
|
||||
must = []
|
||||
must_not = []
|
||||
if type_filter:
|
||||
must.append(qm.FieldCondition(key="type", match=qm.MatchValue(value=type_filter)))
|
||||
if exclude_pinned:
|
||||
must_not.append(qm.FieldCondition(key="pinned", match=qm.MatchValue(value=True)))
|
||||
|
||||
flt = qm.Filter(must=must or None, must_not=must_not or None)
|
||||
|
||||
results = self.client.search(
|
||||
collection_name=COLLECTION,
|
||||
query_vector=query_vector,
|
||||
query_filter=flt if (must or must_not) else None,
|
||||
limit=k,
|
||||
with_payload=True,
|
||||
)
|
||||
return [MemoryPoint.from_qdrant(p) for p in results]
|
||||
|
||||
def count(self) -> int:
|
||||
return self.client.count(collection_name=COLLECTION, exact=True).count
|
||||
@@ -0,0 +1,399 @@
|
||||
"""
|
||||
Migration aus aria-data/brain-import/ → Vector-DB.
|
||||
|
||||
Parst die mitgelieferten Markdown-Dateien (AGENT.md, USER.md, TOOLING.md)
|
||||
und zerlegt sie in atomare Memory-Punkte. Jeder Punkt bekommt:
|
||||
|
||||
source = "import"
|
||||
migration_key = stabiler Identifier (z.B. "agent.md/rule-1") fuer Idempotenz
|
||||
pinned = True
|
||||
|
||||
Beim Re-Run werden vorhandene Punkte mit gleicher migration_key entfernt
|
||||
und neu geschrieben.
|
||||
|
||||
Mapping pro Datei:
|
||||
|
||||
AGENT.md
|
||||
"Identitaet" → 1 Punkt type=identity
|
||||
"Persoenlichkeit" (Intro) → 1 Punkt type=identity
|
||||
"Kern-Eigenschaften" (Liste) → 1 Punkt pro Bullet type=identity
|
||||
"Tool-Freigaben" → 1 Punkt type=tool
|
||||
"Sicherheitsregeln" (Liste) → 1 Punkt pro Bullet type=rule
|
||||
"Arbeitsprinzipien" (Liste) → 1 Punkt pro Bullet type=rule
|
||||
"Dateien an Stefan zurueckgeben"→ 1 Punkt type=skill
|
||||
"Stimme" → 1 Punkt type=tool
|
||||
|
||||
USER.md
|
||||
"Allgemein" (Liste) → 1 Punkt pro Bullet type=preference
|
||||
"Bestaetigung erforderlich" → 1 Punkt type=preference
|
||||
"Autonomes Arbeiten OK fuer" → 1 Punkt type=preference
|
||||
"Tools & Infrastruktur" → 1 Punkt type=preference
|
||||
|
||||
TOOLING.md
|
||||
gesamter Inhalt → 1 Punkt type=tool, title="Tooling-Stack"
|
||||
|
||||
BOOTSTRAP.md ist eine Variante von AGENT.md — wird (vorerst) ignoriert
|
||||
damit keine doppelten Punkte landen.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from memory import Embedder, VectorStore, MemoryPoint
|
||||
from memory.vector_store import COLLECTION
|
||||
from qdrant_client.http import models as qm
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Block:
|
||||
title: str
|
||||
content: str
|
||||
|
||||
|
||||
def _split_h2(md: str) -> List[_Block]:
|
||||
"""Zerlegt Markdown in H2-Bloecke. Inhalt vor dem ersten H2 wird verworfen."""
|
||||
blocks: List[_Block] = []
|
||||
current: Optional[_Block] = None
|
||||
for line in md.splitlines():
|
||||
m = re.match(r"^##\s+(.+?)\s*$", line)
|
||||
if m and not line.startswith("### "):
|
||||
if current:
|
||||
blocks.append(current)
|
||||
current = _Block(title=m.group(1).strip(), content="")
|
||||
continue
|
||||
if current is not None:
|
||||
current.content += line + "\n"
|
||||
if current:
|
||||
blocks.append(current)
|
||||
return blocks
|
||||
|
||||
|
||||
def _split_h3(content: str) -> List[_Block]:
|
||||
"""Zerlegt einen H2-Block in H3-Untersektionen + 'header'-Block davor."""
|
||||
blocks: List[_Block] = []
|
||||
header_lines: List[str] = []
|
||||
current: Optional[_Block] = None
|
||||
for line in content.splitlines():
|
||||
m = re.match(r"^###\s+(.+?)\s*$", line)
|
||||
if m:
|
||||
if current is None and header_lines:
|
||||
blocks.append(_Block(title="_intro", content="\n".join(header_lines).strip()))
|
||||
if current:
|
||||
blocks.append(current)
|
||||
current = _Block(title=m.group(1).strip(), content="")
|
||||
continue
|
||||
if current is None:
|
||||
header_lines.append(line)
|
||||
else:
|
||||
current.content += line + "\n"
|
||||
if current:
|
||||
blocks.append(current)
|
||||
elif header_lines:
|
||||
blocks.append(_Block(title="_intro", content="\n".join(header_lines).strip()))
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_bullets(content: str) -> List[tuple[str, str]]:
|
||||
"""Findet "- **Title** — Body" oder "N. **Title** — Body" Bullets.
|
||||
|
||||
Returns: Liste von (title, full_bullet_text).
|
||||
"""
|
||||
bullets: List[tuple[str, str]] = []
|
||||
current_lines: List[str] = []
|
||||
current_title: Optional[str] = None
|
||||
|
||||
def flush():
|
||||
if current_title and current_lines:
|
||||
bullets.append((current_title, "\n".join(current_lines).strip()))
|
||||
|
||||
for line in content.splitlines():
|
||||
m = re.match(r"^\s*(?:[-*]|\d+\.)\s+\*\*([^*]+?)\*\*\s*[—\-:]?\s*(.*)$", line)
|
||||
if m:
|
||||
flush()
|
||||
current_title = m.group(1).strip()
|
||||
current_lines = [line]
|
||||
continue
|
||||
# Folge-Zeilen mit Einrueckung gehoeren zum aktuellen Bullet
|
||||
if current_title and (line.startswith(" ") or line.startswith("\t") or not line.strip()):
|
||||
current_lines.append(line)
|
||||
continue
|
||||
if current_title and not re.match(r"^\s*(?:[-*]|\d+\.)\s+", line):
|
||||
current_lines.append(line)
|
||||
continue
|
||||
# Neuer Bullet ohne **Title** Format
|
||||
if re.match(r"^\s*(?:[-*]|\d+\.)\s+", line):
|
||||
flush()
|
||||
text = re.sub(r"^\s*(?:[-*]|\d+\.)\s+", "", line).strip()
|
||||
short_title = (text[:60] + "…") if len(text) > 60 else text
|
||||
bullets.append((short_title, line.strip()))
|
||||
current_title = None
|
||||
current_lines = []
|
||||
flush()
|
||||
return bullets
|
||||
|
||||
|
||||
# ─── Pro Datei eine Parser-Funktion ──────────────────────────────────
|
||||
|
||||
def _parse_agent_md(md: str, source_file: str) -> List[MemoryPoint]:
|
||||
points: List[MemoryPoint] = []
|
||||
h2_blocks = _split_h2(md)
|
||||
for h2 in h2_blocks:
|
||||
title = h2.title
|
||||
content = h2.content.strip()
|
||||
if not content:
|
||||
continue
|
||||
|
||||
if title.lower() == "identitaet" or title.lower() == "identität":
|
||||
points.append(_mk(
|
||||
type_="identity", title="ARIA — Identitaet",
|
||||
content=f"## {title}\n\n{content}",
|
||||
category="persoenlichkeit",
|
||||
migration_key=f"{source_file}/identity",
|
||||
))
|
||||
|
||||
elif title.lower() == "persoenlichkeit" or title.lower() == "persönlichkeit":
|
||||
# Intro-Absatz + Kern-Eigenschaften-Liste trennen
|
||||
sub = _split_h3(content)
|
||||
for s in sub:
|
||||
if s.title == "_intro" and s.content.strip():
|
||||
points.append(_mk(
|
||||
type_="identity", title="Persoenlichkeit — Grundsatz",
|
||||
content=s.content.strip(),
|
||||
category="persoenlichkeit",
|
||||
migration_key=f"{source_file}/personality-intro",
|
||||
))
|
||||
elif s.title.lower().startswith("kern"):
|
||||
for idx, (btitle, btext) in enumerate(_extract_bullets(s.content), 1):
|
||||
points.append(_mk(
|
||||
type_="identity", title=f"Eigenschaft: {btitle}",
|
||||
content=btext, category="persoenlichkeit",
|
||||
migration_key=f"{source_file}/personality-trait-{idx}",
|
||||
))
|
||||
|
||||
elif "sicherheitsregel" in title.lower():
|
||||
for idx, (btitle, btext) in enumerate(_extract_bullets(content), 1):
|
||||
points.append(_mk(
|
||||
type_="rule", title=f"Sicherheit: {btitle}",
|
||||
content=btext, category="sicherheit",
|
||||
migration_key=f"{source_file}/security-{idx}",
|
||||
))
|
||||
|
||||
elif "arbeitsprinzipien" in title.lower() or "arbeitsprinzip" in title.lower():
|
||||
for idx, (btitle, btext) in enumerate(_extract_bullets(content), 1):
|
||||
points.append(_mk(
|
||||
type_="rule", title=f"Prinzip: {btitle}",
|
||||
content=btext, category="arbeitsweise",
|
||||
migration_key=f"{source_file}/work-principle-{idx}",
|
||||
))
|
||||
|
||||
elif "tool-freigaben" in title.lower() or "tool freigaben" in title.lower():
|
||||
points.append(_mk(
|
||||
type_="tool", title="Tool-Freigaben — Vollzugriff",
|
||||
content=content, category="infrastruktur",
|
||||
migration_key=f"{source_file}/tool-access",
|
||||
))
|
||||
|
||||
elif "dateien an stefan" in title.lower() or "dateien zurueckgeben" in title.lower() or "dateien zur" in title.lower():
|
||||
points.append(_mk(
|
||||
type_="skill", title="Dateien an User zurueckgeben",
|
||||
content=content, category="ausgabe",
|
||||
migration_key=f"{source_file}/file-return-skill",
|
||||
))
|
||||
|
||||
elif title.lower() == "stimme":
|
||||
points.append(_mk(
|
||||
type_="tool", title="Stimme (F5-TTS)",
|
||||
content=content, category="infrastruktur",
|
||||
migration_key=f"{source_file}/voice",
|
||||
))
|
||||
|
||||
# Permanente Freigaben (in BOOTSTRAP) — als rule
|
||||
elif "freigaben" in title.lower():
|
||||
points.append(_mk(
|
||||
type_="rule", title=title,
|
||||
content=content, category="freigaben",
|
||||
migration_key=f"{source_file}/permissions",
|
||||
))
|
||||
|
||||
else:
|
||||
# Unbekannter Block: als generischer fact ablegen, NICHT pinned
|
||||
logger.info("Unbekannter H2-Block '%s' in %s — als fact (unpinned)", title, source_file)
|
||||
points.append(_mk(
|
||||
type_="fact", title=f"{source_file}: {title}",
|
||||
content=content, pinned=False,
|
||||
migration_key=f"{source_file}/section-{title.lower().replace(' ', '-')}",
|
||||
))
|
||||
return points
|
||||
|
||||
|
||||
def _parse_user_md(md: str, source_file: str) -> List[MemoryPoint]:
|
||||
points: List[MemoryPoint] = []
|
||||
for h2 in _split_h2(md):
|
||||
title = h2.title
|
||||
content = h2.content.strip()
|
||||
if not content:
|
||||
continue
|
||||
# Template-Platzhalter herausfiltern: Beispiel-Zeilen mit <Tag>
|
||||
if "<Beispiel-Tool>" in content or "<Username>" in title:
|
||||
continue
|
||||
if title.lower() == "allgemein":
|
||||
for idx, (btitle, btext) in enumerate(_extract_bullets(content), 1):
|
||||
# Template-Platzhalter ueberspringen
|
||||
if "<z.B." in btext or "<XYZ>" in btext:
|
||||
continue
|
||||
points.append(_mk(
|
||||
type_="preference", title=f"User: {btitle}",
|
||||
content=btext, category="allgemein",
|
||||
migration_key=f"{source_file}/general-{idx}",
|
||||
))
|
||||
else:
|
||||
cat_key = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") or "allgemein"
|
||||
points.append(_mk(
|
||||
type_="preference", title=title,
|
||||
content=content, category=cat_key,
|
||||
migration_key=f"{source_file}/{cat_key}",
|
||||
))
|
||||
return points
|
||||
|
||||
|
||||
def _parse_tooling_md(md: str, source_file: str) -> List[MemoryPoint]:
|
||||
md = md.strip()
|
||||
if not md:
|
||||
return []
|
||||
return [_mk(
|
||||
type_="tool", title="Tooling-Stack (VM)",
|
||||
content=md, category="infrastruktur",
|
||||
migration_key=f"{source_file}/tooling-full",
|
||||
)]
|
||||
|
||||
|
||||
# ─── Helper ─────────────────────────────────────────────────────────
|
||||
|
||||
def _mk(
|
||||
type_: str,
|
||||
title: str,
|
||||
content: str,
|
||||
migration_key: str,
|
||||
pinned: bool = True,
|
||||
category: str = "",
|
||||
) -> MemoryPoint:
|
||||
p = MemoryPoint(
|
||||
id="",
|
||||
type=type_,
|
||||
title=title,
|
||||
content=content.strip(),
|
||||
pinned=pinned,
|
||||
category=category,
|
||||
source="import",
|
||||
tags=[],
|
||||
)
|
||||
# migration_key wird ueber Payload-Index angesprochen — in to_payload manuell anhaengen
|
||||
setattr(p, "_migration_key", migration_key)
|
||||
return p
|
||||
|
||||
|
||||
# ─── Eintrittspunkt ─────────────────────────────────────────────────
|
||||
|
||||
def run_migration(
|
||||
import_dir: Path,
|
||||
store: VectorStore,
|
||||
embedder: Embedder,
|
||||
) -> dict:
|
||||
"""Liest alle .md-Dateien aus import_dir, parst sie, schreibt in DB.
|
||||
|
||||
Idempotent: vorhandene Punkte mit gleicher migration_key werden geloescht
|
||||
und neu geschrieben.
|
||||
|
||||
Returns: {"created": int, "updated": int, "skipped": int, "files": [...]}
|
||||
"""
|
||||
if not import_dir.exists():
|
||||
return {"created": 0, "updated": 0, "skipped": 0, "files": [], "error": f"{import_dir} nicht gefunden"}
|
||||
|
||||
parsers = {
|
||||
"AGENT.md": _parse_agent_md,
|
||||
"BOOTSTRAP.md": _parse_agent_md, # gleicher Parser, ggf. ueberlappende Eintraege
|
||||
"USER.md": _parse_user_md,
|
||||
"USER.md.example": _parse_user_md,
|
||||
"TOOLING.md": _parse_tooling_md,
|
||||
"TOOLING.md.example": _parse_tooling_md,
|
||||
}
|
||||
|
||||
# USER.md hat Vorrang vor USER.md.example
|
||||
file_priority = ["AGENT.md", "BOOTSTRAP.md", "USER.md", "USER.md.example",
|
||||
"TOOLING.md", "TOOLING.md.example"]
|
||||
seen_kinds: set[str] = set() # "USER" / "TOOLING" — nur einmal
|
||||
|
||||
points: List[MemoryPoint] = []
|
||||
processed_files: List[str] = []
|
||||
|
||||
for fname in file_priority:
|
||||
fp = import_dir / fname
|
||||
if not fp.exists():
|
||||
continue
|
||||
kind = fname.split(".")[0] # "AGENT", "BOOTSTRAP", "USER", "TOOLING"
|
||||
# USER.md.example nur wenn USER.md fehlt
|
||||
if kind in ("USER", "TOOLING") and kind in seen_kinds:
|
||||
continue
|
||||
seen_kinds.add(kind)
|
||||
parser = parsers.get(fname)
|
||||
if not parser:
|
||||
continue
|
||||
try:
|
||||
md = fp.read_text(encoding="utf-8")
|
||||
file_points = parser(md, fname)
|
||||
points.extend(file_points)
|
||||
processed_files.append(f"{fname} ({len(file_points)})")
|
||||
logger.info("Migration: %s → %d Punkte", fname, len(file_points))
|
||||
except Exception as exc:
|
||||
logger.exception("Migration: %s fehlgeschlagen", fname)
|
||||
processed_files.append(f"{fname} (FEHLER: {exc})")
|
||||
|
||||
if not points:
|
||||
return {"created": 0, "updated": 0, "skipped": 0, "files": processed_files}
|
||||
|
||||
# Erst alte Migration-Punkte mit gleicher migration_key loeschen
|
||||
migration_keys = [getattr(p, "_migration_key", None) for p in points]
|
||||
migration_keys = [k for k in migration_keys if k]
|
||||
if migration_keys:
|
||||
store.client.delete(
|
||||
collection_name=COLLECTION,
|
||||
points_selector=qm.FilterSelector(filter=qm.Filter(must=[
|
||||
qm.FieldCondition(key="migration_key", match=qm.MatchAny(any=migration_keys))
|
||||
])),
|
||||
)
|
||||
logger.info("Migration: %d alte Punkte mit gleicher migration_key entfernt", len(migration_keys))
|
||||
|
||||
# Embed in Batches
|
||||
texts = [p.content for p in points]
|
||||
vectors = embedder.embed_batch(texts)
|
||||
|
||||
created = 0
|
||||
for p, vec in zip(points, vectors):
|
||||
payload = p.to_payload()
|
||||
mkey = getattr(p, "_migration_key", None)
|
||||
if mkey:
|
||||
payload["migration_key"] = mkey
|
||||
from datetime import datetime, timezone
|
||||
import uuid as _uuid
|
||||
pid = str(_uuid.uuid4())
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
payload["created_at"] = now
|
||||
payload["updated_at"] = now
|
||||
store.client.upsert(
|
||||
collection_name=COLLECTION,
|
||||
points=[qm.PointStruct(id=pid, vector=vec, payload=payload)],
|
||||
)
|
||||
created += 1
|
||||
|
||||
return {
|
||||
"created": created,
|
||||
"files": processed_files,
|
||||
"import_dir": str(import_dir),
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
"""
|
||||
System-Prompt-Bau aus Memory-Punkten.
|
||||
|
||||
Strategie:
|
||||
1. Alle pinned Punkte (Hot Memory) — gruppiert nach Type — in den
|
||||
System-Prompt schreiben. IMMER drin.
|
||||
2. Top-K semantisch aehnliche Punkte (Cold Memory) zur aktuellen
|
||||
User-Nachricht — als "Moeglicherweise relevant" eingehaengt.
|
||||
3. Aktive Skills als kompakte Liste (nur Name + Description) — damit
|
||||
ARIA weiss was sie hat.
|
||||
|
||||
Phase B Punkt 1: nur Hot-Memory-Bau, Skills + Cold-Search kommen
|
||||
mit dem Conversation-Loop in spaeteren Phasen.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import List
|
||||
|
||||
from memory import MemoryPoint
|
||||
|
||||
TYPE_HEADINGS = {
|
||||
"identity": "## Wer du bist",
|
||||
"rule": "## Sicherheitsregeln & Prinzipien",
|
||||
"preference": "## Benutzer-Praeferenzen",
|
||||
"tool": "## Tool-Freigaben",
|
||||
"skill": "## Deine Skills",
|
||||
}
|
||||
|
||||
|
||||
def build_hot_memory_section(pinned: List[MemoryPoint]) -> str:
|
||||
"""Baue den 'IMMER-im-Prompt'-Block aus pinned Punkten."""
|
||||
grouped: dict[str, List[MemoryPoint]] = {}
|
||||
for p in pinned:
|
||||
grouped.setdefault(p.type, []).append(p)
|
||||
|
||||
parts: List[str] = []
|
||||
# Sortier-Reihenfolge: identity → rule → preference → tool → skill → Rest
|
||||
order = ["identity", "rule", "preference", "tool", "skill"]
|
||||
for t in order:
|
||||
items = grouped.pop(t, [])
|
||||
if not items:
|
||||
continue
|
||||
parts.append(TYPE_HEADINGS.get(t, f"## {t}"))
|
||||
for p in items:
|
||||
parts.append(f"### {p.title}")
|
||||
parts.append(p.content.strip())
|
||||
parts.append("")
|
||||
|
||||
# uebrige Types (falls jemand was anderes als pinned markiert)
|
||||
for t, items in grouped.items():
|
||||
parts.append(f"## {t}")
|
||||
for p in items:
|
||||
parts.append(f"### {p.title}")
|
||||
parts.append(p.content.strip())
|
||||
parts.append("")
|
||||
|
||||
return "\n".join(parts).strip()
|
||||
|
||||
|
||||
def build_cold_memory_section(matches: List[MemoryPoint]) -> str:
|
||||
"""Baue 'Moeglicherweise relevant'-Block aus Search-Treffern."""
|
||||
if not matches:
|
||||
return ""
|
||||
lines = ["## Moeglicherweise relevant (aus Gedaechtnis)"]
|
||||
for p in matches:
|
||||
score = f" [score={p.score:.2f}]" if p.score is not None else ""
|
||||
lines.append(f"- **{p.title}**{score}")
|
||||
lines.append(f" {p.content.strip()}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_skills_section(skills: List[dict]) -> str:
|
||||
"""Listet alle Skills (aktiv + deaktiviert) damit ARIA weiss was es gibt
|
||||
und keine doppelt baut. Plus klare Schwelle wann ein Skill sich lohnt."""
|
||||
lines = ["## Deine Skills"]
|
||||
if skills:
|
||||
for s in skills:
|
||||
active = s.get("active", True)
|
||||
marker = "" if active else " [DEAKTIVIERT — kann nicht aufgerufen werden]"
|
||||
lines.append(f"- **{s.get('name', '?')}**{marker} — {s.get('description', '(ohne Beschreibung)')}")
|
||||
lines.append("")
|
||||
lines.append("Wenn ein vorhandener Skill zur Aufgabe passt: nutze ihn via Tool-Call.")
|
||||
else:
|
||||
lines.append("(noch keine Skills vorhanden)")
|
||||
|
||||
lines.append("")
|
||||
lines.append("### Wann lohnt sich ein neuer Skill?")
|
||||
lines.append("")
|
||||
lines.append("**Skills sind IMMER Python** — eigene venv pro Skill mit den noetigen "
|
||||
"pip-Paketen. Kein apt im Skill, kein systemweiter Install. Python deckt "
|
||||
"in der Regel alles ab (yt-dlp, requests, pypdf, pillow, openpyxl, "
|
||||
"static-ffmpeg, beautifulsoup4, …). Falls etwas WIRKLICH nur via apt geht: "
|
||||
"Stefan fragen ob es ins Brain-Dockerfile soll.")
|
||||
lines.append("")
|
||||
lines.append("**Harte Regel — IMMER Skill anlegen wenn:** die Loesung erfordert eine "
|
||||
"pip-Library. Begruendung: Brain-Container hat keinen persistenten State "
|
||||
"ausser /data/skills/. Ohne Skill wuerde der Install bei jedem "
|
||||
"Container-Restart wiederholt.")
|
||||
lines.append("")
|
||||
lines.append("**Sonst — Skill nur wenn alle vier zutreffen:**")
|
||||
lines.append("")
|
||||
lines.append("1. **Wiederkehrend** — die Aufgabe wird realistisch nochmal gestellt. "
|
||||
"Einmal-Faelle (\"wie spaet ist es jetzt\") kein Skill.")
|
||||
lines.append("2. **Nicht-trivial** — mehrere Schritte. Ein einzelner Shell-Befehl "
|
||||
"(`date`, `hostname`, `ls`) ist KEIN Skill — das macht Bash direkt.")
|
||||
lines.append("3. **Parametrisierbar** — der Skill nimmt Eingaben (URL, Datei, Suchbegriff) "
|
||||
"und gibt ein nuetzliches Ergebnis zurueck.")
|
||||
lines.append("4. **Wiederverwendbar als ganzes** — Stefan wuerde es zukuenftig per Name "
|
||||
"ansprechen (\"mach mir den YouTube zu MP3\") statt jedes Mal zu erklaeren.")
|
||||
lines.append("")
|
||||
lines.append("Wenn nichts installiert werden muss UND nicht alle vier zutreffen: einfach "
|
||||
"die Aufgabe loesen ohne Skill anzulegen. Stefan kann jederzeit sagen "
|
||||
"'bau daraus einen Skill'.")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_system_prompt(
|
||||
pinned: List[MemoryPoint],
|
||||
cold: List[MemoryPoint] | None = None,
|
||||
skills: List[dict] | None = None,
|
||||
) -> str:
|
||||
"""Kompletter System-Prompt: Hot + Cold + Skills."""
|
||||
parts = [build_hot_memory_section(pinned)]
|
||||
if skills:
|
||||
parts.append("")
|
||||
parts.append(build_skills_section(skills))
|
||||
if cold:
|
||||
parts.append("")
|
||||
parts.append(build_cold_memory_section(cold))
|
||||
return "\n".join(parts).strip()
|
||||
@@ -0,0 +1,125 @@
|
||||
"""
|
||||
Claude-Aufruf ueber den lokalen Proxy.
|
||||
|
||||
Der Proxy (claude-max-api-proxy) bietet eine OpenAI-kompatible API
|
||||
unter http://proxy:3456/v1/chat/completions. Wir nutzen non-streaming
|
||||
mit einem laengeren Timeout — Claude Code spawnt pro Anfrage einen
|
||||
neuen CLI-Prozess (Cold-Start), das dauert.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
import httpx
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_MODEL = os.environ.get("BRAIN_MODEL", "claude-sonnet-4")
|
||||
PROXY_URL = os.environ.get("PROXY_URL", "http://proxy:3456")
|
||||
PROXY_TIMEOUT_SEC = float(os.environ.get("PROXY_TIMEOUT_SEC", "300"))
|
||||
|
||||
|
||||
class Message(BaseModel):
|
||||
role: str # "system" | "user" | "assistant" | "tool"
|
||||
content: Optional[str] = None
|
||||
tool_calls: Optional[list] = None
|
||||
tool_call_id: Optional[str] = None
|
||||
name: Optional[str] = None # nur fuer role=tool
|
||||
|
||||
|
||||
class ProxyResult(BaseModel):
|
||||
content: str = ""
|
||||
tool_calls: list = [] # je: {"id", "name", "arguments" (dict)}
|
||||
finish_reason: str = ""
|
||||
|
||||
|
||||
class ProxyClient:
|
||||
def __init__(self, base_url: str = PROXY_URL, model: str = DEFAULT_MODEL):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.model = model
|
||||
# Persistente Client-Connection — vermeidet TCP-Handshake bei jedem Call
|
||||
self._client = httpx.Client(timeout=PROXY_TIMEOUT_SEC)
|
||||
|
||||
def chat(self, messages: List[Message], model: Optional[str] = None) -> str:
|
||||
"""Convenience: einfacher Chat ohne Tools. Gibt nur den Reply-String zurueck."""
|
||||
result = self.chat_full(messages, tools=None, model=model)
|
||||
if not result.content:
|
||||
raise RuntimeError("Proxy lieferte leeren content")
|
||||
return result.content
|
||||
|
||||
def chat_full(
|
||||
self,
|
||||
messages: List[Message],
|
||||
tools: Optional[list] = None,
|
||||
model: Optional[str] = None,
|
||||
) -> ProxyResult:
|
||||
"""Full chat — kann Tool-Calls liefern (wenn tools mitgegeben).
|
||||
|
||||
tools-Format ist OpenAI-Style:
|
||||
[{"type":"function","function":{"name":..,"description":..,"parameters":{...}}}, ...]
|
||||
"""
|
||||
url = f"{self.base_url}/v1/chat/completions"
|
||||
# Pydantic-Dumps mit exclude_none damit role=tool ohne tool_calls geht
|
||||
payload = {
|
||||
"model": model or self.model,
|
||||
"messages": [m.model_dump(exclude_none=True) for m in messages],
|
||||
}
|
||||
if tools:
|
||||
payload["tools"] = tools
|
||||
logger.info("Proxy → %s (%d Messages, %d tools, model=%s)",
|
||||
url, len(messages), len(tools or []), payload["model"])
|
||||
try:
|
||||
r = self._client.post(url, json=payload)
|
||||
except httpx.RequestError as exc:
|
||||
raise RuntimeError(f"Proxy unreachable: {exc}") from exc
|
||||
if r.status_code != 200:
|
||||
raise RuntimeError(f"Proxy HTTP {r.status_code}: {r.text[:300]}")
|
||||
try:
|
||||
data = r.json()
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"Proxy invalid JSON: {exc}") from exc
|
||||
|
||||
choices = data.get("choices") or []
|
||||
if not choices:
|
||||
raise RuntimeError(f"Proxy ohne choices: {str(data)[:300]}")
|
||||
|
||||
msg = choices[0].get("message") or {}
|
||||
finish_reason = choices[0].get("finish_reason", "")
|
||||
|
||||
content = msg.get("content") or ""
|
||||
if isinstance(content, list):
|
||||
content = "".join(
|
||||
part.get("text", "") for part in content if isinstance(part, dict) and part.get("type") == "text"
|
||||
)
|
||||
|
||||
tool_calls_raw = msg.get("tool_calls") or []
|
||||
tool_calls = []
|
||||
import json as _json
|
||||
for tc in tool_calls_raw:
|
||||
fn = tc.get("function") or {}
|
||||
args_raw = fn.get("arguments", "{}")
|
||||
args: dict
|
||||
if isinstance(args_raw, dict):
|
||||
args = args_raw
|
||||
else:
|
||||
try:
|
||||
args = _json.loads(args_raw)
|
||||
except Exception:
|
||||
args = {"_raw": args_raw}
|
||||
tool_calls.append({
|
||||
"id": tc.get("id", ""),
|
||||
"name": fn.get("name", ""),
|
||||
"arguments": args,
|
||||
})
|
||||
|
||||
return ProxyResult(content=content or "", tool_calls=tool_calls, finish_reason=finish_reason)
|
||||
|
||||
def close(self):
|
||||
try:
|
||||
self._client.close()
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,14 @@
|
||||
fastapi==0.115.0
|
||||
uvicorn[standard]==0.32.0
|
||||
pydantic==2.9.2
|
||||
httpx==0.27.2
|
||||
websockets==13.1
|
||||
|
||||
# Vector-DB
|
||||
qdrant-client==1.12.1
|
||||
|
||||
# Embeddings (laeuft auf CPU, ~120MB Modell)
|
||||
sentence-transformers==3.2.1
|
||||
|
||||
# Utility
|
||||
python-multipart==0.0.12
|
||||
@@ -0,0 +1,373 @@
|
||||
"""
|
||||
Skill-Manager — Filesystem-Layer fuer ARIAs Faehigkeiten.
|
||||
|
||||
Layout:
|
||||
/data/skills/<name>/
|
||||
skill.json - Manifest
|
||||
README.md - Beschreibung (vom Stil her: was, wann, wie aufrufen)
|
||||
run.sh - Entry-Point (sh, python -m, was auch immer)
|
||||
requirements.txt - optional, fuer local-venv
|
||||
venv/ - automatisch erzeugt bei local-venv
|
||||
bin/ - statische Binaries (fuer local-bin)
|
||||
logs/ - <ts>.json Run-Logs (append-only pro Run)
|
||||
|
||||
Manifest (skill.json):
|
||||
{
|
||||
"name": "youtube2mp3",
|
||||
"description": "Konvertiert YouTube-Video-URL zu MP3",
|
||||
"execution": "local-venv" | "local-bin" | "bash",
|
||||
"entry": "run.sh",
|
||||
"args": [{"name": "url", "required": true}, ...],
|
||||
"requires": {"pip": [...], "binaries": [...]},
|
||||
"active": true,
|
||||
"created_at": "ISO",
|
||||
"updated_at": "ISO",
|
||||
"last_used": null | "ISO",
|
||||
"use_count": 0,
|
||||
"version": "1.0",
|
||||
"author": "aria" | "stefan"
|
||||
}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SKILLS_DIR = Path(os.environ.get("SKILLS_DIR", "/data/skills"))
|
||||
SHARED_UPLOADS = Path("/shared/uploads")
|
||||
|
||||
VALID_EXECUTIONS = {"local-venv", "local-bin", "bash"}
|
||||
NAME_RE = re.compile(r"^[a-zA-Z0-9_-]{2,60}$")
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _safe_name(name: str) -> str:
|
||||
if not isinstance(name, str) or not NAME_RE.match(name):
|
||||
raise ValueError(f"Ungültiger Skill-Name: {name!r}")
|
||||
return name
|
||||
|
||||
|
||||
def _skill_dir(name: str) -> Path:
|
||||
return SKILLS_DIR / _safe_name(name)
|
||||
|
||||
|
||||
# ─── Listing ────────────────────────────────────────────────────────
|
||||
|
||||
def list_skills(active_only: bool = False) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
if not SKILLS_DIR.exists():
|
||||
return out
|
||||
for entry in sorted(SKILLS_DIR.iterdir()):
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
manifest = read_manifest(entry.name)
|
||||
if manifest is None:
|
||||
continue
|
||||
if active_only and not manifest.get("active", True):
|
||||
continue
|
||||
out.append(manifest)
|
||||
return out
|
||||
|
||||
|
||||
def read_manifest(name: str) -> Optional[dict]:
|
||||
try:
|
||||
path = _skill_dir(name) / "skill.json"
|
||||
if not path.exists():
|
||||
return None
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
logger.warning("Manifest lesen %s: %s", name, exc)
|
||||
return None
|
||||
|
||||
|
||||
def write_manifest(name: str, manifest: dict) -> None:
|
||||
d = _skill_dir(name)
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
manifest["updated_at"] = _now()
|
||||
(d / "skill.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def read_readme(name: str) -> str:
|
||||
path = _skill_dir(name) / "README.md"
|
||||
return path.read_text(encoding="utf-8") if path.exists() else ""
|
||||
|
||||
|
||||
# ─── Create / Update / Delete ────────────────────────────────────────
|
||||
|
||||
def create_skill(
|
||||
name: str,
|
||||
description: str,
|
||||
execution: str,
|
||||
entry_code: str,
|
||||
readme: str = "",
|
||||
args: Optional[list] = None,
|
||||
requires: Optional[dict] = None,
|
||||
pip_packages: Optional[list[str]] = None,
|
||||
author: str = "aria",
|
||||
) -> dict:
|
||||
"""Legt einen neuen Skill an. Wirft ValueError bei ungueltigen Inputs.
|
||||
|
||||
entry_code wird je nach execution in run.sh oder run.py geschrieben.
|
||||
Bei local-venv wird sofort eine venv erzeugt + pip_packages installiert.
|
||||
"""
|
||||
name = _safe_name(name)
|
||||
if execution not in VALID_EXECUTIONS:
|
||||
raise ValueError(f"execution muss eines von {VALID_EXECUTIONS} sein")
|
||||
d = _skill_dir(name)
|
||||
if d.exists():
|
||||
raise ValueError(f"Skill '{name}' existiert bereits — erst loeschen oder updaten")
|
||||
|
||||
d.mkdir(parents=True)
|
||||
(d / "logs").mkdir()
|
||||
|
||||
# Entry-File: run.sh oder run.py
|
||||
if execution == "local-venv":
|
||||
entry_path = d / "run.py"
|
||||
entry_path.write_text(entry_code, encoding="utf-8")
|
||||
entry_name = "run.py"
|
||||
(d / "requirements.txt").write_text("\n".join(pip_packages or []) + "\n", encoding="utf-8")
|
||||
else:
|
||||
entry_path = d / "run.sh"
|
||||
# Shebang ergaenzen wenn nicht da
|
||||
content = entry_code if entry_code.startswith("#!") else "#!/usr/bin/env bash\nset -euo pipefail\n" + entry_code
|
||||
entry_path.write_text(content, encoding="utf-8")
|
||||
entry_path.chmod(0o755)
|
||||
entry_name = "run.sh"
|
||||
|
||||
# README
|
||||
(d / "README.md").write_text(readme or f"# {name}\n\n{description}\n", encoding="utf-8")
|
||||
|
||||
manifest = {
|
||||
"name": name,
|
||||
"description": description,
|
||||
"execution": execution,
|
||||
"entry": entry_name,
|
||||
"args": args or [],
|
||||
"requires": requires or {},
|
||||
"active": True,
|
||||
"created_at": _now(),
|
||||
"updated_at": _now(),
|
||||
"last_used": None,
|
||||
"use_count": 0,
|
||||
"version": "1.0",
|
||||
"author": author,
|
||||
}
|
||||
write_manifest(name, manifest)
|
||||
|
||||
# venv aufbauen bei local-venv
|
||||
if execution == "local-venv":
|
||||
try:
|
||||
_setup_venv(d, pip_packages or [])
|
||||
except Exception as exc:
|
||||
# venv-Aufbau fehlgeschlagen → Skill steht trotzdem im Repo, aber inaktiv
|
||||
manifest["active"] = False
|
||||
manifest["setup_error"] = str(exc)[:500]
|
||||
write_manifest(name, manifest)
|
||||
logger.warning("Skill %s: venv-Setup fehlgeschlagen → deaktiviert: %s", name, exc)
|
||||
|
||||
logger.info("Skill erstellt: %s (%s)", name, execution)
|
||||
return manifest
|
||||
|
||||
|
||||
def _setup_venv(skill_dir: Path, pip_packages: list[str]) -> None:
|
||||
venv = skill_dir / "venv"
|
||||
logger.info("venv erstellen: %s", venv)
|
||||
subprocess.run(["python", "-m", "venv", str(venv)], check=True, timeout=120)
|
||||
pip = venv / "bin" / "pip"
|
||||
if pip_packages:
|
||||
subprocess.run([str(pip), "install", "--no-cache-dir", *pip_packages], check=True, timeout=600)
|
||||
|
||||
|
||||
def update_skill(name: str, patch: dict) -> dict:
|
||||
manifest = read_manifest(name)
|
||||
if manifest is None:
|
||||
raise ValueError(f"Skill '{name}' nicht gefunden")
|
||||
allowed = {"description", "args", "requires", "active", "version", "entry"}
|
||||
for k, v in patch.items():
|
||||
if k in allowed:
|
||||
manifest[k] = v
|
||||
write_manifest(name, manifest)
|
||||
return manifest
|
||||
|
||||
|
||||
def delete_skill(name: str) -> None:
|
||||
d = _skill_dir(name)
|
||||
if not d.exists():
|
||||
raise ValueError(f"Skill '{name}' nicht gefunden")
|
||||
shutil.rmtree(d)
|
||||
logger.info("Skill geloescht: %s", name)
|
||||
|
||||
|
||||
# ─── Run ────────────────────────────────────────────────────────────
|
||||
|
||||
def run_skill(name: str, args: Optional[dict] = None, timeout_sec: int = 300) -> dict:
|
||||
"""Fuehrt einen Skill aus. Args werden als ENV-Vars uebergeben
|
||||
(Praefix ARG_, z.B. ARG_URL fuer args["url"]).
|
||||
|
||||
Returns: {ok, exit_code, stdout, stderr, duration_sec, log_path}
|
||||
"""
|
||||
manifest = read_manifest(name)
|
||||
if manifest is None:
|
||||
raise ValueError(f"Skill '{name}' nicht gefunden")
|
||||
if not manifest.get("active", True):
|
||||
raise ValueError(f"Skill '{name}' ist deaktiviert")
|
||||
|
||||
d = _skill_dir(name)
|
||||
entry = manifest.get("entry", "run.sh")
|
||||
exec_mode = manifest.get("execution", "bash")
|
||||
|
||||
env = os.environ.copy()
|
||||
# Skill-Args als ENV-Vars
|
||||
for k, v in (args or {}).items():
|
||||
if not re.match(r"^[a-zA-Z][a-zA-Z0-9_]*$", k):
|
||||
continue
|
||||
env[f"ARG_{k.upper()}"] = str(v)
|
||||
env["SKILL_DIR"] = str(d)
|
||||
env["SHARED_UPLOADS"] = str(SHARED_UPLOADS)
|
||||
|
||||
# Command bauen
|
||||
if exec_mode == "local-venv":
|
||||
python = d / "venv" / "bin" / "python"
|
||||
cmd = [str(python), str(d / entry)]
|
||||
elif exec_mode == "local-bin":
|
||||
# Skill bringt seine bin/ mit — wir prepended sie an den PATH
|
||||
env["PATH"] = f"{d / 'bin'}:{env.get('PATH', '')}"
|
||||
cmd = [str(d / entry)]
|
||||
else: # bash
|
||||
cmd = [str(d / entry)]
|
||||
|
||||
log_id = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
|
||||
log_path = d / "logs" / f"{log_id}.json"
|
||||
|
||||
t0 = time.time()
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd, env=env, cwd=str(d),
|
||||
capture_output=True, text=True, timeout=timeout_sec,
|
||||
)
|
||||
out_text = proc.stdout
|
||||
err_text = proc.stderr
|
||||
exit_code = proc.returncode
|
||||
timed_out = False
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
out_text = exc.stdout or ""
|
||||
err_text = (exc.stderr or "") + f"\n[TIMEOUT {timeout_sec}s]"
|
||||
exit_code = -1
|
||||
timed_out = True
|
||||
duration = time.time() - t0
|
||||
|
||||
# Log schreiben (gekuerzt damit es nicht explodiert)
|
||||
record = {
|
||||
"ts": _now(),
|
||||
"args": args or {},
|
||||
"exit_code": exit_code,
|
||||
"duration_sec": round(duration, 2),
|
||||
"stdout": (out_text or "")[:8000],
|
||||
"stderr": (err_text or "")[:8000],
|
||||
"timed_out": timed_out,
|
||||
}
|
||||
try:
|
||||
log_path.write_text(json.dumps(record, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Stats updaten
|
||||
manifest["last_used"] = _now()
|
||||
manifest["use_count"] = int(manifest.get("use_count", 0)) + 1
|
||||
write_manifest(name, manifest)
|
||||
|
||||
record["ok"] = exit_code == 0
|
||||
record["log_path"] = str(log_path)
|
||||
return record
|
||||
|
||||
|
||||
def list_logs(name: str, limit: int = 50) -> list[dict]:
|
||||
d = _skill_dir(name) / "logs"
|
||||
if not d.exists():
|
||||
return []
|
||||
files = sorted(d.glob("*.json"), reverse=True)[:limit]
|
||||
out: list[dict] = []
|
||||
for f in files:
|
||||
try:
|
||||
data = json.loads(f.read_text(encoding="utf-8"))
|
||||
data["log_id"] = f.stem
|
||||
out.append(data)
|
||||
except Exception:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
# ─── Export / Import ────────────────────────────────────────────────
|
||||
|
||||
def export_skill(name: str) -> bytes:
|
||||
"""Packt einen Skill als tar.gz und gibt die Bytes zurueck.
|
||||
venv und logs werden ausgeschlossen (werden beim Import neu gebaut)."""
|
||||
import io
|
||||
import tarfile
|
||||
d = _skill_dir(name)
|
||||
if not d.exists():
|
||||
raise ValueError(f"Skill '{name}' nicht gefunden")
|
||||
buf = io.BytesIO()
|
||||
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||
for path in d.iterdir():
|
||||
if path.name in ("venv", "logs", "__pycache__"):
|
||||
continue
|
||||
tar.add(path, arcname=f"{name}/{path.name}")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def import_skill(tar_bytes: bytes, overwrite: bool = False) -> dict:
|
||||
"""Importiert einen Skill aus tar.gz. Liefert das Manifest zurueck."""
|
||||
import io
|
||||
import tarfile
|
||||
SKILLS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(fileobj=io.BytesIO(tar_bytes), mode="r:gz") as tar:
|
||||
# Erst Root-Name finden (= Skill-Name)
|
||||
members = tar.getmembers()
|
||||
if not members:
|
||||
raise ValueError("Leeres Archiv")
|
||||
root = members[0].name.split("/", 1)[0]
|
||||
name = _safe_name(root)
|
||||
d = _skill_dir(name)
|
||||
if d.exists():
|
||||
if not overwrite:
|
||||
raise ValueError(f"Skill '{name}' existiert bereits — overwrite=true setzen")
|
||||
shutil.rmtree(d)
|
||||
# Extrahieren — Path-Traversal verhindern
|
||||
for m in members:
|
||||
target = SKILLS_DIR / m.name
|
||||
if not str(target.resolve()).startswith(str(SKILLS_DIR.resolve())):
|
||||
raise ValueError(f"Unsicherer Pfad im Archiv: {m.name}")
|
||||
tar.extractall(SKILLS_DIR)
|
||||
# logs-Verzeichnis anlegen falls fehlte
|
||||
(d / "logs").mkdir(exist_ok=True)
|
||||
# venv neu bauen falls local-venv
|
||||
manifest = read_manifest(name) or {}
|
||||
if manifest.get("execution") == "local-venv":
|
||||
req_file = d / "requirements.txt"
|
||||
pip_packages: list[str] = []
|
||||
if req_file.exists():
|
||||
pip_packages = [l.strip() for l in req_file.read_text().splitlines() if l.strip() and not l.startswith("#")]
|
||||
try:
|
||||
_setup_venv(d, pip_packages)
|
||||
except Exception as exc:
|
||||
logger.warning("Skill-Import %s: venv-Setup fehlgeschlagen: %s", name, exc)
|
||||
manifest["active"] = False
|
||||
manifest["setup_error"] = str(exc)[:500]
|
||||
write_manifest(name, manifest)
|
||||
return manifest
|
||||
Reference in New Issue
Block a user