docs: CLAUDE.md aktualisiert (neue Serveradresse, fehlende Dateien, Multi-Pass, vollstaendige Architektur)
Dieser Commit ist enthalten in:
Claude Dev
278
CLAUDE.md
278
CLAUDE.md
@@ -1,190 +1,195 @@
|
||||
# AegisSight-Monitor
|
||||
|
||||
> OSINT-Monitoringsystem mit KI-gestützter Nachrichtenanalyse
|
||||
> OSINT-Lagemonitoring mit KI-gestützter Nachrichtenanalyse
|
||||
|
||||
## Übersicht
|
||||
|
||||
```yaml
|
||||
projekt: AegisSight-Monitor
|
||||
url: https://osint.intelsight.de
|
||||
beschreibung: "OSINT-basiertes Lagemonitoring mit Claude-KI-Agenten"
|
||||
server: alt (91.99.192.14, User: claude-dev)
|
||||
url: https://monitor.aegis-sight.de
|
||||
server: ssh monitor (178.104.43.177, User: claude-dev)
|
||||
pfad: /home/claude-dev/AegisSight-Monitor
|
||||
datenbank: /mnt/gitea/osint-data/osint.db (geteilt mit AegisSight-Monitor-Verwaltung)
|
||||
quellcode: /home/claude-dev/AegisSight-Monitor/src/
|
||||
datenbank: /mnt/gitea/osint-data/osint.db (SQLite WAL, geteilt mit Verwaltungsportal + Globe)
|
||||
gitea: https://gitea-undso.aegis-sight.de/AegisSight/AegisSight-Monitor
|
||||
git_push_regel: "Jede Aenderung MUSS sofort committed und nach Gitea gepusht werden."
|
||||
service: osint-monitor.service (systemd, Port 8891, Nginx Reverse Proxy)
|
||||
venv: /home/claude-dev/.venvs/osint/
|
||||
status: aktiv
|
||||
venv: /home/claude-dev/.venvs/osint/ (Python 3.12)
|
||||
```
|
||||
|
||||
## Technologie-Stack
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
framework: FastAPI (Python 3.12)
|
||||
datenbank: SQLite (WAL-Modus, aiosqlite)
|
||||
auth: Magic-Link-Login per E-Mail (JWT HS256, 24h Ablauf)
|
||||
framework: FastAPI + Uvicorn
|
||||
datenbank: SQLite WAL (aiosqlite, async)
|
||||
auth: Magic-Link-Login per E-Mail (JWT HS256, 24h)
|
||||
scheduler: APScheduler (Auto-Refresh 1min, Cleanup 1h, Health-Check taeglich 04:00)
|
||||
websocket: FastAPI native (Echtzeit-Updates)
|
||||
ki_agenten: Claude CLI (WebSearch + WebFetch Tools)
|
||||
email: aiosmtplib (Magic Links, Benachrichtigungen)
|
||||
port: 8891 (localhost, Nginx Reverse Proxy)
|
||||
websocket: FastAPI native (Echtzeit-Updates an Clients)
|
||||
ki: Claude CLI als Subprocess (WebSearch + WebFetch Tools)
|
||||
ki_modelle:
|
||||
schnell: CLAUDE_MODEL_FAST (Haiku) — Feed-Selektion, Geoparsing, Chat, QC
|
||||
mittel: CLAUDE_MODEL_MEDIUM (Sonnet) — Entity-Extraktion, Netzwerkanalyse
|
||||
standard: CLI-Default (Opus) — Recherche, Analyse, Faktencheck
|
||||
email: aiosmtplib (smtp.ionos.de:587 TLS)
|
||||
|
||||
frontend:
|
||||
typ: Vanilla JS (kein Framework)
|
||||
typ: Vanilla JS (kein Framework, kein Build-Step)
|
||||
design: AegisSight Dark/Light Theme (Navy/Gold)
|
||||
fonts: Poppins (Titel), Inter (Body)
|
||||
layout: gridstack.js (Drag-and-Drop Dashboard-Kacheln)
|
||||
karte: Leaflet + MarkerCluster
|
||||
echtzeit: WebSocket mit Auto-Reconnect und Ping/Pong
|
||||
```
|
||||
|
||||
## Projektstruktur
|
||||
|
||||
```yaml
|
||||
AegisSight-Monitor/:
|
||||
CLAUDE.md: "Diese Datei"
|
||||
requirements.txt: "Python-Abhaengigkeiten"
|
||||
data/: "Symlink -> /mnt/gitea/osint-data/ (SQLite DB)"
|
||||
logs/: "Anwendungs-Logs (osint-monitor.log)"
|
||||
src/:
|
||||
main.py: "FastAPI App, WebSocketManager, Scheduler, Lifespan, statische Routen"
|
||||
config.py: "Konfiguration (JWT, Claude-Modelle, SMTP, RSS-Feeds, Zeitzone)"
|
||||
auth.py: "JWT erstellen/verifizieren, Magic-Link/Code, get_current_user Dependency"
|
||||
database.py: "SQLite Schema (25+ Tabellen), Migrationen, init_db(), get_db()"
|
||||
models.py: "Pydantic Request/Response-Schemas"
|
||||
source_rules.py: "Domain-Kategorisierung, RSS-Feed-Discovery, Claude-Feed-Bewertung"
|
||||
report_generator.py: "PDF (WeasyPrint) + DOCX (python-docx) Export"
|
||||
|
||||
src/:
|
||||
main.py: "FastAPI App, WebSocketManager, Scheduler (lifespan), statische Routen"
|
||||
config.py: "Konfiguration (JWT, Claude CLI Pfad/Timeout, SMTP, RSS-Default-Feeds, Excluded Sources, Zeitzone)"
|
||||
auth.py: "JWT-Token erstellen/verifizieren, Magic-Link/Code generieren, get_current_user Dependency"
|
||||
database.py: "SQLite Schema (13 Tabellen), Migrationen, init_db()"
|
||||
models.py: "Pydantic Request/Response-Schemas"
|
||||
source_rules.py: "Dynamische Quellen-Regeln aus DB, Domain-Kategorisierung, Feed-Discovery"
|
||||
routers/:
|
||||
auth.py: "Magic-Link-Login, Token-Verify, /api/auth/me"
|
||||
incidents.py: "CRUD Lagen, Refresh, Artikel, Snapshots, Faktenchecks, Export, E-Mail-Abos, Refresh-Log"
|
||||
sources.py: "CRUD Quellen, Discovery (Single/Multi), Domain sperren, Telegram-Validierung"
|
||||
chat.py: "KI-Assistent (Haiku), Injection-Schutz, Tech-Leak-Filter"
|
||||
public_api.py: "API-Key Auth, Globe-Feed (GeoJSON), Globe-Ingest, Snapshot-Abruf"
|
||||
notifications.py: "CRUD Benachrichtigungen, Unread-Count, Mark-Read"
|
||||
feedback.py: "E-Mail-Feedback mit Bild-Anhaengen"
|
||||
tutorial.py: "Tutorial-Fortschritt pro User"
|
||||
|
||||
routers/:
|
||||
auth.py: "Magic-Link-Login: POST /api/auth/magic-link, /verify, /verify-code, GET /api/auth/me"
|
||||
incidents.py: "CRUD Lagen, Artikel, Snapshots, Faktenchecks, Refresh, Export, E-Mail-Subscriptions"
|
||||
sources.py: "CRUD Quellen, Discovery (Single/Multi), Domain sperren/entsperren, Stats"
|
||||
notifications.py: "GET/PUT Benachrichtigungen (Liste, ungelesen, als gelesen markieren)"
|
||||
feedback.py: "POST /api/feedback (Rate-Limited, E-Mail an feedback@aegis-sight.de)"
|
||||
agents/:
|
||||
orchestrator.py: "Queue-basierte Refresh-Steuerung, Research Multi-Pass (3 Durchlaeufe), Retry, Cancel, Credits-Tracking"
|
||||
researcher.py: "WebSearch-Recherche (Standard + 4-Phasen-Tiefenrecherche), Feed-Selektion, Keyword-Extraktion"
|
||||
analyzer.py: "Analyse-Agent (Lagebild/Briefing, Erst- + inkrementell, Inline-Zitate)"
|
||||
factchecker.py: "Faktencheck (Erst/Inkrementell/Zwei-Phasen mit Triage), Claim-Matching, Dedup"
|
||||
geoparsing.py: "Haiku-basierte Ortsextraktion, Geocoding via geonamescache"
|
||||
entity_extractor.py: "Netzwerkanalyse: Entity-Extraktion (Sonnet), Beziehungsanalyse, Dedup"
|
||||
claude_client.py: "Shared Claude CLI Client, Usage-Tracking (Token, Kosten), Rate-Limit-Erkennung"
|
||||
|
||||
agents/:
|
||||
claude_client.py: "Shared Claude CLI Client (JSON-Output, Usage-Tracking: Token, Kosten)"
|
||||
orchestrator.py: "AsyncQueue, Agenten-Pipeline, Cancel, Snapshots, E-Mail-Benachrichtigungen, Quellen-Discovery"
|
||||
researcher.py: "Claude WebSearch Agent (Ad-hoc + Deep Research Modus)"
|
||||
analyzer.py: "Analyse-Agent (Zusammenfassung/Briefing mit Inline-Zitaten)"
|
||||
factchecker.py: "Faktencheck-Agent (Claims gegen unabhaengige Quellen pruefen)"
|
||||
feeds/:
|
||||
rss_parser.py: "RSS-Feed-Parsing (feedparser + httpx), Keyword-Matching, Domain-Cap"
|
||||
telegram_parser.py: "Telethon-basierter Telegram-Parser, Kanal-Validierung"
|
||||
|
||||
feeds/:
|
||||
rss_parser.py: "RSS-Feed Aggregation (dynamisch aus DB, Keyword-Matching)"
|
||||
services/:
|
||||
post_refresh_qc.py: "Post-Refresh Quality Check: Faktencheck-Duplikate, Location-Korrektur"
|
||||
fact_consolidation.py: "Periodisches Haiku-Clustering, Auto-Resolve veralteter Fakten"
|
||||
source_health.py: "Quellen-Health-Checks (Erreichbarkeit, Feed-Validitaet, Stale)"
|
||||
source_suggester.py: "KI-Quellen-Vorschlaege via Haiku"
|
||||
license_service.py: "Lizenz-Pruefung (Org, Ablauf, Nutzer-Limit)"
|
||||
|
||||
services/:
|
||||
license_service.py: "Lizenzpruefung (check_license), Nutzer-Limit, Ablauf-Check"
|
||||
source_health.py: "Quellen-Health-Check Engine (Erreichbarkeit, Feed-Validitaet, Aktualitaet, Duplikate)"
|
||||
source_suggester.py: "KI-gestuetzte Quellen-Vorschlaege via Claude Haiku"
|
||||
middleware/:
|
||||
license_check.py: "Dependencies: require_active_license, require_writable_license"
|
||||
|
||||
middleware/:
|
||||
license_check.py: "FastAPI Dependencies: require_active_license, require_writable_license"
|
||||
email_utils/:
|
||||
sender.py: "Async SMTP Versand"
|
||||
templates.py: "HTML-Templates (Magic-Link, Benachrichtigungen)"
|
||||
rate_limiter.py: "Rate-Limiting Magic-Links"
|
||||
|
||||
migration/:
|
||||
migrate_to_multitenancy.py: "Einmal-Migration: Single-Tenant zu Multi-Tenant"
|
||||
migration/:
|
||||
migrate_to_multitenancy.py: "Einmal-Migration Single->Multi-Tenant"
|
||||
|
||||
email_utils/:
|
||||
sender.py: "Async SMTP E-Mail-Versand (aiosmtplib, TLS)"
|
||||
templates.py: "HTML-E-Mail-Templates (Magic-Link-Login, Incident-Benachrichtigungen)"
|
||||
rate_limiter.py: "Rate-Limiting fuer Magic-Links und Code-Verifizierung"
|
||||
report_templates/:
|
||||
report.html: "HTML-Template fuer PDF/DOCX-Export"
|
||||
|
||||
static/:
|
||||
index.html: "Login-Seite (Magic-Link: E-Mail eingeben, Code eingeben)"
|
||||
dashboard.html: "Hauptdashboard (Sidebar + Grid + Modals)"
|
||||
css/:
|
||||
style.css: "AegisSight Design System (Dark/Light Theme, alle Komponenten)"
|
||||
js/:
|
||||
api.js: "REST-API-Client (fetch-basiert, 30s Timeout, Auto-Redirect bei 401)"
|
||||
app.js: "Hauptlogik: ThemeManager, A11yManager, NotificationCenter, App-Objekt"
|
||||
components.js: "UI-Rendering: Sidebar-Items, Faktenchecks, Evidence-Chips, Toasts, Fortschritt, Quellen"
|
||||
layout.js: "gridstack.js Wrapper (Drag und Resize, localStorage-Persistenz)"
|
||||
ws.js: "WebSocket-Client (Reconnect mit exponential Backoff, Ping/Pong)"
|
||||
static/:
|
||||
index.html: "Login-Seite (Magic-Link)"
|
||||
dashboard.html: "Hauptdashboard (Sidebar + GridStack + Modals)"
|
||||
css/:
|
||||
style.css: "AegisSight Design System (Dark/Light Theme, alle Komponenten)"
|
||||
js/:
|
||||
api.js: "REST-API-Client (fetch, Auth-Header, 30s Timeout)"
|
||||
app.js: "Hauptlogik: ThemeManager, NotificationCenter, App-Objekt"
|
||||
components.js: "UI-Rendering: Sidebar, Faktenchecks, Toasts, Progress-Bar, Karte"
|
||||
chat.js: "Chat-Assistent Widget"
|
||||
layout.js: "gridstack.js Wrapper (Drag/Resize, localStorage)"
|
||||
tutorial.js: "Interaktiver 32-Schritte Rundgang mit Animationen"
|
||||
ws.js: "WebSocket-Client (Reconnect, Ping/Pong)"
|
||||
vendor/:
|
||||
leaflet.js: "Karten-Bibliothek"
|
||||
leaflet.markercluster.js: "Marker-Clustering"
|
||||
```
|
||||
|
||||
## Architektur
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
methode: "Magic-Link per E-Mail (kein Passwort-Login)"
|
||||
flow: "E-Mail eingeben, Code per E-Mail, Code eingeben oder Link klicken, JWT"
|
||||
rate_limiting: "3 Magic-Links pro E-Mail/15min, 5 Fehlversuche Code/E-Mail"
|
||||
multi_tenancy: "JWT enthaelt tenant_id, org_slug, role"
|
||||
|
||||
agenten_pipeline:
|
||||
1_rss: "RSS-Feeds durchsuchen (nur Ad-hoc-Lagen)"
|
||||
2_claude_recherche: "Claude CLI WebSearch (Ad-hoc oder Deep Research)"
|
||||
3_deduplizierung: "URL-Normalisierung + Headline-Aehnlichkeit"
|
||||
4_analyse: "Zusammenfassung/Briefing mit Inline-Zitaten [1][2]"
|
||||
5_faktencheck: "Claims gegen unabhaengige Quellen pruefen"
|
||||
orchestrierung: "Sequentielle AsyncQueue (1 Auftrag gleichzeitig, 3 Retries)"
|
||||
|
||||
incident_typen:
|
||||
adhoc: "Breaking News: RSS + WebSearch, Fliesstext-Summary"
|
||||
research: "Hintergrundrecherche: Deep Research, Markdown-Briefing"
|
||||
adhoc:
|
||||
label: "Live-Monitoring"
|
||||
quellen: "RSS + WebSearch + optional Telegram"
|
||||
analyse: "Fliesstext-Lagebild"
|
||||
faktencheck_status: "confirmed/unconfirmed/contradicted/developing"
|
||||
refresh: "Manuell oder automatisch (Intervall konfigurierbar)"
|
||||
research:
|
||||
label: "Recherche"
|
||||
quellen: "Nur WebSearch 4-Phasen-Tiefenrecherche (kein RSS)"
|
||||
analyse: "Strukturiertes Briefing (Ueberblick, Hintergrund, Akteure, Lage, Einschaetzung, Quellenqualitaet)"
|
||||
faktencheck_status: "established/unverified/disputed/developing"
|
||||
refresh: "Immer manuell, erster Refresh automatisch 3 Durchlaeufe (Multi-Pass)"
|
||||
multi_pass:
|
||||
durchlaeufe: 3
|
||||
labels: ["Breite Erfassung", "Vertiefung", "Konsolidierung"]
|
||||
bedingung: "Nur beim ersten Refresh (kein Summary vorhanden)"
|
||||
cancel: "Zwischen und innerhalb der Durchlaeufe moeglich"
|
||||
|
||||
sidebar:
|
||||
aktive_lagen: "Lagen mit type=adhoc und status=active"
|
||||
aktive_recherchen: "Lagen mit type=research und status=active"
|
||||
archiv: "Alle Lagen mit status=archived (standardmaessig zugeklappt)"
|
||||
zaehler: "Anzahl pro Sektion in Klammern"
|
||||
filter: "Alle / Eigene"
|
||||
refresh_pipeline:
|
||||
1: "Feed-Selektion (Haiku) + dynamische Keywords"
|
||||
2: "Parallel: RSS + WebSearch + optional Telegram"
|
||||
3: "URL-Verifizierung (HEAD-Requests)"
|
||||
4: "Duplikaterkennung (URL + Headline)"
|
||||
5: "Relevanz-Scoring + DB-Dedup"
|
||||
6: "Geoparsing (Haiku + geonamescache)"
|
||||
7: "Parallel: Analyse + Faktencheck"
|
||||
8: "Post-Refresh QC"
|
||||
9: "Notifications (DB + E-Mail + WebSocket)"
|
||||
10: "Credits-Tracking (Token auf Lizenz buchen)"
|
||||
11: "Background: Source-Discovery"
|
||||
|
||||
benachrichtigungen:
|
||||
in_app: "NotificationCenter (Glocke + Badge, DB-persistent, 7 Tage)"
|
||||
email:
|
||||
einstellung: "Pro Lage konfigurierbar (3 Toggles im Lage-Modal)"
|
||||
optionen: "Neues Lagebild, Neue Artikel, Statusaenderung Faktencheck"
|
||||
tabelle: "incident_subscriptions (pro User pro Lage)"
|
||||
versand: "Nach jedem Refresh (ab dem 2.) basierend auf Subscriptions"
|
||||
|
||||
quellenverwaltung:
|
||||
features: "Anlegen, Bearbeiten, Loeschen, Discovery (Multi-Feed), Domain sperren"
|
||||
source_types: "rss_feed, web_source, excluded"
|
||||
|
||||
lizenz_anzeige:
|
||||
header: "Org-Name + Lizenz-Badge (Trial/Annual/Permanent/Abgelaufen)"
|
||||
read_only: "Warnung wenn Lizenz abgelaufen"
|
||||
multi_tenancy: "Volle Mandantentrennung (tenant_id auf allen Tabellen)"
|
||||
|
||||
dashboard_kacheln:
|
||||
lagebild: "Markdown-Zusammenfassung mit klickbaren Zitaten"
|
||||
faktencheck: "Status-Icons, Evidence-Chips, Filter"
|
||||
quellenübersicht: "Aggregiert nach Quellen mit Sprach-Statistik"
|
||||
timeline: "Interaktive Zeitleiste mit Bucketing, Filtern, Suche"
|
||||
|
||||
datenbank_tabellen:
|
||||
organizations: "Multi-Tenancy Organisationen"
|
||||
licenses: "Lizenzen pro Organisation (trial/annual/permanent)"
|
||||
users: "Nutzer (E-Mail, Org, Rolle)"
|
||||
magic_links: "Login-Tokens (10 Min. gueltig)"
|
||||
portal_admins: "Admin-Zugaenge (genutzt von AegisSight-Monitor-Verwaltung)"
|
||||
incidents: "Lagen/Recherchen"
|
||||
articles: "Gesammelte Artikel (original + deutsche Uebersetzung)"
|
||||
fact_checks: "Faktenchecks (claim, status, evidence)"
|
||||
refresh_log: "Refresh-Protokoll (Token-Statistiken, Kosten)"
|
||||
incident_snapshots: "Archivierte Lageberichte"
|
||||
sources: "Quellen-Verwaltung (RSS-Feeds, Web-Quellen, Ausgeschlossene)"
|
||||
source_health_checks: "Health-Check-Ergebnisse (Erreichbarkeit, Feed-Validitaet)"
|
||||
source_suggestions: "KI-Vorschlaege (neue Quellen, Deaktivierung, URL-Fix)"
|
||||
user_excluded_domains: "Per-User ausgeschlossene Domains"
|
||||
notifications: "Persistente In-App-Benachrichtigungen"
|
||||
incident_subscriptions: "E-Mail-Abo-Einstellungen pro User/Lage"
|
||||
|
||||
deployment:
|
||||
service: "systemd osint-monitor.service"
|
||||
restart: "sudo systemctl restart osint-monitor"
|
||||
logs: "tail -f ~/AegisSight-Monitor/logs/osint-monitor.log"
|
||||
status: "systemctl status osint-monitor"
|
||||
- "Lagebild (Markdown + Inline-Zitate)"
|
||||
- "Faktencheck (Status-Icons, Evidence, Filter)"
|
||||
- "Quellenübersicht (nach Domain gruppiert)"
|
||||
- "Timeline (horizontale Achse, Bucketing, Filter)"
|
||||
- "Karte (Leaflet, Kategorie-Marker, Legende)"
|
||||
```
|
||||
|
||||
## Verwandte Projekte
|
||||
## Datenbank (25+ Tabellen)
|
||||
|
||||
```yaml
|
||||
kern: "organizations, licenses, users, magic_links, portal_admins"
|
||||
lagen: "incidents, articles, incident_snapshots, fact_checks, refresh_log"
|
||||
quellen: "sources, source_health_checks, source_suggestions, user_excluded_domains"
|
||||
geo: "article_locations"
|
||||
netzwerk: "network_analyses, network_analysis_incidents, network_entities, network_entity_mentions, network_relations, network_generation_log"
|
||||
system: "notifications, incident_subscriptions, feedback, token_usage_monthly"
|
||||
```
|
||||
|
||||
## Verwandte Projekte (gleicher Server)
|
||||
|
||||
```yaml
|
||||
verwaltungsportal:
|
||||
pfad: /home/claude-dev/AegisSight-Monitor-Verwaltung
|
||||
beschreibung: "Admin-Portal fuer Organisationen, Lizenzen, Nutzer"
|
||||
geteilte_db: /mnt/gitea/osint-data/osint.db
|
||||
url: https://monitor-verwaltung.aegis-sight.de
|
||||
service: verwaltungsportal.service (Port 8892)
|
||||
geteilte_db: ja
|
||||
|
||||
globe:
|
||||
pfad: /home/claude-dev/AegisSight-Globe
|
||||
url: https://globe.aegis-sight.de
|
||||
service: globe.service (Port 8890)
|
||||
geteilte_db: ja
|
||||
|
||||
netzwerkanalyse:
|
||||
pfad: /home/claude-dev/AegisSight-Netzwerkanalyse
|
||||
url: https://netzwerkanalyse.aegis-sight.de
|
||||
service: netzwerkanalyse.service (Port 8893)
|
||||
```
|
||||
|
||||
## Regeln
|
||||
@@ -192,8 +197,9 @@ verwaltungsportal:
|
||||
```yaml
|
||||
regeln:
|
||||
- "Jede Aenderung MUSS sofort committed und nach Gitea gepusht werden"
|
||||
- "Echte Umlaute in UI-Texten verwenden, Umschreibungen in YAML/Code-Kommentaren OK"
|
||||
- "Echte Umlaute in UI-Texten (ue, ae, oe, ss), keine Umschreibungen"
|
||||
- "Keine Passwoerter oder Secrets in den Code committen"
|
||||
- "Service nach Backend-Aenderungen neustarten: sudo systemctl restart osint-monitor"
|
||||
- "Frontend-Aenderungen brauchen keinen Neustart (statische Dateien)"
|
||||
- "Service nach Backend-Aenderungen: sudo systemctl restart osint-monitor"
|
||||
- "Frontend-Aenderungen (HTML/JS/CSS) brauchen keinen Neustart"
|
||||
- "Backup-Dateien (.bak) nicht committen, vor Push loeschen"
|
||||
```
|
||||
|
||||
In neuem Issue referenzieren
Einen Benutzer sperren