Hetzner-Backup/OPERATIONS_GUIDE.md

# V2-Docker Operations Guide

## Deployment

### Prerequisites
- Docker and Docker Compose
- PostgreSQL 13+
- Python 3.8+
- Minimum 4GB RAM
- 20GB disk space

### Initial Setup

```bash
# Clone repository
git clone <repository-url>
cd v2-Docker

# Environment Variables sind bereits in docker-compose.yaml definiert
# Für Produktion: Erstelle .env Datei mit sensiblen Daten

# Start services
docker-compose up -d

# Datenbank wird automatisch initialisiert via init.sql
# Keine manuellen Migrationen erforderlich
```

### Standard-Zugangsdaten

#### Admin Panel
- URL: https://admin-panel-undso.z5m7q9dk3ah2v1plx6ju.com
- User 1: `rac00n` / `1248163264`
- User 2: `w@rh@mm3r` / `Warhammer123!`

#### License Server API
- URL: https://api-software-undso.z5m7q9dk3ah2v1plx6ju.com
- API Key: Muss in Requests mitgesendet werden

### Service Configuration

#### License Server
```yaml
license-server:
  image: v2_lizenzserver:latest
  environment:
    - DATABASE_URL=postgresql://adminuser:supergeheimespasswort@db:5432/meinedatenbank
    - JWT_SECRET=your-secret-jwt-key-here-minimum-32-chars
    - API_KEY=your-api-key-here
    - REDIS_HOST=redis-cache
    - RABBITMQ_HOST=rabbitmq
  # Kein externer Port - nur über Nginx erreichbar
  expose:
    - "8443"
  networks:
    - internal_net
```

#### Admin Panel  
```yaml
admin-panel:
  image: v2_adminpanel:latest
  environment:
    - DATABASE_URL=postgresql://adminuser:supergeheimespasswort@db:5432/meinedatenbank
    - SECRET_KEY=supersecretkey
    - JWT_SECRET=your-secret-jwt-key-here-minimum-32-chars
    - LIZENZSERVER_BASE_URL=http://license-server:8443
    - REDIS_HOST=redis-cache
  # Kein externer Port - nur über Nginx erreichbar
  expose:
    - "5000"
  networks:
    - internal_net
```

#### Nginx Reverse Proxy
```yaml
nginx-proxy:
  image: v2_nginx:latest
  ports:
    - "80:80"    # HTTP (wird auf HTTPS umgeleitet)
    - "443:443"  # HTTPS
  networks:
    - internal_net
  # Routet Requests an interne Services:
  # admin-panel-undso.z5m7q9dk3ah2v1plx6ju.com → admin-panel:5000
  # api-software-undso.z5m7q9dk3ah2v1plx6ju.com → license-server:8443
```

## Monitoring

### Prometheus Configuration

#### Scrape Configs
```yaml
scrape_configs:
  - job_name: 'license-server'
    static_configs:
      - targets: ['license-server:8443']
    metrics_path: /metrics
  
  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']
      
  - job_name: 'redis'
    static_configs:
      - targets: ['redis-exporter:9121']
      
  - job_name: 'nginx'
    static_configs:
      - targets: ['nginx-exporter:9113']
      
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
      
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8081']
```

#### Alert Rules
- License server down: `up{job="license-server"} == 0`
- High error rate: `rate(http_requests_total{status=~"5.."}[5m]) > 0.05`
- Database connections: `pg_stat_database_numbackends > 100`

### Grafana Dashboards

1. **System Overview Dashboard**
   - CPU and memory usage
   - Network traffic
   - Disk usage
   - Container status

2. **License Server Dashboard**
   - Active licenses
   - Heartbeat frequency
   - API response times
   - Error rates

3. **Database Performance Dashboard**
   - Query performance
   - Connection pool status
   - Table sizes
   - Slow queries

### Accessing Monitoring

- Prometheus: http://localhost:9090
- Grafana: http://localhost:3001
  - Default Login: admin/admin
  - Vorkonfigurierte Dashboards:
    - System Overview
    - License Server Metrics  
    - Database Performance
- Alertmanager: http://localhost:9093

### Monitoring Stack Services
- PostgreSQL Exporter: Sammelt DB-Metriken
- Redis Exporter: Sammelt Cache-Metriken
- Node Exporter: System-Level Metriken
- Nginx Exporter: Webserver-Metriken
- cAdvisor: Container-Metriken (Port 8081)

## Maintenance

### Database Maintenance

#### Partition Management
```sql
-- Check existing partitions
SELECT tablename FROM pg_tables 
WHERE tablename LIKE 'license_heartbeats_%' 
ORDER BY tablename;

-- Create future partitions manually
CALL create_monthly_partitions('license_heartbeats', 3);

-- Drop old partitions
DROP TABLE IF EXISTS license_heartbeats_2024_01;
```

#### Backup Procedures

##### Manuelles Backup
```bash
# Full database backup
docker exec db pg_dump -U adminuser meinedatenbank > backup_$(date +%Y%m%d).sql

# Backup specific tables
docker exec db pg_dump -U adminuser -t licenses -t license_activations meinedatenbank > licenses_backup.sql

# Komprimiertes Backup
docker exec db pg_dump -U adminuser meinedatenbank | gzip > backup_$(date +%Y%m%d).sql.gz

# Restore from backup
docker exec -i db psql -U adminuser meinedatenbank < backup_20250619.sql
```

##### Integriertes Backup-System
Das Admin Panel bietet ein eingebautes Backup-System:
1. Login ins Admin Panel
2. Navigiere zu "Backups"
3. Klicke "Create Backup"
4. Backups werden verschlüsselt im Verzeichnis `/backups` gespeichert
5. Download oder Restore direkt über die UI

### Log Management

#### Log Locations

##### Container Logs
```bash
# License Server Logs
docker logs license-server

# Admin Panel Logs  
docker logs admin-panel

# Nginx Logs
docker logs nginx-proxy

# Database Logs
docker logs db
```

##### Persistent Log Volumes
- Nginx Access/Error Logs: Mapped to local `./v2_nginx/logs/`
- Application Logs: Verfügbar über Docker logging driver
- Audit Logs: In der Datenbank (Tabelle `audit_log`)

#### Log Rotation
```bash
# Configure logrotate
/var/log/license-server/*.log {
    daily
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data www-data
}
```

### Performance Optimization

#### Database Tuning
```sql
-- Update statistics
ANALYZE;

-- Reindex tables
REINDEX TABLE licenses;
REINDEX TABLE license_activations;

-- Vacuum tables
VACUUM ANALYZE licenses;
```

#### Resource Limits

Alle Services haben konfigurierte Resource Limits:

```yaml
# License Server
license-server:
  deploy:
    resources:
      limits:
        cpus: '1.0'
        memory: 1G
      reservations:
        cpus: '0.5'
        memory: 512M

# Admin Panel
admin-panel:
  deploy:
    resources:
      limits:
        cpus: '1.0'
        memory: 1G
      reservations:
        cpus: '0.5'
        memory: 512M

# PostgreSQL
db:
  deploy:
    resources:
      limits:
        cpus: '2.0'
        memory: 2G
      reservations:
        cpus: '1.0'
        memory: 1G
```

## Troubleshooting

### Common Issues

#### License Server Not Responding
1. Check container status: `docker ps | grep license-server`
2. View logs: `docker logs license-server --tail 100`
3. Test internal connectivity: 
   ```bash
   docker exec nginx-proxy curl -k https://license-server:8443/health
   ```
4. Verify environment variables:
   ```bash
   docker exec license-server env | grep -E "JWT_SECRET|API_KEY|DATABASE_URL"
   ```
5. Check Nginx routing:
   ```bash
   docker exec nginx-proxy nginx -T | grep api-software
   ```

#### Database Connection Issues
1. Check PostgreSQL status: 
   ```bash
   docker exec db pg_isready -U adminuser -d meinedatenbank
   ```
2. Test connection from service:
   ```bash
   docker exec admin-panel psql postgresql://adminuser:supergeheimespasswort@db:5432/meinedatenbank -c "SELECT 1"
   ```
3. Check network connectivity:
   ```bash
   docker network inspect v2-docker_internal_net
   ```
4. Review PostgreSQL logs: 
   ```bash
   docker logs db --tail 50
   ```

#### High Memory Usage
1. Check container stats: `docker stats`
2. Review memory limits in docker-compose.yml
3. Analyze database queries for optimization
4. Consider scaling horizontally

### Health Checks

```bash
# License server health (über Nginx)
curl -k https://api-software-undso.z5m7q9dk3ah2v1plx6ju.com/health

# Admin panel health (über Nginx)
curl -k https://admin-panel-undso.z5m7q9dk3ah2v1plx6ju.com/

# Database health
docker exec db pg_isready -U adminuser -d meinedatenbank

# Redis health
docker exec redis-cache redis-cli ping

# RabbitMQ health
docker exec rabbitmq rabbitmqctl status

# Monitoring stack
curl http://localhost:9090/-/healthy  # Prometheus
curl http://localhost:3001/api/health  # Grafana
curl http://localhost:9093/-/healthy  # Alertmanager

# Container health overview
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

## Security Considerations

### API Security
- Use strong JWT_SECRET (minimum 32 characters)
- Rotate API keys regularly
- Implement rate limiting
- Use HTTPS in production

### Database Security
- Use strong passwords
- Limit database access
- Enable SSL for connections
- Regular security updates

### Container Security
- Use official base images
- Scan images for vulnerabilities
- Don't run containers as root
- Keep Docker updated

## Scaling Strategies

### Horizontal Scaling

#### Scaling License Server
```bash
# Scale license server instances
docker-compose up -d --scale license-server=3
```

#### Nginx Load Balancing Configuration
```nginx
# In nginx.conf
upstream license_servers {
    least_conn;
    server license-server_1:8443 max_fails=3 fail_timeout=30s;
    server license-server_2:8443 max_fails=3 fail_timeout=30s;
    server license-server_3:8443 max_fails=3 fail_timeout=30s;
    
    # Health checks
    keepalive 32;
}

server {
    server_name api-software-undso.z5m7q9dk3ah2v1plx6ju.com;
    
    location / {
        proxy_pass https://license_servers;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}
```

#### Scaling Considerations
- Redis für Session-Sharing zwischen Instanzen
- RabbitMQ für asynchrone Task-Verteilung
- Sticky Sessions bei Bedarf aktivieren

### Database Scaling
- Read replicas for reporting
- Connection pooling
- Query optimization
- Partitioning for large tables

## Disaster Recovery

### Backup Strategy
- Daily automated backups
- Off-site backup storage
- Test restore procedures
- Document recovery steps

### High Availability
- Database replication
- Service redundancy
- Load balancing
- Automated failover

## Monitoring Best Practices

1. **Set up alerts for critical metrics**
2. **Create runbooks for common issues**
3. **Regular review of dashboards**
4. **Capacity planning based on trends**
5. **Document all custom configurations**