Audit Log Cleanup

MXCP audit logs can grow over time. This guide covers retention policies, manual cleanup, and automated maintenance strategies.

Default Retention

MXCP uses a default retention period of 90 days for endpoint execution logs. Records older than this are automatically removed when you run the cleanup command.

Note: The retention period is currently fixed at 90 days and is not configurable via mxcp-site.yml.

Manual Cleanup

Using mxcp log-cleanup

# Apply retention policy
mxcp log-cleanup

# Preview what would be deleted
mxcp log-cleanup --dry-run

# Specific profile
mxcp log-cleanup --profile production

Output

Applying retention policies...

Deleted records by schema:
  mxcp.endpoints:1: 15234 records

Total records deleted: 15234

Dry-run output:

DRY RUN: Analyzing what would be deleted...

  mxcp.endpoints (retention: 90 days): 15234 records

Total records that would be deleted: 15234

Run without --dry-run to actually delete these records.

JSON output format (--json):

{
  "status": "success",
  "message": "Deleted 15234 records",
  "deleted_per_schema": {
    "mxcp.endpoints:1": 15234
  }
}

Dry-run JSON:

{
  "status": "dry_run",
  "message": "Would delete 15234 records",
  "deleted_per_schema": {
    "mxcp.endpoints:1": 15234
  }
}

Direct DuckDB Cleanup

For advanced scenarios:

# Export logs to DuckDB for analysis
mxcp log --export-duckdb audit_analysis.duckdb

# Query and clean up
duckdb audit_analysis.duckdb << 'EOF'
-- Check log distribution (timestamp is stored as ISO string)
SELECT
  DATE_TRUNC('month', timestamp::TIMESTAMP) as month,
  COUNT(*) as records
FROM audit_logs
GROUP BY month
ORDER BY month;

-- Delete old records
DELETE FROM audit_logs
WHERE timestamp::TIMESTAMP < CURRENT_DATE - INTERVAL '90 days';
EOF

Automated Cleanup

Using Cron

Create a cron job for regular cleanup:

# Edit crontab
crontab -e

# Run cleanup daily at 2 AM
0 2 * * * /usr/local/bin/mxcp log-cleanup --profile production >> /var/log/mxcp/cleanup.log 2>&1

# Run cleanup weekly on Sunday
0 3 * * 0 /usr/local/bin/mxcp log-cleanup --profile production >> /var/log/mxcp/cleanup.log 2>&1

Using systemd Timer

Create /etc/systemd/system/mxcp-log-cleanup.service:

[Unit]
Description=MXCP Audit Log Cleanup
After=network.target

[Service]
Type=oneshot
User=mxcp
Group=mxcp
WorkingDirectory=/opt/mxcp
ExecStart=/usr/local/bin/mxcp log-cleanup --profile production
StandardOutput=journal
StandardError=journal

Create /etc/systemd/system/mxcp-log-cleanup.timer:

[Unit]
Description=Daily MXCP audit log cleanup

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600

[Install]
WantedBy=timers.target

Enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable mxcp-log-cleanup.timer
sudo systemctl start mxcp-log-cleanup.timer

# Check timer status
systemctl list-timers | grep mxcp

Docker Scheduled Cleanup

docker-compose.yml

version: '3.8'

services:
  mxcp:
    build: .
    volumes:
      - audit-logs:/var/log/mxcp

  # Cleanup sidecar
  cleanup:
    image: your-mxcp-image
    volumes:
      - audit-logs:/var/log/mxcp
    command: >
      sh -c "while true; do
        sleep 86400;
        mxcp log-cleanup --profile production;
      done"
    depends_on:
      - mxcp

volumes:
  audit-logs:

Log Rotation

Audit logs are stored as JSONL files and do not have built-in rotation. Use external tools like logrotate to manage file sizes.

Using logrotate

Create /etc/logrotate.d/mxcp:

/var/log/mxcp/*.jsonl {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 mxcp mxcp
    postrotate
        # Signal MXCP to reopen log files
        kill -HUP $(pgrep -f "mxcp serve") 2>/dev/null || true
    endscript
}

Archiving Strategies

Archive to Object Storage

archive-logs.sh

#!/bin/bash
DATE=$(date +%Y%m%d)
ARCHIVE_DIR="/var/log/mxcp/archive"
S3_BUCKET="s3://my-bucket/mxcp-audit"

# Compress old logs
find /var/log/mxcp -name "*.jsonl.*" -mtime +7 -exec gzip {} \;

# Move to archive
mkdir -p $ARCHIVE_DIR
mv /var/log/mxcp/*.gz $ARCHIVE_DIR/

# Upload to S3
aws s3 sync $ARCHIVE_DIR $S3_BUCKET/$DATE/

# Clean up local archive after successful upload
rm -rf $ARCHIVE_DIR/*

Archive to DuckDB

archive-to-duckdb.sh

#!/bin/bash
DATE=$(date +%Y%m%d)
ARCHIVE_DB="/archive/audit-$DATE.duckdb"

# Export current logs
mxcp log --export-duckdb $ARCHIVE_DB --since 30d

# Compress archive
gzip $ARCHIVE_DB

# Clean up old JSONL logs
mxcp log-cleanup

Compliance Considerations

Retention Requirements

Different regulations have different retention requirements:

Regulation	Typical Retention
SOC 2	1 year
HIPAA	6 years
PCI DSS	1 year
GDPR	As needed

Note: The default 90-day retention may not meet all compliance requirements. For longer retention, consider archiving logs before they’re deleted (see Archiving Strategies above).

Secure Deletion

For compliance, ensure secure deletion:

# Overwrite before deletion
shred -vfz -n 3 /var/log/mxcp/audit.jsonl.old

# Or use secure delete tools
srm -sz /var/log/mxcp/audit.jsonl.old

Audit Trail of Deletions

Log cleanup operations:

cleanup-with-audit.sh

#!/bin/bash
TIMESTAMP=$(date -Iseconds)
RECORDS_BEFORE=$(wc -l < /var/log/mxcp/audit.jsonl)

mxcp log-cleanup --profile production

RECORDS_AFTER=$(wc -l < /var/log/mxcp/audit.jsonl)
DELETED=$((RECORDS_BEFORE - RECORDS_AFTER))

echo "$TIMESTAMP: Deleted $DELETED audit records" >> /var/log/mxcp/cleanup-audit.log

Best Practices

1. Plan for Compliance Requirements

The default 90-day retention period works for most operational use cases. For compliance requirements that need longer retention:

# Archive before cleanup
mxcp log --export-duckdb /archive/audit-$(date +%Y%m%d).duckdb

# Then run cleanup
mxcp log-cleanup

2. Monitor Log Growth

Alert on unusual growth:

# Check log size
LOG_SIZE=$(stat -f%z /var/log/mxcp/audit.jsonl 2>/dev/null || stat -c%s /var/log/mxcp/audit.jsonl)
if [ $LOG_SIZE -gt 1073741824 ]; then  # 1GB
    echo "WARNING: Audit log exceeds 1GB"
fi

3. Test Cleanup Before Production

# Always dry-run first
mxcp log-cleanup --dry-run --profile production

# Review what will be deleted
# Then run actual cleanup
mxcp log-cleanup --profile production

4. Archive Before Deletion

Keep archives for compliance:

# Archive first
mxcp log --export-duckdb /archive/audit-$(date +%Y%m%d).duckdb

# Then cleanup
mxcp log-cleanup

5. Regular Maintenance Schedule

• Daily: Run cleanup with retention policy
• Weekly: Archive old logs to cold storage
• Monthly: Verify retention compliance
• Quarterly: Review retention requirements

Troubleshooting

”Log file locked"

# Check if MXCP is writing to the log
lsof /var/log/mxcp/audit.jsonl

# Wait for writes to complete or
# send SIGHUP to close and reopen
kill -HUP $(pgrep -f "mxcp serve")

"Permission denied"

# Check permissions
ls -la /var/log/mxcp/

# Fix permissions
sudo chown mxcp:mxcp /var/log/mxcp/*.jsonl
sudo chmod 640 /var/log/mxcp/*.jsonl

"Disk full”

# Emergency cleanup
mxcp log-cleanup --profile production

# Or remove oldest archives
rm /var/log/mxcp/audit.jsonl.*.gz

# Check disk space
df -h /var/log/mxcp

Next Steps

• Auditing - Audit log configuration
• Monitoring - Log analysis
• Deployment - Production setup