Self-Hosted Installation

Deploy Engram in your own infrastructure with full control and privacy

Self-Hosted Installation

Deploy Engram in your own infrastructure for maximum control, privacy, and customization. This guide covers installation using Docker and Kubernetes.

Prerequisites

Before you begin, ensure you have:

  • Docker 20.10+ and Docker Compose
  • At least 4GB RAM available
  • 50GB storage space
  • PostgreSQL 14+ (for metadata storage)
  • Redis 6+ (for caching)

Quick Start with Docker Compose

The fastest way to get Engram running is with our pre-configured Docker Compose setup:

# Clone the repository
git clone https://github.com/engram/self-hosted.git
cd self-hosted

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start all services
docker-compose up -d

Environment Configuration

Edit your .env file with the following settings:

# Database Configuration
POSTGRES_USER=engram
POSTGRES_PASSWORD=your_secure_password
POSTGRES_DB=engram

# Redis Configuration
REDIS_URL=redis://redis:6379

# Engram Configuration
ENGRAM_SECRET_KEY=your_secret_key_here
ENGRAM_ADMIN_EMAIL=admin@yourcompany.com
ENGRAM_ADMIN_PASSWORD=secure_admin_password

# Vector Database
QDRANT_URL=http://qdrant:6333
QDRANT_API_KEY=your_qdrant_api_key

# Storage
STORAGE_BACKEND=local
STORAGE_PATH=/data/engram

Manual Installation

1. Vector Database Setup

First, set up Qdrant for vector storage:

# Pull and run Qdrant
docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -v qdrant_data:/qdrant/storage \
  qdrant/qdrant:latest

2. PostgreSQL Setup

Set up PostgreSQL for metadata storage:

# Create PostgreSQL container
docker run -d \
  --name engram-postgres \
  -e POSTGRES_USER=engram \
  -e POSTGRES_PASSWORD=your_password \
  -e POSTGRES_DB=engram \
  -p 5432:5432 \
  -v postgres_data:/var/lib/postgresql/data \
  postgres:14

3. Redis Setup

Set up Redis for caching:

# Create Redis container
docker run -d \
  --name engram-redis \
  -p 6379:6379 \
  -v redis_data:/data \
  redis:alpine

4. Engram Core Service

Deploy the main Engram service:

# Pull Engram image
docker pull engram/core:latest

# Run Engram
docker run -d \
  --name engram-core \
  --link engram-postgres \
  --link engram-redis \
  --link qdrant \
  -e DATABASE_URL=postgresql://engram:password@engram-postgres:5432/engram \
  -e REDIS_URL=redis://engram-redis:6379 \
  -e QDRANT_URL=http://qdrant:6333 \
  -p 8080:8080 \
  engram/core:latest

Configuration

Memory Settings

Configure memory allocation in your docker-compose.yml:

services:
  engram-core:
    deploy:
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 1G

Security Configuration

For production deployments, ensure:

  1. SSL/TLS: Configure HTTPS with proper certificates
  2. Authentication: Set up API key authentication
  3. Network Security: Use firewalls and VPNs
  4. Data Encryption: Enable encryption at rest and in transit

Performance Tuning

Optimize performance based on your workload:

# In docker-compose.yml
environment:
  # Vector search performance
  QDRANT_INDEXING_THRESHOLD: 20000
  
  # Memory management
  ENGRAM_MAX_MEMORY_SIZE: "2GB"
  ENGRAM_CACHE_TTL: 3600
  
  # Parallel processing
  ENGRAM_WORKER_THREADS: 4

Monitoring and Maintenance

Health Checks

Monitor your Engram deployment:

# Check service health
curl http://localhost:8080/health

# Check vector database status
curl http://localhost:6333/cluster

Backup and Recovery

Set up automated backups:

# Backup PostgreSQL
docker exec engram-postgres pg_dump -U engram engram > backup.sql

# Backup Qdrant data
docker exec qdrant tar -czf - /qdrant/storage > qdrant_backup.tar.gz

Updates

Keep your installation up to date:

# Pull latest images
docker-compose pull

# Restart services with new images
docker-compose up -d

Scaling

For high-traffic deployments, consider:

  1. Horizontal Scaling: Run multiple Engram instances behind a load balancer
  2. Database Clustering: Use PostgreSQL clustering for high availability
  3. Vector Database Scaling: Scale Qdrant with multiple nodes
  4. Caching: Implement distributed Redis caching

Troubleshooting

Common issues and solutions:

Service Won't Start

# Check logs
docker-compose logs engram-core

# Verify network connectivity
docker exec engram-core ping qdrant

Memory Issues

# Monitor memory usage
docker stats

# Increase memory limits in docker-compose.yml

Performance Problems

# Check database connections
docker exec engram-postgres pg_stat_activity

# Monitor vector database performance
curl http://localhost:6333/metrics

Next Steps