# StackAI Auto CLI

Your Tools • Your Data • Your Rules

Enterprise License 99.9% Uptime HIPAA + ISO Compliant

Enterprise-grade command-line interface for auto-deploying, managing, and maintaining StackAI's platform on customer infrastructure. ## 🚀 Overview The **StackAI CLI** orchestrates a complete 50+ service Docker deployment for enterprise AI infrastructure. Built for Fortune 500 customers requiring strict data sovereignty, it provides: * **Complete bootstrap** from bare Ubuntu to production in < 1 hour * **License-key encrypted secrets** with AES-256-GCM encryption * **Self-service diagnostics** with automated support bundle generation * **Compliant** security architecture **Architecture**: 4-layer stack with 50+ services across databases, authentication, AI backends, and frontend. *** ## 📋 Prerequisites ### System Requirements | Component | Minimum Version | Recommended | | ------------------ | --------------- | ----------- | | **Ubuntu** | 24.04 LTS | 24.04 LTS | | **Docker** | 24.0+ | 27.0+ | | **Docker Compose** | 2.20+ | 2.29+ | | **RAM** | 32GB | 64GB+ | | **CPU** | 8 cores | 16+ cores | | **Storage** | 500GB SSD | 1TB+ NVMe | ### Install Docker ```bash curl -fsSL https://get.docker.com | sh && sudo usermod -aG docker $USER && newgrp docker ``` > Full instructions: > **Note:** Run `stackai` as your normal user, not with `sudo`. Config and license live under `~/.config/stackai/`, and on RHEL / Amazon Linux `sudo`'s `secure_path` excludes `/usr/local/bin` so `sudo stackai` resolves to "command not found". The docker-group step above is what lets your user talk to the docker socket without `sudo`. ### Required Credentials * **StackAI License Key** (provided by StackAI Account Manager) * **Registry Credentials** (provided by StackAI Account Manager) * **Domain Name** (for production deployments with TLS certificates) ## Installation ### Automated Installation ```bash # Download and install the CLI + deployment bundle # The installer will prompt for your StackAI License Key curl -sSf https://install.stack.ai | bash # Verify installation stackai --version ``` On Linux, the installer now defaults to the static `musl` build to avoid host glibc version issues. Set `STACKAI_LIBC=gnu` if you need to force the dynamic GNU build. **What gets installed:** * `stackai` binary in `/usr/local/bin/` (or `~/.local/bin/`) * Complete deployment bundle in `~/.config/stackai/` (or custom directory via `--data-dir`) containing: * `docker-compose.yaml` (50+ services) * Configuration templates * Version manifests * SSL certificate templates * **License key** saved to your shell configuration (`~/.bashrc`, `~/.zshrc`, etc.) as `STACKAI_LICENSE_KEY` **Note**: The installer prompts for your StackAI License Key and saves it to your shell config. This allows `stackai init` to automatically use it without prompting again. *** ## 🏠 Quick Start Deploy StackAI in 3 commands: ### 1. Complete Bootstrap ```bash # License key is automatically detected from STACKAI_LICENSE_KEY env var # (set by the installer in your shell config) stackai init # Or provide license explicitly (overrides env var) stackai init --license YOUR_LICENSE_KEY # Force reinitialize existing installation stackai init --force ``` **What happens:** * ✅ **License collection** (from env var, CLI arg, or interactive prompt) * ✅ **System prerequisites validation** (Docker, Docker Compose - always checked) * ✅ **Directory structure creation** (`~/.config/stackai/` or custom via `--data-dir`) * ✅ **Interactive configuration collection** (IP/domain, registry credentials, service URLs) * ✅ **Encrypted secrets vault generation** (license-key protected) * ✅ **Environment configuration** (`.env` file with all secrets) * ✅ **Docker registry authentication** (using stored credentials) * ✅ **Service orchestration setup** (docker-compose.yaml) **Note**: All configuration is collected interactively. The license key can come from the `STACKAI_LICENSE_KEY` environment variable (set by the installer) or be provided via `--license` flag. All other values are prompted for during initialization. ### 2. Deploy Services ```bash stackai system pull stackai deploy start ``` ### 3. Verify Deployment ```bash stackai deploy status ``` **Expected output:** ``` [🟢] Layer 1: Databases healthy ✓ (mongodb, weaviate, redis) [🟢] Layer 2: Supabase healthy ✓ (postgres, auth, realtime, studio) [🟢] Layer 3: Backend healthy ✓ (stackend, temporal) [🟢] Layer 4: Frontend healthy ✓ (nginx, stackweb) Dashboard: https://your-domain.com API: https://your-domain.com/api Studio: https://your-domain.com/studio ``` *** ## 📖 Command Reference ### Core Commands | Command | Purpose | Key Options | | ------------------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | **`stackai init`** | Interactive initialization for a StackAI deployment | `--license`, `--force`, `--skip-pull`, `--data-dir` | | **`stackai deploy start`** | Start Docker services | `[SERVICE]`, `--build` | | **`stackai deploy stop`** | Stop Docker services | `[SERVICE]` | | **`stackai deploy restart`** | Restart Docker services | `[SERVICE]`, `--force` | | **`stackai deploy status`** | Show service status | — | | **`stackai deploy logs`** | View container logs | `[SERVICE]`, `-f/--follow`, `-n/--tail ` | | **`stackai config domains`** | Manage domain URLs | — | | **`stackai config tls`** | TLS certificate management | `list`, `generate`, `letsencrypt [--email ] [--staging]`, `install --cert --key [--ca ]`, `renew` | | **`stackai config secrets`** | View/manage encrypted secrets | `--reveal` | | **`stackai config registry`** | Docker registry authentication | `login`, `logout`, `status` | | **`stackai config saml`** | SAML authentication configuration | `enable`, `status`, `disable`, `add-provider `, `list-providers`, `delete-provider ` | | **`stackai config sso`** | SSO domain registration | `register [ROLE]`, `status` | | **`stackai config env`** | Generate/update `.env` configuration | — | | **`stackai system backup`** | Create configuration and data backup | `-n/--name `, `-l/--list` | | **`stackai system restore`** | Restore from backup | `` | | **`stackai system update`** | Update StackAI platform version | `-v/--version `, `--skip-checksum` | | **`stackai system migrate`** | Run database migrations | — | | **`stackai system pull`** | Pull Docker images | — | | **`stackai system build`** | Build-related operations for services | `[SERVICE]`, `--no-cache` | | **`stackai system cleanup`** | Remove stopped containers and unused images | `-y/--yes` | | **`stackai system prune`** | Aggressive cleanup including volumes (DESTRUCTIVE) | `-y/--yes` | | **`stackai system templates`** | Manage workflow templates | `init`, `update` | | **`stackai system releases`** | Show or set release | `list`, `set ` | | **`stackai system import`** | Import volumes and env vars from old deployment | `--force`, `--dry-run` | | **`stackai diagnose doctor`** | Run system health checks | — | | **`stackai diagnose support`** | Generate sanitized support bundle | — | | **`stackai diagnose info`** | Show deployment information | — | ### Detailed Command Reference #### `stackai init` Interactive initialization of a StackAI deployment with encrypted customer configuration. **What it does:** * **License collection** (from `STACKAI_LICENSE_KEY` env var, `--license` flag, or interactive prompt) * **System prerequisites validation** (Docker, Docker Compose - always checked) * Collects customer configuration interactively (public host, registry credentials, service URLs with derived defaults) * Stores everything encrypted in vault using license key * Performs automatic Docker registry authentication * Generates secure secrets and environment configuration * Never asks for credentials again after initial setup ```bash stackai init [OPTIONS] OPTIONS: --license StackAI license key (optional: checks STACKAI_LICENSE_KEY env var first) --force Overwrite existing deployment --skip-pull Skip pulling Docker images (you can run `stackai system pull` later) --data-dir Custom installation directory (default: ~/.config/stackai/) INTERACTIVE PROMPTS: StackAI License Key (if not in env var or --license flag) Public host (IP or domain) where StackAI will be accessible (default: detected IP or 127.0.0.1) Azure Registry Username Azure Registry Password Web URL (press Enter to use derived default: http://:3000) API URL (press Enter to use derived default: http://:8000) Supabase URL (press Enter to use derived default: http://:8800) S3 URL (press Enter to use derived default: http://:9000) Temporal URL (press Enter to use derived default: http://:7233) ENVIRONMENT VARIABLES: STACKAI_LICENSE_KEY License key (automatically set by installer in shell config) STACKAI_ROOT_DIR Custom installation directory (alternative to --data-dir) STACKAI_SKIP_PULL If set, behaves like `--skip-pull` ``` **License Key Priority:** 1. `--license` CLI flag (highest priority) 2. `STACKAI_LICENSE_KEY` environment variable (set by installer) 3. Interactive prompt (if neither above is provided) **Custom Installation Directory:** If your home directory has limited space, use `--data-dir` to install to a different filesystem: ```bash sudo stackai init --data-dir /mnt/data/stackai --license ``` The custom path is persisted to `/etc/stackai/config.yaml` (following the Docker/K3s convention), so all subsequent commands automatically use it — no need to pass `--data-dir` again. Resolution priority: 1. `--data-dir` CLI flag (highest priority) 2. `STACKAI_ROOT_DIR` environment variable 3. `/etc/stackai/config.yaml` (persisted from init) 4. `~/.config/stackai/` (default) **Important**: All other commands (`deploy`, `config`, `system`, `diagnose`) require `stackai init` to be completed first. They will show an error if initialization hasn't been done. #### Service Deployment Commands **`stackai deploy start`** Start Docker Compose services with optional rebuild. ```bash stackai deploy start [SERVICE] [--build] ``` **`stackai deploy stop`** Stop Docker Compose services. ```bash stackai deploy stop [SERVICE] ``` **`stackai deploy restart`** Restart Docker Compose services with optional force recreate. ```bash stackai deploy restart [SERVICE] [--force] ``` **`stackai deploy status`** Show comprehensive service health status with indicators. ```bash stackai deploy status ``` **`stackai deploy logs`** View container logs with follow and tail options. ```bash stackai deploy logs [SERVICE] [-f|--follow] [-n|--tail ] ``` #### Configuration Commands **`stackai config domains`** Configure domain URLs for the deployment. ```bash stackai config domains ``` **`stackai config tls`** Manage SSL/TLS certificate configuration used by the reverse proxy. ```bash stackai config tls # Same as: list stackai config tls list stackai config tls generate # Self-signed (development) stackai config tls letsencrypt --email you@company.com # Let's Encrypt wildcard (production) stackai config tls letsencrypt --email you@company.com --staging # Test with staging first stackai config tls install --cert /path/to/cert.pem --key /path/to/key.pem # Custom certs stackai config tls renew # Renewal guidance ``` **Let's Encrypt (recommended for production):** The `letsencrypt` subcommand obtains a trusted wildcard certificate via DNS-01 challenge: 1. Creates an ACME account with Let's Encrypt 2. Requests a wildcard certificate (e.g., `*.cli.dev.stack.ai`) 3. Displays one or two TXT records to create in your DNS provider 4. Verifies DNS propagation, submits the challenge, and installs the certificate 5. Prompts to restart nginx to apply Certificates are valid for **90 days**. Set a reminder to renew by running the command again. If the web URL uses the bare base domain (e.g., `cli.dev.stack.ai` instead of `web.cli.dev.stack.ai`), the certificate automatically includes both the base domain and wildcard. **`stackai config secrets`** View license-key encrypted secrets (optionally revealed). ```bash stackai config secrets [--reveal] # --reveal shows unmasked values ``` **`stackai config registry`** Manage Docker registry authentication using stored credentials from vault. ```bash stackai config registry # Default action: login stackai config registry login # Authenticate with registry using stored credentials stackai config registry logout # Logout from registry stackai config registry status # Check authentication status ``` **Note**: Registry credentials are automatically configured during `stackai init` and stored encrypted in the vault. **`stackai config saml`** Configure SAML identity providers and authentication. ```bash stackai config saml enable # Enable SAML authentication stackai config saml status # Show SAML configuration status stackai config saml disable # Disable SAML authentication stackai config saml add-provider # Add identity provider stackai config saml list-providers # List configured providers stackai config saml delete-provider # Delete provider ``` **`stackai config sso`** Manage SSO domain registrations. ```bash stackai config sso register [ROLE] # Register SSO domain (ROLE is optional positional arg) stackai config sso status # Show SSO status ``` **`stackai config env`** Generate or update .env configuration file. ```bash stackai config env ``` #### System Management Commands **`stackai system backup`** Create comprehensive backup of configuration, databases, and version information. **Features:** * Creates compressed `.tar.gz` archive with checksum verification * Backs up PostgreSQL and MongoDB databases * Includes configuration files (`.env`, `docker-compose.yaml`, `config/` directory) * Includes release/version information (`.release`, `versions.json` when present) * Includes encrypted secrets vault (`secrets.vault`) * Generates manifest with metadata and component status * Automatic backup before updates and version changes ```bash stackai system backup [-n|--name ] [-l|--list] ``` **Options:** * `-n, --name `: Custom backup name (default: auto-generated timestamp) * `-l, --list`: List all available backups instead of creating one **Backup includes:** * PostgreSQL database dump (`postgresql.sql`) * MongoDB archive (`mongodb.archive`) * Configuration files (sanitized) * Version information * Encrypted secrets vault * Complete `config/` directory * Manifest with metadata and checksum **`stackai system restore`** Restore deployment from backup archive with safety checks. **Features:** * Verifies backup checksum before restore * Creates safety backup of current state before restore * Checks version compatibility * Restores all configuration, databases, and version information * Validates backup integrity ```bash stackai system restore ``` **Restore process:** 1. Verifies backup checksum 2. Creates safety backup of current state 3. Checks version compatibility 4. Stops services 5. Restores configuration files 6. Restores databases 7. Restores version information 8. Restarts services **`stackai system update`** Update StackAI platform (CLI + services) to latest or specific version. **Features:** * Automatic backup before update * CLI self-update capability * Version-aware updates with pinned subservice versions * Caching mechanism for downloaded releases * **Mandatory checksum verification** (prevents supply chain attacks) * Atomic binary replacement ```bash stackai system update [-v|--version ] [--skip-checksum] ``` **Options:** * `-v, --version `: Update to specific version (e.g., `1.0.9`) * `--skip-checksum`: Skip checksum verification (NOT RECOMMENDED - only for air-gapped environments) * Without `--version`: Updates to version set via `stackai system releases set`, or latest if not set **Security Note**: Checksum verification is **mandatory by default** to prevent supply chain attacks. The update will fail if checksums cannot be verified. Only use `--skip-checksum` in air-gapped environments where checksum files are unavailable. **Update process:** 1. Creates automatic backup 2. Resolves target version (from flag, `.release` file, or latest) 3. Downloads release tarball (with cache support) 4. **Verifies checksum** (mandatory unless `--skip-checksum` is used) 5. Updates configuration files (`docker-compose.yaml`, `versions.json`) 6. Updates CLI binary atomically 7. Updates `.release` file 8. Pulls new Docker images 9. Restarts services **Version resolution priority:** 1. `--version` flag (if provided) 2. `.release` file (if set via `stackai system releases set`) 3. Latest version from S3 **Checksum verification:** * Checksums are fetched from `https://install.stack.ai/v{VERSION}/sha256sums` * Falls back to `https://install.stack.ai/latest/sha256sums` if versioned file not found * Update fails if checksum cannot be verified (unless `--skip-checksum` is used) * Air-gapped environments: Use `--skip-checksum` flag with explicit warning **`stackai system migrate`** Run database migrations and template synchronization. ```bash stackai system migrate ``` **`stackai system pull`** Pull latest Docker images from registry. ```bash stackai system pull ``` **`stackai system build`** Build custom Docker images for services. ```bash stackai system build [SERVICE] [--no-cache] ``` **`stackai system cleanup`** Remove stopped containers and unused images. ```bash stackai system cleanup [-y|--yes] ``` **`stackai system prune`** Destructive cleanup including volumes (removes all data). ```bash stackai system prune [-y|--yes] ``` **`stackai system templates`** Manage workflow templates and configurations. ```bash stackai system templates init stackai system templates update ``` **`stackai system releases`** Manage StackAI platform releases with subservice version tracking. **Features:** * Display current installed release * List all available releases with subservice mappings * Set target release for future updates * Automatic backup before release changes * Shows pinned subservice versions (stackend, stackweb, stackrepl) ```bash stackai system releases # Show current release (default) stackai system releases list # List available releases stackai system releases set # Set target release ``` **Release information:** * **CLI Binary Version**: Version of the running CLI binary * **Installed Release**: Version from `.release` file (what's currently deployed) * **Subservice Versions**: Pinned image tags for stackend, stackweb, stackrepl **Release management:** * `stackai system releases`: Shows current installed release and subservice versions * `stackai system releases list`: Lists all available releases from `versions.json` with their subservice mappings * `stackai system releases set `: Sets target release in `.release` file (creates backup automatically) **Note**: Setting a release doesn't immediately update the platform. Run `stackai system update` to apply the release change. **`stackai system import`** Import volumes, configuration, and environment variables from an old stackai-auto-docker deployment into the new CLI structure. **Use Case**: Migrate from the old multi-compose repository structure to the unified CLI. **⚠️ Prerequisites**: The CLI must be installed and initialized BEFORE running import. The import command requires `docker-compose.yaml`, `versions.json`, and `.release` files which are created by the installer and `stackai init`. **Features:** * Near-zero downtime migration with staged sync * Automatic discovery of source volumes and env files * Per-service `.env` files merged into single unified `.env` * Secrets automatically stored in encrypted vault * Version variables (`STACKEND_VERSION`, `STACKREPL_VERSION`) automatically injected * Preflight checks (disk space, permissions, Docker, CLI prerequisites) * Progress bars and file verification * Dry-run mode to preview changes * Automatic file ownership fix when running with sudo * Automatic service stop/start for data integrity ```bash # Run from the old stackai-auto-docker root directory cd /path/to/old/stackai-auto-docker # Use sudo to access root-owned Docker volume data sudo stackai system import # Import with sudo for root-owned volumes sudo stackai system import --dry-run # Preview what would be copied sudo stackai system import --force # Overwrite existing target files ``` **Why sudo?** Docker volumes (MongoDB, PostgreSQL, etc.) are owned by root. The import command needs elevated privileges to read these files. When running with sudo, the CLI automatically: 1. Writes to the original user's directory (`/home/ubuntu/.config/stackai/` or custom `--data-dir`) instead of root's 2. Fixes file ownership after import so subsequent commands work without sudo **Volume Mappings:** | Old Path | New Path (relative to data dir) | | ----------------------------------------------- | ------------------------------- | | `mongodb/mongodb_data/` | `data/mongodb/` | | `weaviate/weaviate_data/` | `data/weaviate/` | | `supabase/volumes/db/data/supabase/postgresql/` | `data/supabase/postgresql/` | | `supabase/volumes/storage/` | `data/supabase/storage/` | | `temporalio/volumes/postgres/` | `data/temporal/` | | `supabase/volumes/db/*.sql` | `config/supabase/db/` | | `stackend/*.toml` | `config/stackend/` | | `nginx/*.conf` | `config/nginx/` | **Environment Variable Import:** | Old Env Files | New Location | | ------------------------ | --------------------------------- | | `mongodb/.env` | `~/.config/stackai/.env` (merged) | | `supabase/.env` | `~/.config/stackai/.env` (merged) | | `stackend/.env` | `~/.config/stackai/.env` (merged) | | `stackweb/.env` | `~/.config/stackai/.env` (merged) | | `weaviate/.env` | `~/.config/stackai/.env` (merged) | | `unstructured/.env` | `~/.config/stackai/.env` (merged) | | Secrets (180+ variables) | `~/.config/stackai/secrets.vault` | **Import Flow:** 1. **Prerequisites Check**: Verifies CLI is initialized (`docker-compose.yaml`, `versions.json`, `.release` exist) 2. **Discovery**: Scans current directory for known volume paths and env files 3. **Preflight**: Checks disk space, permissions, running services 4. **Stage 1 Sync**: Copies data while source may still be running 5. **Service Stop**: Stops source services for data integrity 6. **Stage 2 Sync**: Final sync to catch any changes 7. **Verification**: Confirms all files were copied correctly 8. **Env Import**: Merges per-service `.env` files, stores secrets in vault 9. **Version Injection**: Adds `STACKEND_VERSION` and `STACKREPL_VERSION` based on `.release` 10. **Ownership Fix**: Changes file ownership back to original user (when running with sudo) **Large Data Handling (50GB+):** The import is optimized for large production datasets: * **Staged sync**: Minimizes downtime by copying most data while services run * **8MB buffered I/O**: Large files (>100MB) use buffered copy for efficiency * **Delta sync**: Stage 2 only copies files that changed during Stage 1 * **Time estimates**: Shows estimated import time based on data size * **Progress tracking**: Per-volume progress bars during copy For very large datasets (100GB+): * Expect \~10-15 minutes per 100GB on SSD * Stage 2 (downtime) is typically <1 minute for most workloads * Monitor disk I/O during import for optimal performance **⚠️ Critical: Stop Old Services Before Import** For database integrity, stop the old deployment BEFORE importing: ```bash # Stop old deployment first cd /path/to/old/stackai-auto-docker docker compose down # Then run import sudo stackai system import --force ``` If databases are running during import, journal files may be corrupted causing startup failures. #### Diagnostic Commands **`stackai diagnose doctor`** Run comprehensive system health checks. ```bash stackai diagnose doctor ``` **`stackai diagnose support`** Generate comprehensive sanitized support bundle for asynchronous troubleshooting. **Features:** * Collects diagnostic data from all 50+ services * Automatic sanitization of sensitive information * Version information included * Error analysis and pattern detection * Resource usage per container * Network diagnostics * Complete configuration files (sanitized) ```bash stackai diagnose support ``` **Support bundle includes:** * **System Information**: OS, Docker versions, hostname, uptime, disk usage * **Version Information**: CLI version, StackAI version, Docker image tags, `versions.json` * **Configuration**: Sanitized `.env`, `docker-compose.yaml`, complete `config/` directory structure * **Container Status**: Health checks, resource usage, restart counts * **Logs**: Last 1000 lines from all 50+ services (sanitized) * **Error Analysis**: Error/warning counts per service, patterns, summary * **Resource Usage**: CPU, memory, disk, network I/O per container * **Network Diagnostics**: Network interfaces, Docker networks, database connectivity tests * **Diagnostics**: Health check results **Security & Privacy:** * All passwords, secrets, keys, and tokens automatically redacted * Email addresses anonymized * No database contents or user data included * Safe to send to support team **Output**: `~/.config/stackai/support/stackai-support-YYYYMMDD_HHMMSS.tar.gz` **Send to**: `support@stack-ai.com` **`stackai diagnose info`** Show detailed deployment and system information. ```bash stackai diagnose info ``` ### Global Options ```bash --help Show help --version Show version --debug Enable debug output (global) --data-dir Custom installation directory (env: STACKAI_ROOT_DIR) ``` **Note**: Working directory defaults to `~/.config/stackai/`. Use `--data-dir` or the `STACKAI_ROOT_DIR` environment variable to specify a custom directory (e.g., when the home directory has limited space). The custom path is persisted to `/etc/stackai/config.yaml` during init so all subsequent commands find it automatically. *** ## 🏗️ Architecture & Services StackAI deploys across **4 layers** with **50+ services**: ### Layer 1: Databases (Foundation) | Service | Purpose | Port | Storage | | ------------ | ---------------------- | --------- | ---------------- | | **mongodb** | Document database | 27017 | `data/mongodb/` | | **weaviate** | Vector database for AI | 9090 | `data/weaviate/` | | **redis** | Cache & sessions | 6379 | `data/redis/` | | **minio** | Object storage | 9000/9001 | `data/minio/` | ### Layer 2: Supabase (Auth & Data) | Service | Purpose | Port | Storage | | --------------------- | ----------------------- | ---- | -------------------------- | | **supabase-db** | PostgreSQL database | 5432 | `data/supabase/postgresql` | | **supabase-auth** | Authentication service | 9999 | | | **supabase-kong** | API gateway | 8000 | | | **supabase-realtime** | Real-time subscriptions | 4000 | | | **supabase-storage** | File storage API | 5000 | | | **supabase-studio** | Admin dashboard | 3000 | | | **supabase-pooler** | Connection pooling | 4000 | | ### Layer 3: AI Backend (Processing) | Service | Purpose | Port | | ------------------ | ---------------------- | ---- | | **stackend** | LLM API server | 8000 | | **stackrepl** | Interactive REPL | — | | **unstructured** | Document processing | 9099 | | **temporal-db** | Temporal database | — | | **temporal** | Workflow orchestration | 8081 | | **temporal-admin** | Temporal admin tools | — | | **temporal-ui** | Temporal web UI | — | ### Layer 4: Frontend (User Interface) | Service | Purpose | Port | | ------------ | ------------------- | ------ | | **stackweb** | Next.js application | 3000 | | **nginx** | Reverse proxy + TLS | 80/443 | **All services are orchestrated together - individual enable/disable is not supported.** *** ## 🔐 Security Architecture ### License-Key Encrypted Vault **Enterprise-grade encryption with customer isolation:** * **AES-256-GCM encryption** with Argon2 key derivation * **License-key based**: Each customer's vault is locked to their unique license * **HIPAA/ISO compliant** for healthcare and regulated industries * **Headless server compatible** (no user sessions required) **Security guarantees:** * ✅ Zero plaintext secrets in configuration files * ✅ Customer data isolation (license-key boundaries) * ✅ Audit trail (license hash validation) * ✅ Air-gapped compatible (no external dependencies) ### Secret Management ```bash # View masked secrets stackai config secrets # Reveal actual values (use with caution) stackai config secrets --reveal ``` **What's encrypted:** * Database passwords (MongoDB, PostgreSQL, Redis) * JWT secrets and API keys * Supabase service keys * Weaviate authentication keys * SMTP credentials * SAML private keys * Encryption keys for various services ### File Permissions & Access Control > Paths shown use the default `~/.config/stackai/`. If you used `--data-dir`, substitute your custom path. ``` ~/.config/stackai/ ├── secrets.vault # 600 (owner read/write only) ├── .env # 600 (decrypted for docker-compose) ├── docker-compose.yaml # 644 (orchestration) └── data/ # 755 (persistent storage) ``` ### Registry Security * **Docker credential helper** manages registry authentication * **No plaintext passwords** in configuration files * **Credentials isolated** from application secrets *** ## 🔧 Configuration Management ### Directory Structure ``` ~/.config/stackai/ # Default working directory (customizable via --data-dir) ├── docker-compose.yaml # Service orchestration (generated) ├── .env # Environment variables (decrypted) ├── .release # Current StackAI platform release (e.g. 1.0.11) ├── secrets.vault # Encrypted secrets (license-key protected) ├── config/ # Template files │ ├── versions.json # Version manifests │ ├── nginx/ # Reverse proxy templates │ ├── supabase/ # Supabase configuration │ └── stackend/ # Backend configuration ├── backups/ # Backup archives │ ├── backup_YYYYMMDD_HHMMSS.tar.gz │ └── backup_YYYYMMDD_HHMMSS.sha256 ├── support/ # Support bundles │ └── stackai-support-YYYYMMDD_HHMMSS.tar.gz └── data/ # Persistent storage (100GB+) ├── mongodb/ # Document storage ├── weaviate/ # Vector embeddings ├── supabase/ # PostgreSQL data ├── minio/ # Object storage └── redis/ # Cache data ``` ### Configuration Files #### `.env` - Runtime Environment ```bash # Generated by stackai init - DO NOT EDIT STACKAI_LICENSE=your-license-key-hash MONGO_INITDB_ROOT_PASSWORD=decrypted-password POSTGRES_PASSWORD=decrypted-password JWT_SECRET=decrypted-jwt-secret # ... ~50 environment variables ``` ### Domain Configuration ```bash # Configure primary domain stackai config domains # Expected: your-domain.com # Services will be available at: # - https://your-domain.com (StackWeb) # - https://api.your-domain.com (StackEnd) # - https://studio.your-domain.com (Supabase Studio) ``` ### TLS/SSL Management ```bash # Obtain Let's Encrypt wildcard certificate (recommended for production) stackai config tls letsencrypt --email you@company.com # Install existing SSL certificates stackai config tls install --cert /path/to/cert.pem --key /path/to/key.pem # List configured certificates stackai config tls list # Configure domains (HTTP by default) stackai config domains ``` **Note:** `stackai init` generates **self-signed** certificates by default (browsers will warn). For production, use `stackai config tls letsencrypt` to obtain a trusted wildcard certificate from Let's Encrypt via DNS-01 challenge, or install your own certificates with `stackai config tls install`. *** ## 🚀 Deployment Lifecycle ### Production Deployment ```bash # 1. Install CLI (prompts for license key, saves to shell config) curl -sSf https://install.stack.ai | bash # 2. Bootstrap everything (license auto-detected from env var) stackai init # 3. Pre-pull images (reduces startup time) stackai system pull # 4. Deploy with health checks stackai deploy start # 5. Verify all services stackai deploy status # 6. Generate support baseline stackai diagnose support ``` ### Blue-Green Updates ```bash # Update to latest version (automatic backup created) stackai system update # Or update to specific version stackai system update --version 1.0.9 # Or set target release first, then update stackai system releases set 1.0.9 stackai system update # Air-gapped environments (NOT RECOMMENDED for production) stackai system update --skip-checksum # Verify health stackai deploy status ``` **Update features:** * Automatic backup created before update * CLI self-update capability * Version-aware updates with pinned subservice versions * Caching for faster subsequent updates * **Mandatory checksum verification** for security (prevents supply chain attacks) * `--skip-checksum` flag for air-gapped environments only ### Disaster Recovery ```bash # List available backups stackai system backup --list # Stop services stackai deploy stop # Restore from backup (creates safety backup automatically) stackai system restore ~/.config/stackai/backups/backup_20250115_143022.tar.gz # Restart services stackai deploy start # Verify integrity stackai deploy status ``` **Restore features:** * Checksum verification before restore * Safety backup of current state created automatically * Version compatibility checking * Complete state restoration (config, databases, versions) *** ## 📚 Common Customer Scenarios ### Scenario 1: First-Time Installation **Goal**: Deploy StackAI from scratch on a new Ubuntu server ```bash # 1. Install CLI (prompts for license key) curl -sSf https://install.stack.ai | bash # 2. Initialize (license auto-detected from env var) stackai init # 3. Pull images (reduces startup time) stackai system pull # 4. Start services stackai deploy start # 5. Verify deployment stackai deploy status ``` **Expected time**: < 1 hour ### Scenario 2: Service Not Starting **Symptom**: A specific service keeps crashing or won't start ```bash # 1. Check service status stackai deploy status # 2. View service logs stackai deploy logs --tail 200 # 3. Check for errors stackai deploy logs | grep -i error # 4. Run health diagnostics stackai diagnose doctor # 5. If issue persists, generate support bundle stackai diagnose support ``` ### Scenario 3: Updating to New Version **Goal**: Update StackAI platform to latest version safely ```bash # 1. Check current release stackai system releases # 2. List available releases stackai system releases list # 3. Update (automatic backup created) stackai system update # 4. Verify services are healthy stackai deploy status # 5. If issues occur, restore from backup stackai system backup --list stackai system restore ``` **Note**: Automatic backup is created before update. Checksum verification is mandatory for security. ### Scenario 4: Registry Authentication Issues **Symptom**: Cannot pull Docker images, getting "unauthorized" errors ```bash # 1. Verify license key is set echo $STACKAI_LICENSE_KEY # 2. Check registry authentication status stackai config registry status # 3. Re-authenticate with registry stackai config registry login # 4. Try pulling images again stackai system pull # 5. If still failing, reinitialize (requires re-entering config) stackai init --force ``` ### Scenario 5: Backup and Restore **Goal**: Create backup before major changes, restore if needed ```bash # Create backup stackai system backup --name before-update-$(date +%Y%m%d) # List available backups stackai system backup --list # Restore from backup (creates safety backup automatically) stackai deploy stop stackai system restore ~/.config/stackai/backups/backup_YYYYMMDD_HHMMSS.tar.gz stackai deploy start stackai deploy status ``` ### Scenario 6: Migrating from Old Deployment (stackai-auto-docker) **Goal**: Migrate from old stackai-auto-docker multi-compose structure to the new unified CLI **⚠️ Important**: The CLI must be installed and initialized BEFORE importing data. ```bash # ============================================================ # STEP 1: Install CLI on the server (if not already installed) # ============================================================ # The installer prompts for your license key and saves it to ~/.config/stackai/license curl -sSf https://install.stack.ai | bash # ============================================================ # STEP 2: Initialize CLI (configure domains, credentials) # ============================================================ # License is automatically read from ~/.config/stackai/license stackai init # ============================================================ # STEP 3: CRITICAL - Stop old deployment BEFORE import # ============================================================ # Database files MUST NOT be in use during import to avoid corruption cd /path/to/old/stackai-auto-docker docker compose down # Verify all containers are stopped docker ps # Should show no stackai containers running # ============================================================ # STEP 4: Preview what will be migrated (optional but recommended) # ============================================================ sudo stackai system import --dry-run # ============================================================ # STEP 5: Perform the import (must use sudo for root-owned volumes) # ============================================================ sudo stackai system import --force # ============================================================ # STEP 6: Start services with new CLI # ============================================================ stackai deploy start # ============================================================ # STEP 7: Verify health # ============================================================ stackai deploy status # ============================================================ # STEP 8: Test the deployment # ============================================================ # Access the web UI at your configured domain # Check that data was migrated successfully ``` **Why sudo for import?** Docker volumes (MongoDB, PostgreSQL, Weaviate, etc.) are owned by root because Docker runs containers as root by default. The import command needs elevated privileges to read these files. The CLI automatically: * Detects `SUDO_USER` and writes to the correct user directory (`/home/ubuntu/.config/stackai/`) * Fixes file ownership after import so all files are owned by the original user * Allows subsequent commands to run without sudo **What gets migrated:** | Data Type | Source Path | Target Path | | ---------------- | ------------------------------ | --------------------------------------------- | | MongoDB data | `mongodb/mongodb_data/` | `~/.config/stackai/data/mongodb/` | | PostgreSQL data | `supabase/volumes/db/data/` | `~/.config/stackai/data/supabase/postgresql/` | | Weaviate vectors | `weaviate/weaviate_data/` | `~/.config/stackai/data/weaviate/` | | MinIO storage | `supabase/volumes/storage/` | `~/.config/stackai/data/supabase/storage/` | | Temporal DB | `temporalio/volumes/postgres/` | `~/.config/stackai/data/temporal/` | | Stackend configs | `stackend/*.toml` | `~/.config/stackai/config/stackend/` | | Nginx configs | `nginx/*.conf` | `~/.config/stackai/config/nginx/` | | Environment vars | `*/.env` (7 files) | `~/.config/stackai/.env` (merged) | | Secrets | Various API keys | `~/.config/stackai/secrets.vault` (encrypted) | **What the import does automatically:** 1. ✅ Copies all data volumes to the new structure 2. ✅ Merges 7 separate `.env` files into one unified file 3. ✅ Stores 180+ secret variables in encrypted vault 4. ✅ Injects version variables (`STACKEND_VERSION`, `STACKREPL_VERSION`) 5. ✅ Fixes file ownership when running with sudo 6. ✅ Verifies file integrity after copy **⚠️ Common Issues:** | Issue | Cause | Solution | | --------------------------- | ---------------------------------- | -------------------------------------------------- | | "CLI not initialized" error | Import run before `stackai init` | Run `stackai init` first | | MongoDB won't start | Database was running during import | Stop old deployment, clear data, re-import | | Permission denied on vault | Ownership not fixed | Run `sudo chown -R $USER:$USER ~/.config/stackai/` | | Port already in use | Old containers still running | Run `docker compose down` in old directory | **Rollback**: The old deployment is NOT deleted. If migration fails, you can restart the old deployment: ```bash cd /path/to/old/stackai-auto-docker docker compose up -d ``` ### Scenario 7: Troubleshooting Production Issues **Goal**: Diagnose and resolve production problems ```bash # 1. Quick health check stackai diagnose doctor # 2. Check all service statuses stackai deploy status # 3. Monitor logs in real-time stackai deploy logs --follow # 4. Check specific service logs stackai deploy logs --tail 500 # 5. Generate comprehensive support bundle stackai diagnose support # 6. Send support bundle to support@stack-ai.com ``` ### Scenario 8: Air-Gapped Deployment **Goal**: Deploy/update in environment without internet access ```bash # For updates in air-gapped environments: # 1. Download release tarball on connected machine # 2. Transfer to air-gapped server # 3. Update with checksum skip (NOT RECOMMENDED for production) stackai system update --skip-checksum # For initial deployment: # 1. Transfer installation bundle manually # 2. Run installer locally # 3. Initialize with manual configuration stackai init ``` **Security Warning**: Skipping checksum verification reduces security. Only use in air-gapped environments where checksum files are unavailable. ### Scenario 9: Disk Space Issues **Symptom**: Running out of disk space ```bash # 1. Check disk usage df -h # 2. Check backup directory size du -sh ~/.config/stackai/backups/ # 3. List backups stackai system backup --list # 4. Remove old backups manually if needed rm ~/.config/stackai/backups/backup_OLD_DATE.tar.gz # 5. Clean Docker resources stackai system cleanup # 6. Aggressive cleanup (DESTRUCTIVE - removes volumes) stackai system prune --yes ``` *** ## 🔍 Observability & Support ### Health Monitoring ```bash # Comprehensive health check stackai diagnose doctor # Service status with health indicators stackai deploy status # Real-time logs stackai deploy logs --follow # Service-specific logs stackai deploy logs stackend --tail 100 ``` ### Support Bundle Generation **When issues occur, generate a comprehensive sanitized support bundle:** ```bash stackai diagnose support ``` **What's included:** * **System Information**: OS, Docker versions, hostname, uptime, disk usage * **Version Information**: CLI version, StackAI version, Docker image tags, version mappings * **Configuration**: Sanitized `.env`, `docker-compose.yaml`, complete `config/` directory * **Container Status**: Health checks, resource usage (CPU/memory/disk), restart counts * **Logs**: Last 1000 lines from all 50+ services (automatically sanitized) * **Error Analysis**: Error/warning counts per service, patterns, summary * **Resource Usage**: CPU, memory, disk, network I/O per container * **Network Diagnostics**: Network interfaces, Docker networks, database connectivity tests * **Diagnostics**: Health check results **Security & Privacy:** * All passwords, secrets, keys, tokens automatically redacted * Email addresses anonymized * No database contents or user data * Safe to send to support team **Output**: `~/.config/stackai/support/stackai-support-YYYYMMDD_HHMMSS.tar.gz` **Send to**: `support@stack-ai.com` *** ## 🆘 Troubleshooting Guide ### Quick Diagnostics When encountering issues, start with these diagnostic commands: ```bash # 1. Check overall system health stackai diagnose doctor # 2. View service status stackai deploy status # 3. Check specific service logs stackai deploy logs --tail 100 # 4. Generate support bundle (for support team) stackai diagnose support ``` ### Troubleshooting Common Issues | Issue | Symptom | Solution | | | -------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Registry Auth Failed** | `unauthorized` or `authentication failed` when pulling images |

1. Verify license key: echo $STACKAI\_LICENSE\_KEY
2. Re-authenticate: stackai config registry login
3. If still failing, reinitialize: stackai init --force

| | | **Checksum Verification Failed** | `Checksum verification required but checksum not found` during update |

Normal environments: Check internet connectivity, verify is accessible
Air-gapped: Use stackai system update --skip-checksum (not recommended for production)

| | | **Checksum Mismatch** | `Checksum mismatch! Expected: X, Got: Y` |

CRITICAL: Do not proceed. This indicates corrupted or tampered download.
1. Delete cached file: rm \~/.cache/stackai/stackai-v\*.tar.gz
2. Retry update: stackai system update
3. If persists, contact support with stackai diagnose support

| | | **Port Conflicts** | `port is already allocated` or services won't start |

1. Find conflicting process: sudo netstat -tlnp | grep :PORT
2. Stop conflicting service or change port in docker-compose.yaml
3. Common conflicts: 80, 443, 3000, 5432, 27017

| | **Disk Space** | `no space left on device` or deployment fails |

1. Check disk usage: df -h
2. Ensure 500GB+ free in your data directory (default: \~/.config/stackai/, or use --data-dir to point to a larger filesystem)
3. Clean old backups: stackai system backup --list, remove old ones
4. Clean Docker: stackai system cleanup

| | | **Memory Issues** | Services crash with OOM (Out of Memory) |

1. Verify RAM: free -h (need 64GB+ for production)
2. Check container memory: docker stats
3. Reduce concurrent services if needed
4. Consider larger instance

| | | **SSL/TLS Errors** | HTTPS not working, certificate errors |

1. Obtain Let's Encrypt cert: stackai config tls letsencrypt --email
2. Or install custom certs: stackai config tls install --cert /path/to/cert.pem --key /path/to/key.pem
3. Verify domain DNS points to server IP
4. Check nginx logs: stackai deploy logs nginx

| | | **Database Connection Issues** | `connection refused` to PostgreSQL/MongoDB |

1. Check database status: stackai deploy status
2. View database logs: stackai deploy logs supabase-db or stackai deploy logs mongodb
3. Verify databases are healthy: stackai diagnose doctor
4. Check if databases are starting: stackai deploy logs --follow

| | | **Services Not Starting** | Containers exit immediately or show unhealthy |

1. Check service logs: stackai deploy logs \
2. Run health check: stackai diagnose doctor
3. Verify configuration: stackai config secrets
4. Check Docker: docker ps -a to see exit codes
5. Generate support bundle: stackai diagnose support

| | | **License Key Mismatch** | `License key does not match vault` |

1. Verify correct license: echo $STACKAI\_LICENSE\_KEY
2. If license changed, reinitialize: stackai init --force
3. Warning: Reinitializing will require re-entering all configuration

| | | **Update Fails** | Update process stops or errors |

1. Check logs: stackai deploy logs
2. Verify backup exists: stackai system backup --list
3. Restore from backup if needed: stackai system restore \
4. Generate support bundle: stackai diagnose support

| | | **Vault Decryption Failed** | `Decryption failed` or `Invalid vault format` |

1. Verify license key matches: echo $STACKAI\_LICENSE\_KEY
2. Check vault file exists: ls -la \~/.config/stackai/secrets.vault
3. If vault corrupted, reinitialize: stackai init --force
4. Note: Reinitializing requires re-entering all secrets

| | | **Docker Permission Denied** | `permission denied` when running commands |

1. Add user to docker group: sudo usermod -aG docker $USER
2. Log out and back in, or: newgrp docker
3. Verify: docker ps should work without sudo

| | | **Docker Not Running** | `Cannot connect to Docker daemon` |

1. Start Docker: sudo systemctl start docker
2. Enable auto-start: sudo systemctl enable docker
3. Verify: docker ps

| | | **Configuration Not Found** | `StackAI has not been initialized` |

Run initialization: stackai init
Note: Requires license key (from env var or prompt) and interactive configuration

| | | **Backup/Restore Issues** | Backup fails or restore doesn't work |

1. Verify disk space: df -h (need space for backup + restore)
2. Check backup integrity: sha256sum \~/.config/stackai/backups/backup\_\*.tar.gz
3. Verify services are stopped before restore: stackai deploy stop
4. Check restore logs for specific errors

| | | **Network Connectivity** | Cannot pull images or download updates |

1. Check internet: curl -I
2. Verify DNS: nslookup install.stack.ai
3. Check firewall rules for outbound HTTPS (443)
4. For air-gapped: Use --skip-checksum flag and manual image transfer

| | | **Version Mismatch** | Services show different versions than expected |

1. Check current release: stackai system releases
2. List available releases: stackai system releases list
3. Set target release: stackai system releases set \
4. Update: stackai system update

| | **When to Contact Support:** * Critical security issues (checksum mismatches, unauthorized access) * Data loss or corruption * Services consistently failing after troubleshooting * Performance issues affecting production * Any issue requiring immediate resolution **Before Contacting Support:** 1. Generate support bundle: `stackai diagnose support` 2. Document the issue and steps to reproduce 3. Include relevant logs: `stackai deploy logs ` 4. Send support bundle to: `support@stack-ai.com` *** ## 📊 Performance & Scaling ### Resource Requirements by Scale | Deployment Size | RAM | CPU | Storage | Services | | -------------------- | ----- | -------- | ------- | ------------ | | **Development** | 32GB | 8 cores | 500GB | All services | | **Production Small** | 64GB | 16 cores | 1TB | All services | | **Production Large** | 128GB | 32 cores | 2TB | All services | ### Monitoring Commands ```bash # System resources stackai diagnose info # Service health stackai deploy status # Performance metrics (via logs) stackai deploy logs --follow | grep -E "(ERROR|WARN|performance)" ``` ### Backup Strategy **Automated backups:** ```bash # Create backup (automatic before updates/version changes) stackai system backup # Create named backup stackai system backup --name daily-$(date +%Y%m%d) # List available backups stackai system backup --list ``` **Backup features:** * Creates compressed `.tar.gz` archive with checksum * Includes PostgreSQL and MongoDB databases * Includes all configuration files and version information * Includes encrypted secrets vault * Automatic backup before updates and version changes * Backup manifest with metadata **Backup location**: `~/.config/stackai/backups/` **Note**: Backups include databases and configuration. Data volumes (Weaviate, MinIO) are large (100GB+) and not included by default. *** ## 🆘 Support & Resources ### Getting Help **Priority Support Channels:** * 📧 **Enterprise Support**: (24/7 for Fortune 500) * 📞 **Account Manager**: Direct line for urgent issues * 💬 **Documentation**: ### Support Bundle Workflow ```bash # 1. Generate bundle stackai diagnose support # 2. Describe the issue echo "Issue: Services failing to start after update" > issue.txt # 3. Send both files to support # support-bundle-*.tar.gz + issue.txt ``` ## 📈 Version Management ### Current Versions | Component | Version | Release Date | | ------------- | ------- | ------------ | | **CLI** | 0.12.0 | Dec 2025 | | **StackEnd** | v1.0.12 | Dec 2025 | | **StackWeb** | v1.0.12 | Dec 2025 | | **StackRepl** | v1.0.0 | Oct 2025 | ### Update Process ```bash # Update to latest version (automatic backup created) stackai system update # Update to specific version stackai system update --version 1.0.9 # Air-gapped environments (skip checksum verification) stackai system update --skip-checksum # List available releases with subservice mappings stackai system releases list # Set target release for future update stackai system releases set 1.0.9 # Show current release and subservice versions stackai system releases ``` **Security Note**: Checksum verification is mandatory by default. Updates will fail if checksums cannot be verified, protecting against supply chain attacks. Only use `--skip-checksum` in air-gapped environments where checksum files are unavailable. **Version management:** * Each CLI release is bundled with specific subservice versions (stackend, stackweb, stackrepl) * Version mappings stored in `versions.json` * Current installed version tracked in `.release` file * Updates are version-aware and pin subservice versions automatically *** ## ⚖️ Compliance & Security ### Compliance Standards * ✅ **HIPAA** - Healthcare data protection * ✅ **ISO 27001** - Information security management * ✅ **SOC 2 Type II** - Security, availability, confidentiality * ✅ **GDPR** - Data protection and privacy ### Security Features * 🔐 **License-key encryption** (AES-256-GCM) * 🚫 **Air-gapped compatible** (zero external calls) * 🔒 **Zero plaintext secrets** in configuration * 👥 **Customer isolation** (license-based boundaries) * 📊 **Audit trails** (all operations logged) * 🔄 **Secure updates** (signed releases) ### Penetration Testing **Recommended security assessments:** * Container image scanning * Network segmentation verification * Secret management validation * Access control auditing --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.stackai.com/stackai-auto-cli/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.