DevCLI

Advanced Features

Deep dive into devcli’s powerful features for production-ready process management.

Table of Contents

Health Checks

Automatically monitor app health and trigger restarts when unhealthy.

Types of Health Checks

HTTP Health Check

Sends HTTP requests to check if service is responding.

health_check:
  http:
    url: http://localhost:3000/health
    expected_status: 200
    timeout_secs: 5

Features:

  • Configurable URL and expected status
  • Timeout protection
  • Follows redirects (up to 5)
  • Supports HTTPS

Best For: Web services, APIs, frontends

TCP Health Check

Checks if TCP port is accepting connections.

health_check:
  tcp:
    host: localhost
    port: 5432
    timeout_secs: 10

Best For: Databases, message queues, any TCP service

Command Health Check

Runs custom command to determine health.

health_check:
  command:
    cmd: curl -f http://localhost:3000/health
    timeout_secs: 5

Exit code 0 = healthy, non-zero = unhealthy

Best For: Complex health logic, custom protocols

Health Check Behavior

Monitor Cycle:

  1. Monitor runs health checks every 3 seconds
  2. Failure increments failure counter
  3. Success resets failure counter to 0
  4. After 3 consecutive failures, restart is triggered

Example Timeline:

00:00 - Check #1: ✓ healthy (failures: 0)
00:03 - Check #2: ✗ failed  (failures: 1)
00:06 - Check #3: ✗ failed  (failures: 2)
00:09 - Check #4: ✗ failed  (failures: 3) → RESTART TRIGGERED

Health Check Best Practices

1. Implement Proper Health Endpoints

// Express.js example
app.get('/health', (req, res) => {
  // Check database connection
  if (!db.isConnected()) {
    return res.status(503).json({ status: 'unhealthy', reason: 'db' });
  }

  // Check critical dependencies
  if (!redis.ping()) {
    return res.status(503).json({ status: 'unhealthy', reason: 'redis' });
  }

  res.json({ status: 'healthy' });
});

2. Use Appropriate Timeouts

# Fast service
health_check:
  http:
    timeout_secs: 2

# Slow service (database warmup)
health_check:
  tcp:
    timeout_secs: 30

3. Combine with Restart Policies

health_check:
  http:
    url: http://localhost:3000/health
restart_policy:
  max_restarts: 5
  restart_window_secs: 300

Restart Policies

Automatically restart crashed or unhealthy processes.

Basic Configuration

restart_policy:
  max_restarts: 5           # Maximum restart attempts
  restart_window_secs: 300  # Within this time window (5 minutes)
  backoff_secs: 5           # Initial backoff delay

Restart Triggers

1. Process Crashes (non-zero exit code):

Process exited with code 1 → automatic restart

2. Health Check Failures (3 consecutive):

Health check failed 3 times → automatic restart

Exponential Backoff

Prevents restart loops by increasing delay:

Attempt 1: Wait 5 seconds
Attempt 2: Wait 10 seconds
Attempt 3: Wait 20 seconds
Attempt 4: Wait 40 seconds
Attempt 5: Wait 80 seconds

Formula: delay = backoff_secs * 2^(attempt-1)

Max backoff: Capped at 300 seconds (5 minutes)

Restart Window

Restart counter resets after the window expires:

restart_policy:
  max_restarts: 3
  restart_window_secs: 60  # 1 minute window

Scenario:

  • 00:00 - Restart #1
  • 00:20 - Restart #2
  • 00:40 - Restart #3
  • 01:10 - Window expired, counter resets to 0
  • 01:15 - Restart #1 (new window starts)

Strategies by Service Type

Web Service (needs to be available):

restart_policy:
  max_restarts: 10
  restart_window_secs: 600  # 10 minutes
  backoff_secs: 3

Background Worker (can tolerate downtime):

restart_policy:
  max_restarts: 5
  restart_window_secs: 300
  backoff_secs: 10

Database (critical, but restart is expensive):

restart_policy:
  max_restarts: 3
  restart_window_secs: 600
  backoff_secs: 30

Development Service (aggressive restart):

restart_policy:
  max_restarts: 20
  restart_window_secs: 600
  backoff_secs: 1

Metrics & Monitoring

Real-time insights into process behavior and performance.

Viewing Metrics

CLI (Formatted):

devcli metrics

HTTP API (Raw JSON):

curl http://localhost:9090/metrics | jq

Metrics Categories

Process Metrics

{
  "processes": {
    "total_processes": 5,
    "running_processes": 3,
    "stopped_processes": 2,
    "total_restarts": 12,
    "restarts_last_hour": 3,
    "health_check_success_rate": 0.98,
    "exit_code_distribution": {
      "0": 8,
      "1": 3,
      "137": 1
    },
    "apps": [
      {
        "project": "awesome-project",
        "name": "api",
        "status": "running",
        "uptime_seconds": 3600,
        "restart_count": 2,
        "health_status": "healthy"
      }
    ]
  }
}

System Metrics

{
  "system": {
    "monitor_uptime_seconds": 7200,
    "monitor_loop_iterations": 2400,
    "devcli_version": "0.1.0",
    "total_apps_configured": 12
  }
}

Performance Metrics

{
  "performance": {
    "avg_startup_time_ms": 450.5,
    "avg_health_check_duration_ms": 42.3,
    "avg_restart_duration_ms": 1023.7,
    "recent_operations": [
      {
        "operation": "start",
        "app": "api",
        "duration_ms": 523,
        "timestamp": "2024-02-08T10:30:00Z",
        "success": true
      }
    ]
  }
}

Metrics Use Cases

1. Monitoring Dashboards:

# Feed to Prometheus/Grafana
curl -s http://localhost:9090/metrics | \
  jq '.processes.total_restarts' | \
  promtool push

2. Alerting:

# Alert if restart rate too high
RESTARTS=$(curl -s http://localhost:9090/metrics | jq '.processes.restarts_last_hour')
if [ "$RESTARTS" -gt 10 ]; then
  notify-send "High restart rate: $RESTARTS/hour"
fi

3. Performance Analysis:

# Track startup times
curl -s http://localhost:9090/metrics | \
  jq '.performance.avg_startup_time_ms'

Structured Logging

Production-grade logging with JSON format and structured fields.

Log Files

Location: ~/.devcli/logs/

Files:

  • devcli.YYYY-MM-DD.json - Structured JSON logs (daily rotation)
  • <project>_<app>_<date>.log - App-specific logs

JSON Log Format

{
  "timestamp": "2024-02-08T10:30:15.123Z",
  "level": "INFO",
  "fields": {
    "app_name": "api",
    "project": "awesome-project",
    "environment": "docker",
    "duration_ms": 523
  },
  "target": "devcli_core::commands::start",
  "span": {
    "name": "start_command",
    "app_count": 1
  },
  "message": "Starting application"
}

Log Levels

Set via RUST_LOG environment variable:

# Info level (default)
RUST_LOG=info devcli start api

# Debug level
RUST_LOG=debug devcli start api

# Trace level (very verbose)
RUST_LOG=trace devcli start api

# Module-specific
RUST_LOG=devcli_core::commands::start=debug devcli start api

Querying Logs with jq

All errors:

cat ~/.devcli/logs/devcli.*.json | \
  jq 'select(.level == "ERROR")'

Specific app:

cat ~/.devcli/logs/devcli.*.json | \
  jq 'select(.fields.app_name == "api")'

Slow operations (> 1 second):

cat ~/.devcli/logs/devcli.*.json | \
  jq 'select(.fields.duration_ms > 1000)'

Dependency Management

Automatic dependency resolution and ordered startup.

Defining Dependencies

frontend:
  dependencies:
    - api

api:
  dependencies:
    - database
    - redis

Dependency Resolution

Topological Sort ensures correct order:

devcli start frontend

Start Order:

  1. database
  2. redis
  3. api
  4. frontend

Transitive Dependencies

Dependencies of dependencies are automatically included:

frontend → api → database
frontend → api → redis

Result: Starting frontend also starts api, database, and redis.

Circular Dependency Detection

# INVALID: circular dependency
api:
  dependencies: [worker]

worker:
  dependencies: [api]

devcli start api
# Error: Circular dependency detected: api → worker → api

Skipping Dependencies

# Start without dependencies
devcli start api --skip-deps

Use Cases:

  • Dependencies already running
  • Testing in isolation
  • Manual dependency management

Environment Management

Manage environment-specific configuration across dev/qa/prod.

Stage Concept

A stage represents a deployment environment:

  • dev - Local development
  • qa - QA environment
  • staging - Pre-production
  • prod - Production

Env File Mapping

env_files:
  dev:
    local: .env.dev
    docker: .env.docker.dev
  qa:
    docker: .env.qa
    k8s: k8s/qa/env.yaml
  prod:
    k8s: k8s/prod/env.yaml

Usage

# Use dev stage (local context)
devcli start api --stage dev

# Use qa stage (docker context)
devcli start api --stage qa --env docker

# Use prod stage (k8s context)
devcli start api --stage prod --env k8s

Docker Integration

Env files automatically injected:

commands:
  docker:
    start: docker compose up api

env_files:
  dev:
    docker: .env.dev

Becomes:

docker compose --env-file .env.dev up api

Best Practices

1. Separate env files by stage:

.env.dev       # Local development
.env.qa        # QA
.env.staging   # Staging
.env.prod      # Production (never commit!)

2. Use .gitignore:

.env.prod
.env.*.local
*.secret

3. Document required variables:

# .env.example
DATABASE_URL=
API_KEY=
SECRET_KEY=

TUI Interface

Beautiful terminal UI for viewing logs in real-time.

Launching TUI

devcli ui

Features

  • Real-time log streaming
  • Syntax highlighting for code and JSON
  • Multi-pane view for multiple apps
  • ANSI color support for colored logs
  • Automatic scrolling to latest logs

Keyboard Shortcuts

  • ↑/↓ or j/k - Scroll up/down
  • Page Up/Page Down - Page scroll
  • Home/End or g/G - Jump to top/bottom
  • Tab - Switch between apps
  • Ctrl+C or q - Quit
  • h or ? - Show help

Process Lifecycle

Understanding how devcli manages processes.

State Tracking

File: ~/.devcli/state.json

Tracks all managed processes:

{
  "processes": [
    {
      "app_name": "api",
      "pid": 12345,
      "project": "awesome-project",
      "environment": "local",
      "start_time": "2024-02-08T10:00:00Z",
      "restart_count": 2,
      "last_exit_code": null
    }
  ]
}

Spawner Architecture

devcli start api
      │
      ├─ Spawn internal-spawner (detached)
      │      │
      │      └─ Spawn actual app process
      │             │
      │             └─ Capture stdout/stderr
      │             └─ Write to log file
      │
      └─ Return immediately (non-blocking)

Internal Spawner:

  • Runs as separate process
  • Captures app output
  • Writes to log files
  • Reports exit codes
  • Survives parent process exit

Monitor Lifecycle

Auto-start:

devcli start api
# → Starts monitor daemon automatically if not running

Auto-exit:

Monitor: No processes remaining
Monitor: Exiting

Manual control:

# Check if running
ps aux | grep "devcli monitor --daemon"

# Stop monitor (stops all apps)
devcli stop --all

Cleanup

Dead process cleanup:

# Manual cleanup
devcli monitor

# Output:
# Cleaned up 2 dead process(es):
#   - old-api
#   - crashed-worker

Automatic cleanup:

  • Monitor runs cleanup on each cycle
  • Removes processes with no PID
  • Updates state.json

See Also