# Administration

# Background Job Processing

<div style="text-align: justify;">

VeloxFactory processes background work — thumbnail generation, nightly purge routines, print task cleanup — through a Redis-backed queue managed by <a href="https://laravel.com/docs/horizon" target="_blank">Laravel Horizon</a>. Horizon runs as a single supervised process and manages its own worker pool internally. It replaces the simple `queue:work` approach with dynamic scaling, real-time monitoring, and a built-in dashboard.

---

<h3 style="color: #203671; margin-top: 2.2em;">Prerequisites</h3>

Horizon has two hard requirements beyond the base VeloxFactory stack: a running Redis instance and the `php-redis` PHP extension. Neither is optional — Horizon will refuse to start without both.

<h4 style="color: #203671; margin-top: 1.4em;">Redis</h4>

Redis can be installed natively or run as a Docker container. Both work equally well; the choice depends on your infrastructure preferences.

**Native installation:**

```bash
apt install redis-server
systemctl enable redis-server
systemctl start redis-server
```

**Docker container:**

```bash
docker run -d \
  --name redis \
  --restart unless-stopped \
  -p 127.0.0.1:6379:6379 \
  redis:alpine
```

The container binds to `127.0.0.1` only — Redis is not exposed to the network, which is the correct default for a single-server deployment.

Whichever method you choose, verify connectivity before proceeding:

```bash
redis-cli ping
# Expected: PONG
```

<h4 style="color: #203671; margin-top: 1.4em;">PHP Redis Extension</h4>

VeloxFactory is configured to use `phpredis` as the Redis client. The extension must be installed and active for PHP:

```bash
apt install php-redis
systemctl restart php8.2-fpm   # adjust version to match your PHP installation
```

Confirm the extension is loaded:

```bash
php -m | grep redis
# Expected: redis
```

<h4 style="color: #203671; margin-top: 1.4em;">.env Configuration</h4>

With Redis running and the extension installed, update your `.env` to activate Redis as the queue driver:

```dotenv
QUEUE_CONNECTION=redis

REDIS_CLIENT=phpredis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
```

---

<h3 style="color: #203671; margin-top: 2.2em;">Web Server</h3>

VeloxFactory requires a web server that routes all requests through Laravel's `public/index.php` entry point. Both Apache2 and nginx are supported. The Horizon dashboard at `/horizon` and the Reverb WebSocket endpoint require no special routing rules — they are handled by Laravel and PHP-FPM like any other request, with one exception: Reverb needs a WebSocket proxy pass.

<h4 style="color: #203671; margin-top: 1.4em;">Apache2</h4>

Enable `mod_rewrite` before configuring the vhost:

```bash
a2enmod rewrite proxy proxy_http proxy_wstunnel
systemctl restart apache2
```

Virtual host configuration:

```apache
<VirtualHost *:80>
    ServerName veloxfactory.example.com
    DocumentRoot /var/www/veloxfactory/public

    <Directory /var/www/veloxfactory/public>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    # Reverb WebSocket proxy
    ProxyPreserveHost On
    ProxyPass /app ws://127.0.0.1:8080/app
    ProxyPassReverse /app ws://127.0.0.1:8080/app

    ErrorLog ${APACHE_LOG_DIR}/veloxfactory-error.log
    CustomLog ${APACHE_LOG_DIR}/veloxfactory-access.log combined
</VirtualHost>
```

Laravel's bundled `.htaccess` in `public/` handles the rewrite rules — no additional configuration needed for URL routing.

<h4 style="color: #203671; margin-top: 1.4em;">nginx</h4>

```nginx
server {
    listen 80;
    server_name veloxfactory.example.com;
    root /var/www/veloxfactory/public;

    index index.php;

    # Laravel URL routing
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    # PHP-FPM
    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 120;
    }

    # Reverb WebSocket proxy
    location /app {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

    # Block dotfile access
    location ~ /\.(?!well-known).* {
        deny all;
    }
}
```

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong>Adjust the PHP-FPM socket path to match your PHP version.</strong> On systems with multiple PHP versions installed, the socket is typically at <code>/var/run/php/php8.2-fpm.sock</code>. Verify with <code>ls /var/run/php/</code>.
</div>

---

<h3 style="color: #203671; margin-top: 2.2em;">Queues</h3>

VeloxFactory uses two queues with distinct priorities:

<table style="width: 100%; border-collapse: collapse;">
  <thead>
    <tr style="border-top: 1px solid #e6e8ef; border-bottom: 1px solid #e6e8ef;">
      <th style="text-align: left; padding: 6px 10px; white-space: nowrap;">Queue</th>
      <th style="text-align: left; padding: 6px 10px;">Jobs</th>
      <th style="text-align: left; padding: 6px 10px;">Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>high</code></td>
      <td style="padding: 6px 10px;">Thumbnail generation</td>
      <td style="padding: 6px 10px;">Dispatched on-demand when a report is rendered. Processed with highest priority — workers on this queue are never blocked by maintenance routines.</td>
    </tr>
    <tr>
      <td style="padding: 6px 10px; white-space: nowrap;"><code>default</code></td>
      <td style="padding: 6px 10px;">Nightly purge jobs</td>
      <td style="padding: 6px 10px;">Scheduled automatically at midnight. Cleans up expired Report History Records, orphaned files, and completed Print Tasks according to the configured retention policies.</td>
    </tr>
  </tbody>
</table>

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong>The two queues run in separate worker pools.</strong> A long-running purge job on the <code>default</code> queue cannot delay thumbnail generation on the <code>high</code> queue — they never compete for the same worker.
</div>

---

<h3 style="color: #203671; margin-top: 2.2em;">Horizon Supervisors</h3>

Horizon manages workers through internal supervisors — process groups, each responsible for one queue. The system-level Supervisor (Supervisord) only ever manages the single Horizon master process; Horizon itself handles everything below that.

<table style="width: 100%; border-collapse: collapse;">
  <thead>
    <tr style="border-top: 1px solid #e6e8ef; border-bottom: 1px solid #e6e8ef;">
      <th style="text-align: left; padding: 6px 10px; white-space: nowrap;">Supervisor</th>
      <th style="text-align: left; padding: 6px 10px;">Queue</th>
      <th style="text-align: left; padding: 6px 10px;">Balancing</th>
      <th style="text-align: left; padding: 6px 10px;">Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>supervisor-rendering</code></td>
      <td style="padding: 6px 10px;"><code>high</code></td>
      <td style="padding: 6px 10px;">Auto</td>
      <td style="padding: 6px 10px;">Scales worker processes dynamically based on queue depth — up to 10 in production. Job timeout: 120 seconds.</td>
    </tr>
    <tr>
      <td style="padding: 6px 10px; white-space: nowrap;"><code>supervisor-default</code></td>
      <td style="padding: 6px 10px;"><code>default</code></td>
      <td style="padding: 6px 10px;">Simple</td>
      <td style="padding: 6px 10px;">Fixed worker count. Runs with lower CPU priority (<code>nice 10</code>) — purge jobs are maintenance work and should not compete with rendering for system resources. Job timeout: 300 seconds.</td>
    </tr>
  </tbody>
</table>

---

<h3 style="color: #203671; margin-top: 2.2em;">System Supervisor Configuration</h3>

Supervisord keeps the Horizon master process alive and restarts it automatically on failure. The following blocks replace the previous `queue:work`-based configuration entirely.

```ini
; Horizon — manages all VeloxFactory queue workers internally
[program:veloxfactory-horizon]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan horizon
user=www-data
process_name=%(program_name)s
numprocs=1
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=TERM
startsecs=3
stopwaitsecs=600
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-horizon.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"

; Reverb — WebSocket server for real-time events
[program:veloxfactory-reverb]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan reverb:start
user=www-data
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=INT
startsecs=3
stopwaitsecs=60
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-reverb.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"

; Scheduler — runs Laravel's task schedule every minute
[program:veloxfactory-schedule]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan schedule:work
user=www-data
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=INT
startsecs=3
stopwaitsecs=60
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-schedule.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"
```

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong><code>stopsignal=TERM</code> is required for Horizon.</strong> SIGTERM triggers a graceful shutdown — Horizon finishes any in-flight jobs before stopping its workers. Using SIGKILL or SIGINT instead will interrupt running jobs mid-execution and may leave Report History Records in an incomplete state.
</div>

After updating the configuration:

```bash
supervisorctl reread
supervisorctl update
supervisorctl status
```

---

<h3 style="color: #203671; margin-top: 2.2em;">The Dashboard</h3>

The Horizon dashboard is available at `/horizon`. It is restricted to users with the `global:admin` permission — the same permission required to manage users and system-wide settings.

The dashboard provides a real-time view of the entire queue system: pending and completed jobs, throughput metrics, worker counts per supervisor, and a full log of failed jobs with their stack traces. Failed jobs can be retried directly from the dashboard without any CLI access.

---

<h3 style="color: #203671; margin-top: 2.2em;">Graceful Shutdown & Deployments</h3>

When Horizon is stopped — for deployments, configuration changes, or server maintenance — it finishes any jobs currently in flight before exiting. The Supervisord configuration allows up to 600 seconds for this drain, which comfortably covers the longest expected job timeouts.

```bash
# Stop Horizon gracefully (in-flight jobs will finish first)
supervisorctl stop veloxfactory-horizon

# Restart after a deployment
supervisorctl restart veloxfactory-horizon
```

Never force-kill the Horizon process during a deployment. Always use `supervisorctl stop` or `supervisorctl restart` — both send SIGTERM and wait for the drain period.

</div>