# Messenger Workers — Notifications & Webhooks

This document describes how to set up and operate the Symfony Messenger workers
that process deferred notifications (email, WhatsApp, push) and outbound customer
webhooks asynchronously.

Supervisor templates: [`deploy/supervisor/`](../deploy/supervisor/).

---

## Why workers are needed

All notification events (receipt status, guide status, alerts, etc.) are dispatched
through the `notifications` Messenger transport (a `messenger_messages` table in MySQL).
Without a running worker, messages accumulate in the table and are never processed.

Flow:

```
HTTP request → defer() → kernel.terminate → MessageBus
→ messenger_messages (MySQL) → worker: messenger:consume notifications
→ NotificationDispatchMessageHandler → notify() → email / WhatsApp / push
```

---

## Why workers are needed — Webhooks outbound

Customer webhook events (`warehouse.*`, `shipment.*`, `package.*`, etc.) are enqueued as
`WebhookPrepareMessage` and `WebhookDeliveryMessage` on the **`webhooks`** transport.
Without a running worker, messages accumulate in `messenger_messages` and are never
delivered to the client's HTTPS endpoint.

Flow (detail: `documentation/specs/backend-webhooks-flow-diagram.md`):

```
HTTP request → WebhookDispatcher → MessageBus
→ messenger_messages (queue_name=webhooks)
→ worker: messenger:consume webhooks
→ WebhookPrepareMessageHandler → WebhookDeliveryMessageHandler → POST + HMAC
```

**Production requirement:** `WEBHOOK_SECRET_ENCRYPTION_KEY` must be set in `.env.local`
so the delivery handler can decrypt the webhook signing secret and attach
`X-Webhook-Signature`. Without it, delivery is skipped (logged as error).

---

## Requirements

Before setting up workers, confirm:

1. **Code deployed** with the `notifications` and `webhooks` transports in `config/packages/messenger.yaml`.
2. **PHP >= 8.2** (Symfony 7). The project requires PHP 8.2+; PHP 7.x will fail with parse errors.
3. **`MESSENGER_TRANSPORT_DSN`** set in `.env` / `.env.local` (typically `doctrine://default`).
4. **`messenger_messages` table** created via `messenger:setup-transports` (once per instance/database).
5. **Webhooks only:** `WEBHOOK_SECRET_ENCRYPTION_KEY` in `.env.local` (32-byte key, base64). Required for HMAC signing on delivery. See `documentation/specs/backend-webhooks-b4-hmac.md`.

Verify transports after deploy:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console debug:messenger
```

You should see `async`, `notifications`, **`webhooks`**, and `failed` in the receiver list.

---

## Endpoints that use Messenger

These HTTP endpoints call `DeferredNotificationDispatcher::defer()` during the request.
At `kernel.terminate`, each deferred job is dispatched as a `NotificationDispatchMessage`
to the `notifications` transport. A running worker is **required** for delivery.

### Receipt (`/api/receipt`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/receipt/create-status` | `receipt_status` | Status change (email / push / WhatsApp) |
| `POST` | `/api/receipt/delivered` | `receipt_status` | Receipt delivered (when not external) |
| `POST` | `/api/receipt/from-label-photo` | `receipt_batch` | OCR receipt creation |
| `POST` | `/api/receipt/send-notifications` | `receipt_batch` | Manual batch notification by receipt IDs |
| `POST` | `/api/receipt/send-emails` | `receipt_custom_email` | Custom email with user-supplied body/subject |
| `POST` | `/api/receipt/create` | `alert_received` | When tracking matches pending alerts |

### Guide (`/api/guide`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/guide/create-status` | `guide_status` / `guide_delivered` | Status change; `guide_delivered` when step is delivered |
| `POST` | `/api/guide/delivered` | `guide_delivered` | Guide delivered (signature + close images in email) |
| `POST` | `/api/guide/send-notifications` | `guide_batch` | Manual batch notification by guide IDs |
| `POST` | `/api/guide/send-emails` | `guide_custom_email` | Custom email with user-supplied body/subject |

### Warehouse Receipt (`/api/warehouse-receipt`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/warehouse-receipt/create-status` | `warehouse_status` | Status change (not on "delivered" step) |
| `POST` | `/api/warehouse-receipt/from-label-photo` | `warehouse_batch` | OCR whrec creation |
| `POST` | `/api/warehouse-receipt/send-notifications` | `warehouse_batch` | Manual batch notification by warehouse receipt IDs |
| `POST` | `/api/warehouse-receipt/send-emails` | `warehouse_custom_email` | Custom email with user-supplied body/subject |

### Alert (`/api/alert`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/alert/create` | `system_users` | System users (push / email) |

### Pickup (`/api/pickup`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/pickup/create` | `system_users` | System users push *(pickup email is sent synchronously via a separate service)* |

### Reempack (`/api/reempack`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/reempack/create` | `system_users` | System users push |

### Prepack (`/api/prepack`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/prepack/create` | `alert_received` | When tracking matches pending alerts |

### Twilio (`/api/twilio`)

| Method | Route | Messenger event | Notification |
|---|---|---|---|
| `POST` | `/api/twilio/send` | `twilio_whatsapp` | Manual WhatsApp dispatch by action + document ID |

**Total: 17 HTTP endpoints** use Messenger for notifications.

### Summary by Messenger event

| Event | Triggered by |
|---|---|
| `receipt_status` | `POST /api/receipt/create-status`, `POST /api/receipt/delivered` |
| `receipt_batch` | `POST /api/receipt/from-label-photo`, `POST /api/receipt/send-notifications` |
| `receipt_custom_email` | `POST /api/receipt/send-emails` |
| `guide_status` | `POST /api/guide/create-status` (non-delivered step) |
| `guide_delivered` | `POST /api/guide/delivered`, `POST /api/guide/create-status` (delivered step) |
| `guide_batch` | `POST /api/guide/send-notifications` |
| `guide_custom_email` | `POST /api/guide/send-emails` |
| `warehouse_status` | `POST /api/warehouse-receipt/create-status` |
| `warehouse_batch` | `POST /api/warehouse-receipt/from-label-photo`, `POST /api/warehouse-receipt/send-notifications` |
| `warehouse_custom_email` | `POST /api/warehouse-receipt/send-emails` |
| `alert_received` | `POST /api/receipt/create`, `POST /api/prepack/create` |
| `system_users` | `POST /api/alert/create`, `POST /api/pickup/create`, `POST /api/reempack/create` |
| `twilio_whatsapp` | `POST /api/twilio/send` |

### Related endpoints that do NOT use Messenger

These call `NotificationService::notify()` **synchronously** in the same HTTP request.
They do **not** enqueue to `messenger_messages` and do **not** require a worker:

| Method | Route | Event |
|---|---|---|
| `POST` | `/api/warehouse-receipt/delivered` | Direct email/push (no queue) |
| `POST` | `/api/reempack/create-strict` | Direct reempack email (no `defer`) |

---

## PHP version on cPanel / multi-PHP servers

On cPanel servers, **multiple PHP versions coexist**. The default CLI binary is often
PHP 7.x and must **not** be used for this project.

| Binary | Typical version | Use for TrackingPremium? |
|---|---|---|
| `/usr/local/bin/php` | PHP 7.3 | **No** |
| `/opt/cpanel/ea-php83/root/usr/bin/php` | PHP 8.3 | **Yes** |

Check versions:

```bash
/usr/local/bin/php -v                                    # often 7.3 — do not use
/opt/cpanel/ea-php83/root/usr/bin/php -v                   # should be 8.2+
```

**Always use the full path to PHP 8.3** in Supervisor configs and CLI commands.
Do not change the global `/usr/local/bin/php` symlink — other legacy apps may depend on it.

Find the cPanel account user (PHP-FPM pool owner):

```bash
ps aux | grep php-fpm | grep -v grep | head -3
id <cpanel_user>
```

---

## First-time setup (per instance)

Run **once per Symfony instance** (demo, prod, etc.), from the project directory,
using PHP 8.3:

```bash
cd /home/<cpanel_user>/demomultitrackapi   # or multitrackapi for prod

sudo -u <cpanel_user> /opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:setup-transports
```

This creates the `messenger_messages` table in **that instance's database**.
If the table already exists it is a no-op.

Test the worker manually before Supervisor:

```bash
sudo -u <cpanel_user> bash -lc 'cd /home/<cpanel_user>/demomultitrackapi && /opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:consume notifications -vv --limit=1'
```

If it starts without errors (waiting for messages or processing one), the instance is ready.

---

## Starting workers with Supervisor

### Install Supervisor

**Debian / Ubuntu:**

```bash
sudo apt update && sudo apt install -y supervisor
sudo systemctl enable supervisor && sudo systemctl start supervisor
```

**AlmaLinux 8 / RHEL / CentOS (cPanel):**

```bash
sudo dnf install -y epel-release
sudo dnf install -y supervisor
sudo systemctl enable supervisord
sudo systemctl start supervisord
```

Config file locations differ by OS:

| OS | Program config directory |
|---|---|
| Debian / Ubuntu | `/etc/supervisor/conf.d/` |
| AlmaLinux / RHEL | `/etc/supervisord.d/` |

### Demo instance (example: demomultitrackapi)

Create `/etc/supervisord.d/messenger-notifications-dev.ini`:

```ini
[program:messenger-notifications-dev]
command=/opt/cpanel/ea-php83/root/usr/bin/php /home/aab42b5/demomultitrackapi/bin/console messenger:consume notifications --time-limit=3600 --memory-limit=256M --limit=1000 -vv
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
stopsignal=TERM
stopwaitsecs=30
stdout_logfile=/var/log/supervisor/messenger-notifications-dev.log
stderr_logfile=/var/log/supervisor/messenger-notifications-dev-err.log
user=aab42b5
directory=/home/aab42b5/demomultitrackapi
```

Replace `aab42b5` and paths with the actual cPanel user and project directory.

### Prod instance (example: multitrackapi)

When prod is deployed, create a **separate** program (different name, path, logs):

`/etc/supervisord.d/messenger-notifications-prod.ini`:

```ini
[program:messenger-notifications-prod]
command=/opt/cpanel/ea-php83/root/usr/bin/php /home/aab42b5/multitrackapi/bin/console messenger:consume notifications --time-limit=3600 --memory-limit=256M --limit=1000 -vv
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
stopsignal=TERM
stopwaitsecs=30
stdout_logfile=/var/log/supervisor/messenger-notifications-prod.log
stderr_logfile=/var/log/supervisor/messenger-notifications-prod-err.log
user=aab42b5
directory=/home/aab42b5/multitrackapi
```

Each instance needs its own workers because each has its own codebase and (usually) its own database.

### Webhooks workers (demo instance)

Create `/etc/supervisord.d/messenger-webhooks-dev.ini` (or copy from
[`deploy/supervisor/messenger-webhooks-dev.ini.example`](../deploy/supervisor/messenger-webhooks-dev.ini.example)):

```ini
[program:messenger-webhooks-dev]
command=/opt/cpanel/ea-php83/root/usr/bin/php /home/aab42b5/demomultitrackapi/bin/console messenger:consume webhooks --time-limit=3600 --memory-limit=256M --limit=1000 -vv
process_name=%(program_name)s_%(process_num)02d
numprocs=2
autostart=true
autorestart=true
stopsignal=TERM
stopwaitsecs=30
stdout_logfile=/var/log/supervisor/messenger-webhooks-dev.log
stderr_logfile=/var/log/supervisor/messenger-webhooks-dev-err.log
user=aab42b5
directory=/home/aab42b5/demomultitrackapi
```

Use `numprocs=2` for webhooks (lower volume than notifications). Increase if
`messenger:stats` shows sustained backlog on the `webhooks` queue.

### Webhooks workers (prod instance)

`/etc/supervisord.d/messenger-webhooks-prod.ini` (template:
[`deploy/supervisor/messenger-webhooks-prod.ini.example`](../deploy/supervisor/messenger-webhooks-prod.ini.example)):

```ini
[program:messenger-webhooks-prod]
command=/opt/cpanel/ea-php83/root/usr/bin/php /home/aab42b5/multitrackapi/bin/console messenger:consume webhooks --time-limit=3600 --memory-limit=256M --limit=1000 -vv
process_name=%(program_name)s_%(process_num)02d
numprocs=2
autostart=true
autorestart=true
stopsignal=TERM
stopwaitsecs=30
stdout_logfile=/var/log/supervisor/messenger-webhooks-prod.log
stderr_logfile=/var/log/supervisor/messenger-webhooks-prod-err.log
user=aab42b5
directory=/home/aab42b5/multitrackapi
```

### Log directory

```bash
sudo mkdir -p /var/log/supervisor
sudo chown <cpanel_user>:<cpanel_user> /var/log/supervisor
```

### Activate

```bash
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start "messenger-notifications-dev:*"
sudo supervisorctl start "messenger-webhooks-dev:*"
sudo supervisorctl status
```

Expected output — all processes **RUNNING**:

```
messenger-notifications-dev:messenger-notifications-dev_00   RUNNING
messenger-notifications-dev:messenger-notifications-dev_01   RUNNING
...
messenger-webhooks-dev:messenger-webhooks-dev_00             RUNNING
messenger-webhooks-dev:messenger-webhooks-dev_01             RUNNING
```

### Options explained

| Option | Value | Reason |
|---|---|---|
| `--time-limit=3600` | 1 hour | Graceful restart to prevent memory drift |
| `--memory-limit=256M` | 256 MB | Kill worker if it leaks beyond this |
| `--limit=1000` | 1000 messages | Recycle after 1000 jobs |
| `numprocs=4` | 4 workers | Throughput ~100–200 notifications/s; increase if queue grows |
| `numprocs=2` (webhooks) | 2 workers | Typical outbound webhook load; increase if `webhooks` queue backs up |
| `directory=` | project root | Ensures `.env` and relative paths resolve correctly |
| `user=` | cPanel account user | Must match PHP-FPM owner; not `apache`/`www-data` on cPanel |

---

## First-time setup — Webhooks (per instance)

Run once per Symfony instance after deploying code that includes the `webhooks` transport:

```bash
cd /home/<cpanel_user>/demomultitrackapi   # or multitrackapi for prod

# 1. Ensure messenger table / transports exist
sudo -u <cpanel_user> /opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:setup-transports

# 2. HMAC signing key (generate once per environment)
php -r "echo 'WEBHOOK_SECRET_ENCRYPTION_KEY=' . base64_encode(random_bytes(32)) . PHP_EOL;"
# Paste output into .env.local on the server

# 3. Verify transports include webhooks
/opt/cpanel/ea-php83/root/usr/bin/php bin/console debug:messenger

# 4. Install Supervisor program (see Webhooks workers above), then:
sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl start "messenger-webhooks-dev:*"   # or messenger-webhooks-prod:*

# 5. Smoke test
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:stats
```

Smoke test: enable a webhook in the admin panel, trigger an event (e.g. create a standalone
package), confirm the client endpoint receives the POST and the `webhooks` queue drains.

Test the worker manually before Supervisor:

```bash
sudo -u <cpanel_user> bash -lc 'cd /home/<cpanel_user>/demomultitrackapi && /opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:consume webhooks -vv --limit=1'
```

## After every deploy

Workers hold the code in memory. After a code deploy, signal them to restart gracefully.
The Bitbucket pipeline does this **automatically** via the post-deploy webhook (see below).

For manual restarts (from each instance directory, with PHP 8.3):

```bash
cd /home/<cpanel_user>/demomultitrackapi
/opt/cpanel/ea-php83/root/usr/bin/php bin/console cache:clear --no-warmup
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:stop-workers
```

Supervisor will auto-restart them with the new code (**both** `notifications` and
`webhooks` programs, if registered).

If the Messenger config changed (new transports, routing), also run:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:setup-transports
```

---

## Post-deploy hook (automatic — called by Bitbucket)

The Bitbucket pipeline calls a token-protected endpoint after each FTP deploy to run
`cache:clear --no-warmup` and `messenger:stop-workers` without requiring SSH access.

### How it works

```
Bitbucket (after FTP) → POST /webhooks/deploy/post-deploy
                         Authorization: Bearer DEPLOY_HOOK_TOKEN
                         → 202 Accepted (immediate)
                         → cache:clear + messenger:stop-workers (deferred, after response)
                         → Supervisor restarts all messenger:consume workers (notifications + webhooks)
```

The endpoint lives under `/webhooks/…` which already has `security: false` in
`config/packages/security.yaml`. No JWT is required.

### Required env vars

**On each server** — add to `.env.local` (do not commit the real value):

```bash
# Generate a unique token per environment:
openssl rand -hex 32

# Then set it in .env.local:
DEPLOY_HOOK_TOKEN=<the generated value>
```

**In Bitbucket** — configure per deployment environment
(Repository settings → Deployments → select the environment):

| Environment | Variable | Value |
|---|---|---|
| `staging` | `DEPLOY_HOOK_URL` | `https://demo-api.tudominio.com/webhooks/deploy/post-deploy` |
| `staging` | `DEPLOY_HOOK_TOKEN` | same value as in demo `.env.local` (mark as **secured**) |
| `stagingfh` | `DEPLOY_HOOK_URL` | URL of the FH demo instance |
| `stagingfh` | `DEPLOY_HOOK_TOKEN` | token for FH demo `.env.local` |
| `production` | `DEPLOY_HOOK_URL` | `https://api-prod.tudominio.com/webhooks/deploy/post-deploy` |
| `production` | `DEPLOY_HOOK_TOKEN` | same value as in prod `.env.local` (mark as **secured**) |

Use different tokens per environment. If the demo token leaks, prod stays safe.

### Test manually (from any terminal)

```bash
curl -i -X POST \
  -H "Authorization: Bearer <your-token>" \
  https://demo-api.tudominio.com/webhooks/deploy/post-deploy
```

Expected response:

```json
HTTP/1.1 202 Accepted
{
  "success": true,
  "scheduled": true,
  "actions": ["cache:clear", "messenger:stop-workers"],
  "scheduledAt": "2026-05-23T12:00:00+00:00"
}
```

### Troubleshooting

| HTTP code | Cause | Fix |
|---|---|---|
| `202` | OK | — |
| `401` | Token mismatch or missing `Authorization` header | Check `DEPLOY_HOOK_TOKEN` matches on both sides |
| `503` | `DEPLOY_HOOK_TOKEN` is empty on the server | Set the value in `.env.local` |
| `404` | Route not found / firewall misconfigured | Verify `/webhooks/deploy/post-deploy` is in the `webhooks` firewall pattern |

After hitting the endpoint, workers restart within ~30 s. Check:

```bash
sudo supervisorctl status
sudo tail -20 /var/log/supervisor/messenger-notifications-prod.log
sudo tail -20 /var/log/supervisor/messenger-webhooks-prod.log
```

### Manual fallback

If the webhook is unavailable, use `scripts/post-deploy-messenger.sh` directly on the server
or run the commands manually as shown in the "After every deploy" section above.

---

## Monitoring

Check queue depth:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:stats

# Or directly in MySQL:
SELECT queue_name, COUNT(*) FROM messenger_messages GROUP BY queue_name;
-- Example: notifications vs webhooks backlog
SELECT queue_name, COUNT(*) AS pending FROM messenger_messages
WHERE delivered_at IS NULL GROUP BY queue_name;
```

Worker logs:

```bash
sudo tail -f /var/log/supervisor/messenger-notifications-dev.log
sudo tail -f /var/log/supervisor/messenger-notifications-dev-err.log
sudo tail -f /var/log/supervisor/messenger-webhooks-dev.log
sudo tail -f /var/log/supervisor/messenger-webhooks-dev-err.log
```

Supervisor status:

```bash
sudo supervisorctl status
```

Raise an alert if `notifications` queue depth exceeds **5 000** — this indicates
workers are down or delivery is backlogged (e.g. SMTP or Twilio rate-limiting).

Raise an alert if `webhooks` queue depth exceeds **500** — outbound HTTP delivery
may be stalled (worker down, client endpoints failing, or ngrok/URL misconfiguration).

---

## Failed messages

Messages that fail all 3 retry attempts land in the `failed` queue.

```bash
# View failed messages
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:failed:show

# Retry a specific message
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:failed:retry <id>

# Retry all failed messages
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:failed:retry

# Remove all failed messages (use with caution)
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:failed:remove --all
```

---

## Scaling

| Scenario | Action |
|---|---|
| Queue grows steadily | Increase `numprocs` in Supervisor config |
| Email/Twilio rate limits | Reduce `numprocs` or add throttling at the transport level |
| Multi-server | Point all servers at the same MySQL DB; Messenger uses `FOR UPDATE SKIP LOCKED` to avoid double-processing |
| Migrate to Redis | Change `MESSENGER_TRANSPORT_DSN=redis://...` — no code changes needed |
| Demo + prod on same server | Separate Supervisor programs, one per instance; each with its own DB |

---

## Troubleshooting

### `BACKOFF` / `Exited too quickly`

Check stderr log:

```bash
sudo tail -30 /var/log/supervisor/messenger-notifications-dev-err.log
```

Common causes below.

### `PHP Parse error: unexpected '=' in autoload_runtime.php`

Supervisor is using **PHP 7.x** instead of PHP 8.3.

Fix — verify and update the `command=` line:

```bash
sudo grep command /etc/supervisord.d/messenger-notifications-dev.ini
```

Must use `/opt/cpanel/ea-php83/root/usr/bin/php`, **not** `/usr/local/bin/php`:

```bash
sudo sed -i 's|/usr/local/bin/php|/opt/cpanel/ea-php83/root/usr/bin/php|g' /etc/supervisord.d/messenger-notifications-dev.ini
sudo supervisorctl reread && sudo supervisorctl update && sudo supervisorctl restart "messenger-notifications-dev:*"
```

### `Invalid user name apache`

On cPanel, use the cPanel account user (e.g. `aab42b5`), not `apache` or `www-data`:

```bash
ps aux | grep php-fpm | grep -v grep | head -1   # first column is the user
```

### `The receiver "notifications" does not exist`

The deployed code does not include the `notifications` transport yet. Deploy the branch
with `config/packages/messenger.yaml` updates, then:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console cache:clear
/opt/cpanel/ea-php83/root/usr/bin/php bin/console debug:messenger
```

Do **not** consume `async` for notification messages — they route to `notifications`.

### `The receiver "webhooks" does not exist`

The deployed code does not include the `webhooks` transport yet. Deploy the branch
with `config/packages/messenger.yaml` updates, then:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console cache:clear
/opt/cpanel/ea-php83/root/usr/bin/php bin/console debug:messenger
```

Do **not** consume `notifications` for webhook messages — they route to `webhooks`.

### Webhooks queue grows but nothing is delivered

The `webhooks` worker is not running. Register `messenger-webhooks-*` in Supervisor
(see [Webhooks workers](#webhooks-workers-demo-instance)) and start it:

```bash
sudo supervisorctl start "messenger-webhooks-prod:*"
```

### Log: `Webhook delivery skipped: missing signing secret`

Set `WEBHOOK_SECRET_ENCRYPTION_KEY` in `.env.local`, or regenerate the webhook
credential in the admin panel if secrets were encrypted with a different key.

### Log: `Webhook delivery skipped: unable to decrypt signing secret`

`WEBHOOK_SECRET_ENCRYPTION_KEY` changed after secrets were stored. Regenerate the
webhook credential in the admin panel (one-time plain `whsec_*` shown again).

### Queue drains but no webhook reaches the client

The worker is running; check delivery logs (`var/log/prod.log` or Supervisor stderr).
Common causes: client URL returns 4xx/5xx, ngrok tunnel down, or invalid HTTPS URL
in webhook config.

### `Table '...messenger_messages' doesn't exist`

Run setup for that instance:

```bash
cd /home/<cpanel_user>/demomultitrackapi
sudo -u <cpanel_user> /opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:setup-transports
```

### Queue drains but no email arrives

The worker is running; the issue is likely data/config (missing email template, no
recipients, SMTP settings). Check:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php bin/console messenger:failed:show
```

### `Module 'ssh2' already loaded`

Harmless PHP warning on cPanel. Does not block the worker.

---

## Quick reference (demo server)

| Item | Value |
|---|---|
| OS | AlmaLinux 8.10 + cPanel |
| PHP (workers) | `/opt/cpanel/ea-php83/root/usr/bin/php` |
| cPanel user | `aab42b5` |
| Demo path | `/home/aab42b5/demomultitrackapi` |
| Prod path | `/home/aab42b5/multitrackapi` |
| Supervisor config (notifications) | `/etc/supervisord.d/messenger-notifications-*.ini` |
| Supervisor config (webhooks) | `/etc/supervisord.d/messenger-webhooks-*.ini` |
| Supervisor templates (repo) | `deploy/supervisor/*.ini.example` |
| Consume command (notifications) | `messenger:consume notifications` |
| Consume command (webhooks) | `messenger:consume webhooks` |