A Flask/Gunicorn webhook server that listens for GitHub push events and runs deploy scripts for each service. nginx proxies HTTPS traffic to it from deploy.customstack.nyc.
GitHub → deploy.customstack.nyc (nginx, Docker) → host:9000 (gunicorn) → scripts/<service>.sh
nginx runs in Docker on the webnet network. Gunicorn runs directly on the host via systemd, managed by Pacemaker for HA failover across thing-1 and thing-2.
| Path | Description |
|---|---|
/opt/webhook-deploy/app.py |
Flask app — verifies HMAC-SHA256 signatures, dispatches to scripts |
/opt/webhook-deploy/scripts/<service>.sh |
Per-service deploy scripts |
/opt/webhook-deploy/venv/ |
Python virtualenv |
/opt/webhook-deploy/ha-webhook-deploy.service |
Canonical systemd unit file (copy to /etc/systemd/system/ on each node) |
/etc/sysconfig/ha-webhook-deploy |
Environment file — contains WEBHOOK_SECRET |
/mnt/ha-shared/nginx-proxy/nginx.conf |
nginx config — deploy.customstack.nyc server block |
/mnt/ha-shared/nginx-proxy/docker-compose.yml |
nginx Docker Compose — includes extra_hosts for host access |
Defined in app.py — requests to /hook/<service> are accepted for:
accent-api, accent-portal, accent-prodcustomstack, ipod-repair, weather-agentcourse, joshkenney-blog, dolandental, 207dentalnginx runs in Docker and cannot reach 127.0.0.1 on the host. Two changes enable host access:
/mnt/ha-shared/nginx-proxy/docker-compose.yml — adds host.docker.internal DNS:
extra_hosts:
- "host.docker.internal:host-gateway"
nginx.conf — deploy.customstack.nyc server block proxies to host:
location /hook/ {
proxy_pass http://host.docker.internal:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 720s;
}
Gunicorn must bind to 0.0.0.0 (not 127.0.0.1) so Docker can reach it:
ExecStart=/opt/webhook-deploy/venv/bin/gunicorn \
--bind 0.0.0.0:9000 \
--workers 2 \
--timeout 660 \
app:app
The canonical unit file lives at /opt/webhook-deploy/ha-webhook-deploy.service. After any changes, copy it to both nodes:
sudo cp /opt/webhook-deploy/ha-webhook-deploy.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl restart ha-webhook-deploy
ssh thing-1 "sudo cp /opt/webhook-deploy/ha-webhook-deploy.service /etc/systemd/system/ && sudo systemctl daemon-reload"
The service is disabled on both nodes — Pacemaker controls when it starts.
ha-webhook-deploy is a Pacemaker resource. It runs on one active node at a time. After changes:
sudo pcs resource cleanup ha-webhook-deploy
sudo pcs status
Failover test:
sudo pcs node standby thing-2 # forces failover to thing-1
sudo pcs status
sudo pcs node unstandby thing-2
/etc/sysconfig/ha-webhook-deploy must exist on both nodes with identical contents:
WEBHOOK_SECRET=<secret>
Must be mode 600. To sync to thing-1:
sudo cat /etc/sysconfig/ha-webhook-deploy | ssh thing-1 'cat > /tmp/ha-webhook-env'
ssh thing-1 'sudo install -m 600 /tmp/ha-webhook-env /etc/sysconfig/ha-webhook-deploy && rm /tmp/ha-webhook-env'
In each repo's Settings → Webhooks → Add webhook:
| Field | Value |
|---|---|
| Payload URL | https://deploy.customstack.nyc/hook/<service> |
| Content type | application/json |
| Secret | value from WEBHOOK_SECRET in /etc/sysconfig/ha-webhook-deploy |
| Which events | Just the push event |
| Active | checked |
Only pushes to refs/heads/main trigger a deploy — others return 200 {"status":"skipped"}.
https://deploy.customstack.nyc/hook/<service>WEBHOOK_SECRETref == refs/heads/main (ignores other branches)/opt/webhook-deploy/scripts/<service>.sh:
git pull origin maindocker compose build <service>docker compose push <service> (to jgmelon:5002)docker compose up -d --no-deps <service>Watch in real time after pushing to main:
sudo journalctl -u ha-webhook-deploy -f
GitHub also shows delivery status and response under Settings → Webhooks → Recent Deliveries.
# 403 = nginx→gunicorn working, signature required (good)
curl -s -o /dev/null -w "%{http_code}" -X POST https://deploy.customstack.nyc/hook/customstack
# Full end-to-end test with valid signature
SECRET=$(sudo grep WEBHOOK_SECRET /etc/sysconfig/ha-webhook-deploy | cut -d= -f2)
PAYLOAD='{"ref":"refs/heads/main"}'
SIG="sha256=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | cut -d" " -f2)"
curl -s -w "\n%{http_code}" -X POST https://deploy.customstack.nyc/hook/customstack \
-H "Content-Type: application/json" \
-H "X-Hub-Signature-256: $SIG" \
-d "$PAYLOAD"
# Expected: {"status":"ok"} 200