¶ HA Cluster
thing-1 and thing-2 are Dell PowerEdge R240 servers configured as a high-availability Pacemaker/Corosync cluster for hosting web services and applications.
¶ Hardware Specifications
- Model: Dell PowerEdge R240
- Network: 10GbE connectivity
- Storage: Shared NFS storage via Synology NAS (192.168.1.246:/volume1/ha-cluster)
¶ Network Configuration
- thing-1: 192.168.1.21
- thing-2: 192.168.1.22
- cluster_vip: 192.168.1.100 (floating virtual IP)
¶ Cluster Software Stack
- OS: Rocky Linux 9
- Cluster: Pacemaker 2.1.10 + Corosync
- Container Runtime: Docker with docker-compose
- Container Registry: jgmelon:5002
¶ Shared Storage
- Mount Point:
/mnt/ha-shared - NFS Server: 192.168.1.246:/volume1/ha-cluster
- Options: vers=3,rw
- Secondary Mount:
/mnt/senuti(media library)
¶ Cluster Resources
¶ Infrastructure Resources
- cluster_vip (ocf
IPaddr2) - Floating IP address for service access - ha-nfs-mount (ocfFilesystem) - Primary shared storage mount
- ha-nfs-senuti-mount (ocfFilesystem) - Media library mount
¶ Application Resources (systemd)
All application resources are managed as systemd services wrapped by Pacemaker:
-
ha-web-containers - Docker Compose stack containing:
- nginx-reverse-proxy (ports 80, 443)
- accent-api, accent-portal, accent-prod, accent-dev
- customstack, pricing
- joshkenney-app, stdftd-app
- ipod-repair
- wordpress
- weather-agent
-
ha-mongodb - MongoDB instance (port 27018)
-
ha-navidrome - Music streaming service (port 4533)
-
ha-mailcow - Mail server stack with
docker-compose.override.ymlfor clustering config -
ha-wiki - Wiki.js documentation (port 3000, wiki.customstack.nyc)
¶ Resource Location Preferences
By default, all resources prefer to run on thing-1. Resources automatically failover to thing-2 if thing-1 becomes unavailable.
¶ SSL Certificates
Let's Encrypt certificates stored in /mnt/ha-shared/letsencrypt/ and shared across both nodes.
¶ Common Operations
¶ Check cluster status
sudo pcs status
¶ Put node in standby (maintenance mode)
sudo pcs node standby thing-1
# or
sudo pcs node standby thing-2
¶ Remove node from standby
sudo pcs node unstandby thing-1
# or
sudo pcs node unstandby thing-2
¶ Move a resource manually
sudo pcs resource move <resource-name> thing-2
sudo pcs resource clear <resource-name>
¶ Restart a service
sudo pcs resource restart <resource-name>
¶ View resource configuration
sudo pcs resource config <resource-name>
¶ View constraints
sudo pcs constraint location config
sudo pcs constraint colocation
¶ Container Deployment Workflow
¶ Deploying Individual Services (e.g., accent-dev)
- Pull changes to local repo
cd /mnt/ha-shared/web-containers/accent-dev
git pull
- Build the image locally
docker build accent-dev
- Push to registry
docker push jgmelon:5002/accent-dev:latest
- Restart the service
sudo pcs resource restart ha-web-containers
¶ Quick Update Workflow
# On development machine or thing-1
cd /mnt/ha-shared/web-containers
docker-compose build accent-dev
docker tag web-containers-accent-dev:latest jgmelon:5002/accent-dev:latest
docker push jgmelon:5002/accent-dev:latest
# On cluster
sudo pcs resource restart ha-web-containers
¶ Service Access
¶ Public Services
- Web Services: https:// via nginx reverse proxy on cluster_vip
- Mail: https://mail.customstack.nyc (via ha-mailcow)
- Music: htps://ipods.ipodrepair.nyc
- Wiki: https://wiki.customstack.nyc
¶ Internal Services
- MongoDB: 192.168.1.100:27018
- Container Registry: jgmelon:5002
¶ Maintenance Notes
- All persistent data stored in
/mnt/ha-shared/ - Container configurations in
/mnt/ha-shared/web-containers/ - Both nodes must have identical systemd service files in
/etc/systemd/system/ - Run
systemctl daemon-reloadafter service file changes on both nodes - Use standby mode for planned maintenance on a node
¶ Disaster Recovery
In case of total cluster failure:
- NFS data on Synology remains intact
- Systemd service files can be restored from
/etc/systemd/system/ha-*.service - Let's Encrypt certificates stored in
/mnt/ha-shared/letsencrypt/ - Docker images stored in local registry jgmelon:5002