Differences

This shows you the differences between two versions of the page.

--- syntheses:compute-host-stack [16/07/2026 03:54] – created - external edit 127.0.0.1
+++ syntheses:compute-host-stack [15/10/2026 05:26] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-====== Compute Host Stack ======
-===== Notes =====
-===== Summary =====
-====== Compute Host Stack ======
-===== Overview =====
-The compute host stack runs on physical servers (Xen/KVM hosts) and manages guest VPS lifecycle. It consists of software from multiple repos:
-  * **''vps/vps'' → ''product/website/HostDaemon/''** — Main .NET daemon (also duplicated here)
-  * **''vps/vps'' → ''infrastructure/compute/''** — Shell scripts, configs, and services deployed to hosts
-  * **''vps/compute'' (project 22)** — Separate repo with HostDaemon, qemu-network, routed, etc.
-  * **''vps/qemu-network'' (project 20)** — iptables/netns firewall rules
-===== Architecture =====
-<code>
-Customer Action
-    ↓
-WebApi / PanelSite / ClassicPanel
-    ↓
-Domain validation → NHibernate persistence
-    ↓
-HostCommand created (Infrastructure/compute or product/website/HostDaemon)
-    ↓
-HostDaemon (on compute host) receives & executes command
-    ↓
-Guest lifecycle: start/stop/reinstall/migrate/firewall/backup
-    ↓
-qemu-network/mammothvps.sh applies iptables with claim_lock
-</code>
-===== Key Components =====
-==== HostDaemon (product/website/HostDaemon/) ====
-.NET daemon running on each compute host. Handles guest lifecycle commands:
-<code>
-product/website/HostDaemon/
-├── Applets/               — Host command applets (sub-commands)
-├── Application/           — Application logic
-├── GuestConfiguration/    — Guest config generation (Xen/KVM)
-├── Handlers/              — Host command handlers
-├── Helpers/               — Utility classes
-├── Hosted/                — Background services
-├── LoggingContext/        — Logging
-├── Properties/            — Assembly metadata
-├── Scripts/               — Shell scripts deployed to hosts
-├── Services/              — Host services
-├── config/                — HostDaemon config
-├── BlockingHostCommandHandler.cs
-├── HostCommandExecutor.cs — Executes commands on hosts
-├── HostCommandHandler.cs  — Command handler interface
-├── HostShell.cs           — Shell abstraction for host commands
-├── SshShell.cs            — SSH-based host shell
-├── Program.cs             — Entry point
-└── appsettings.json       — Configuration
-</code>
-Key classes:
-  * **HostCommand** — The command object sent from WebApi to HostDaemon
-  * **HostCommandExecutor** — Executes commands, manages ordering
-  * **HostCommandHandler** — Interface for handling specific command types
-  * **HostShell / SshShell** — Abstraction for executing commands on hosts
-==== qemu-network (vps/qemu-network, project 20) ====
-iptables/netns firewall on compute hosts:
-  * **Script:** ''mammothvps.sh'' — Applies iptables rules with ''claim_lock''
-  * Uses claim_lock "iptables" held during entire apply phase
-  * Serialises across ALL guests on a host (one customer's slow apply blocks others)
-  * Critical bug surface: unbounded firewall rules cause Cartesian product explosion
-==== infrastructure/compute/ (vps/vps) ====
-Scripts and configs deployed to compute hosts:
-<code>
-infrastructure/compute/
-├── etc/                   — Host configuration files
-├── hostdaemon/            — HostDaemon deployment
-├── ovmf/                  — UEFI firmware files
-├── qemu-network/          — Firewall scripts
-├── routed/                — Routing daemon
-├── scripts/               — Shell scripts (backup, monitoring, etc.)
-├── services/              — Systemd/init service definitions
-├── smtbalance/            — SMTP load balancing
-├── xm/                    — Xen configuration
-├── xmconfig/              — Xen config tools
-└── xm-for-kvm/            — Xen-to-KVM migration
-</code>
-==== vps/compute (project 22) ====
-Separate repo for compute host programs:
-<code>
-vps/compute/
-├── hostdaemon/            — HostDaemon (duplicate of product/website/HostDaemon)
-├── ovmf/                  — UEFI firmware
-├── power-limiter/         — Power management
-├── qemu-network/          — iptables/netns firewall
-├── routed/                — Routing daemon
-├── scripts/               — Host scripts
-├── services/              — Service definitions
-├── smtbalance/            — SMTP balancing
-├── xm/                    — Xen
-├── xmconfig/              — Xen config
-└── xm-for-kvm/            — Xen-to-KVM migration
-</code>
-===== Guest Lifecycle Commands =====
-Commands flow from WebApi → HostCommand → HostDaemon:
-  - **Power operations** — start, stop, restart, reinstall
-  - **Network** — firewall rules, interface changes, VPC assignment
-  - **Migration** — live migration between hosts
-  - **Backup** — backup/restore operations
-  - **Configuration** — DNS, reverse DNS, hostname changes
-  - **Maintenance** — scheduled maintenance windows
-===== Firewall Rule Flow (Critical Path) =====
-This is the most complex and bug-prone path:
-  - Customer submits firewall rules via WebApi/Panel/CLI
-  - **Domain validation** → ''ValidateAddressListAttribute'' validates each address (NOT collection count)
-  - **NHibernate** persists ''GuestFirewallRule'' entities
-  - **GuestManagementService.UpdateGuestNetwork()** → sends HostCommand to HostDaemon
-  - **HostDaemon** iterates all firewall rules
-  - **OperatingSystem.XenConfiguration()** → builds FIREWALLRULES xenConfig string
-  - **Guest.GetIptablesEntries()** → aggregates across ALL rules
-  - **GuestFirewallRule.GetIptablesEntries()** → 4-level Cartesian loop (source × dest × protocol × port)
-  - **qemu-network/mammothvps.sh** → applies iptables with claim_lock
-===== Xen vs KVM =====
-  * ''xm/'' — Xen configuration (older platform)
-  * ''xm-for-kvm/'' — Xen-to-KVM migration tooling
-  * ''ovmf/'' — UEFI firmware (for KVM)
-  * Migration from Xen to KVM is in progress
-===== Infrastructure/QEMU =====
-  * **build.sh** — QEMU build script
-  * **debian-12.cfg** — Debian 12 QEMU configuration
-  * **harness.sh** — QEMU test harness
-  * **README.md** — Documentation
-===== Key Bug Surfaces =====
-==== claim_lock (qemu-network) ====
-  * ''claim_lock "iptables"'' held during entire apply phase
-  * One customer's massive firewall ruleset blocks ALL other guests on the same host
-  * Known issue #1640
-==== Unbounded firewall rules ====
-  * ''GuestFirewallRule'' — No cap on SourceAddresses/DestinationAddresses count
-  * ''ValidateAddressListAttribute'' — Per-item only, no collection count check
-  * ''GuestFirewallRule.GetIptablesEntries()'' — 4-level nested loop
-  * ''Guest.GetIptablesEntries()'' — No size limit on aggregation
-==== HostCommand serialization ====
-  * HostDaemon processes commands sequentially per guest
-  * Slow commands (massive firewall apply) block subsequent commands
-===== Related =====
-==== Related Pages ====
-  * [[vps-monorepo-structure|VPS Monorepo - Structure Overview]]
-  * [[binarylane-architecture-panels-customer-entry-points|BinaryLane Architecture - Panels & Customer Entry Points]]
-  * [[customer-side-bug-finding-methodology|Customer-Side Bug Finding Methodology]]
-  * [[dotnet-solution-structure|.NET Solution - product/website Structure]]