User Tools

Site Tools


syntheses:compute-host-stack

This is an old revision of the document!


Compute Host Stack

Notes

Summary

Compute Host Stack

Overview

The compute host stack runs on physical servers (Xen/KVM hosts) and manages guest VPS lifecycle. It consists of software from multiple repos:

  • vps/vps → product/website/HostDaemon/ — Main .NET daemon (also duplicated here)
  • vps/vps → infrastructure/compute/ — Shell scripts, configs, and services deployed to hosts
  • vps/compute (project 22) — Separate repo with HostDaemon, qemu-network, routed, etc.
  • vps/qemu-network (project 20) — iptables/netns firewall rules

Architecture

Customer Action
    ↓
WebApi / PanelSite / ClassicPanel
    ↓
Domain validation → NHibernate persistence
    ↓
HostCommand created (Infrastructure/compute or product/website/HostDaemon)
    ↓
HostDaemon (on compute host) receives & executes command
    ↓
Guest lifecycle: start/stop/reinstall/migrate/firewall/backup
    ↓
qemu-network/mammothvps.sh applies iptables with claim_lock

Key Components

HostDaemon (product/website/HostDaemon/)

.NET daemon running on each compute host. Handles guest lifecycle commands:

product/website/HostDaemon/
├── Applets/               — Host command applets (sub-commands)
├── Application/           — Application logic
├── GuestConfiguration/    — Guest config generation (Xen/KVM)
├── Handlers/              — Host command handlers
├── Helpers/               — Utility classes
├── Hosted/                — Background services
├── LoggingContext/        — Logging
├── Properties/            — Assembly metadata
├── Scripts/               — Shell scripts deployed to hosts
├── Services/              — Host services
├── config/                — HostDaemon config
├── BlockingHostCommandHandler.cs
├── HostCommandExecutor.cs — Executes commands on hosts
├── HostCommandHandler.cs  — Command handler interface
├── HostShell.cs           — Shell abstraction for host commands
├── SshShell.cs            — SSH-based host shell
├── Program.cs             — Entry point
└── appsettings.json       — Configuration

Key classes:

  • HostCommand — The command object sent from WebApi to HostDaemon
  • HostCommandExecutor — Executes commands, manages ordering
  • HostCommandHandler — Interface for handling specific command types
  • HostShell / SshShell — Abstraction for executing commands on hosts

qemu-network (vps/qemu-network, project 20)

iptables/netns firewall on compute hosts:

  • Script: mammothvps.sh — Applies iptables rules with claim_lock
  • Uses claim_lock “iptables” held during entire apply phase
  • Serialises across ALL guests on a host (one customer's slow apply blocks others)
  • Critical bug surface: unbounded firewall rules cause Cartesian product explosion

infrastructure/compute/ (vps/vps)

Scripts and configs deployed to compute hosts:

infrastructure/compute/
├── etc/                   — Host configuration files
├── hostdaemon/            — HostDaemon deployment
├── ovmf/                  — UEFI firmware files
├── qemu-network/          — Firewall scripts
├── routed/                — Routing daemon
├── scripts/               — Shell scripts (backup, monitoring, etc.)
├── services/              — Systemd/init service definitions
├── smtbalance/            — SMTP load balancing
├── xm/                    — Xen configuration
├── xmconfig/              — Xen config tools
└── xm-for-kvm/            — Xen-to-KVM migration

vps/compute (project 22)

Separate repo for compute host programs:

vps/compute/
├── hostdaemon/            — HostDaemon (duplicate of product/website/HostDaemon)
├── ovmf/                  — UEFI firmware
├── power-limiter/         — Power management
├── qemu-network/          — iptables/netns firewall
├── routed/                — Routing daemon
├── scripts/               — Host scripts
├── services/              — Service definitions
├── smtbalance/            — SMTP balancing
├── xm/                    — Xen
├── xmconfig/              — Xen config
└── xm-for-kvm/            — Xen-to-KVM migration

Guest Lifecycle Commands

Commands flow from WebApi → HostCommand → HostDaemon:

  1. Power operations — start, stop, restart, reinstall
  2. Network — firewall rules, interface changes, VPC assignment
  3. Migration — live migration between hosts
  4. Backup — backup/restore operations
  5. Configuration — DNS, reverse DNS, hostname changes
  6. Maintenance — scheduled maintenance windows

Firewall Rule Flow (Critical Path)

This is the most complex and bug-prone path:

  1. Customer submits firewall rules via WebApi/Panel/CLI
  2. Domain validation → ValidateAddressListAttribute validates each address (NOT collection count)
  3. NHibernate persists GuestFirewallRule entities
  4. GuestManagementService.UpdateGuestNetwork() → sends HostCommand to HostDaemon
  5. HostDaemon iterates all firewall rules
  6. OperatingSystem.XenConfiguration() → builds FIREWALLRULES xenConfig string
  7. Guest.GetIptablesEntries() → aggregates across ALL rules
  8. GuestFirewallRule.GetIptablesEntries() → 4-level Cartesian loop (source × dest × protocol × port)
  9. qemu-network/mammothvps.sh → applies iptables with claim_lock

Xen vs KVM

  • xm/ — Xen configuration (older platform)
  • xm-for-kvm/ — Xen-to-KVM migration tooling
  • ovmf/ — UEFI firmware (for KVM)
  • Migration from Xen to KVM is in progress

Infrastructure/QEMU

  • build.sh — QEMU build script
  • debian-12.cfg — Debian 12 QEMU configuration
  • harness.sh — QEMU test harness
  • README.md — Documentation

Key Bug Surfaces

claim_lock (qemu-network)

  • claim_lock “iptables” held during entire apply phase
  • One customer's massive firewall ruleset blocks ALL other guests on the same host
  • Known issue #1640

Unbounded firewall rules

  • GuestFirewallRule — No cap on SourceAddresses/DestinationAddresses count
  • ValidateAddressListAttribute — Per-item only, no collection count check
  • GuestFirewallRule.GetIptablesEntries() — 4-level nested loop
  • Guest.GetIptablesEntries() — No size limit on aggregation

HostCommand serialization

  • HostDaemon processes commands sequentially per guest
  • Slow commands (massive firewall apply) block subsequent commands
syntheses/compute-host-stack.1784174059.txt.gz · Last modified: by 127.0.0.1

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki