Differences

This shows you the differences between two versions of the page.

--- syntheses:vps-create-flow [22/07/2026 00:29] – Expand VPS create flow with source-verified action sub-points bkirk
+++ syntheses:vps-create-flow [03/08/2026 23:16] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-====== VPS Creation / Provisioning Flow ======
-This page traces the source-backed flow for creating a VPS from the customer/API request through to HostDaemon provisioning on the compute host.
-===== Scope =====
-  * Main repo: ''vps/vps'' project 1
-  * Main application area: ''product/website''
-  * Compute execution: HostDaemon and host command handlers
-  * Related repo: ''vps/compute'' project 22, for deployed compute-host programs and scripts
-===== Short Version =====
-  - The API accepts a create-server request and validates the customer, plan, OS, network, billing state, and requested options.
-  - Domain services create a ''Guest'' in the database with status ''Building''.
-  - Provisioning services assign the lease, host, network, primary interface/IPs, backup schedule, and default firewall state.
-  - ''GuestManagementService.CreateGuest()'' creates a ''CreateGuestCommand''.
-  - HostDaemon polls the ''HostCommand'' table, marks the command started, and runs ''CreateGuestCommandHandler''.
-  - On success, HostDaemon sets the guest status to ''Active'', commits the guest, records system actions/host logs, sends mail, updates network members, and marks the command completed.
-  - On failure, HostDaemon attempts cleanup/refund/cancellation and leaves the guest suspended or terminated depending on where the failure occurred.
-===== API and Domain Flow =====
-==== 1. API request ====
-''product/website/WebApi/Services/GuestPlanApiService.cs''
-''GuestPlanApiService.CreateServer()'' (line 111) is the entry point for server creation via the public API.
-  * fetches the customer via ''CustomerResourceService.GetCustomer()'' — loads the full customer entity including subscription state and billing history.
-  * checks ''customer.Subscribed'' (line 114) — throws ''ValidationException'' if the customer lacks verified payment details, unless the signup flow explicitly bypasses this via ''requireSubscribed = false''.
-  * calls ''ValidateCreate()'' — validates the incoming DTO against plan availability, guest limits, and field constraints.
-  * calls ''GetGuestCreationParams()'' (line 118) — resolves facility, plan, operating system, network, and extras from the DTO into a ''GuestCreationParameters'' struct.
-  * wraps the guest creation inside ''_transactionRunner.Transaction()'' (line 126) — ensures all domain changes are atomic.
-  * calls ''_guestCreationService.CreateNewGuestDomain()'' (line 128) — performs the actual domain-level guest creation (see next step).
-  * flushes the NHibernate session (line 132) — forces an INSERT so the guest gets a database ID and generated ''ApiId'' before returning.
-  * after the transaction, if the command is ''IInvoiceBlockableCommand'' and the blocking invoice ''RequiresPayment()'', calls ''_invoiceService.TryProcessInvoice(blockingInvoice, null)'' (line 152) — attempts immediate payment using stored card or customer credit. This happens outside the domain transaction. If payment fails, the command still enters the queue — HostDaemon will block on the unpaid invoice later.
-  * returns an ''ActionDTO'' wrapping the ''CreateGuestCommand'' so the API caller can poll build progress.
-==== 2. Guest/domain creation ====
-''product/website/Domain/Services/GuestCreationService.cs''
-''GuestCreationService.CreateNewGuestDomain()'' (line 46) orchestrates domain-level guest creation:
-  * ''_invoiceService.CustomerCanPurchase()'' (line 48) — blocks if the customer has overdue invoices. Throws ''ValidationException("Guest creation is not permitted due to overdue invoices.")''.
-  * guest limit check (line 51) — compares ''customer.GuestCount'' against ''customer.GuestLimit''; throws if at or above limit.
-  * private IP validation (line 54) — if a specific VPC private IP was requested, validates it belongs to the network via ''_networkService.IsValidPrivateIp()'';
-  * split NIC validation (line 57) — ''SplitNics'' requires a private (VPC) network; throws if ''parameters.SplitNics && !parameters.Network.IsPrivate'';
-  * ''_guestFactory.InitializeNewGuest()'' (line 60) — creates the in-memory guest entity with status ''Building'', sets plan/OS/host placeholders, generates a unique name and rescue password;
-  * ''_validationService.ValidateOrThrow(guest)'' (line 61) — runs NHibernate/DataAnnotations validation on the guest entity;
-  * applies customer-facing options (lines 64-68): sets ''PortBlock'', ''Ipv6Enabled'', ''SplitNics'', ''NextDurationInMonths'';
-  * referral handling (lines 69-70): looks up the referrer from ''ReferrerCode'' via ''ReferrerHelper'', applies promotional code;
-  * ''_guestProvisioningService.ProvisionNewGuest()'' (line 73) — performs the resource allocations (host, network, IPs, lease, backup, firewall);
-  * if a specific private IP was requested, overrides the randomly assigned one (line 76): ''guest.PrimaryInterface.PrivateIp = parameters.PrivateIp'';
-  * ''_guestService.Commit(guest)'' (line 79) — persists the fully configured guest to the database;
-  * referral commit (line 81): if a referrer was found, creates and commits a referral record;
-  * fraud review gate (lines 87-91): if ''customer.GuestCount == 9'' (this is their 10th server), marks the customer as ''Unreviewed'' with reason "10th server created" and writes a system action — this triggers manual review for high-volume creation.
-  * ''_guestManagementService.CreateGuest()'' (line 95) — wraps the guest in a ''CreateGuestCommand'' with password/SSH/script/user-data options and queues it via ''HostCommandService.AddHostCommand()'';
-==== 3. Initial guest state ====
-''product/website/Domain/Services/GuestFactory.cs''
-''GuestFactory.InitializeNewGuest()'' (line 19) creates the in-memory guest entity:
-  * determines ''netnsEnabled'' from ''customer.NetnsCompatible'' — if the customer is not netns-compatible, the guest is created with ''NetnsEnabled = false'' even if the host supports it;
-  * generates a unique name via ''_guestService.GenerateUniqueName()'';
-  * sets initial placeholders: ''Host = Host.Unset'', ''Storage = Storage.Unset'', ''CurrentCpuModel = CpuModel.Unset'' — these are ''Unset'' sentinel values until the host assigns real resources during provisioning;
-  * sets ''Hostname'' from the customer-supplied value, or falls back to a generated name via ''Guest.GetAlternativeName()'';
-  * generates a rescue password via ''_guestService.GenerateRescuePassword()'';
-  * ''guest.InitializeFromOperatingSystem()'' — copies OS-specific defaults (SSH port, remote username, etc.) from the selected operating system entity;
-  * ''guest.SetStatus(GuestStatus.Building)'' (line 44) — the first status the guest sees in the database;
-==== 4. Provisioning assignments ====
-''product/website/Domain/Services/GuestProvisioningService.cs''
-''ProvisionNewGuest()'' (line 23) performs all resource allocations before the command enters the queue:
-  - ''_guestLeaseCreationService.CreateNewLeaseForGuest()'' (line 26): creates a new ''Lease'' and ''LeaseTerm'' entities linked to the guest, which generates the invoice that will block command execution until paid.
-  - ''_hostService.AddGuestToAvailableHost()'' (line 30): selects a compute host from the available pool in the requested facility and assigns it to the guest; considers host capacity, supported OS, and storage type.
-  - ''_networkService.AddGuestToNetwork()'' (line 33): assigns the guest to the selected network zone; this is a logical assignment — the physical IP allocation happens next.
-  - ''_guestService.AssignPrimaryInterfaceToGuest()'' (line 36): creates the primary network interface, allocates a public IPv4 address (and IPv6 if enabled), and persists the interface.
-  - ''_guestService.AssignAdditionalInterfaces()'' (line 37): creates additional NICs for VPC/split-NIC configurations.
-  - ''_backupScheduleService.SetupGuestBackupSchedule()'' (line 38): configures daily/weekly/monthly backup and offsite backup state based on guest extras.
-  - ''SetupDefaultFirewall()'' (line 39, static method): clears any inherited firewall rules and initialises the default allow/deny state for a fresh guest.
-===== HostCommand / HostDaemon Flow =====
-==== 5. CreateGuestCommand is queued ====
-''product/website/Domain/Services/GuestManagementService.cs''
-''CreateGuest()'' (line 174) wraps the guest entity in a host command:
-  * creates the ''CreateGuestCommand'' via ''_hostCommandFactory.CreateCommand<CreateGuestCommand>(guest)'';
-  * attaches customer-facing options: ''EmailVPSPassword'', ''WithPassword'' (password override), ''Script'' (deployment script), ''ScriptInput'', ''WithSshKeys'' (list of SSH key IDs), ''UserData'';
-  * calls ''_hostCommandService.AddHostCommand()'' which persists the command to the ''HostCommand'' table with status = not started, not completed.
-''product/website/Domain/Entities/CreateGuestCommand.cs''
-The command declares six customer-visible build steps (lines 21-26):
-  - Requesting server resources — connects to host shell, validates banned status, checks disk doesn't already exist
-  - Allocating virtual disk — creates the virtual disk on the compute host; issues licenses early so activation completes before OS install
-  - Installing operating system — triggers the OS installer on the compute host; configures time machine if enabled
-  - Starting server — boots the VM on the compute host
-  - Configuring operating system — waits for guest ping (guest is responsive), installs SSH keys, runs deployment script/user-data, sets password
-  - Processing invoice — finalises the build: sets status to Active, writes system actions, sends customer emails, updates network members
-It implements ''IInvoiceBlockableCommand'' (line 7). Its blocking invoice is ''Guest.LatestLeaseTerm.Invoice'' (line 49). This is the lease-term invoice created during provisioning — HostDaemon will not begin the build until this invoice is paid (for prepaid) or the customer is subscribed (for postpaid).
-==== 6. HostDaemon picks up the command ====
-''product/website/HostDaemon/Hosted/HostCommandThread.cs''
-HostDaemon runs a polling loop that repeatedly:
-  - calls ''GetNextCommand()'' (line 65) — opens a database transaction, calls ''broker.GetNextCommand()'' to find the next unstarted, unblocked command, then calls ''service.TryStartCommand()'' to atomically mark it started. ''TryStartCommand()'' checks ''IsBlockedBy()'' and ''IsDuplicateOf()'' guards to prevent concurrent execution of the same command type on the same guest.
-  - measures query duration via the transaction scope (lines 81-93) — if obtaining a database connection took longer than ''MaxQueryDuration'' (default 15s pool timeout), the command is skipped to avoid starving already-running handlers of database connections.
-  - starts the command in a worker thread via ''new Thread(new ProcessCommand(...).Run).Start()'' (line 120) — or as a separate process on Windows hosts.
-Adaptive rest behaviour (lines 58-66): after each iteration, the thread sleeps for a duration proportional to how long the database query took. If the database is responsive, threads spin up quickly. If the connection pool is contended, threads slow down. If no command was found, rests for at least ''DefaultRestDuration'' to avoid tight-loop polling.
-==== 7. HostDaemon creates the guest ====
-''product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs''
-''CreateGuestCommandHandler'' extends ''BlockingHostCommandHandler<CreateGuestCommand>'' (line 18). The base class handles the invoice gate.
-''Handle()'' (line 64) performs the actual host-side build in six steps:
-  - Step 0 — invoice gate (line 72): calls ''base.Handle(command)''. The base class checks ''command.BlockingInvoice()'' via ''BlockedOnInvoice()'' (line 42 of BlockingHostCommandHandler). For prepaid invoices, blocks until ''invoice.Paid''. For postpaid invoices, only blocks if ''!customer.Subscribed''. If blocked, reschedules with a 1-minute retry or cancels after 14 days. Returns ''false'' if blocked — the handler exits without touching the guest.
-  - Step 1 — Requesting server resources (lines 75-77): connects to the compute host's shell via ''ConnectShellToGuest()''; checks if the customer is banned via ''CheckForBannedCustomer()'' (checks ''BannedCountries'' config and customer ''Reviewed'' status); aborts via ''AbortIfDiskExists()'' if a disk already exists for this guest (prevents duplicate builds).
-  - Step 2 — Allocating virtual disk (line 80): calls ''CreateVirtualDisk()'' which creates the disk image on the compute host storage. Then immediately issues licenses via ''ChangeLicensesFromTo()'' (line 83) — this happens early so license activation completes before the OS is installed.
-  - Step 3 — Installing operating system (line 86): calls ''InstallOperatingSystem()'' which triggers the OS installer on the compute host. If ''guest.TimeMachine'' is enabled, configures the time machine via ''_xenManagementHelper.ConfigureTimeMachine()'' (line 89).
-  - Step 4 — Starting server (line 94): calls ''StartServer()'' which boots the VM on the compute host.
-  - Step 5 — Configuring operating system (line 97): calls ''ConfigureOperatingSystem()'' which waits for the guest to respond to ping (throws ''PingException'' if unresponsive, handled by ''HandlePingException()''), then installs SSH keys, runs the deployment script/user-data, and sets the root password.
-  - Step 6 — Processing invoice (lines 100-102): calls ''FinalizeSuccessfulCreate()'' which does three things: (a) ''FlagGuestAsCreated()'' — sets ''GuestStatus.Active'' and commits; (b) ''NotifyCustomer()'' — emails VPS password (if requested and auto-generated), welcome email (if customer was reviewed and hasn't received one), and service level notification (if service level has a cost); (c) ''UpdateNetwork()'' — notifies other guests in the same network about the new member.
-On success, ''FlagGuestAsCreated()'' (line 222) runs:
-<code>
-command.Guest.SetStatus(GuestStatus.Active);
-_guestService.Commit(command.Guest);
-</code>
-So the source-backed transition is ''Building'' → ''Active'', not ''Running''.
-==== 8. Command completion ====
-''product/website/HostDaemon/ProcessCommand.cs''
-''ProcessCommand.Run()'' (line 44) is the worker that executes any host command, not just create:
-  * loads the command from the database via ''broker.GetById()'' (line 56) — if this fails (e.g., connection timeout), enqueues a ''FailedCommand'' for later recovery;
-  * resolves the correct handler via ''uow.GetHandlerFor(command)'' (line 70);
-  * calls ''handler.Handle()'' (line 74);
-  * on success: calls ''command.SetCompleted()'' (line 75), then ''broker.Commit()'' (line 102) — marks the command completed/succeeded in the database;
-  * calls ''handler.Completed()'' (line 112) — handler-specific post-completion cleanup;
-  * calls ''broker.Commit()'' again (line 113) — persists any changes from ''Completed()'';
-  * if the command failed without throwing (succeeded=false), saves a structured ''ExceptionSummary'' with log output (lines 123-131);
-  * if any commit fails, enqueues a ''FailedCommand'' for the ''FailedCommandQueue'' to retry.
-There is no separate HTTP callback from HostDaemon to the API. HostDaemon updates the shared database directly via NHibernate/brokers. The API reads the command state from the same database when the customer polls build progress.
-===== Billing Behaviour =====
-Two separate billing concerns operate at different stages:
-  * **Immediate API-side payment attempt:** After the guest is created and the command queued, ''GuestPlanApiService.CreateServer()'' calls ''InvoiceService.TryProcessInvoice()'' if the blocking invoice ''RequiresPayment()'' (line 152). This attempts payment immediately — uses customer credit first, generates a credit card deposit if needed, calls ''invoice.MarkAsPaid()'' on success.
-  * **HostDaemon invoice gate:** Before any build work begins, ''BlockingHostCommandHandler.Handle()'' checks ''BlockedOnInvoice()'' (line 18). For prepaid invoices, blocks until ''invoice.Paid''. For postpaid invoices, only blocks if ''!customer.Subscribed''. This means postpaid customers with active subscriptions can build immediately without upfront payment.
-''InvoiceService.TryProcessInvoice()'' (line 75) does:
-  * refreshes the customer entity via ''_customerBroker.Refresh()'' (line 84) — ensures the ''Reviewed'' flag is current in case the account was approved during a long-running creation;
-  * returns ''true'' early if ''!invoice.RequiresPayment()'' (line 87);
-  * marks the invoice ready for sending (line 97) — triggers the "Payable Now" email if payment hasn't been made yet;
-  * for zero-dollar invoices, calls ''HandleInvoiceProcessingSuccess()'' and returns ''true'' (line 100);
-  * if insufficient customer credit and the customer has a stored card, generates a credit card deposit via ''_depositService.GenerateDepositForInvoiceFromCreditCard()'' (line 106);
-  * if credit-only mode was requested and there's insufficient credit, returns ''false'' without marking the invoice sent (preserves sent state to avoid duplicate "Payable Now" emails while customer is on a payment gateway page);
-  * on success, calls ''invoice.MarkAsPaid()'' (line 269) and returns ''true''.
-HostDaemon does not "finalise" the invoice during successful provisioning. The create command uses the invoice as a blocker. The API/payment path pays it before or while the command waits.
-===== Failure and Recovery Behaviour =====
-==== Unpaid invoice / pending payment ====
-''product/website/HostDaemon/BlockingHostCommandHandler.cs''
-If the blocking invoice is not paid:
-  * ''BlockedOnInvoice()'' (line 42): returns ''true'' if the invoice exists and is unpaid (for prepaid) or the customer is not subscribed (for postpaid).
-  * ''RescheduleOrCancel()'' (line 63): calls ''ShouldCancel()'' (line 60) which checks if ''invoice.IsOverdue || (invoice.Cancelled && invoice.IsFinal)'' or if ''SystemTime.Now() - command.Created >= PENDING_PAYMENT_MAX_TIME'' (14 days, line 30).
-  * If should cancel: calls the handler's ''Cancel()'' method.
-  * Otherwise: calls ''Reschedule()'' (line 70) which sets ''command.RetryAfter(PENDING_PAYMENT_WAIT_TIME, ...)'' — a 1-minute wait (line 30: ''new TimeSpan(0, 1, 0)'').
-For create guest cancellation, ''CreateGuestCommandHandler.Cancel()'' (line 314):
-  * ends the lease via ''guest.EndCurrentLease()'';
-  * cancels the invoice via ''_invoiceService.CancelInvoice(invoice)'' if not already cancelled;
-  * removes allocated resources via ''_guestService.RemoveAllResourcesFromGuest(guest)'';
-  * sets guest status to ''Terminated'' (line 329);
-  * commits the guest;
-  * sends a guest-creation-cancelled email via ''_guestCreationCancelledMailer.Send(guest)'' if the cancellation wasn't already triggered by someone else.
-==== Exception during provisioning ====
-''CreateGuestCommandHandler.HandleException()'' (line 272):
-  * logs the failure via ''_logger.Debug()'' with exception details;
-  * writes a ''CreateGuestFailure'' system action via ''_actionService.Write()'' — includes hostname, plan, OS, and the exception;
-  * clears ''command.ResultData'' (line 277) — removes the generated root password from the command result data so it can't be recovered;
-  * attempts cleanup (lines 280-284): calls ''_xenManagementHelper.UncleanPoweroffGuest(guest)'' and ''_xenManagementHelper.RemovePrimaryDisk(guest)''. If cleanup fails, logs the error but does not rethrow — the guest is already in a failed state.
-  * inside a transaction (line 286): cancels and refunds the invoice via ''_invoiceService.CancelInvoiceAndRefundTransaction(invoice)'' if one exists; ends the current lease via ''guest.EndCurrentLease()''; cancels the guest via ''guest.CancelGuest(true, "Creation failed.")''; sets guest status to ''Suspended'' (line 298); commits the guest.
-  * revokes any licenses via ''_guestManagementService.ChangeLicensesFromTo()'' (line 304).
-The code comment (lines 307-310) explains that failed creates used to delete the guest entirely, but now leave it suspended so normal automatic termination can clean it up.
-==== Failure while recording command state ====
-''product/website/HostDaemon/Hosted/FailedCommandQueue.cs''
-If command loading or committing fails, ''ProcessCommand'' enqueues a ''FailedCommand''. ''FailedCommandQueue.Run()'' later processes the queue:
-  * loads the command;
-  * runs handler abort logic;
-  * calls ''command.SetFailed()'' (line 56) with the exception and log output;
-  * commits the state via ''broker.Commit()'' (line 64).
-If the failure handler itself fails for non-database reasons, it logs and proceeds rather than retrying indefinitely.
-===== Data / State Transitions =====
-^ Object ^ Initial / queued state ^ Successful state ^ Failure state ^
-| ''Guest.Status'' | ''Building'' | ''Active'' | ''Suspended'' for provisioning exception; ''Terminated'' for invoice-cancel path |
-| ''HostCommand'' | not started / not completed | completed, succeeded | completed/failed where failure can be recorded |
-| ''Invoice'' | lease-term invoice, may require payment | paid by ''TryProcessInvoice()'' when payment succeeds | cancelled/refunded on provisioning failure or cancellation path |
-| ''Guest.Host'' | assigned before command execution | retained | resources may be removed/cancelled |
-| ''Guest.Interfaces'' | primary/additional assigned before command execution | retained | resources may be removed/cancelled |
-===== Support-Relevant Failure Points =====
-  * Customer has unpaid/overdue invoices — blocked at API level by ''CustomerCanPurchase()'' or at HostDaemon level by ''BlockedOnInvoice()''.
-  * Customer has no verified payment details / is not subscribed — throws at API level if ''customer.Subscribed'' is false.
-  * Customer is at guest limit — checked at both API and domain level; throws ''ValidationException'' with "please contact support".
-  * Plan is inactive or not listable — caught by ''ValidateCreate()'' at API level.
-  * No host/network/IP is available — ''AddGuestToAvailableHost()'' or ''AddGuestToNetwork()'' will fail if no capacity; the exception propagates up and cancels the guest.
-  * HostCommand is blocked by another active command — ''TryStartCommand()'' checks ''IsBlockedBy()'' and ''IsDuplicateOf()'' guards.
-  * Payment cannot be processed — ''TryProcessInvoice()'' returns false; command enters queue but blocks at HostDaemon level.
-  * HostDaemon cannot create disk, install OS, start guest, or reach guest by ping — any of these throws and triggers ''HandleException()''.
-  * HostDaemon/DB commit fails — the ''FailedCommandQueue'' picks up the command and retries the commit.
-===== Source Files =====
-  * ''product/website/WebApi/Services/GuestPlanApiService.cs''
-  * ''product/website/Domain/Services/GuestCreationService.cs''
-  * ''product/website/Domain/Services/GuestFactory.cs''
-  * ''product/website/Domain/Services/GuestProvisioningService.cs''
-  * ''product/website/Domain/Services/GuestManagementService.cs''
-  * ''product/website/Domain/Entities/CreateGuestCommand.cs''
-  * ''product/website/Domain/Services/HostCommandService.cs''
-  * ''product/website/Persistence/HostCommandBroker.cs''
-  * ''product/website/HostDaemon/Hosted/HostCommandThread.cs''
-  * ''product/website/HostDaemon/ProcessCommand.cs''
-  * ''product/website/HostDaemon/BlockingHostCommandHandler.cs''
-  * ''product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs''
-  * ''product/website/Domain/Services/InvoiceService.cs''
-  * ''product/website/HostDaemon/Hosted/FailedCommandQueue.cs''
-===== Remaining Questions =====
-  * Which customer-visible status label maps to internal ''GuestStatus.Active'' in the current panel/API?
-  * Which operational job performs automatic cleanup/termination of suspended failed-create guests, and after what delay?
-  * Which logs/queries should support use first when a customer reports a server stuck in ''Building''?
-===== Related Wiki Pages =====
-  * [[compute-host-stack|Compute Host Stack]]
-  * [[compute-repo-structure|vps/compute Repo Structure]]
-  * [[infrastructure-services|Infrastructure Services]]
-  * [[wiki:jargon|Jargon / Reference Index]]