This is an old revision of the document!
Table of Contents
VPS Creation / Provisioning Flow
This page traces the source-backed flow for creating a VPS from the customer/API request through to HostDaemon provisioning on the compute host.
Scope
- Main repo:
vps/vpsproject 1. - Main application area:
product/website. - Compute execution: HostDaemon and host command handlers.
- Related repo:
vps/computeproject 22, for deployed compute-host programs and scripts.
Short Version
- The API accepts a create-server request and validates the customer, plan, OS, network, billing state, and requested options.
- Domain services create a
Guestin the database with statusBuilding. - Provisioning services assign the lease, host, network, primary interface/IPs, backup schedule, and default firewall state.
GuestManagementService.CreateGuest()creates aCreateGuestCommand.- HostDaemon polls the
HostCommandtable, marks the command started, and runsCreateGuestCommandHandler. - On success, HostDaemon sets the guest status to
Active, commits the guest, records actions/logs, sends mail, updates network members, and marks the command completed. - On failure, HostDaemon attempts cleanup/refund/cancellation and leaves the guest
SuspendedorTerminateddepending on where the failure occurred.
API and Domain Flow
1. API Request
Source: product/website/WebApi/Services/GuestPlanApiService.cs
Entry point: GuestPlanApiService.CreateServer().
What it does:
CustomerResourceService.GetCustomer()- Loads the customer entity, including subscription state and billing history.
customer.Subscribedcheck- Blocks creation if the customer lacks verified payment details.
- Signup flow can bypass this by calling with
requireSubscribed = false.
ValidateCreate()- Checks purchase eligibility, guest limits, DTO fields, and plan availability.
GetGuestCreationParams()- Resolves facility, plan, operating system, network, and extras into
GuestCreationParameters.
_transactionRunner.Transaction()- Wraps domain guest creation so the main database changes are atomic.
_guestCreationService.CreateNewGuestDomain()- Performs the domain-level guest creation described in the next step.
- NHibernate flush
- Forces the guest insert so the API ID exists before the response is returned.
_invoiceService.TryProcessInvoice()- Runs after the domain transaction if the command has a blocking invoice that
RequiresPayment(). - Attempts immediate payment using customer credit or stored payment details.
- If payment fails, the command still enters the queue and HostDaemon blocks on the unpaid invoice later.
Result: the API returns an action DTO for the created host command so the caller can poll build progress.
2. Guest/Domain Creation
Source: product/website/Domain/Services/GuestCreationService.cs
Entry point: GuestCreationService.CreateNewGuestDomain().
What it does:
_invoiceService.CustomerCanPurchase()- Blocks customers with overdue invoices.
- Guest limit check
- Compares
customer.GuestCountwithcustomer.GuestLimit.
- Private IP validation
- If a VPC private IP was requested, validates that it belongs to the requested network.
- Split NIC validation
- Requires a private network when
SplitNicsis requested.
_guestFactory.InitializeNewGuest()- Creates the in-memory guest entity.
- Sets initial status to
Building. - Sets plan/OS placeholders, generated name, hostname, and rescue password.
_validationService.ValidateOrThrow(guest)- Runs entity validation before provisioning resources.
- * _validationService.ValidateOrThrow(guest) runs Validator.TryValidateObject(validateAllProperties: true) — DataAnnotation attributes plus Guest.Validate(). It does not recurse into nested objects. Checks:
Guest.Validate() (IValidatableObject) OS fits config: memory & storage meet the selected OS minimums (CanAccomodateOperatingSystem).
[ValidateHostname] on Hostname Not null. No trailing dot; no consecutive dots (..). No leading/trailing hyphen on labels (^-, -$, -., .-). Valid chars only: ^[A-Za-z0-9.-]*$, non-empty.
[CustomerHostnameUnique] on Hostname Hostname unique per customer (GetByHostnameForCustomer); fails “Hostname in use by another server” if a different guest owns it. Also fails if null/empty.
- Option application
- Applies port block, IPv6, split NICs, SSH keys, scripts/user-data, and duration settings.
- Referral and fraud-review handling
- Applies referral information where present.
- Marks the customer for review on the 10th server creation path.
_guestProvisioningService.ProvisionNewGuest()- Allocates lease, host, network, interfaces, backup schedule, and firewall state.
_guestService.Commit(guest)- Persists the configured guest.
_guestManagementService.CreateGuest()- Queues the HostDaemon create command.
Result: a fully allocated database guest exists and a create command is queued.
3. Initial Guest State
Source: product/website/Domain/Services/GuestFactory.cs
Entry point: GuestFactory.InitializeNewGuest().
What it sets:
Host = Host.Unset,Storage = Storage.Unset, andCurrentCpuModel = CpuModel.Unsetuntil provisioning assigns real resources.- A unique generated name and customer-facing hostname.
- Rescue password and OS-specific defaults through
guest.InitializeFromOperatingSystem(). NetnsEnabledbased on customer compatibility.- Initial status:
guest.SetStatus(GuestStatus.Building);
Result: the guest starts life as Building, not Running or Active.
4. Provisioning Assignments
Source: product/website/Domain/Services/GuestProvisioningService.cs
Entry point: ProvisionNewGuest().
Allocation sequence:
GuestLeaseCreationService.CreateNewLeaseForGuest()- Creates lease and lease-term records.
- Creates the invoice that can block command execution.
HostService.AddGuestToAvailableHost()- Selects and assigns a compute host in the requested facility.
NetworkService.AddGuestToNetwork()- Assigns the guest to the selected network zone.
GuestService.AssignPrimaryInterfaceToGuest()- Creates the primary interface.
- Allocates public IPv4 and IPv6 where applicable.
GuestService.AssignAdditionalInterfaces()- Adds VPC or split-NIC interfaces.
BackupScheduleService.SetupGuestBackupSchedule()- Sets backup and offsite backup schedule state.
SetupDefaultFirewall()- Clears or initialises firewall state for a fresh guest.
Result: the guest has database-side host, lease, network, interface, backup, and firewall assignments before HostDaemon starts the VM build.
HostCommand / HostDaemon Flow
5. CreateGuestCommand Is Queued
Source: product/website/Domain/Services/GuestManagementService.cs
Entry point: CreateGuest().
What it does:
- Creates
CreateGuestCommandthrough_hostCommandFactory.CreateCommand<CreateGuestCommand>(guest). - Attaches build inputs:
EmailVPSPassword.- Password override through
WithPassword. - Deployment script and script input.
- SSH key IDs.
- User-data.
- Calls
HostCommandService.AddHostCommand().- Persists the command to the
HostCommandtable. - Leaves it not started and not completed until HostDaemon picks it up.
Source: product/website/Domain/Entities/CreateGuestCommand.cs
Customer-visible command steps:
- Requesting server resources.
- Allocating virtual disk.
- Installing operating system.
- Starting server.
- Configuring operating system.
- Processing invoice.
Invoice blocker:
CreateGuestCommandimplementsIInvoiceBlockableCommand.- Its blocking invoice is
Guest.LatestLeaseTerm.Invoice. - HostDaemon will not build prepaid guests until that invoice is paid.
- For postpaid customers, the invoice gate depends on subscription state.
6. HostDaemon Picks Up The Command
Source: product/website/HostDaemon/Hosted/HostCommandThread.cs
Polling sequence:
GetNextCommand()- Opens a database transaction.
- Calls
broker.GetNextCommand()to find the next unstarted, unblocked command. - Calls
service.TryStartCommand()to mark it started atomically.
TryStartCommand()guards- Checks
IsBlockedBy(). - Checks
IsDuplicateOf(). - Prevents concurrent execution of incompatible commands for the same guest.
- Query-duration guard
- If acquiring a database connection takes too long, the command is skipped for that pass.
- This avoids starving already-running handlers when the pool is contended.
- Worker start
- Starts
new Thread(new ProcessCommand(…).Run).Start(). - Some Windows host paths can run commands in a separate process.
Polling behaviour: HostDaemon adapts its sleep duration based on database responsiveness. If no command is found, it rests long enough to avoid tight-loop polling.
7. HostDaemon Creates The Guest
Source: product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs
Entry point: CreateGuestCommandHandler.Handle().
Base class: BlockingHostCommandHandler<CreateGuestCommand> handles the invoice gate before host-side work begins.
Build sequence:
- Step 0: Invoice gate
- Calls
base.Handle(command). - Checks
command.BlockingInvoice()throughBlockedOnInvoice(). - For prepaid invoices, blocks until
invoice.Paid. - For postpaid invoices, blocks only when the customer is not subscribed.
- If blocked, returns without touching the guest.
- Step 1: Request server resources
- Connects to the compute host shell through
ConnectShellToGuest(). - Checks banned-customer conditions.
- Aborts if a disk already exists for this guest.
- Step 2: Allocate virtual disk
- Calls
CreateVirtualDisk(). - Issues licences early through
ChangeLicensesFromTo()so activation can complete before OS install.
- Step 3: Install operating system
- Calls
InstallOperatingSystem()on the compute host. - Configures time machine if enabled.
- Step 4: Start server
- Boots the VM on the compute host.
- Step 5: Configure operating system
- Waits for the guest to respond to ping.
- Installs SSH keys.
- Runs deployment script/user-data.
- Sets the password.
- Step 6: Finalise successful create
- Calls
FlagGuestAsCreated(). - Sends relevant customer emails.
- Updates network membership state.
Successful status transition:
command.Guest.SetStatus(GuestStatus.Active); _guestService.Commit(command.Guest);
Result: the source-backed transition is Building to Active, not Running.
8. Command Completion
Source: product/website/HostDaemon/ProcessCommand.cs
Entry point: ProcessCommand.Run().
What it does:
- Loads the command from the database with
broker.GetById(). - Resolves the correct handler through
uow.GetHandlerFor(command). - Calls
handler.Handle(). - On success:
- Calls
command.SetCompleted(). - Commits the command.
- Calls
handler.Completed(). - Commits again for post-completion changes.
- On handler failure without an exception:
- Saves structured
ExceptionSummaryand log output.
- On load or commit failure:
- Enqueues a
FailedCommandfor later recovery.
Result: HostDaemon writes command state directly to the shared database. There is no separate HTTP callback to the API in this path.
Billing Behaviour
API-Side Payment Attempt
Source: product/website/Domain/Services/InvoiceService.cs
Entry point: InvoiceService.TryProcessInvoice().
What it does:
- Refreshes the customer entity.
- Returns early if the invoice does not require payment.
- Marks the invoice ready for sending.
- Handles zero-dollar invoices.
- Uses customer credit first.
- Generates a credit-card deposit if needed and possible.
- Preserves invoice sent state in credit-only failure cases to avoid duplicate payable-now emails.
- Calls
invoice.MarkAsPaid()on success.
HostDaemon Invoice Gate
Source: product/website/HostDaemon/BlockingHostCommandHandler.cs
What it does:
- Checks the command's blocking invoice before the handler performs host-side work.
- Blocks prepaid invoices until paid.
- Allows subscribed postpaid customers through when the invoice is not final and unpaid.
- Reschedules unpaid commands with
RetryAfter(PENDING_PAYMENT_WAIT_TIME, …). - Cancels commands when the invoice is overdue, cancelled/final, or the command has waited at least 14 days.
Important distinction: successful provisioning does not appear to finalise the invoice. The invoice is a blocker for the create command; the API/payment path pays it before or while HostDaemon waits.
Failure and Recovery Behaviour
Unpaid Invoice / Pending Payment
Source: product/website/HostDaemon/BlockingHostCommandHandler.cs
Behaviour:
BlockedOnInvoice()returns true when the relevant invoice/subscription condition blocks execution.RescheduleOrCancel()decides whether to retry or cancel.- Retry path uses a 1-minute wait.
- Cancel path runs the handler's
Cancel()method.
Create guest cancellation: CreateGuestCommandHandler.Cancel() does the following:
- Ends the lease.
- Cancels the invoice when needed.
- Removes allocated resources from the guest.
- Sets guest status to
Terminated. - Commits the guest.
- Sends a guest-creation-cancelled email when appropriate.
Exception During Provisioning
Source: product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs
Entry point: HandleException().
What it does:
- Logs the failure.
- Writes a
CreateGuestFailuresystem action. - Clears
command.ResultDataso the generated root password is not recoverable from the failed command. - Attempts cleanup:
- unclean guest poweroff;
- primary disk removal.
- In a transaction:
- cancels and refunds the invoice where applicable;
- ends the current lease;
- cancels the guest;
- sets guest status to
Suspended; - commits the guest.
- Revokes licences.
Important distinction: failed creates are left suspended for later automatic cleanup rather than being deleted immediately.
Failure While Recording Command State
Source: product/website/HostDaemon/Hosted/FailedCommandQueue.cs
Behaviour:
ProcessCommandenqueues aFailedCommandwhen command loading or committing fails.FailedCommandQueue.Run()later:- loads the command;
- runs handler abort logic;
- calls
command.SetFailed()with exception/log output; - commits the state.
- If failure handling itself fails for non-database reasons, it logs and proceeds rather than retrying indefinitely.
Data / State Transitions
| Object | Initial / queued state | Successful state | Failure state |
|---|---|---|---|
Guest.Status | Building | Active | Suspended for provisioning exception; Terminated for invoice-cancel path |
HostCommand | not started / not completed | completed, succeeded | completed/failed where failure can be recorded |
Invoice | lease-term invoice, may require payment | paid by TryProcessInvoice() when payment succeeds | cancelled/refunded on provisioning failure or cancellation path |
Guest.Host | assigned before command execution | retained | resources may be removed/cancelled |
Guest.Interfaces | primary/additional assigned before command execution | retained | resources may be removed/cancelled |
Support-Relevant Failure Points
- Customer has unpaid or overdue invoices.
- API path:
CustomerCanPurchase()can block creation. - HostDaemon path:
BlockedOnInvoice()can block execution.
- Customer has no verified payment details or is not subscribed.
- API path checks
customer.Subscribed.
- Customer is at guest limit.
- Checked at API and domain level.
- Plan is inactive or not listable.
- Caught by
ValidateCreate().
- No host, network, or IP is available.
- Allocation services fail before HostDaemon starts the build.
- HostCommand is blocked by another active command.
TryStartCommand()checks blocking and duplicate guards.
- Payment cannot be processed.
TryProcessInvoice()returns false; HostDaemon later blocks on the invoice.
- HostDaemon cannot create disk, install OS, start guest, or reach guest by ping.
- Handler exception path suspends the guest and attempts cleanup.
- HostDaemon or database commit fails.
FailedCommandQueueattempts to record command failure later.
Source Files
product/website/WebApi/Services/GuestPlanApiService.csproduct/website/Domain/Services/GuestCreationService.csproduct/website/Domain/Services/GuestFactory.csproduct/website/Domain/Services/GuestProvisioningService.csproduct/website/Domain/Services/GuestManagementService.csproduct/website/Domain/Entities/CreateGuestCommand.csproduct/website/Domain/Services/HostCommandService.csproduct/website/Persistence/HostCommandBroker.csproduct/website/HostDaemon/Hosted/HostCommandThread.csproduct/website/HostDaemon/ProcessCommand.csproduct/website/HostDaemon/BlockingHostCommandHandler.csproduct/website/HostDaemon/Handlers/CreateGuestCommandHandler.csproduct/website/Domain/Services/InvoiceService.csproduct/website/HostDaemon/Hosted/FailedCommandQueue.cs
Remaining Questions
- Which customer-visible status label maps to internal
GuestStatus.Activein the current panel/API? - Which operational job performs automatic cleanup/termination of suspended failed-create guests, and after what delay?
- Which logs/queries should support use first when a customer reports a server stuck in
Building?
