User Tools

Site Tools


syntheses:vps-create-flow

This is an old revision of the document!


VPS Creation / Provisioning Flow

This page traces the source-backed flow for creating a VPS from the customer/API request through to HostDaemon provisioning on the compute host.

Scope

  • Main repo: vps/vps project 1.
  • Main application area: product/website.
  • Compute execution: HostDaemon and host command handlers.
  • Related repo: vps/compute project 22, for deployed compute-host programs and scripts.

Short Version

  1. The API accepts a create-server request and validates the customer, plan, OS, network, billing state, and requested options.
  2. Domain services create a Guest in the database with status Building.
  3. Provisioning services assign the lease, host, network, primary interface/IPs, backup schedule, and default firewall state.
  4. GuestManagementService.CreateGuest() creates a CreateGuestCommand.
  5. HostDaemon polls the HostCommand table, marks the command started, and runs CreateGuestCommandHandler.
  6. On success, HostDaemon sets the guest status to Active, commits the guest, records actions/logs, sends mail, updates network members, and marks the command completed.
  7. On failure, HostDaemon attempts cleanup/refund/cancellation and leaves the guest Suspended or Terminated depending on where the failure occurred.

API and Domain Flow

1. API Request

Source: product/website/WebApi/Services/GuestPlanApiService.cs

Entry point: GuestPlanApiService.CreateServer().

What it does:

  • CustomerResourceService.GetCustomer()
    • Loads the customer entity, including subscription state and billing history.
  • customer.Subscribed check
    • Blocks creation if the customer lacks verified payment details.
    • Signup flow can bypass this by calling with requireSubscribed = false.
  • ValidateCreate()
    • Checks purchase eligibility, guest limits, DTO fields, and plan availability.
  • GetGuestCreationParams()
    • Resolves facility, plan, operating system, network, and extras into GuestCreationParameters.
  • _transactionRunner.Transaction()
    • Wraps domain guest creation so the main database changes are atomic.
  • _guestCreationService.CreateNewGuestDomain()
    • Performs the domain-level guest creation described in the next step.
  • NHibernate flush
    • Forces the guest insert so the API ID exists before the response is returned.
  • _invoiceService.TryProcessInvoice()
    • Runs after the domain transaction if the command has a blocking invoice that RequiresPayment().
    • Attempts immediate payment using customer credit or stored payment details.
    • If payment fails, the command still enters the queue and HostDaemon blocks on the unpaid invoice later.

Result: the API returns an action DTO for the created host command so the caller can poll build progress.

2. Guest/Domain Creation

Source: product/website/Domain/Services/GuestCreationService.cs

Entry point: GuestCreationService.CreateNewGuestDomain().

What it does:

  • _invoiceService.CustomerCanPurchase()
    • Blocks customers with overdue invoices.
  • Guest limit check
    • Compares customer.GuestCount with customer.GuestLimit.
  • Private IP validation
    • If a VPC private IP was requested, validates that it belongs to the requested network.
  • Split NIC validation
    • Requires a private network when SplitNics is requested.
  • _guestFactory.InitializeNewGuest()
    • Creates the in-memory guest entity.
    • Sets initial status to Building.
    • Sets plan/OS placeholders, generated name, hostname, and rescue password.
  • _validationService.ValidateOrThrow(guest)
    • Runs entity validation before provisioning resources.
  • Option application
    • Applies port block, IPv6, split NICs, SSH keys, scripts/user-data, and duration settings.
  • Referral and fraud-review handling
    • Applies referral information where present.
    • Marks the customer for review on the 10th server creation path.
  • _guestProvisioningService.ProvisionNewGuest()
    • Allocates lease, host, network, interfaces, backup schedule, and firewall state.
  • _guestService.Commit(guest)
    • Persists the configured guest.
  • _guestManagementService.CreateGuest()
    • Queues the HostDaemon create command.

Result: a fully allocated database guest exists and a create command is queued.

3. Initial Guest State

Source: product/website/Domain/Services/GuestFactory.cs

Entry point: GuestFactory.InitializeNewGuest().

What it sets:

  • Host = Host.Unset, Storage = Storage.Unset, and CurrentCpuModel = CpuModel.Unset until provisioning assigns real resources.
  • A unique generated name and customer-facing hostname.
  • Rescue password and OS-specific defaults through guest.InitializeFromOperatingSystem().
  • NetnsEnabled based on customer compatibility.
  • Initial status:
guest.SetStatus(GuestStatus.Building);

Result: the guest starts life as Building, not Running or Active.

4. Provisioning Assignments

Source: product/website/Domain/Services/GuestProvisioningService.cs

Entry point: ProvisionNewGuest().

Allocation sequence:

  1. GuestLeaseCreationService.CreateNewLeaseForGuest()
    • Creates lease and lease-term records.
    • Creates the invoice that can block command execution.
  2. HostService.AddGuestToAvailableHost()
    • Selects and assigns a compute host in the requested facility.
  3. NetworkService.AddGuestToNetwork()
    • Assigns the guest to the selected network zone.
  4. GuestService.AssignPrimaryInterfaceToGuest()
    • Creates the primary interface.
    • Allocates public IPv4 and IPv6 where applicable.
  5. GuestService.AssignAdditionalInterfaces()
    • Adds VPC or split-NIC interfaces.
  6. BackupScheduleService.SetupGuestBackupSchedule()
    • Sets backup and offsite backup schedule state.
  7. SetupDefaultFirewall()
    • Clears or initialises firewall state for a fresh guest.

Result: the guest has database-side host, lease, network, interface, backup, and firewall assignments before HostDaemon starts the VM build.

HostCommand / HostDaemon Flow

5. CreateGuestCommand Is Queued

Source: product/website/Domain/Services/GuestManagementService.cs

Entry point: CreateGuest().

What it does:

  • Creates CreateGuestCommand through _hostCommandFactory.CreateCommand<CreateGuestCommand>(guest).
  • Attaches build inputs:
    • EmailVPSPassword.
    • Password override through WithPassword.
    • Deployment script and script input.
    • SSH key IDs.
    • User-data.
  • Calls HostCommandService.AddHostCommand().
    • Persists the command to the HostCommand table.
    • Leaves it not started and not completed until HostDaemon picks it up.

Source: product/website/Domain/Entities/CreateGuestCommand.cs

Customer-visible command steps:

  1. Requesting server resources.
  2. Allocating virtual disk.
  3. Installing operating system.
  4. Starting server.
  5. Configuring operating system.
  6. Processing invoice.

Invoice blocker:

  • CreateGuestCommand implements IInvoiceBlockableCommand.
  • Its blocking invoice is Guest.LatestLeaseTerm.Invoice.
  • HostDaemon will not build prepaid guests until that invoice is paid.
  • For postpaid customers, the invoice gate depends on subscription state.

6. HostDaemon Picks Up The Command

Source: product/website/HostDaemon/Hosted/HostCommandThread.cs

Polling sequence:

  1. GetNextCommand()
    • Opens a database transaction.
    • Calls broker.GetNextCommand() to find the next unstarted, unblocked command.
    • Calls service.TryStartCommand() to mark it started atomically.
  2. TryStartCommand() guards
    • Checks IsBlockedBy().
    • Checks IsDuplicateOf().
    • Prevents concurrent execution of incompatible commands for the same guest.
  3. Query-duration guard
    • If acquiring a database connection takes too long, the command is skipped for that pass.
    • This avoids starving already-running handlers when the pool is contended.
  4. Worker start
    • Starts new Thread(new ProcessCommand(…).Run).Start().
    • Some Windows host paths can run commands in a separate process.

Polling behaviour: HostDaemon adapts its sleep duration based on database responsiveness. If no command is found, it rests long enough to avoid tight-loop polling.

7. HostDaemon Creates The Guest

Source: product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs

Entry point: CreateGuestCommandHandler.Handle().

Base class: BlockingHostCommandHandler<CreateGuestCommand> handles the invoice gate before host-side work begins.

Build sequence:

  1. Step 0: Invoice gate
    • Calls base.Handle(command).
    • Checks command.BlockingInvoice() through BlockedOnInvoice().
    • For prepaid invoices, blocks until invoice.Paid.
    • For postpaid invoices, blocks only when the customer is not subscribed.
    • If blocked, returns without touching the guest.
  2. Step 1: Request server resources
    • Connects to the compute host shell through ConnectShellToGuest().
    • Checks banned-customer conditions.
    • Aborts if a disk already exists for this guest.
  3. Step 2: Allocate virtual disk
    • Calls CreateVirtualDisk().
    • Issues licences early through ChangeLicensesFromTo() so activation can complete before OS install.
  4. Step 3: Install operating system
    • Calls InstallOperatingSystem() on the compute host.
    • Configures time machine if enabled.
  5. Step 4: Start server
    • Boots the VM on the compute host.
  6. Step 5: Configure operating system
    • Waits for the guest to respond to ping.
    • Installs SSH keys.
    • Runs deployment script/user-data.
    • Sets the password.
  7. Step 6: Finalise successful create
    • Calls FlagGuestAsCreated().
    • Sends relevant customer emails.
    • Updates network membership state.

Successful status transition:

command.Guest.SetStatus(GuestStatus.Active);
_guestService.Commit(command.Guest);

Result: the source-backed transition is Building to Active, not Running.

8. Command Completion

Source: product/website/HostDaemon/ProcessCommand.cs

Entry point: ProcessCommand.Run().

What it does:

  • Loads the command from the database with broker.GetById().
  • Resolves the correct handler through uow.GetHandlerFor(command).
  • Calls handler.Handle().
  • On success:
    • Calls command.SetCompleted().
    • Commits the command.
    • Calls handler.Completed().
    • Commits again for post-completion changes.
  • On handler failure without an exception:
    • Saves structured ExceptionSummary and log output.
  • On load or commit failure:
    • Enqueues a FailedCommand for later recovery.

Result: HostDaemon writes command state directly to the shared database. There is no separate HTTP callback to the API in this path.

Billing Behaviour

API-Side Payment Attempt

Source: product/website/Domain/Services/InvoiceService.cs

Entry point: InvoiceService.TryProcessInvoice().

What it does:

  • Refreshes the customer entity.
  • Returns early if the invoice does not require payment.
  • Marks the invoice ready for sending.
  • Handles zero-dollar invoices.
  • Uses customer credit first.
  • Generates a credit-card deposit if needed and possible.
  • Preserves invoice sent state in credit-only failure cases to avoid duplicate payable-now emails.
  • Calls invoice.MarkAsPaid() on success.

HostDaemon Invoice Gate

Source: product/website/HostDaemon/BlockingHostCommandHandler.cs

What it does:

  • Checks the command's blocking invoice before the handler performs host-side work.
  • Blocks prepaid invoices until paid.
  • Allows subscribed postpaid customers through when the invoice is not final and unpaid.
  • Reschedules unpaid commands with RetryAfter(PENDING_PAYMENT_WAIT_TIME, …).
  • Cancels commands when the invoice is overdue, cancelled/final, or the command has waited at least 14 days.

Important distinction: successful provisioning does not appear to finalise the invoice. The invoice is a blocker for the create command; the API/payment path pays it before or while HostDaemon waits.

Failure and Recovery Behaviour

Unpaid Invoice / Pending Payment

Source: product/website/HostDaemon/BlockingHostCommandHandler.cs

Behaviour:

  • BlockedOnInvoice() returns true when the relevant invoice/subscription condition blocks execution.
  • RescheduleOrCancel() decides whether to retry or cancel.
  • Retry path uses a 1-minute wait.
  • Cancel path runs the handler's Cancel() method.

Create guest cancellation: CreateGuestCommandHandler.Cancel() does the following:

  • Ends the lease.
  • Cancels the invoice when needed.
  • Removes allocated resources from the guest.
  • Sets guest status to Terminated.
  • Commits the guest.
  • Sends a guest-creation-cancelled email when appropriate.

Exception During Provisioning

Source: product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs

Entry point: HandleException().

What it does:

  • Logs the failure.
  • Writes a CreateGuestFailure system action.
  • Clears command.ResultData so the generated root password is not recoverable from the failed command.
  • Attempts cleanup:
    • unclean guest poweroff;
    • primary disk removal.
  • In a transaction:
    • cancels and refunds the invoice where applicable;
    • ends the current lease;
    • cancels the guest;
    • sets guest status to Suspended;
    • commits the guest.
  • Revokes licences.

Important distinction: failed creates are left suspended for later automatic cleanup rather than being deleted immediately.

Failure While Recording Command State

Source: product/website/HostDaemon/Hosted/FailedCommandQueue.cs

Behaviour:

  • ProcessCommand enqueues a FailedCommand when command loading or committing fails.
  • FailedCommandQueue.Run() later:
    • loads the command;
    • runs handler abort logic;
    • calls command.SetFailed() with exception/log output;
    • commits the state.
  • If failure handling itself fails for non-database reasons, it logs and proceeds rather than retrying indefinitely.

Data / State Transitions

Object Initial / queued state Successful state Failure state
Guest.Status Building Active Suspended for provisioning exception; Terminated for invoice-cancel path
HostCommand not started / not completed completed, succeeded completed/failed where failure can be recorded
Invoice lease-term invoice, may require payment paid by TryProcessInvoice() when payment succeeds cancelled/refunded on provisioning failure or cancellation path
Guest.Host assigned before command execution retained resources may be removed/cancelled
Guest.Interfaces primary/additional assigned before command execution retained resources may be removed/cancelled

Support-Relevant Failure Points

  • Customer has unpaid or overdue invoices.
    • API path: CustomerCanPurchase() can block creation.
    • HostDaemon path: BlockedOnInvoice() can block execution.
  • Customer has no verified payment details or is not subscribed.
    • API path checks customer.Subscribed.
  • Customer is at guest limit.
    • Checked at API and domain level.
  • Plan is inactive or not listable.
    • Caught by ValidateCreate().
  • No host, network, or IP is available.
    • Allocation services fail before HostDaemon starts the build.
  • HostCommand is blocked by another active command.
    • TryStartCommand() checks blocking and duplicate guards.
  • Payment cannot be processed.
    • TryProcessInvoice() returns false; HostDaemon later blocks on the invoice.
  • HostDaemon cannot create disk, install OS, start guest, or reach guest by ping.
    • Handler exception path suspends the guest and attempts cleanup.
  • HostDaemon or database commit fails.
    • FailedCommandQueue attempts to record command failure later.

Source Files

  • product/website/WebApi/Services/GuestPlanApiService.cs
  • product/website/Domain/Services/GuestCreationService.cs
  • product/website/Domain/Services/GuestFactory.cs
  • product/website/Domain/Services/GuestProvisioningService.cs
  • product/website/Domain/Services/GuestManagementService.cs
  • product/website/Domain/Entities/CreateGuestCommand.cs
  • product/website/Domain/Services/HostCommandService.cs
  • product/website/Persistence/HostCommandBroker.cs
  • product/website/HostDaemon/Hosted/HostCommandThread.cs
  • product/website/HostDaemon/ProcessCommand.cs
  • product/website/HostDaemon/BlockingHostCommandHandler.cs
  • product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs
  • product/website/Domain/Services/InvoiceService.cs
  • product/website/HostDaemon/Hosted/FailedCommandQueue.cs

Remaining Questions

  • Which customer-visible status label maps to internal GuestStatus.Active in the current panel/API?
  • Which operational job performs automatic cleanup/termination of suspended failed-create guests, and after what delay?
  • Which logs/queries should support use first when a customer reports a server stuck in Building?
syntheses/vps-create-flow.1784681017.txt.gz · Last modified: by bkirk

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki