Differences

This shows you the differences between two versions of the page.

--- syntheses:vps-create-flow [22/07/2026 00:43] – Format VPS create flow to match DokuWiki style guide bkirk
+++ syntheses:vps-create-flow [03/08/2026 23:16] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-====== VPS Creation / Provisioning Flow ======
-This page traces the source-backed flow for creating a VPS from the customer/API request through to HostDaemon provisioning on the compute host.
-===== Scope =====
-  * Main repo: ''vps/vps'' project 1.
-  * Main application area: ''product/website''.
-  * Compute execution: HostDaemon and host command handlers.
-  * Related repo: ''vps/compute'' project 22, for deployed compute-host programs and scripts.
-===== Short Version =====
-  - The API accepts a create-server request and validates the customer, plan, OS, network, billing state, and requested options.
-  - Domain services create a ''Guest'' in the database with status ''Building''.
-  - Provisioning services assign the lease, host, network, primary interface/IPs, backup schedule, and default firewall state.
-  - ''GuestManagementService.CreateGuest()'' creates a ''CreateGuestCommand''.
-  - HostDaemon polls the ''HostCommand'' table, marks the command started, and runs ''CreateGuestCommandHandler''.
-  - On success, HostDaemon sets the guest status to ''Active'', commits the guest, records actions/logs, sends mail, updates network members, and marks the command completed.
-  - On failure, HostDaemon attempts cleanup/refund/cancellation and leaves the guest ''Suspended'' or ''Terminated'' depending on where the failure occurred.
-===== API and Domain Flow =====
-==== 1. API Request ====
-''Source:'' ''product/website/WebApi/Services/GuestPlanApiService.cs''
-''Entry point:'' ''GuestPlanApiService.CreateServer()''.
-''What it does:''
-  * ''CustomerResourceService.GetCustomer()''
-    * Loads the customer entity, including subscription state and billing history.
-  * ''customer.Subscribed'' check
-    * Blocks creation if the customer lacks verified payment details.
-    * Signup flow can bypass this by calling with ''requireSubscribed = false''.
-  * ''ValidateCreate()''
-    * Checks purchase eligibility, guest limits, DTO fields, and plan availability.
-  * ''GetGuestCreationParams()''
-    * Resolves facility, plan, operating system, network, and extras into ''GuestCreationParameters''.
-  * ''_transactionRunner.Transaction()''
-    * Wraps domain guest creation so the main database changes are atomic.
-  * ''_guestCreationService.CreateNewGuestDomain()''
-    * Performs the domain-level guest creation described in the next step.
-  * NHibernate flush
-    * Forces the guest insert so the API ID exists before the response is returned.
-  * ''_invoiceService.TryProcessInvoice()''
-    * Runs after the domain transaction if the command has a blocking invoice that ''RequiresPayment()''.
-    * Attempts immediate payment using customer credit or stored payment details.
-    * If payment fails, the command still enters the queue and HostDaemon blocks on the unpaid invoice later.
-''Result:'' the API returns an action DTO for the created host command so the caller can poll build progress.
-==== 2. Guest/Domain Creation ====
-''Source:'' ''product/website/Domain/Services/GuestCreationService.cs''
-''Entry point:'' ''GuestCreationService.CreateNewGuestDomain()''.
-''What it does:''
-  * ''_invoiceService.CustomerCanPurchase()''
-    * Blocks customers with overdue invoices.
-  * Guest limit check
-    * Compares ''customer.GuestCount'' with ''customer.GuestLimit''.
-  * Private IP validation
-    * If a VPC private IP was requested, validates that it belongs to the requested network.
-  * Split NIC validation
-    * Requires a private network when ''SplitNics'' is requested.
-  * ''_guestFactory.InitializeNewGuest()''
-    * Creates the in-memory guest entity.
-    * Sets initial status to ''Building''.
-    * Sets plan/OS placeholders, generated name, hostname, and rescue password.
-  * ''_validationService.ValidateOrThrow(guest)''
-    * Runs entity validation before provisioning resources.
-  * Option application
-    * Applies port block, IPv6, split NICs, SSH keys, scripts/user-data, and duration settings.
-  * Referral and fraud-review handling
-    * Applies referral information where present.
-    * Marks the customer for review on the 10th server creation path.
-  * ''_guestProvisioningService.ProvisionNewGuest()''
-    * Allocates lease, host, network, interfaces, backup schedule, and firewall state.
-  * ''_guestService.Commit(guest)''
-    * Persists the configured guest.
-  * ''_guestManagementService.CreateGuest()''
-    * Queues the HostDaemon create command.
-''Result:'' a fully allocated database guest exists and a create command is queued.
-==== 3. Initial Guest State ====
-''Source:'' ''product/website/Domain/Services/GuestFactory.cs''
-''Entry point:'' ''GuestFactory.InitializeNewGuest()''.
-''What it sets:''
-  * ''Host = Host.Unset'', ''Storage = Storage.Unset'', and ''CurrentCpuModel = CpuModel.Unset'' until provisioning assigns real resources.
-  * A unique generated name and customer-facing hostname.
-  * Rescue password and OS-specific defaults through ''guest.InitializeFromOperatingSystem()''.
-  * ''NetnsEnabled'' based on customer compatibility.
-  * Initial status:
-<code>
-guest.SetStatus(GuestStatus.Building);
-</code>
-''Result:'' the guest starts life as ''Building'', not ''Running'' or ''Active''.
-==== 4. Provisioning Assignments ====
-''Source:'' ''product/website/Domain/Services/GuestProvisioningService.cs''
-''Entry point:'' ''ProvisionNewGuest()''.
-''Allocation sequence:''
-  - ''GuestLeaseCreationService.CreateNewLeaseForGuest()''
-    * Creates lease and lease-term records.
-    * Creates the invoice that can block command execution.
-  - ''HostService.AddGuestToAvailableHost()''
-    * Selects and assigns a compute host in the requested facility.
-  - ''NetworkService.AddGuestToNetwork()''
-    * Assigns the guest to the selected network zone.
-  - ''GuestService.AssignPrimaryInterfaceToGuest()''
-    * Creates the primary interface.
-    * Allocates public IPv4 and IPv6 where applicable.
-  - ''GuestService.AssignAdditionalInterfaces()''
-    * Adds VPC or split-NIC interfaces.
-  - ''BackupScheduleService.SetupGuestBackupSchedule()''
-    * Sets backup and offsite backup schedule state.
-  - ''SetupDefaultFirewall()''
-    * Clears or initialises firewall state for a fresh guest.
-''Result:'' the guest has database-side host, lease, network, interface, backup, and firewall assignments before HostDaemon starts the VM build.
-===== HostCommand / HostDaemon Flow =====
-==== 5. CreateGuestCommand Is Queued ====
-''Source:'' ''product/website/Domain/Services/GuestManagementService.cs''
-''Entry point:'' ''CreateGuest()''.
-''What it does:''
-  * Creates ''CreateGuestCommand'' through ''_hostCommandFactory.CreateCommand<CreateGuestCommand>(guest)''.
-  * Attaches build inputs:
-    * ''EmailVPSPassword''.
-    * Password override through ''WithPassword''.
-    * Deployment script and script input.
-    * SSH key IDs.
-    * User-data.
-  * Calls ''HostCommandService.AddHostCommand()''.
-    * Persists the command to the ''HostCommand'' table.
-    * Leaves it not started and not completed until HostDaemon picks it up.
-''Source:'' ''product/website/Domain/Entities/CreateGuestCommand.cs''
-''Customer-visible command steps:''
-  - Requesting server resources.
-  - Allocating virtual disk.
-  - Installing operating system.
-  - Starting server.
-  - Configuring operating system.
-  - Processing invoice.
-''Invoice blocker:''
-  * ''CreateGuestCommand'' implements ''IInvoiceBlockableCommand''.
-  * Its blocking invoice is ''Guest.LatestLeaseTerm.Invoice''.
-  * HostDaemon will not build prepaid guests until that invoice is paid.
-  * For postpaid customers, the invoice gate depends on subscription state.
-==== 6. HostDaemon Picks Up The Command ====
-''Source:'' ''product/website/HostDaemon/Hosted/HostCommandThread.cs''
-''Polling sequence:''
-  - ''GetNextCommand()''
-    * Opens a database transaction.
-    * Calls ''broker.GetNextCommand()'' to find the next unstarted, unblocked command.
-    * Calls ''service.TryStartCommand()'' to mark it started atomically.
-  - ''TryStartCommand()'' guards
-    * Checks ''IsBlockedBy()''.
-    * Checks ''IsDuplicateOf()''.
-    * Prevents concurrent execution of incompatible commands for the same guest.
-  - Query-duration guard
-    * If acquiring a database connection takes too long, the command is skipped for that pass.
-    * This avoids starving already-running handlers when the pool is contended.
-  - Worker start
-    * Starts ''new Thread(new ProcessCommand(...).Run).Start()''.
-    * Some Windows host paths can run commands in a separate process.
-''Polling behaviour:'' HostDaemon adapts its sleep duration based on database responsiveness. If no command is found, it rests long enough to avoid tight-loop polling.
-==== 7. HostDaemon Creates The Guest ====
-''Source:'' ''product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs''
-''Entry point:'' ''CreateGuestCommandHandler.Handle()''.
-''Base class:'' ''BlockingHostCommandHandler<CreateGuestCommand>'' handles the invoice gate before host-side work begins.
-''Build sequence:''
-  - Step 0: Invoice gate
-    * Calls ''base.Handle(command)''.
-    * Checks ''command.BlockingInvoice()'' through ''BlockedOnInvoice()''.
-    * For prepaid invoices, blocks until ''invoice.Paid''.
-    * For postpaid invoices, blocks only when the customer is not subscribed.
-    * If blocked, returns without touching the guest.
-  - Step 1: Request server resources
-    * Connects to the compute host shell through ''ConnectShellToGuest()''.
-    * Checks banned-customer conditions.
-    * Aborts if a disk already exists for this guest.
-  - Step 2: Allocate virtual disk
-    * Calls ''CreateVirtualDisk()''.
-    * Issues licences early through ''ChangeLicensesFromTo()'' so activation can complete before OS install.
-  - Step 3: Install operating system
-    * Calls ''InstallOperatingSystem()'' on the compute host.
-    * Configures time machine if enabled.
-  - Step 4: Start server
-    * Boots the VM on the compute host.
-  - Step 5: Configure operating system
-    * Waits for the guest to respond to ping.
-    * Installs SSH keys.
-    * Runs deployment script/user-data.
-    * Sets the password.
-  - Step 6: Finalise successful create
-    * Calls ''FlagGuestAsCreated()''.
-    * Sends relevant customer emails.
-    * Updates network membership state.
-''Successful status transition:''
-<code>
-command.Guest.SetStatus(GuestStatus.Active);
-_guestService.Commit(command.Guest);
-</code>
-''Result:'' the source-backed transition is ''Building'' to ''Active'', not ''Running''.
-==== 8. Command Completion ====
-''Source:'' ''product/website/HostDaemon/ProcessCommand.cs''
-''Entry point:'' ''ProcessCommand.Run()''.
-''What it does:''
-  * Loads the command from the database with ''broker.GetById()''.
-  * Resolves the correct handler through ''uow.GetHandlerFor(command)''.
-  * Calls ''handler.Handle()''.
-  * On success:
-    * Calls ''command.SetCompleted()''.
-    * Commits the command.
-    * Calls ''handler.Completed()''.
-    * Commits again for post-completion changes.
-  * On handler failure without an exception:
-    * Saves structured ''ExceptionSummary'' and log output.
-  * On load or commit failure:
-    * Enqueues a ''FailedCommand'' for later recovery.
-''Result:'' HostDaemon writes command state directly to the shared database. There is no separate HTTP callback to the API in this path.
-===== Billing Behaviour =====
-==== API-Side Payment Attempt ====
-''Source:'' ''product/website/Domain/Services/InvoiceService.cs''
-''Entry point:'' ''InvoiceService.TryProcessInvoice()''.
-''What it does:''
-  * Refreshes the customer entity.
-  * Returns early if the invoice does not require payment.
-  * Marks the invoice ready for sending.
-  * Handles zero-dollar invoices.
-  * Uses customer credit first.
-  * Generates a credit-card deposit if needed and possible.
-  * Preserves invoice sent state in credit-only failure cases to avoid duplicate payable-now emails.
-  * Calls ''invoice.MarkAsPaid()'' on success.
-==== HostDaemon Invoice Gate ====
-''Source:'' ''product/website/HostDaemon/BlockingHostCommandHandler.cs''
-''What it does:''
-  * Checks the command's blocking invoice before the handler performs host-side work.
-  * Blocks prepaid invoices until paid.
-  * Allows subscribed postpaid customers through when the invoice is not final and unpaid.
-  * Reschedules unpaid commands with ''RetryAfter(PENDING_PAYMENT_WAIT_TIME, ...)''.
-  * Cancels commands when the invoice is overdue, cancelled/final, or the command has waited at least 14 days.
-''Important distinction:'' successful provisioning does not appear to finalise the invoice. The invoice is a blocker for the create command; the API/payment path pays it before or while HostDaemon waits.
-===== Failure and Recovery Behaviour =====
-==== Unpaid Invoice / Pending Payment ====
-''Source:'' ''product/website/HostDaemon/BlockingHostCommandHandler.cs''
-''Behaviour:''
-  * ''BlockedOnInvoice()'' returns true when the relevant invoice/subscription condition blocks execution.
-  * ''RescheduleOrCancel()'' decides whether to retry or cancel.
-  * Retry path uses a 1-minute wait.
-  * Cancel path runs the handler's ''Cancel()'' method.
-''Create guest cancellation:'' ''CreateGuestCommandHandler.Cancel()'' does the following:
-  * Ends the lease.
-  * Cancels the invoice when needed.
-  * Removes allocated resources from the guest.
-  * Sets guest status to ''Terminated''.
-  * Commits the guest.
-  * Sends a guest-creation-cancelled email when appropriate.
-==== Exception During Provisioning ====
-''Source:'' ''product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs''
-''Entry point:'' ''HandleException()''.
-''What it does:''
-  * Logs the failure.
-  * Writes a ''CreateGuestFailure'' system action.
-  * Clears ''command.ResultData'' so the generated root password is not recoverable from the failed command.
-  * Attempts cleanup:
-    * unclean guest poweroff;
-    * primary disk removal.
-  * In a transaction:
-    * cancels and refunds the invoice where applicable;
-    * ends the current lease;
-    * cancels the guest;
-    * sets guest status to ''Suspended'';
-    * commits the guest.
-  * Revokes licences.
-''Important distinction:'' failed creates are left suspended for later automatic cleanup rather than being deleted immediately.
-==== Failure While Recording Command State ====
-''Source:'' ''product/website/HostDaemon/Hosted/FailedCommandQueue.cs''
-''Behaviour:''
-  * ''ProcessCommand'' enqueues a ''FailedCommand'' when command loading or committing fails.
-  * ''FailedCommandQueue.Run()'' later:
-    * loads the command;
-    * runs handler abort logic;
-    * calls ''command.SetFailed()'' with exception/log output;
-    * commits the state.
-  * If failure handling itself fails for non-database reasons, it logs and proceeds rather than retrying indefinitely.
-===== Data / State Transitions =====
-^ Object ^ Initial / queued state ^ Successful state ^ Failure state ^
-| ''Guest.Status'' | ''Building'' | ''Active'' | ''Suspended'' for provisioning exception; ''Terminated'' for invoice-cancel path |
-| ''HostCommand'' | not started / not completed | completed, succeeded | completed/failed where failure can be recorded |
-| ''Invoice'' | lease-term invoice, may require payment | paid by ''TryProcessInvoice()'' when payment succeeds | cancelled/refunded on provisioning failure or cancellation path |
-| ''Guest.Host'' | assigned before command execution | retained | resources may be removed/cancelled |
-| ''Guest.Interfaces'' | primary/additional assigned before command execution | retained | resources may be removed/cancelled |
-===== Support-Relevant Failure Points =====
-  * Customer has unpaid or overdue invoices.
-    * API path: ''CustomerCanPurchase()'' can block creation.
-    * HostDaemon path: ''BlockedOnInvoice()'' can block execution.
-  * Customer has no verified payment details or is not subscribed.
-    * API path checks ''customer.Subscribed''.
-  * Customer is at guest limit.
-    * Checked at API and domain level.
-  * Plan is inactive or not listable.
-    * Caught by ''ValidateCreate()''.
-  * No host, network, or IP is available.
-    * Allocation services fail before HostDaemon starts the build.
-  * HostCommand is blocked by another active command.
-    * ''TryStartCommand()'' checks blocking and duplicate guards.
-  * Payment cannot be processed.
-    * ''TryProcessInvoice()'' returns false; HostDaemon later blocks on the invoice.
-  * HostDaemon cannot create disk, install OS, start guest, or reach guest by ping.
-    * Handler exception path suspends the guest and attempts cleanup.
-  * HostDaemon or database commit fails.
-    * ''FailedCommandQueue'' attempts to record command failure later.
-===== Source Files =====
-  * ''product/website/WebApi/Services/GuestPlanApiService.cs''
-  * ''product/website/Domain/Services/GuestCreationService.cs''
-  * ''product/website/Domain/Services/GuestFactory.cs''
-  * ''product/website/Domain/Services/GuestProvisioningService.cs''
-  * ''product/website/Domain/Services/GuestManagementService.cs''
-  * ''product/website/Domain/Entities/CreateGuestCommand.cs''
-  * ''product/website/Domain/Services/HostCommandService.cs''
-  * ''product/website/Persistence/HostCommandBroker.cs''
-  * ''product/website/HostDaemon/Hosted/HostCommandThread.cs''
-  * ''product/website/HostDaemon/ProcessCommand.cs''
-  * ''product/website/HostDaemon/BlockingHostCommandHandler.cs''
-  * ''product/website/HostDaemon/Handlers/CreateGuestCommandHandler.cs''
-  * ''product/website/Domain/Services/InvoiceService.cs''
-  * ''product/website/HostDaemon/Hosted/FailedCommandQueue.cs''
-===== Remaining Questions =====
-  * Which customer-visible status label maps to internal ''GuestStatus.Active'' in the current panel/API?
-  * Which operational job performs automatic cleanup/termination of suspended failed-create guests, and after what delay?
-  * Which logs/queries should support use first when a customer reports a server stuck in ''Building''?
-===== Related Wiki Pages =====
-  * [[syntheses:compute-host-stack|Compute Host Stack]]
-  * [[syntheses:compute-repo-structure|vps/compute Repo Structure]]
-  * [[syntheses:infrastructure-services|Infrastructure Services]]
-  * [[wiki:jargon|Jargon / Reference Index]]
-  * [[wiki:style-guide|DokuWiki Style Guide]]