This is an old revision of the document!
Table of Contents
Customer-Side Bug Finding Methodology
Notes
Summary
Customer-Side Bug Finding Methodology
Purpose
Identify bugs that a customer (authenticated user of the panel, API, or CLI) can trigger, from a customer's perspective. NOT internal admin bugs or ops issues.
Rule: External vs Internal
EXTERNAL (customer-triggerable):
- Any endpoint in WebApi, PanelSite, or ClassicPanel that accepts customer input
- Any validation layer that processes customer-provided data
- Any domain entity that customers can modify (Guest, FirewallRule, DomainName, etc.)
- The chain: Customer → WebApi/Panel/CLI → Domain validation → NHibernate → HostCommand → HostDaemon → iptables
INTERNAL (not customer-triggerable):
- AdminSite (staff-only)
- HostDaemon running on compute hosts (customers can't SSH to these)
- StatsDaemon, Icinga, Tron, Ansible
- Internal ops scripts and provisioning tooling
- Third-party forks
How to Find Customer-Triggerable Bugs
Pattern 1: Unbounded Input Collection
Search for foreach loops over customer-controlled collections without .Take() or count caps.
- Example found: GuestFirewallRule.GetIptablesEntries() — 4-level nested loop over source/dest/protocol/port with no bounds
- Where to look: Every POST/PUT endpoint that accepts a
List<T>from customer input
Pattern 2: Per-Item Validation Without Per-Collection Validation
Search for validators that check individual items but not the collection itself.
- Example found: ValidateAddressListAttribute validates each address but not how many addresses are in the list
- Where to look: Domain/Validators/ directory, especially attributes used on collection properties
Pattern 3: Cartesian Product Expansion
Search for nested foreach loops that multiply input sizes.
- Formula: If input A has N items and input B has M items, the loop runs N×M times
- Where to look: Any method that iterates over multiple customer-provided collections
- Key file: GuestFirewallRule.cs lines 78-86 (source × destination × protocol × port)
Pattern 4: Lock Hold During Expensive Operations
Search for lock statements or claim_lock patterns that hold during long-running operations.
- Example found: qemu-network/mammothvps.sh uses claim_lock “iptables” held during the entire apply phase
- Impact: One customer's slow operation blocks all other guests on the same host
- Where to look: HostDaemon, qemu-network scripts, any code with “lock” or “claim”
Pattern 5: Missing Idempotency on Mutations
Search for POST endpoints that don't check for duplicate execution.
- Impact: Double-clicking “reinstall” or “restart” could cause unexpected state
- Where to look: WebApi controllers, ClassicPanel POST endpoints
Pattern 6: Race Conditions in Multi-Step Transactions
Search for operations that span multiple steps without atomic guarantees.
- Examples found: ApproveAccount/ProcessDomainNameCommand race (#1635), PayPal register-then-capture (#1863)
- Where to look: Domain/Commands/ files, especially those handling money or domains
Pattern 7: String Parsing Without Size Limits
Search for Split(), Parse(), Deserialize() on customer input without size checks.
- Impact: Memory exhaustion from oversized input
- Where to look: GuestFirewallRule.FromJson(), FromDictionary(), any endpoint accepting raw JSON strings
Pattern 8: Missing Null/Default Checks on Nullable Collections
Search for .Count or .Any() on collections that could be null.
- Where to look: Entity classes, especially those with virtual collections
Quick Search Commands for Bug Hunting
From the vps/vps repo, search for these patterns:
# Unbounded foreach loops in customer-facing code grep -rn "foreach.*FirewallRule\|foreach.*guest\|foreach.*customer" product/website/Domain/ product/website/WebApi/ product/website/ClassicPanel/ # Per-item validation without count limits grep -rn "ValidateAddressList\|ValidationResult" product/website/Domain/Validators/ # Nested loops (potential Cartesian products) grep -rn "foreach.*foreach" product/website/Domain/Entities/ # Lock/claim patterns grep -rn "claim_lock\|lock.*iptables\|lock.*guest" product/website/HostDaemon/ infrastructure/ # Missing .Take() or count caps on collections grep -rn "\.Add(.*rule\|\.Add(.*customer" product/website/Domain/Services/ # POST endpoints without idempotency grep -rn "Verb.Post\|HttpPost" product/website/ClassicPanel/Controllers/ product/website/WebApi/Controllers/ | grep -v "Get\|Idempotent\|Check"
High-Value Targets for Customer-Side Bug Hunting
Firewall Rules (highest ROI)
- Attack vector: Submit massive firewall ruleset via API or ClassicPanel import
- Impact: Stalls all guests on the same compute host (known issue #1640)
- What to test: Create a guest, add 1000+ firewall rules with 100+ addresses each via API. Monitor host load and other guest network operations.
Guest Creation/Provisioning
- Attack vector: Create many guests simultaneously, or create with unusual configurations
- Impact: Race conditions in provisioning (#1863), stale state (#1884)
- What to test: Concurrent guest creation with same payment method, unusual OS/plan combinations
Domain Registration
- Attack vector: Register domain + create account simultaneously
- Impact: ApproveAccount/ProcessDomainNameCommand race (#1635, #1863)
- What to test: Simultaneous account creation with domain registration, rapid cancel/re-register
Payment/Billing
- Attack vector: Modify payment method while Mpay is slow/unavailable
- Impact: Token-delete timeout (#1811), billing on wrong card (#1891)
- What to test: Kill Mpay service mid-transaction, rapid card add/Remove cycles
Guest Power Operations
- Attack vector: Rapid start/stop/reinstall cycles
- Impact: State corruption during HostCommand processing
- What to test: Power cycle guest 10x rapidly while it's being migrated
DNS/Forward DNS
- Attack vector: Add many DNS records, or malformed DNS entries
- Impact: DNS propagation failures, record loss
- What to test: 500+ DNS records per domain, unusual record types
Testing Strategy
- Isolated environment: Use a single guest on a known host (check via API which host a guest is on)
- Observe side effects: Monitor OTHER guests on the same host during your test
- Measure, don't just observe: Note response times, error rates, CPU usage
- Reproduce deterministically: If it happens once, can you make it happen again?
- Document: Record exact steps, inputs, and observed behavior
What NOT to Test
- Anything requiring admin panel access
- Anything requiring SSH to compute hosts
- Production customer data
- Anything that could cause data loss without approval
- Database schema changes
