User Tools

Site Tools


syntheses:customer-side-bug-finding-methodology

This is an old revision of the document!


Customer-Side Bug Finding Methodology

Notes

Summary

Customer-Side Bug Finding Methodology

Purpose

Identify bugs that a customer (authenticated user of the panel, API, or CLI) can trigger, from a customer's perspective. NOT internal admin bugs or ops issues.

Rule: External vs Internal

EXTERNAL (customer-triggerable):

  • Any endpoint in WebApi, PanelSite, or ClassicPanel that accepts customer input
  • Any validation layer that processes customer-provided data
  • Any domain entity that customers can modify (Guest, FirewallRule, DomainName, etc.)
  • The chain: Customer → WebApi/Panel/CLI → Domain validation → NHibernate → HostCommand → HostDaemon → iptables

INTERNAL (not customer-triggerable):

  • AdminSite (staff-only)
  • HostDaemon running on compute hosts (customers can't SSH to these)
  • StatsDaemon, Icinga, Tron, Ansible
  • Internal ops scripts and provisioning tooling
  • Third-party forks

How to Find Customer-Triggerable Bugs

Pattern 1: Unbounded Input Collection

Search for foreach loops over customer-controlled collections without .Take() or count caps.

  • Example found: GuestFirewallRule.GetIptablesEntries() — 4-level nested loop over source/dest/protocol/port with no bounds
  • Where to look: Every POST/PUT endpoint that accepts a List<T> from customer input

Pattern 2: Per-Item Validation Without Per-Collection Validation

Search for validators that check individual items but not the collection itself.

  • Example found: ValidateAddressListAttribute validates each address but not how many addresses are in the list
  • Where to look: Domain/Validators/ directory, especially attributes used on collection properties

Pattern 3: Cartesian Product Expansion

Search for nested foreach loops that multiply input sizes.

  • Formula: If input A has N items and input B has M items, the loop runs N×M times
  • Where to look: Any method that iterates over multiple customer-provided collections
  • Key file: GuestFirewallRule.cs lines 78-86 (source × destination × protocol × port)

Pattern 4: Lock Hold During Expensive Operations

Search for lock statements or claim_lock patterns that hold during long-running operations.

  • Example found: qemu-network/mammothvps.sh uses claim_lock “iptables” held during the entire apply phase
  • Impact: One customer's slow operation blocks all other guests on the same host
  • Where to look: HostDaemon, qemu-network scripts, any code with “lock” or “claim”

Pattern 5: Missing Idempotency on Mutations

Search for POST endpoints that don't check for duplicate execution.

  • Impact: Double-clicking “reinstall” or “restart” could cause unexpected state
  • Where to look: WebApi controllers, ClassicPanel POST endpoints

Pattern 6: Race Conditions in Multi-Step Transactions

Search for operations that span multiple steps without atomic guarantees.

  • Examples found: ApproveAccount/ProcessDomainNameCommand race (#1635), PayPal register-then-capture (#1863)
  • Where to look: Domain/Commands/ files, especially those handling money or domains

Pattern 7: String Parsing Without Size Limits

Search for Split(), Parse(), Deserialize() on customer input without size checks.

  • Impact: Memory exhaustion from oversized input
  • Where to look: GuestFirewallRule.FromJson(), FromDictionary(), any endpoint accepting raw JSON strings

Pattern 8: Missing Null/Default Checks on Nullable Collections

Search for .Count or .Any() on collections that could be null.

  • Where to look: Entity classes, especially those with virtual collections

Quick Search Commands for Bug Hunting

From the vps/vps repo, search for these patterns:

# Unbounded foreach loops in customer-facing code
grep -rn "foreach.*FirewallRule\|foreach.*guest\|foreach.*customer" product/website/Domain/ product/website/WebApi/ product/website/ClassicPanel/
 
# Per-item validation without count limits
grep -rn "ValidateAddressList\|ValidationResult" product/website/Domain/Validators/
 
# Nested loops (potential Cartesian products)
grep -rn "foreach.*foreach" product/website/Domain/Entities/
 
# Lock/claim patterns
grep -rn "claim_lock\|lock.*iptables\|lock.*guest" product/website/HostDaemon/ infrastructure/
 
# Missing .Take() or count caps on collections
grep -rn "\.Add(.*rule\|\.Add(.*customer" product/website/Domain/Services/
 
# POST endpoints without idempotency
grep -rn "Verb.Post\|HttpPost" product/website/ClassicPanel/Controllers/ product/website/WebApi/Controllers/ | grep -v "Get\|Idempotent\|Check"

High-Value Targets for Customer-Side Bug Hunting

Firewall Rules (highest ROI)

  • Attack vector: Submit massive firewall ruleset via API or ClassicPanel import
  • Impact: Stalls all guests on the same compute host (known issue #1640)
  • What to test: Create a guest, add 1000+ firewall rules with 100+ addresses each via API. Monitor host load and other guest network operations.

Guest Creation/Provisioning

  • Attack vector: Create many guests simultaneously, or create with unusual configurations
  • Impact: Race conditions in provisioning (#1863), stale state (#1884)
  • What to test: Concurrent guest creation with same payment method, unusual OS/plan combinations

Domain Registration

  • Attack vector: Register domain + create account simultaneously
  • Impact: ApproveAccount/ProcessDomainNameCommand race (#1635, #1863)
  • What to test: Simultaneous account creation with domain registration, rapid cancel/re-register

Payment/Billing

  • Attack vector: Modify payment method while Mpay is slow/unavailable
  • Impact: Token-delete timeout (#1811), billing on wrong card (#1891)
  • What to test: Kill Mpay service mid-transaction, rapid card add/Remove cycles

Guest Power Operations

  • Attack vector: Rapid start/stop/reinstall cycles
  • Impact: State corruption during HostCommand processing
  • What to test: Power cycle guest 10x rapidly while it's being migrated

DNS/Forward DNS

  • Attack vector: Add many DNS records, or malformed DNS entries
  • Impact: DNS propagation failures, record loss
  • What to test: 500+ DNS records per domain, unusual record types

Testing Strategy

  1. Isolated environment: Use a single guest on a known host (check via API which host a guest is on)
  2. Observe side effects: Monitor OTHER guests on the same host during your test
  3. Measure, don't just observe: Note response times, error rates, CPU usage
  4. Reproduce deterministically: If it happens once, can you make it happen again?
  5. Document: Record exact steps, inputs, and observed behavior

What NOT to Test

  • Anything requiring admin panel access
  • Anything requiring SSH to compute hosts
  • Production customer data
  • Anything that could cause data loss without approval
  • Database schema changes
syntheses/customer-side-bug-finding-methodology.1784174059.txt.gz · Last modified: by 127.0.0.1

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki