What is a delta storm and why test it?

A "delta storm" describes a high-change workload pattern that drives unexpectedly large incremental uploads: thousands of small-file churns, frequent rewrites of the same files, cycles of large-file rewrites, or repeated metadata-only changes. Simulating these situations in a safe lab helps confirm changed-block efficiency, bandwidth shaping, retention behavior and whether policies or exclusions need tuning.

Test design principles

  • Non‑destructive: run tests in temporary folders and avoid production data or any destructive OS changes.
  • Repeatable: use scripted steps so results can be reproduced and compared.
  • Measurable: record local changed bytes and compare them with bytes uploaded as reported by the backup client or admin console.
  • Isolated: run when other backup activity is idle; disable other scheduled jobs for the test machine.

Before you start — checklist

  • Create a dedicated test account or use a non-production Windows VM.
  • Ensure AgooCloud client is installed and idle; note current cloud quota usage.
  • Temporarily pause unrelated backup schedules on the test device to avoid noise.
  • Open the agent UI or admin console so you can see per-device upload statistics and version counts.
  • Note that live database copies are not application-consistent unless you take application-aware exports or VSS snapshots. Do not assume file copies equal application-consistent backups.

How to measure delta volume

  1. Record baseline cloud usage and agent-upload totals from the AgooCloud admin console or device log.
  2. Measure local changed bytes during each recipe using a PowerShell drive scan: for example, get total bytes under the test folder before and after changes using Get-ChildItem and Summation of Length (run locally on Windows).
  3. Run the recipe and wait until the agent shows the corresponding upload(s) are complete.
  4. Record the increase in cloud-upload bytes from the client or console and compare to local changed bytes.

Compute a simple efficiency ratio: uploaded_bytes / local_changed_bytes. Lower ratios are better: they mean the client uploaded less than the raw changed data (changed-block or dedupe savings). Use the comparison to decide accept/reject against thresholds below.

Acceptance thresholds (heuristic guides)

  • Large-file partial rewrites: expect significant savings. Typical target: uploaded_bytes < 30% of local_changed_bytes when only a small portion of a large file was modified.
  • Large-file full rewrites: expect uploaded_bytes ≈ local_changed_bytes (no savings) — this is expected when the whole file changes.
  • Many small-file churns: changed-block techniques offer little benefit; expect uploaded_bytes to roughly equal local_changed_bytes plus metadata overhead. Watch object-count and API overhead more than percent savings.
  • Metadata-only changes (timestamps/ACLs): expect minimal upload. If you see large uploads, investigate client settings, metadata policies and VSS interactions.

These are guidelines. Use them as triggers to investigate rather than absolute pass/fail gates.

Recipe A — Thousands of small-file churn

  1. Create a sandbox folder: %TEMP%\agoodelta_test_smallfiles.
  2. Generate 5,000 small files: use PowerShell to create files ~4KB each with random content. Keep all files in the same folder (flat) and in nested folders (to test tree traversal).
  3. Run a baseline backup (trigger incremental) and wait for it to finish.
  4. Perform churn: touch or rewrite 2,000 random files by overwriting them (e.g., append a short line or rewrite content).
  5. Wait for the agent to finish uploading and record uploaded bytes and count of uploaded objects in the console.

Expected client behavior: many small objects uploaded; metadata and per-file overhead may be proportionally high. Changed-block savings are minimal. Use this test to validate file-level parallelism, small-file handling and API-rate behaviors.

Recipe B — Frequent rewrites of the same large file

  1. Create a sandbox file: %TEMP%\agoodelta_test_large\bigfile.bin sized 500MB (or a size large enough to demonstrate block savings).
  2. Trigger an initial backup so the file baseline is recorded.
  3. Simulate frequent small edits: open the file and overwrite several small regions (e.g., modify bytes at offsets every 10MB) 10–20 times. Use a PowerShell loop to open a FileStream, Seek and Write a few KB to specified offsets. This rewrites only parts of the file.
  4. Wait for the agent to upload deltas and record upload size.

Expected behavior: if your backup client uses changed-chunk uploads, the uploaded bytes should correspond roughly to the sum of modified blocks, not the whole file. If the client uploads the whole 500MB each time, changed-block is not effective for this file and needs investigation.

Recipe C — Large-file full rewrite cycles

  1. Using the same large file, perform an overwrite of the entire file content (create a fresh file with new random bytes) multiple times and run increments between overwrites.
  2. Record uploaded bytes for each full rewrite.

Expected behavior: each full rewrite will upload roughly the size of the file. This validates that full-file changes lead to full deltas, which is normal.

Recipe D — Metadata-only churn

  1. Take a group of files and change modification timestamps and attributes without changing content: in PowerShell, set (Get-Item file).LastWriteTime = (Get-Date).
  2. Also test ACL-only changes carefully if you have permission (use Set-Acl sparingly in lab). Run incremental backup and record uploads.

Expected behavior: a well-tuned client uploads minimal bytes; excessive uploads indicate the agent treats metadata changes as content changes or an application is rewriting files behind the scenes.

Monitoring checkpoints during each recipe

  • Agent status and per-device upload totals in AgooCloud console.
  • Local IO: Task Manager or Resource Monitor to ensure the test is the dominant IO source.
  • Network throughput: Windows Performance Monitor counters or Task Manager network tab.
  • Object counts and API errors in the agent log; retry storms create additional overhead.
  • Cloud quota and retention changes: confirm new versions appear and older versions are preserved per policy.

Troubleshooting checklist and decision matrix

  • Symptom: Uploaded bytes are close to or greater than local_changed_bytes when you expected savings.
    • Likely causes: changed-chunk disabled, files compressed or encrypted at the application layer, client configured to upload full files on change.
    • Action: check client settings, test with an uncompressed large file, review logs for changed-chunk errors, consult AgooCloud console for per-file behavior.
  • Symptom: Metadata-only changes cause large uploads.
    • Likely causes: agent treats metadata as content or an agent pre-scan rewrites files; antivirus or indexing service may rewrite metadata.
    • Action: isolate with a process monitor, temporarily exclude the test folder from antivirus, retest. If metadata triggers remain, raise with support including diagnostic logs.
  • Symptom: Many small-file uploads saturate the WAN.
    • Likely causes: large object counts, insufficient batching or throttling.
    • Action: consider retention/selection changes, increase local replica to absorb churn (local-first), or schedule test windows and bandwidth limits.
  • Symptom: Changed-block efficiency varies unpredictably across files.
    • Likely causes: file-level compression, encrypted containers, or filesystems with dedupe that change block alignment.
    • Action: test with plain files, document file types that resist changed-block savings and consider alternate protection patterns (periodic full copies + incremental deltas).

Safe verification steps after testing

  • Perform an incremental restore of a small subset of test files to confirm versions and ability to restore.
  • Check that retention headers and version counts in the console reflect test activity.
  • Revert test account or VM to a known baseline (delete sandbox folders) to avoid accidental long‑term retention costs.

Final notes and tradeoffs

Delta‑storm testing clarifies how your incremental strategy behaves under real-world churn. Small-file storms increase object-count and metadata overhead; changed-chunk shines for repeated small edits inside large files; metadata churn should be lightweight. Use these recipes to tune client policies, exclusions, bandwidth shaping and local replica strategies. Always pair these functional tests with restore verification and remember: copying a live database is not a substitute for application-aware exports or VSS‑aware snapshots.

If you run into unclear results, gather the agent logs, a short test plan and measurements, then open a support ticket so the backup vendor can examine the diagnostic bundle and advise next steps.