Overview

Some files and file types are effectively invisible to changed‑chunk or delta transfers: a small edit can make an entirely new set of bytes, removing the benefit of deduplication and causing large uploads and fast quota consumption. For small businesses using Windows backups, spotting these "dedup‑unfriendly‑files" early lets you choose safer options—exclude, archive locally, or change how those files are produced and protected.

Why this matters

Changed‑chunk transfers and dedupe reduce bandwidth and cloud quota by only sending new or changed blocks. Files that are compressed, already encrypted, or are large monolithic binaries often have high entropy; small changes produce large new deltas. That drives repeated large uploads, larger quota usage, and longer restores.

Common categories of dedup‑unfriendly files

  • Compressed archives: .zip, .7z, .rar, .tar.gz, .iso. These are already compressed and resist further delta reduction.
  • Encrypted containers and client‑side encrypted blobs: Any file encrypted before backup is usually unique per change.
  • Large monolithic binaries and disk images: virtual disks (.vhdx, .vmdk), frequent full disk images and large installers.
  • Some media and CAD files: certain codecs or proprietary formats (video masters, raw camera, large CAD assemblies) store changes poorly for chunking.
  • Database and mailbox files taken outside an application‑consistent snapshot: live copies can change unpredictably—see AgooCloud guidance on Outlook PST/OST for safe options.

Quick, pragmatic scans on Windows

Start with lightweight discovery: find large files and group by extension to identify likely offenders.

Explorer search examples (enter in any folder search box):

  • Archives & large files:
    ext:zip OR ext:7z OR ext:rar size:>100MB
  • Disk images and VMs:
    ext:vhd OR ext:vhdx OR ext:vmdk size:>200MB

PowerShell to list the largest files under C:\Users and group by extension (run in an elevated prompt if you need access):

Get-ChildItem -Path 'C:\Users' -Recurse -File -ErrorAction SilentlyContinue | Select-Object FullName,Length,@{Name='Ext';Expression={$_.Extension.ToLower()}} | Where-Object { $_.Length -gt 100MB } | Group-Object Ext | Sort-Object @{Expression={$_.Group | Measure-Object Length -Sum};Descending=$true} | Select Name,@{Name='Files';Expression={$_.Count}},@{Name='TotalMB';Expression={[math]::Round((($_.Group | Measure-Object Length -Sum).Sum /1MB),1)}}

Entropy heuristic to detect compressed or encrypted files

High entropy (closer to 8 bits per byte) usually means the file is compressed or encrypted. A practical approach is to sample the first 64–256 KB of a candidate file and compute Shannon entropy. Use this as a heuristic—it's not a proof of encryption, but it helps prioritise files for further review.

# sample PowerShell outline (conceptual) function Get-FileEntropy { param($path,$sampleKB=64) # reads sample and computes Shannon entropy } # For each candidate file, compute entropy and report size and path

Note: the block above is an outline. If you run a script that reads files, run it with appropriate permissions and test on non-critical files. Use sampling so scans stay fast.

Estimate quota impact safely

To understand real quota cost, run a controlled test:

  1. Pick a representative folder of suspected files (copy it to an isolated test machine if possible).
  2. Record current cloud quota usage and create a clear baseline.
  3. Force a backup of just that folder and watch the uploaded bytes and resulting quota change.
  4. Make a small, realistic change to one file and rerun a backup to observe incremental upload size.

This test shows whether small edits cause large uploads, proving delta inefficiency in practice.

Safe options and tradeoffs

  • Exclude non‑critical archives: exclude raw installers, received zip attachments, and transient exports. Risk: losing a copy of that file unless archived elsewhere.
  • Keep large blobs local-only: AgooCloud local backups do not consume cloud quota—use a local NAS or server‑routed copy for archival blobs and keep cloud backups for critical smaller files.
  • Switch to application‑aware snapshots: for databases and mailstores, use VSS or application export so only consistent deltas are captured. See AgooCloud guidance on Outlook PST/OST for safe capture.
  • Avoid client‑side encryption for bulk non‑sensitive data: client‑side encryption increases uniqueness and quota usage because the service cannot dedupe encrypted data. Use it where privacy demands it, and accept higher quota; for voluminous non‑sensitive archives consider storing them unencrypted locally and cloud‑backing only critical data.
  • Adopt retention tiers and offload: older large files can be pruned from cloud retention and kept on lower‑cost local media.

Risk checklist before excluding or moving files

  • Is the file legally or contractually required to be retained offsite?
  • Could exclusion prevent a future restore of an important item (e.g., unique project files inside an archive)?
  • Are there operational procedures that rely on that file being in the standard backup set?
  • Have you tested restores from the alternate location (local archive, NAS)?
  • Have you documented exclusions and informed affected users?

Sample lightweight policy (starter)

  • Exclude archives >500 MB unless tagged as "keep‑in‑cloud".
  • Store VM disks and periodic disk images to a local replica only; include them in cloud backup only when application‑consistent and infrequent (e.g., monthly image).
  • Require client‑side encryption for files in \Sensitive\ but leave large media archives unencrypted and local.
  • Run weekly "quota impact" tests for the top three offending folders and review findings monthly.

Verify and monitor

After changes, verify backups and restores. Perform a selective restore of a few files from each affected folder, and schedule periodic recovery drills for critical items. Monitor quota trends and incremental upload statistics so you spot regressions quickly.

Troubleshooting

  • If quota still spikes, rerun the targeted test to isolate which file produced the large upload.
  • Check whether application updates or file format changes made previously dedupe‑friendly files unfriendly.
  • Review agent logs for repeated full‑file uploads and escalate with a captured example if needed.

Further reading and next steps

For Outlook store safety, see our guidance on backing up PST/OST. For measuring real backup throughput and planning seeding strategies, see the field benchmark guide. Use those tests to quantify the effect of any changes you make to exclusions, retention or encryption policies.

Identifying and managing dedup‑unfriendly files is a practical way to control cloud quota, reduce bandwidth use and improve restore speed. Use discovery, small controlled tests and a conservative risk checklist to make changes without losing recoverability.