Skip to main content

Case Study: Solving Real-Life Problems in "Constant Time"

The "Disk Full" warning on a Mac isn't just a storage issue; it's a bottleneck for productivity. I recently faced a 52GB bottleneck in the form of old mobile backups and media archives. While most users would reach for a paid tool or spend hours manually clicking "Delete," I decided to treat it as an engineering problem.

This deep dive covers how I used Agentic AI to build a secure, high-performance deduplication engine that handles large-scale data migration with lightning-fast efficiency.

1. The Operational Flow​

Before deleting a single byte, we needed a 100% reliable safety framework. We used a Copy-Verify-Store pattern.

Safe Migration Pipeline:

  1. SECURE COPY (rsync) -> Get data to Seagate safely.
  2. VERIFICATION GATE -> Check if every byte made it over.
  3. RECLAIM SPACE -> Delete the local Mac copy only after verification.
  4. DEDUPLICATION -> Clean up the Seagate drive in "Instant Lookup" mode.

2. The Deduplication Engine: Under the Hood​

The core challenge was identifying duplicates across the entire Seagate drive (over 1TB of data) without waiting hours for file comparisons.

The Two-Phase Optimization​

We didn't just compare every file to every other file. That would take forever. Instead, we used a smarter approach:

  1. Phase 1: The Quick Scan (Metadata): We look at the "File Size" first. If two files have different sizes, they are guaranteed to be different. We "skip" hashing them.
  2. Phase 2: The Fingerprint (SHA-256): For files that have the exact same size, we generate a unique "Digital Fingerprint."

The "Instant Lookup" ($O(1)$) Advantage​

The real magic happens with Hashmaps. By storing these fingerprints in an in-memory dictionary, we turned the duplicate check into an "Instant Lookup."

  • Traditional Way: Compare File A against every other file in the library one by one. This is $O(n^2)$, meaning if you have 10,000 files, you might perform 100,000,000 comparisons.
  • Engineering Way: Put all fingerprints in a "Master List." When we see a new file, we don't scan the list; we jump directly to the answer. This check takes the same amount of time whether you have 10 files or 10,000,000 files. This is what engineers call "Constant Time" or $O(1)$.
# Our Memory-based "Master List"
fingerprints = {} 
for path in potential_duplicates:
    # This lookup is O(1) - it doesn't wait!
    if unique_id in fingerprints:
        mark_as_duplicate(path)

3. Why In-Memory is the Key​

Even with 100,000 files, storing their SHA-256 signatures (64-character strings) only consumes about ~7MB of RAM. Since modern Macs have 8GB or 16GB of RAM, this script effectively uses 0.001% of your memory.

By keeping everything in-memory rather than writing results to a slow database, we achieved a scan speed limited only by the reading speed of the hard drive itself.

4. The Human Impact​

By extending this to the entire hard disk, I wasn't just fixing a "full disk" warning; I was cleaning up years of technical debt.

  • Manual Effort: ~6 hours of risky work.
  • The AI Engine: 10 minutes, $0, and 100% precision.

Conclusion​

Data structures aren't just for coding interviews. Whether it's "Instant Lookups" or resumable data transfers, the engineering mindset turns overwhelming chores into simple, automated wins.