A short clip shows up online with no title or link. Someone replies, "source?" and the thread goes quiet. Reverse image search feels like the obvious move - but it usually searches images that have already been published as standalone images on the web. A random frame cut from the middle of a video may never have been indexed, so there's nothing for the usual tools to find.
That gap is the motivation behind SauceMatch, a reverse video search engine that tries to find where a clip came from. Its index contains about 32 million frames pulled from more than four million videos, and it runs on one home PC. It's also a revealing systems project: the hard parts aren't only picking a model, but keeping false matches under control, searching data bigger than RAM, and making what the browser does line up with what the server returns.
This article walks through the design and the trade-offs. It covers frame indexing, visual fingerprints, disk-backed FAISS search, geometric verification, and a browser pipeline built with WebAssembly. SauceMatch is an 18+ service; here, the focus is strictly on how the system works and where it still falls short.
Why video search starts with frames
A video can be treated like a collection of still images. The pipeline is simple enough to explain at a high level:
- Collect representative frames from videos that can be indexed.
- Turn each frame into a visual fingerprint.
- Fingerprint frames from a query clip or screenshot.
- Find nearby indexed fingerprints and group results by video.
- Verify promising candidates before calling them matches.
That grouping step matters. The same clip gets reposted to multiple sites, and those copies can be useful evidence - not duplicates to toss out. SauceMatch groups copies under a single video result and can show multiple links. If one link is dead, another may still work.
Public preview images provide the raw material
The index is built from preview images published by video sites - things like listing thumbnails and frames shown on hover. They're ordinary images, which makes them usable for indexing even when the full video isn't available to pull directly.
Frames are stored as WebP images with the longest side capped at 512 pixels. A hash of decoded pixels deduplicates identical frames, so a frame reused across reposts can be stored once and associated with multiple videos.
Not every repeat is helpful, though. Studio logos, intro cards, and placeholders show up across lots of unrelated uploads. A frame associated with more than five videos is ignored at query time. It's a simple filter, but it stops common branding from causing lots of "matches" that really aren't.
Copy detection is different from semantic similarity
Models like CLIP are built to represent what an image depicts. That's useful for semantic search, but it's not the same as detecting an edited copy of a specific frame. Two images of similar people, rooms, or compositions can end up near each other even if they come from different videos.
SauceMatch uses SSCD, a copy-detection model from Meta based on ResNet-50. It turns a frame into a 512-dimensional vector and is trained so the same image and edited versions of it land close together. It's designed to tolerate cropping, resizing, color shifts, text overlays, and recompression. The vectors are L2-normalized and compared with cosine similarity.
All of that gives you a good "likely candidates" step - but it doesn't prove identity. Different videos shot in the same room can still produce close vectors. So SauceMatch doesn't treat nearest-neighbour similarity as the final answer. Instead, it adds a second, more expensive verification check before labeling anything a match.
How the index searches beyond RAM
SauceMatch uses FAISS IVF-PQ. At this scale, the question isn't only "which approximate nearest-neighbour method is fast?" You also have to ask: how much memory does it use, what recall do you lose, and what happens when the index doesn't fit in RAM?
The current setup uses 16,384 inverted lists and probes 1,536 lists per query frame. It retrieves up to 2,000 candidates per frame, then reranks them using 8-bit scalar-quantized copies of the vectors. Those reranking rows take 512 bytes each. The index gets extended with new frames every 12 hours, and storage is reported at about 0.6 KB per vector per build.
One tested alternative was an HNSW coarse quantizer. In the author's tests, it dropped recall by about 15% with inner product, so the system sticks with a flat quantizer instead. The point isn't "HNSW is bad." It's that popular components don't automatically stay good after you change the metric, data, and recall goals.
Memory mapping solved capacity, not latency
Once the index grew beyond available RAM, "just add more memory" wasn't an option. The server was changed to memory-map the FAISS inverted lists and the arrays with quantized codes and IDs. Instead of loading the entire index into private process memory, the OS manages mapped pages as file cache and can reclaim them when other memory is needed.
That cut the server's private memory from 18.1 GiB down to 5.3 GiB. But it introduced a nasty side effect: cold searches got slow. Reranking needs to read thousands of small rows spread across a multi-gigabyte file. A 30-frame query took about 18.3 seconds when those pages weren't already cached.
The key insight was that IVF search already tells you which rows the reranker will need. So rather than letting the server touch scattered rows one by one and pay the page-fault cost along the way, the server asks the OS to prefetch the pages first. On Windows, it uses PrefetchVirtualMemory; on Linux, madvise with MADV_WILLNEED can request similar read-ahead.
After prefetching, cold-search latency dropped from 18.3 seconds to 1.1 seconds. Warm searches were around 0.65 seconds. The search algorithm didn't change - the order of storage access did.
It's a useful operational lesson: if a workload gets slower after moving data to disk, look at the access pattern before assuming the only fix is more hardware. Prefetching can help when the application knows what pages it's about to read. It's not magic: the result depends on the storage and the OS, and on how scattered those reads really are.
On Windows, the project also had to handle index replacement. A mapped file can block deletion if the old build is still being served. Opening mapped files with delete sharing lets the builder replace old index files while the active server keeps working.
How the system verifies a match
Approximate search is great for finding candidates. It's not proof that a candidate came from the same source video. SauceMatch adds a geometric check using SIFT features and a homography, specifically to separate a transformed copy from a merely similar image.
For each candidate, the system detects up to 4,000 SIFT keypoints in both frames, pairs descriptors, and filters pairs using Lowe's ratio test. It then estimates a homography with MAGSAC. Practically speaking, the check asks whether enough corresponding features support a single transformation that maps the query frame onto the indexed frame.
That "single transformation" requirement is important. It can explain cropping, scaling, and text overlays placed on top of an image. A bunch of isolated lookalike features isn't enough - the correspondences have to agree on a shared geometric story.
The final confidence score, from 0 to 100, combines multiple signals:
- How many keypoint correspondences agree with the transformation.
- The inlier proportion among the candidate correspondences.
- How evenly inliers are distributed across the frame.
- Pixel-level checks on the overlapping region, including perceptual hash, histogram correlation, and SSIM.
Thresholds were chosen with held-out tests. In the project's tests, genuine reposts didn't fall below 28 agreeing keypoints, while different videos shot on the same set stayed under the acceptance thresholds. Still, those are observations from a test set - not a promise that every future match behaves identically.
The UI keeps two result types separate. A Match has passed geometric verification. A Similar result looks close, but it didn't pass that proof step - so it's explicitly unverified. If nothing passes, the page says nothing was found rather than showing uncertain results as if they were verified and charging for certainty.
That restraint is part of the engineering. In a search tool, false confidence can be worse than a miss. Nearest-neighbour scores are fine for candidate generation; identity claims need stronger evidence.
Why search runs in the browser
At first, the server handled the full search. The design later moved frame selection, fingerprinting, and geometric verification into the visitor's browser for two reasons: the user's video usually doesn't need to leave their device, and a home server shouldn't have to run a neural network for every query.
In the browser, a worker selects frames and runs SSCD using ONNX Runtime Web. It uses WebGPU when available and WebAssembly when it isn't. The browser sends fingerprints to the API, then runs geometric checks against candidate thumbnails loaded from a CDN. Less capable devices (or browsers where the client-side path fails) can fall back to server processing. That fallback may send selected frames to the server - so "the file stays local" describes the normal browser path, not an absolute guarantee for every device and failure case.
Quantization needs difficult test cases
A full-precision model is too expensive to download for a web page, so the model was quantized. Several options were tested, with very different results on dark frames:
- Full int8 scored 0.78 cosine similarity against the server fingerprint - too low for reliable equivalence.
- Static QDQ quantization scored 0.69.
- A plain fp16 graph produced NaNs on WebGPU.
- Weight-only int8 (keeping the first convolution, layer 1, and biases in fp32) produced a 25.5 MB model with cosine similarity of at least 0.9977 against the server.
The lesson isn't "one quantization method is always best." It's that average inputs can hide problems. Dark frames exposed failures that ordinary examples didn't. Test the cases that stress your pipeline, then compare outputs against the reference implementation.
ONNX Runtime's WebAssembly backend can use threads when the page is cross-origin isolated. That means using SharedArrayBuffer and adding the right response headers, including Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: credentialless. In the reported tests, enabling threads cut frame processing from about 420 ms to about 140 ms. These headers affect page behaviour with cross-origin resources, so they need to be tested along with things like bot checks.
Reuse the matcher instead of reimplementing it
The browser and server need to agree on whether a candidate is a match. Instead of rewriting the OpenCV logic in JavaScript, the project compiled the same C++ code paths to WebAssembly using Emscripten and runs them in a pool of workers.
The supporting image operations needed the same kind of alignment. The browser version reproduces the Python pipeline's grayscale conversion, fixed-point LANCZOS resize, unnormalized DCT, and the median calculation used for perceptual hashes. A parity script runs both implementations on the same image pairs and compares answers. Small floating-point differences can still affect SIFT descriptor rounding - and occasionally a keypoint - so identical source code doesn't automatically mean identical results across platforms.
Video decoding had its own "same result, different path" problem. Seeking repeatedly with a <video> element was slow on phones because each seek required decoding forward from a previous keyframe. A worker-based reader was built using mp4box.js to demux and WebCodecs to decode frame runs. To avoid changing search results, it was tested against the previous reader in headless Chrome, including frame positions and JPEG bytes. The faster path is limited to Chromium browsers and supported files; unusual files fall back to the standard video element rather than failing outright.
Frame selection also has to deal with screenshots and messy clips. The scanner checks up to 160 positions, dropping black, blurred, and near-duplicate frames, and tries to crop black bars or blurred side backgrounds. It detects app screenshots, isolates the video region when possible, and can resample around a promising timestamp - or retry mirrored frames when the first pass finds nothing.
What works, what fails, and what to learn
The infrastructure is intentionally compact. Static pages run on Cloudflare Pages. The API runs on a home PC behind Cloudflare Tunnel, so there's no exposed inbound port. Thumbnails live in Bunny Storage and are served with Cloudflare caching. The same machine fingerprints new frames on a GPU.
It isn't a zero-maintenance setup, though. The GPU driver occasionally resets, and the builder doesn't always know why. When the CUDA context fails, the server exits with code 75 and a supervisor restarts it. The index is rebuilt every 12 hours; the previous build keeps serving until the new one is ready. It's a practical availability approach: publish the new index only when it's ready, instead of taking search offline during the build.
The project also draws clear boundaries around use. It's for adults and lawful use, not for identifying private people. Videos or individual frames can be removed on request. Creators can also search in the other direction - looking for reposts of their own work and pursuing takedowns. Because results may involve sensitive material, the 18+ restriction and removal process are part of the product.
The limitations are real, too. Search can only find content represented in the index. In the first ten days of October covered by the project's report, 12% of searches ended in a verified match; a lot of misses were simply because the video wasn't indexed yet. Public previews make indexing possible, but sites that don't publish accessible previews are harder to cover.
There's also a known false-positive case. In tests, roughly 1-2% of matches were wrong when two different videos used the same room and overlay. The geometry can agree in that situation. The project identified another possible improvement: an additional photometric check after homography estimation, but it wasn't implemented yet.
A few design choices carry over beyond this specific service:
- Use copy detection for "is this the same image after editing?" rather than semantic embeddings for "does this look similar?"
- Treat approximate retrieval as candidate generation, not a verdict.
- When data outgrows memory, measure page faults and I/O patterns before buying hardware.
- Validate quantized models on difficult inputs, not only representative ones.
- When two implementations must agree, sharing or compiling the same implementation often beats maintaining separate rewrites.
- Make fallbacks explicit. A browser path can improve privacy and reduce server load, but device limitations still need a reliable alternative.
For more detail on the original build, see the project write-up. SauceMatch is available at saucematch.com; it's 18+ and intended for lawful use.
Conclusion
A reverse video search engine isn't just an embedding model pointed at a database. It's a pipeline: collect frames that are worth indexing, retrieve candidates efficiently, verify them with independent evidence, and handle the messy reality of browsers and limited hardware.
The most transferable idea is the split between "find" and "prove." Fingerprints make large-scale search practical; geometric verification makes the results more trustworthy. Memory mapping and prefetching keep the index workable on limited hardware, and browser-side processing reduces routine server load - without pretending every device will follow the same path.
If you're building a visual search system, start by defining what counts as a match and how you'll measure false positives. Then build the smallest pipeline you can use to test that definition on real, difficult inputs. For the project details and context, read the original SauceMatch build article.