How ValEye Counts Pills
ValEye counts pills from a single overhead photo, entirely on your device. The vision pipeline runs in your browser via OpenCV compiled to WebAssembly, works offline once installed, and takes roughly a quarter-second per image. Photos stay on your device — unless you explicitly tick “Share this photo” on a result, which contributes that one photo (plus your corrected count) to ValEye’s test suite to improve future counting.
The problem sounds easy — "count the blobs" — but real trays have touching pills, shadows, white pills on white counters, engraved trays, and piles. The pipeline below is classical computer vision (no neural network), tuned against a test corpus of real pharmacy-style photos and synthetic stress tests with exact known counts.
The pipeline
Loading the vision engine to render the stage demos below — these are computed live in your browser, right now, by the same code the app uses…
Estimate the background
The border of the photo is assumed to be tray/counter. Its median color becomes the background model — no fixed color or special tray required.
Color-distance segmentation with shadow damping
Every pixel gets a distance-from-background score. The luminance component is down-weighted only when a pixel is darker than the background at the same hue — the signature of a cast shadow. White pills on a white counter keep full weight, so they still register. Otsu's method picks the foreground threshold automatically.
Second-mode rescue
If the tray holds both strongly-colored and faint pills, Otsu splits "colored vs everything" and drops the faint ones. We re-run Otsu on just the leftover background; anything it reveals is admitted only if it is a compact, pill-shaped piece (its area must be explainable by its thickness — a disk, not a smudge).
Surface refinement
A blob covering more than ~12% of the frame is a plate or tray, not a pill. It gets re-segmented against its own dominant color so the pills sitting on it pop out. An accept-test guards the other direction: if re-segmenting yields crevice fragments instead of several pill-like pieces (i.e. the blob was actually a pile of same-colored pills), the refinement is reverted.
Distance transform & markers
The distance transform gives each foreground pixel its distance to the nearest background pixel — pill centers become peaks, and a blob's peak is its half-width. Pill-scale blobs seed one marker per peak cluster (60% of the blob's own peak); blobs thicker than a pill (piles) instead use local maxima, spaced by the estimated pill radius, so each buried pill center still gets a seed.
Watershed
The watershed algorithm floods outward from the markers and draws ridge lines where floods meet — which is exactly the boundary between two touching pills. This is the classic separator for touching convex objects.
Filter, split, number
Regions are filtered by an absolute size floor (relative to the image), a relative
floor (vs the median region), and a minimum thickness — killing specks, rims, and
engraving lines. A region much larger than the median (a merged cluster the watershed
missed) contributes round(area / median) pills and is badged as a range,
e.g. 12–14. Every counted pill gets a numbered badge in the overlay.
Pixel mass: the counting rule that won
A pharmacy count is one medication, so every pill in frame has the same
area in pixels. That turns counting into arithmetic: after segmentation, each
blob's pixel mass should be an integer multiple of one pill's mass — a lone pill is
1.0×, two touching pills are ~2.0×, a chain of five is ~5.0×. The unit mass is
estimated iteratively (start at the median blob, divide each blob by its rounded
multiple, re-take the median), and the count is
Σ round(blobArea / unitArea).
We A/B-tested three algorithms head-to-head on the corpus
(node tools/count-cli.mjs testdata --ab):
| Variant | Idea | Pass ≤10% | Median error |
|---|---|---|---|
| mass (default) | pixel mass ÷ unit area | 14/16 | 0% |
| baseline | watershed hybrid markers | 10/16 | 10% |
| sized | size-derived markers + area split of watershed regions | 4/16 | 35% |
Watershed still runs — it draws the pill boundaries and badge positions for the
overlay — but the number comes from pixel mass. The failed sized
experiment taught us why: dividing watershed regions by area amplifies marker
noise, while dividing raw blobs (whose outlines come straight from
segmentation) is stable.
Current results
| Scenario | Result |
|---|---|
| Scattered same-size pills, overhead, plain surface (the intended use) | exact in testing — 118/118 kraft scatter, 30/30 touching chains, 3/3 minimal |
| White pills on white background | 15/15 via shadow-aware color distance |
| 120-pill single-layer scatter with touching | 116/120 (~3% low) |
| Motion blur | ±50% — blur merges pills with their shadows |
| Heavy vignette, angled/perspective shots, piles with buried pills | out of scope — retake from directly above, single layer, even light |
Test methodology
The test bench runs the full
pipeline in your browser over the corpus: real CC-licensed photos (with hand-counted
expectations) plus synthetic stress tests rendered with exact known counts —
touching chains, tiny/large pills, gradients, sensor noise, motion blur, vignettes,
white-on-white. Grading: pass within max(1, 10%), warn
within 30%, fail beyond. The same suite runs headless via
node tools/count-cli.mjs testdata --ab for algorithm A/B comparisons.
Stack
Static PWA (no build step, no server): vanilla JS + OpenCV.js 4.9 (single-file WASM, ~10 MB, cached by a service worker for offline use). The counting module is environment-agnostic and runs identically in Node for headless regression testing.