So You Want to Do Medical Computer Vision
Medical imaging is nothing like standard CV datasets. You’re not resizing 224x224 JPEGs of cats and dogs. You’re dealing with whole-slide images (WSIs) that are 50,000 x 50,000 pixels and weigh multiple gigabytes. A pathology slide scanned at 40x magnification is about 100,000 x 80,000 pixels. That’s 8 billion pixels in one image.
You can’t throw that into a ResNet. You can’t even load it into RAM.
The Whole-Slide Image Problem
A WSI is a pyramid of resolutions. The scanner captures the slide at max magnification, then down-samples into multiple levels so you can zoom in and out without loading the entire gigapixel image.
Level 0: 100,000 x 80,000 px (40x)
Level 1: 50,000 x 40,000 px (20x)
Level 2: 25,000 x 20,000 px (10x)
Level 3: 12,500 x 10,000 px (5x)
Level 4: 6,250 x 5,000 px (2.5x)
The file format is usually SVS, TIFF, or DICOM. These aren’t handled by standard image libraries.
Reading WSIs Without Crashing Your Machine
OpenSlide reads multi-resolution medical image formats without loading the whole thing into memory:
import openslide # Library purpose-built for multi-resolution medical image formats (SVS, TIFF, DICOM)
slide = openslide.OpenSlide("patient_123.svs") # Opens the file handle without loading pixel data — the pyramid index is read lazily
print(f"Levels: {slide.level_count}") # Number of resolution levels in the pyramid (typically 5–6 for a 40x scan)
print(f"Level 0 dimensions: {slide.level_dimensions[0]}") # Full-res dimensions at 40x — e.g. (100000, 80000); just a tuple, no memory allocated
thumbnail = slide.get_thumbnail((1024, 1024)) # Reads a downsampled preview by selecting the appropriate pyramid level internally — useful for quick UI previews
region = slide.read_region((20000, 30000), 0, (1024, 1024)) # Reads a 1024x1024 pixel region from level 0 at (x=20000, y=30000) — only this patch is decoded into RAM
Read specific regions. Never the whole file.
The Tiling Pipeline
Since you can’t feed a 100,000px image into a neural network, you chop it into tiles (patches). Usually 256x256 or 512x512 at 20x magnification.
But you don’t tile the whole slide — most of it is empty background. First do tissue detection:
import openslide # WSI reader that navigates the multi-resolution pyramid without loading the full slide
import numpy as np # Used for pixel-level thresholding and coordinate arithmetic on the tissue mask
from PIL import Image # Converts OpenSlide's RGBA region into a standard RGB PIL image for downstream use
def extract_tissue_tiles(svs_path, tile_size=256, threshold=0.1):
# tile_size=256: standard patch size for WSI models (balances context vs batch size)
# threshold=0.1: at least 10% of the patch must contain tissue to keep it
slide = openslide.OpenSlide(svs_path) # Opens the slide handle — only metadata is loaded, not pixel data
level = slide.level_count - 1 # Uses the lowest-resolution pyramid level for the tissue mask (fastest to read)
thumbnail = slide.read_region((0, 0), level, slide.level_dimensions[level])
# Reads the full downsampled thumbnail — e.g. 6250x5000 instead of 100000x80000
thumbnail = thumbnail.convert("RGB") # OpenSlide returns RGBA; strip alpha for NumPy processing
thumbnail_np = np.array(thumbnail) # Convert PIL image to a (H, W, 3) array for thresholding
gray = np.mean(thumbnail_np, axis=2) # Average across RGB channels to get a single-channel intensity map
tissue_mask = gray < 220 # Binary mask: pixels darker than 220 (on 0–255 scale) are considered tissue, not background glass
scale = slide.level_dimensions[0][0] / slide.level_dimensions[level][0]
# Downscale factor from level 0 to the thumbnail level (e.g. 16x for 5 levels down)
tiles = [] # Accumulator for valid (tissue-containing) PIL Image patches
step = int(tile_size * scale) # Step size at the thumbnail level corresponding to one tile at level 0
for y in range(0, tissue_mask.shape[0] * int(scale) - tile_size, tile_size):
# Iterate y in level-0 pixel coordinates; shape[0] * scale recovers the full-res height
for x in range(0, tissue_mask.shape[1] * int(scale) - tile_size, tile_size):
# Iterate x in level-0 coordinates; ensures tiles cover the entire slide area
tx, ty = int(x / scale), int(y / scale) # Map level-0 coordinates back to thumbnail coordinates for the mask lookup
if tx >= tissue_mask.shape[1] or ty >= tissue_mask.shape[0]:
continue # Skip tiles whose mapped position falls outside the thumbnail bounds
patch = tissue_mask[ty:ty + 1, tx:tx + 1] # Sample a single-pixel row from the mask at the tile's location
if np.mean(patch) > threshold: # If the fraction of tissue pixels in this mask sample exceeds 10%, read the full tile
tile = slide.read_region((x, y), 0, (tile_size, tile_size))
# Reads only the needed 256x256 region from level 0 — no surrounding pixels decoded
tiles.append(tile) # Append the PIL Image tile to the results list
return tiles # Returns only tissue-bearing patches; empty-background tiles are discarded
This is simplified. In practice you’d use OTSU thresholding or contour-based filtering.
How Many Tiles Per Slide?
A typical biopsy slide at 20x with 50% tissue coverage, 256x256 tiles:
- ~15,000 to 40,000 tiles per slide
A study with 1,000 slides = 15 to 40 million tiles. That’s not “load into a DataFrame and train.” That’s “build a data pipeline.”
Stain Normalization — The Silent Model Killer
Slides from different hospitals look completely different. H&E stain at one lab might look purple. At another lab, it’s pink. At a third, it’s faded.
If you train on Hospital A data and deploy on Hospital B slides, your accuracy drops off a cliff. Not a minor issue — your model goes from 96% to 52%.
The standard fix is Macenko’s method. It separates the H&E stains into individual channels, then matches the stain appearance to a target image:
import numpy as np # All stain normalization is linear algebra on optical density vectors
def stain_normalize(source, target, alpha=1, beta=0.15):
# source: tile from a hospital with unknown stain appearance
# target: reference tile whose stain appearance we want to match
# alpha, beta: percentile trimming params (unused in this simplified version)
source_od = -np.log(np.maximum(source / 255.0, 1e-6))
# Convert source RGB [0-255] to optical density: OD = -log(I / I_0)
# OD space makes stain separation linear because Beer-Lambert law is additive
target_od = -np.log(np.maximum(target / 255.0, 1e-6))
# Same conversion for the target reference slide
source_reshape = source_od.reshape((-1, 3)) # Flatten spatial dims (H*W, 3) so each row is a pixel's OD vector
target_reshape = target_od.reshape((-1, 3)) # Same reshape for target to compute stain vectors per channel
source_mask = np.mean(source_reshape, axis=1) > 0.05 # Keep only pixels whose mean OD > 0.05 — filters out glass background (low OD)
target_mask = np.mean(target_reshape, axis=1) > 0.05 # Same foreground mask for the target; stain vectors should come from tissue only
source_foreground = source_reshape[source_mask] # Foreground-only pixel OD matrix for SVD — glass pixels would distort the stain basis
target_foreground = target_reshape[target_mask] # Target foreground; both matrices are (N_fg, 3) where N_fg is foreground pixel count
_, _, source_V = np.linalg.svd(source_foreground, full_matrices=False)
# SVD on source foreground: V (3x3) rows are the principal stain directions
# First two rows correspond to Hematoxylin (blue/purple) and Eosin (pink/red)
_, _, target_V = np.linalg.svd(target_foreground, full_matrices=False)
# SVD on target foreground: target's own stain basis to map into
source_stains = source_od @ source_V.T # Project every pixel's OD vector into the source stain basis (H&E space)
target_stains = target_od @ target_V.T # Project every pixel's OD vector into the target stain basis
for i in range(3): # Match the mean and std of each stain channel (H, E, residual) to the target
source_std = np.std(source_stains[:, i]) # Standard deviation of the i-th stain component in the source
target_std = np.std(target_stains[:, i]) # Target standard deviation — the spread we want to match
source_mean = np.mean(source_stains[:, i]) # Mean of the i-th stain component in the source
target_mean = np.mean(target_stains[:, i]) # Target mean — the intensity centroid we shift toward
source_stains[:, i] = (source_stains[:, i] - source_mean) * \
(target_std / source_std) + target_mean
# Z-score the source stain, then scale by target std and shift by target mean
# This aligns the statistical distribution of H&E intensities between labs
normalized_od = source_stains @ target_V # Project the adjusted stain coefficients back into RGB OD space via target basis
normalized = np.exp(-normalized_od).reshape(source.shape)
# Invert Beer-Lambert: OD -> transmission -> RGB; reshape back to original (H, W, 3)
return np.clip(normalized * 255, 0, 255).astype(np.uint8)
# Scale OD back to [0-255], clip to valid range, and return uint8 for model ingestion
Normalize every tile before feeding it to the model. The model sees consistent stain appearance regardless of lab.
Deep Learning-Based Normalization
StainGAN and CycleGAN-based approaches exist. They work better but cost more compute. Macenko or Reinhard are good enough for most projects.
The Data Pipeline for WSI
class WSIDataset(Dataset): # Extends PyTorch Dataset — yields tiles on demand without holding pixel data in memory
def __init__(self, slide_paths, tile_size=256, transform=None):
self.slide_paths = slide_paths # List of SVS/TIFF file paths; each can be 1–2 GB on disk
self.tile_size = tile_size # Standard 256x256 or 512x512 at 20x magnification
self.transform = transform # Augmentation pipeline (flip, rotate, color jitter, stain normalize)
self.tiles = [] # Flat list of (slide_path, x, y) tuples — pixel data is never stored here
self._precompute_tiles() # Runs tissue detection on all slides upfront; slow but avoids repeated mask computation
def _precompute_tiles(self): # Scans every slide once to build the coordinate index
for path in self.slide_paths: # Iterate over all SVS files in the cohort (hundreds to thousands)
slide = openslide.OpenSlide(path) # Open handle — reads pyramid metadata but no pixel data yet
for x, y in self._tissue_coords(slide): # Generator yielding (x, y) top-left corners of tissue-bearing regions
self.tiles.append((path, x, y)) # Store only the reference and coordinate — ~24 bytes per tile vs ~1 MB for the pixels
def __getitem__(self, idx): # Called by the DataLoader for each batch item
path, x, y = self.tiles[idx] # Look up the precomputed coordinate for this sample index
slide = openslide.OpenSlide(path) # Open the slide handle again (or fetch from LRU cache in production)
tile = slide.read_region((x, y), 0, (self.tile_size, self.tile_size))
# Lazy decode only the 256x256 region at level 0 — no adjacent pixels loaded
tile = tile.convert("RGB") # OpenSlide returns RGBA; strip alpha channel for 3-channel model input
if self.transform: # Apply stain normalization + augmentation if provided
tile = self.transform(tile) # Usually a torchvision Compose: ToTensor -> MacenkoNormalize -> RandomFlip
return tile # Returns a single PIL Image (or Tensor if transform converts it) for the batch collator
Store coordinates, not pixels. A dataset of 1,000 slides has ~20 million tile coordinates. Storing those plus a slide path reference is a few hundred MB. Storing all pixels would be 4 TB.
What Actually Goes Wrong
1. Slide scanning artifacts. Tissue folds, bubbles, out-of-focus regions, ink marks. Your model learns “black triangle = cancer” because all cancer slides have ink marks.
def detect_artifacts(tile): # Filters out tiles with blur (out-of-focus scan) or excessive edges (tissue fold/ink)
laplacian = cv2.Laplacian(np.array(tile), cv2.CV_64F) # Laplacian edge detection — high-frequency content indicates sharp focus
focus_score = laplacian.var() # Variance of the Laplacian: low variance means the tile is blurry (no fine detail)
gray = cv2.cvtColor(np.array(tile), cv2.COLOR_RGB2GRAY) # Single-channel intensity for edge detection with Canny
edges = cv2.Canny(gray, 50, 150) # Canny with low=50, high=150 — tuned for H&E stain edge contrast
edge_density = np.mean(edges > 0) # Fraction of pixels that are edge pixels in the binary Canny output
if focus_score < 100 or edge_density > 0.3: # Blur threshold: < 100 means out-of-focus (common in WSI scanning artifacts)
# Edge-threshold: > 0.3 means tissue fold, ink mark, or bubble (too many sharp boundaries)
return False # Reject this tile — it would confuse the model with non-tissue features
return True # Accept the tile as valid tissue for training or inference
2. Label noise. Pathology labels are made by humans. Inter-observer agreement is around 70-85%. Account for it — soft labels, label smoothing, or multiple reviewers.
3. Memory management. Don’t keep 100 slides open. Each OpenSlide handle uses significant memory. Use an LRU cache:
from functools import lru_cache # lru_cache imported but not used in this snippet — could decorate get_slide for automatic eviction
class SlideCache: # Manual LRU-like cache for OpenSlide handles to cap OS file descriptor usage
def __init__(self, max_slides=10): # max_slides=10: typical physical memory limit — each handle reserves significant RAM for the pyramid index
self.max_slides = max_slides # Upper bound on concurrently open slide handles
self.cache = {} # Dict mapping file path -> OpenSlide object; insertion order tracks recency
def get_slide(self, path): # Request a slide by path; returns cached handle if available
if path not in self.cache: # Cache miss — need to open a new handle
if len(self.cache) >= self.max_slides: # At capacity: evict the oldest (first inserted) slide handle
oldest = next(iter(self.cache)) # dict insertion order is preserved in Python 3.7+; first key is least recently opened
self.cache[oldest].close() # Close the OpenSlide file handle to release OS file descriptors and allocated memory
del self.cache[oldest] # Remove the evicted entry from the dict
self.cache[path] = openslide.OpenSlide(path) # Open a new handle — reads the pyramid index but doesn't decode pixels
return self.cache[path] # Return the (possibly just-opened) slide handle for region reads
4. Magnification consistency. Your dataset might mix 20x and 40x slides. Standardize to one magnification before training.
The Real Recommendation
Starting a medical imaging project:
- Use OpenSlide for reading. Nothing else handles SVS/TIFF properly.
- Tile at 20x, 256x256. Community standard.
- Do stain normalization on every tile before training. Macenko is fine.
- Precompute tile coordinates, not pixels.
- Make a tissue mask. Don’t waste compute on blank glass.
- Add focus filtering.
- Expect 30-50% of your time to be data pipeline.
Medical CV is not harder than regular CV. It’s just different. You’re not fighting architecture choices — you’re fighting file formats, storage costs, and inconsistent stains.
Comments
Leave a comment