Multi-Stage CV Pipeline

The Problem

A client needed an object recognition system for industrial quality inspection. The challenge: hundreds of fine-grained product categories that a single detector couldn't handle accurately. Products looked similar at the category level but had subtle differences that mattered for quality decisions.

Additionally, the system had to run on edge devices: inference latency under 100ms, no cloud dependency, and models that fit in under 200MB of memory.

The Architecture

Why It's Hard

• Detect-then-classify reduces combinatorial complexity. Instead of one model handling N categories, the detector handles localization and the classifiers each handle a manageable subset. This makes training faster and accuracy higher.
• Edge deployment constraints are real. ONNX export, model quantization (INT8), and careful memory management were essential. The full pipeline including pre/post-processing had to fit within a tight latency budget.
• Data imbalance across categories. Some product types appeared 100x more frequently than others. Custom loss weighting, data augmentation, and occasional synthetic data generation were needed.
• Model update strategy. When new products are added, you don't want to retrain everything. The staged architecture allows adding a new fine-classifier without touching the detector or other classifier stages.

Technical Stack

• YOLOv8 + SSD: object detection, trained on custom dataset with domain-specific augmentations
• PyTorch + TensorFlow: custom classifier training with ResNet/EfficientNet backbones across both frameworks
• ONNX Runtime: inference optimization for edge deployment
• OpenCV: image preprocessing, crop extraction, and post-processing
• Model quantization: INT8 quantization reducing model size by 4x with minimal accuracy loss
• Custom evaluation harness: per-stage metrics, confusion matrices, and latency profiling

Inside the System

Two sample outputs. Both show the same two-stage behaviour: detection proposes regions, then the classifier decides what each region actually is. The second example adds a cropping pass, so every detected face is classified as its own input.

What I'd Do Differently

• Explore YOLO-NAS or RT-DETR. Newer architectures offer better accuracy-latency tradeoffs than YOLOv8 for edge deployment.
• Add a rejection class. When the classifier is uncertain, route to human review instead of forcing a prediction. This matters in quality inspection.
• Implement active learning. Automatically flag low-confidence predictions for human labeling and model improvement over time.
• Use Triton Inference Server. For multi-model orchestration on edge, Triton would simplify model loading, versioning, and batching.

Key Takeaways

The detect-then-classify pattern is underrated. Splitting the problem into stages gives you better accuracy, easier maintenance, and the flexibility to update one stage without touching others. It's now my default approach for any multi-class vision problem.