Research · Shipped to production · 2026
DetectIQ
My capstone: a fine-tune of Meta's SAM3 segmentation foundation model with LoRA that segments bridge and infrastructure damage at the pixel level — then shipped behind a FastAPI + React inspection platform. Upload a photo, get back a colored damage mask with per-class coverage.
Bridge inspection is manual, slow and subjective.
Structural damage assessment still leans on the expert eye: is that a crack, corrosion, or spalling? Two inspectors can label the same photo differently, and annotating thousands of images by hand takes weeks.
The goal: feed in an image and get back a pixel-level mask for each damage type — repeatable, fast and scalable. DetectIQ turns that research model into a product an inspector can actually use.
Two repos, one system: train the model, then serve it.
The work is split into a training pipeline that produces a small LoRA adapter, and a product that ships that adapter to real users behind a clean detector interface.
- Model repo. SAM3 base + LoRA adapters fine-tuned on DACL10K for 19 classes; output is an ~8 MB adapter (
adapter_model.safetensors). - Product repo. FastAPI backend + React 19 frontend: upload an image, analyze in the background, return a colored mask and per-class coverage.
- Detector seam. The whole app talks to one
BaseDetectorinterface — switching from a zero-dependency mock to the real model is a one-line config flag. - Ships live. React on Vercel, FastAPI on a Hugging Face Docker Space, Neon Postgres and Cloudinary for data and images.
Training only 0.25% of 842M parameters.
SAM3 stays frozen; small LoRA adapters are added to the attention projections
(q_proj, v_proj) only. Less memory, faster training, and the
foundation model stays intact — while a text prompt per class drives the segmentation.
| Base model | facebook/sam3 · 842M |
| Trainable (LoRA) | 2.1M · 0.25% |
| Rank / Alpha / Dropout | 8 / 16 / 0.1 |
| Target modules | q_proj, v_proj |
| Loss | BCE·0.5 + Dice·0.5 |
| Hardware / Epochs | A100 40GB · 10 |
| Image → mask | 1024 → 288² |
The results: loss fell, IoU climbed.
Across 975 validation images the model reaches a 0.56 mean IoU over 19 classes, with no overfitting (train and validation plateau together at epoch 9–10). Performance tracks how clean a class's geometry is: structural components score highest, thin ambiguous damage lowest.
Strongest: components with clear geometry — PEquipment (0.81), Drainage (0.75). Weakest: Cavity (0.19), visually ambiguous and irregular. That gap is the clearest next target for more data and targeted augmentation.
From request to mask, with a swappable model seam.
Uploads return fast and analysis runs in the background. The app never imports the heavy ML stack unless the real detector is switched on, so the service boots instantly in mock mode and connecting the trained model is a single environment variable.
Upload & queue
React SPA sends the image; FastAPI stores it (Cloudinary), writes a Photo row and returns immediately while a BackgroundTask runs the analysis.
19 text prompts, one pass each
The image is preprocessed once; each class name is passed as a text prompt, and the model returns a per-class segmentation logit.
Sigmoid, threshold, filter
A mask is made where probability > 0.5; classes covering under 1% of the frame are dropped as noise. Coverage and confidence are recorded per class.
Colored mask returned
An original-vs-mask panel with a legend is uploaded to Cloudinary; the dominant class and total damage coverage are written back to the record.