AI Technology Brief
Detection, segmentation, and tracking can organize construction evidence - if the source remains visible
StructuredLayer evaluated a broader construction visual-evidence architecture beyond one detector family. Grounding DINO, SAM 2, SAMURAI, YOLO-World, Grounded SAM, Florence-2, OWL-ViT, semantic segmentation, enhancement, and specialist classifiers can prepare boxes, masks, labels, captions, and tracks. This is architecture evaluation, not completed deployment or autonomous inspection.
01 / Independently verifiable claims
Begin with what the technology and standards actually support.
- Grounding DINO and OWL-ViT support open-vocabulary detection using text concepts; YOLO-World is a separate real-time open-vocabulary detector.
- SAM 2 supports prompted segmentation in images and video; SAMURAI adds motion-aware tracking logic around SAM 2.1 rather than replacing it with a standalone foundation checkpoint.
- Grounded SAM is a composed pipeline combining grounding and segmentation components, not one model. Every component and checkpoint must be versioned.
- Florence-2 is a general vision model supporting multiple prompted tasks. Semantic Segment Anything, Mask2Former, RAM-Grounded-SAM, YOLOX, mask makers, clothing, hair, face, foreground, and content classifiers solve narrower tasks with different datasets and licences.
- Boxes, masks, labels, tracks, and captions are different evidence objects. Catalogue speed, popularity, and commercial-use summaries are not deployment evidence.
02 / The practical distinction
Detection, segmentation, tracking, and whole-image classification are different tasks.
Select the task from the accepted business output and preserve the source representation.
Bounding boxes
Grounding DINO, YOLO-World, OWL-ViT, YOLOX, and related detectors locate candidate objects and classes.
Pixel masks
SAM 2, Grounded SAM, Mask2Former, semantic segmentation, foreground, clothing, and hair tools produce selected or labeled regions.
Temporal tracks
SAM 2 video and SAMURAI follow prompted objects across frames; tracking errors can accumulate during occlusion and scene change.
Scene or content labels
Florence-2 and specialist classifiers can label or describe images, but labels depend on task, training data, and threshold.
03 / Operating architecture
Preserve every component and evidence object in a client-owned visual record.
Do not return only an annotated image. Store the original, prompt, raw regions, masks, tracks, confidence, versions, and review state.
Source media
Project, image or video, frame, timestamp, fingerprint, quality, permission, sensitivity, and retention.
Task pipeline
Detector, segmenter, tracker, classifier, prompts, points, boxes, thresholds, component versions, and device.
Evidence output
Classes, scores, boxes, masks, tracks, frame ranges, captions, crops, warnings, and raw serialization.
Reviewed record
Correction, accepted observation, exception, issue or annotation, reviewer, downstream action, and audit.
04 / Required records
A composed visual pipeline requires component-level provenance.
Source
Original file, checksum, project, frame, time, capture, rights, quality, and location policy.
Prompt
Text classes, negative definitions, click points, initial box, reference frame, task, and expected result.
Components
Grounder, detector, segmenter, tracker, classifier, enhancer, model cards, checkpoints, licences, and order.
Candidate
Box, mask, track, label, score, frame, source link, overlap, conflict, and uncertainty.
Exception
Miss, false alarm, identity switch, drift, occlusion, prompt ambiguity, quality, owner, and resolution.
Decision
Reviewer correction, accepted record, issue, annotation, use, approval, timestamp, and history.
05 / Construction example
Use the narrowest visual evidence object that answers the construction question.
People retain safety, quality, progress, commercial, and professional authority.
Text-prompt detection
Locate candidate barriers, equipment, signs, openings, or materials from approved class prompts for review.
Prompted masks
Create masks for visible surfaces, access zones, objects, or annotation preparation without treating pixels as physical quantities.
Video tracking
Track an approved object through footage for review while exposing identity switches, occlusion, lost tracks, and frame ranges.
Document and scene parsing
Use Florence-2 or specialist pipelines to prepare captions, regions, and annotations tied to the original image.
06 / Deterministic controls
Control prompts, versions, thresholds, privacy, and downstream release.
Class definitions
Define positive, negative, ambiguous, occluded, small, and visually similar examples.
Component registry
Verify repository, checkpoint, licence, lifecycle, input, output, and compatibility for every pipeline stage.
Source-visible review
Show original frames with selectable boxes, masks, tracks, scores, prompts, and warnings.
Privacy minimization
Prohibit unnecessary identity, protected-attribute, workforce, neighboring-property, or hidden surveillance uses.
No physical inference
Do not convert pixel extent into quantity, distance, condition severity, or compliance without calibrated methods.
Human authority
Block autonomous safety, defect, progress, payment, discipline, compliance, and professional conclusions.
07 / Failure analysis
Pipeline composition can multiply errors and hide which component failed.
Prompt ambiguity
Text labels, synonyms, class boundaries, and context can shift detections.
Mask leakage
Segmentation may include background, omit thin parts, merge objects, or fail at boundaries.
Tracking drift
Tracks can switch identity, disappear, or attach to a similar object after occlusion.
Component mismatch
Boxes, points, image size, coordinates, and preprocessing may not align across models.
Dataset transfer
General datasets do not represent client-specific trades, cameras, objects, and conditions.
Flattened output
An annotated preview can hide raw scores, masks, missing detections, and component versions.
08 / Deployment and cost
A composed pipeline needs reproducible packaging and replaceable components.
Private pipeline
Control media, weights, components, GPU, logs, access, updates, and storage.
Managed components
Verify retention, training use, subprocessors, region, terms, limits, cost, and exit.
Edge tracking
Reduce transfer but manage device identity, health, storage, updates, and fallback.
Client-owned evidence
Keep source files, prompts, outputs, corrections, decisions, and metrics outside model services.
- Media capture, transfer, frame extraction, storage, retention, and deletion
- Prompt and class design, annotation, masks, tracks, ground truth, and review
- GPU, serving, model weights, composed dependencies, scaling, monitoring, and updates
- Inference, retries, identity switches, exceptions, corrections, and downstream integration
- Privacy, licensing, contracts, security, incident response, and workforce governance
- Regression testing, component replacement, drift, and complete cost per accepted observation
09 / Evaluation
Evaluate boxes, masks, tracks, and downstream outcomes separately.
- Detection precision, recall, misses, false alarms, calibration, and class confusion
- Mask overlap and boundary quality against accepted annotations
- Track continuity, identity switches, lost tracks, recovery, and frame accuracy
- Performance by camera, view, lighting, weather, scale, occlusion, clutter, and quality
- Prompt stability and disagreement across synonyms and pipeline components
- Reviewer correction, critical errors, privacy, latency, and complete accepted-outcome cost
- Failure, version change, licensing, reproducibility, backup, and recovery
- Errors reaching safety, quality, progress, payment, compliance, or workforce decisions
10 / Controlled pilot
Prove the operating boundary before expanding it.
Choose one output
Select boxes, masks, or tracks for one approved low-consequence class set.
Freeze components
Record every repository, checkpoint, prompt, threshold, and preprocessing step.
Build ground truth
Include normal, ambiguous, occluded, small, crowded, and adverse examples.
Preserve raw evidence
Store source frames, raw outputs, masks, tracks, scores, and warnings.
Keep read-only
Create candidates and review queues only.
Set stop conditions
Stop for critical misses, drift, privacy harm, licence uncertainty, or excessive correction.
11 / StructuredLayer recommendation
Use detection, segmentation, and tracking as source-linked annotation layers, not as substitutes for inspection or project truth.
Start with one evidence object and one class set. Preserve the original media, every component version, prompts, boxes, masks, tracks, scores, and corrections. Expand only after representative construction testing, privacy and licence review, stable failure behavior, and accountable human approval.
12 / Primary sources
Capability, governance, and implementation claims remain inspectable.
IDEA Research
Grounding DINO repository
Meta
SAM 2 repository
SAMURAI
SAMURAI repository
AILab-CVC
YOLO-World repository
IDEA Research
Grounded Segment Anything
Microsoft
Florence-2 model card
OWL-ViT model card
NIST
AI Risk Management Framework
Sources reviewed 22 July 2026. Technology capabilities, laws, guidance, terms, and pricing can change.