OpenSenseNova/SenseNova-Vision

TL;DR: SenseNova-Vision (SenseTime, Apache-2.0) reframes essentially all of classic computer vision — detection, OCR, keypoints, segmentation, depth, normals, multi-view point maps, camera pose — as native text/image generation from a single 7B unified multimodal model, with no task-specific heads or decoders. They released the weights, the 50M-example instruction-response corpus, inference/eval/training pipelines, and a tech report. The numbers are genuinely competitive with specialists on detection and depth, weaker on segmentation and multi-view geometry — a real data point in the "one generative model for all of vision" debate.

The idea

The bet is simple: a unified multimodal model (UMM) that can already generate text and images natively has all the output machinery vision needs. Symbolic outputs — categories, boxes, points, OCR strings, keypoints, camera parameters — are just text generation. Dense spatial outputs — segmentation masks, depth maps, surface normals, multi-view point maps — are just image generation. Compositional tasks (e.g., grounded conversation with masks) interleave both. Natural-language instructions plus optional visual prompts (points, boxes, scribbles) specify the task, target region, output schema, and decoding convention, so outputs remain decodable against standard benchmarks.

Training starts from an off-the-shelf pretrained UMM ("7B-MoT" — Mixture-of-Transformers, Bagel-lineage) and fine-tunes primarily on the SenseNova-Vision Corpus — 50M heterogeneous CV annotations converted into instruction-response examples — with auxiliary multimodal data to preserve general capabilities. The interesting downstream property: task variants are definable in language beyond fixed benchmark schemas, since the schema lives in the prompt, not the architecture.

What's actually released

Substantial: 7B-MoT weights, the 50M corpus, inference code and Gradio demo, benchmark/eval harness (including multi-view reconstruction and camera pose, added mid-July), a training pipeline adapted to open datasets, and end-to-end data prep docs. Tech report on arXiv. Apache-2.0 throughout.

Evidence

Strongest showing is structured understanding via text generation. It beats both prior unified models and detection specialists on several detection benchmarks:

Depth/normals via image generation are competitive with dedicated geometry models: NYUv2 AbsRel 4.0 / δ1 98.1 vs DepthAnything's 4.3/98.1 and MoGe-2's 3.5/98.0; best-in-table normals on ScanNet and NYUv2.

The unified formulation costs something elsewhere. Segmentation trails the specialist X-SAM (panoptic PQ 48.8 vs 54.7; RefCOCO 81.3 vs 85.1), though it leads on reasoning segmentation (63.2 val) and box-interactive (73.9). Multi-view geometry is respectable but clearly behind feedforward geometry models (ETH3D reconstruction accuracy 0.301 vs VGGT 0.177; CO3Dv2 AUC@30 80.1 vs DepthAnything3's 91.8) — notable mainly because a generative UMM does this at all.

Caveats

• The Grounding DINO baseline is the small Swin-T variant; specialist comparisons are somewhat favorable picks. In the "Vision Banana" comparison table their own DIODE depth δ1 (76.4 vs 91.7) is a visible weak spot.
• Heavy hardware: 1×80GB GPU for the demo, 8×80GB for the full benchmark, and 16–256+ 80GB GPUs for training. Not an edge model, and generation-based inference will be far slower than a DETR or SAM head per image — no latency numbers are given.
• The README doesn't specify the base UMM or how dense targets are encoded/decoded (RGB-coded depth? diffusion vs AR image head?) — that's in the tech report.

Try it

git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
cd SenseNova-Vision
bash setup.sh sensenova-vision && conda activate sensenova-vision

# one-off inference: segment "person" in an example image
bash scripts/run_sensenova_vision.sh inference binary_seg "person" examples/images/2.jpg

# or the Gradio demo (needs ~80GB GPU)
MODEL_PATH=/path/to/SenseNova-Vision-7B-MoT bash scripts/run_sensenova_vision.sh demo

Worth reading alongside the tech report (arXiv:2607.06560). Even if you never deploy it, the released 50M decodable instruction-response corpus is likely the most reusable artifact here.