Vision & multimodal papers
What AI researchers are reading: the papers the Hugging Face community picks each day, and new arXiv submissions in the directions we follow. Titles and abstracts are arXiv’s; every paper links to arXiv.
Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training
Emre Kutlusan, Yongmin Li
Foundation models pretrained on large natural-image collections are increasingly adapted for medical image segmentation, in the expectation that they will replace models trained for each task. Most comparisons, however, use data from the training distribution and a single task-specific baseline whose configuration…
Toward Foundation Detection Models via Example-Based Grounding in EM-DETR
Sheethal Bhat, Bogdan Georgescu, Awais Mansoor et al.
DeepLearning (DL)-based Computer-Aided Diagnostic (CAD) systems have shown great promise in supporting clinicians, but their performance heavily depends on the availability of large, well-annotated datasets. In medical lesion detection, the scarcity of comprehensive, stan dardized annotations and the heterogeneity…
Emergent Biological Capabilities in a Foundation Model for Molecular Interactions
Adi Mashiach
Foundation models trained on large unlabeled corpora develop emergent capabilities, tasks that go beyond the training objective. We extend this paradigm to multimodal biology with Bonbon, a foundation model for molecular interactions trained on approximately 3.2 trillion tokens of paired protein-ligand sequences.…
Measuring Robustness and Efficiency in a Connectome-Constrained Fly Visual System Model: Linking an Adversarial and a Perceptual Illusion
Alexandre Garcia-Duran, Alejandro Rodriguez-Garcia, Manuel Molano-Mazón et al.
Biological visual systems compute reliably under tight metabolic and wiring constraints, whereas artificial vision systems consume far more energy and can be sensitive to adversarial perturbations. Whether networks with brain-like wiring are more robust at lower computational cost is unclear. We address this by…
BoviShift: A Six-Year Multi-Attribute Audit of Visual Cattle Phenotyping Across Animals and Time
Mobasshir Bhuiya Shagor, Nilambar Halder Tonmoy, Md Saef Ullah Miah
Computer vision offers a scalable, non-contact means of estimating livestock traits, but correlated images and video frames make reliable evaluation difficult. Image-random splitting can place views of the same animal in both training and test sets, while deployment further requires transfer to animals captured in…
C$^2$-Routing: Complementary Cropping and Compression for Efficient Vision-Language Inference
Guorui Lu, Qinyu Chen
Visual Question Answering (VQA) imposes substantial computational overhead on vision-language models (VLMs), motivating model-agnostic input reduction methods that can be readily applied across diverse VLMs, including in black-box settings. Simply enforcing a visual-token cap reduces computation, but it can discard…
Bayesian inverse problems and epidemic-threshold uncertainty for a heterogeneous SIS reaction-diffusion model
Duc-Lam Duong
We study Bayesian inference of persistence from early observations of a heterogeneous susceptible--infected--susceptible (SIS) reaction--diffusion model with standard incidence. We show that the leading prevalence dynamics depend on the unknown coefficients only through the difference between the transmission and…
UGG: Unified Generative Geometry Model
Amani Kiruga, Zhiyi Li, Ruojin Cai et al.
Understanding 4D scenes is a core challenge in visual intelligence, requiring models to jointly capture scene geometry, appearance, and dynamics from partial observations. However, existing approaches are subdivided into feedforward models for direct geometric prediction, and generative models for visual synthesis.…
Metric-Aligned Group Relative Policy Optimization for Explainable Deepfake Detection
Luka Dragar, Peter Peer, Vitomir Štruc et al.
Deepfake detectors are commonly evaluated on real/fake classification alone, but real-world applications require explanations of the visual evidence behind each prediction. Joint vision-language models struggle with authenticity assessment, producing "real" narratives for manipulated images. To address this, we…
One Policy for 100 Household Tasks: Our Solution for the 2026 BEHAVIOR Challenge
Ilia Larchenko, Gleb Zarin, Atindra Nair et al.
We describe our solution for the 2026 BEHAVIOR Challenge: 100 long-horizon household tasks in the OmniGibson simulator. We placed 9th on the final leaderboard, the only independent team in the top 10. The core of the solution is a single vision-language-action (VLA) model, one checkpoint for all 100 tasks, built on…
OnlyAct: Extracting Action Representations from Foundation Models for World Models
Runjia Li, Minghao Chen, Junyu Xie et al.
Video world models promise general-purpose interactive simulators, but their training is limited by action supervision: action labels are scarce, fragmented across incompatible specifications, and do not scale with the unlabeled video pretraining corpus. A common workaround jointly trains an action encoder to…
Blurring the Block Boundaries: Overlapping Block Denoising for Efficient Diffusion Language Models
Amir Mann, Jonathan Fhima, Shaked Brody et al.
Diffusion Language Models (DLMs) have emerged as effective alternatives to autoregressive models for latency-critical tasks, enabling faster token generation. Recent advances employ block diffusion with self-editing capabilities, where each block refines tokens iteratively before advancing to the next one. Yet each…
What Factors Matter for Collection-Level Counting with Human-AI Collaboration?
Seoyun Jeong, Logan Lawrence, Owen L. Martin et al.
Many scientific and policy decisions require estimating the total number of target objects across a large image collection. Exhaustive human labeling is often infeasible, while raw model predictions can be biased, especially under domain shift. Recent human-AI estimation methods combine imperfect model predictions…
From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
Guoqiang Liang, Yiqi Lin, Yanzhe Chen et al.
Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural…
Forward-Value Discretization: Faster Stochastic Diffusion Sampling Without High-Order Approximation
Yuchen Jiao, Na Li, Changxiao Cai et al.
Stochastic diffusion samplers often require substantially more neural function evaluations (NFEs) than deterministic samplers to achieve comparable sample quality. Although higher-order discretization accelerates deterministic sampling, its benefits for stochastic sampling remains unclear. We establish matching…
Multimodal Abstention That Preserves Acquired Correct Answers
Jinyuan Zhang
Adding abstention to a task-adapted multimodal model can discard correct answers, including capabilities acquired during adaptation. We formulate abstention repair as maximizing correct abstention on unanswerable questions under a budget on newly lost correct answers. We propose ReFrain (Regression-Free…
TaoMate-R1: Temporal Role Specialization for Streaming Multi-Reference Video Generation
Qijun Gan, Chenwei Zhang, Xinyang Li et al.
Streaming multi-reference video generation establishes a scene from specified people and objects, then continues it under changing instructions. We introduce TaoMate-R1, a streaming multi-reference generator that organizes adaptation parameters through temporal role specialization. It assigns separate low-rank…
A Cross-Scenario Survey and Analysis of Self-Evolving Multimodal Agents
Jingjing Jiang
Recent work shows that multimodal agents increasingly self-improve without recurring human annotation. However, each self-improvement mechanism is designed and evaluated within a single scenario, leaving its preconditions unverified under cross-scenario transfer. This survey investigates the structural source of…
Location Matters: Sparse Latent-Variance Attacks for Face Privacy against Diffusion Personalization
Shuting Yang, Shuxin Li, Linyan Dai et al.
Text-to-image diffusion models can learn and reproduce an individual's identity from only a few reference photos, creating risks of unauthorized and misleading content generation. Proactive defenses address this risk by adding adversarial perturbations to facial images before release, thereby disrupting identity…
ViGA-FLAME: Aligning Latent Visual and Geometric Representations for Robust 3D Head Expression Estimation
Wen Hao He, Jiacheng Wu, Jie Chen
Estimating 3D head expression from a single image requires mapping visual appearance to underlying facial geometry. We propose ViGA-FLAME, a two-stage framework that aligns semantic visual features with a structured geometric latent space. First, a Parametric-Geometric Variational Autoencoder (PG-VAE) learns a…
Few-Step Streaming Video Generation via Condition Shifting
Deyuan Liu, Jungang Li, Kaiwen Zheng et al.
Few-step sampling reduces the cost of streaming video generation. Distribution Matching Distillation (DMD) supervises few-step generators with a separate fake-score network, which adds training memory and computation and must track the evolving generator. We extend APEX to streaming video by using condition…
PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation
Huiyi Chen, Weitai Kang, Shicheng Fan et al.
As text-to-video (T2V) models are increasingly positioned as world simulators, it is essential to evaluate whether their videos conform to physical commonsense. However, existing benchmarks rely on annotators who rate each video with a single score based on intuition, yielding ratings that are subjective and do not…
Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies
Chubin Zhang, Bo An
Memory-augmented vision-language-action (VLA) policies are meant to remember what happened. Many of them, however, fill memory by time: they sample frames uniformly across the episode and stamp each frame with its time step. Because time passes even when nothing happens, such a memory may not tell waiting from…
EVAThink: Evidence Sufficiency-Guided Training for Adaptive Thinking in Video Understanding
Changti Wu, Ruizhe Fan, Bin Wang et al.
Under a fixed frame budget, video understanding with vision-language models (VLMs) depends on whether the sampled frames provide sufficient evidence for the question. Sufficient frame evidence supports direct answering, whereas less informative observations may benefit from reasoning that connects the available…
Visual Similarity in Adult Word Extension and Artificial Word Learning
Patricio M. Vera
When two objects share a name, how much does their visual similarity explain, and what do the teaching examples add? First, we reanalyzed trial-level data from a large adult word-extension study (Lewis & Frank, 2018), computing each test image's distance to the examples shown in the space of a frozen…
Lipschitz-Controlled Flow Matching for Robust Image Restoration
Sedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner et al.
Flow-matching models can produce high-quality image restorations, but their learned dynamics may amplify small input perturbations and make reconstructions unstable. We introduce LipsFlower to reduce this amplification. We derive stability bounds for Flow-from-Y, which initializes the ordinary differential equation…
dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang et al.
Discrete autoregressive modeling, notably through next-token prediction, has become a cornerstone of modern language models. Extending this paradigm to vision allows visual and language models to converge on a unified framework, thereby fostering shared methodology and infrastructure. This work investigates a…
PhysOPSD: Code as Privilege for Physically Plausible Video Generation
Harold Haodong Chen, Rui Peng, Lingteng Qiu et al.
While modern video generative models achieve remarkable visual fidelity, they remain noticeably brittle when generating physically plausible dynamics and complex interactions. Targeted post-training is crucial, yet supervised fine-tuning suffers from offline state mismatch, and reinforcement learning is hindered by…
Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
Yuji Nozawa, Yu-Chieh Lin, Youyang Ng
Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly…
EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving
Fenghai Li, Bingxuan Li, Jiawen Zhang et al.
Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual…
WorldFoundry: Towards Unified Infrastructure for World Intelligence
Juanxi Tian, Jiadong Hou, Baiang Li et al.
Visual generative models increasingly blur the line between synthesizing observations and simulating environments. Yet visual plausibility alone does not constitute successful world modeling: compelling outputs may lack persistent state, physical dynamics, or causally consistent responses to interactions. We define…
JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation
Xiangjun Tang, Jianqi Chen, Peter Wonka
Generating high-fidelity voxels while keeping their lightweight nature remains an open problem in computer vision. Current generators are either high-fidelity but computationally heavy, or lightweight but of lower fidelity. Hierarchical methods keep the lightweight nature of voxels. However, their coarse levels,…
InstantHMR : Real-Time Human Mesh Recovery on Your Phone
Mohamed Adjel, Kahina Chalabi, Vincent Bonnet
Real-time 3D human mesh recovery from a single image on mobile devices is a challenging problem with many applications, including robotics, biomechanics, AR/VR, and exergames. Current state-of-the-art models are either too slow to run efficiently on mobile devices or sacrifice accuracy for speed. We present…
FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li et al.
Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity…
Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds
Chenhao Zheng, Zixian Ma, Kai Li et al.
Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive…
Stochastic Deep Supervision for Gallery-Consistent Early-Exit Face Recognition
Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira et al.
Vision Transformers (ViTs) perform strongly in face recognition (FR), yet their fixed-depth inference limits deployment in resource-constrained settings. We propose SD-ViT, a training framework that gives a usable exit at every block depth by making every depth produce discriminative embeddings, for budget-selected…