M³-EDIT: An Agentic Framework for Multi-modal, Multi-reference, Multi-task Editing

Advait Gupta, Rishie Raj, Ruohan Gao

M³-EDIT is an agentic framework for multi-modal, multi-reference, multi-task editing that addresses the critical grounding failures of existing agents, which struggle to interpret complex, cross-modal instructions. To solve this, M³-EDIT employs a novel two-stage architecture that strictly decouples perception from action: an initial Analysis & Grounding stage constructs a verified, machine-readable map of all inputs called the Unified Editing Canvas (UEC), and a subsequent Manipulation & Execution stage operates on this fully-grounded canvas to generate and execute a precise, step-by-step editing plan. This design enables robust, compositional workflows and allows for granular, step-by-step quality control, a crucial advantage over monolithic models. The framework’s effectiveness is validated on M³-BENCH, our new benchmark of 110 complex, multi-reference tasks designed to test the limits of agentic multimodal reasoning.

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Gabriel Sarch, Zhuang Liu

AgentVQA is a unified benchmark for agentic visual understanding that systematically evaluates the agentic capabilities of Vision-Language Models (VLMs). It addresses the need for a comprehensive evaluation framework by unifying 14 challenging datasets across five critical domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Understanding. To ensure standardized and scalable analysis, AgentVQA reformulates diverse and dynamic tasks, such as web navigation and gameplay, into a unified multiple-choice question (MCQ) format. The benchmark is designed to be challenging, featuring “hard negatives” that are generated through a VLM-assisted pipeline and manually reviewed to ensure they are plausible yet incorrect, thereby testing for fine-grained understanding.

FaSTA*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing [Project]

Advait Gupta, Rishie Raj, Dang Nguyen, Tianyi Zhou

FaSTA* is a neurosymbolic online learning, tool-use agent with fast-slow planning for complex multi-turn image editing tasks. It decomposes a task into subtasks and calls a sequence of AI tools to address each subtask. By learning a library of frequently used subroutines (subsequences of tools), it can rely on fast planning for most subtasks, and occasionally, lazily activate slow planning (which requires A* search) for rare and challenging subtasks that the learned library of subroutines cannot handle.

CoSTA*: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing [Project]

Advait Gupta, NandaKiran Velaga, Dang Nguyen, Tianyi Zhou

CoSTA* is a hybrid agent for multi-turn image editing that combines LLM-based reasoning with A* search for cost-efficient tool selection, balancing cost and quality. Unlike text-to-image models like Stable Diffusion and DALLE-3, which struggle with complex edits and retaining input image details, and agents like GenArtist, CLOVA, and VisProg, which perform poorly on multimodal, multi-step tasks, CoSTA* constructs an optimal toolpath using LLM-guided hierarchical planning and A* search. It dynamically adapts by refining tool effectiveness through real-time feedback from a VLM, ensuring robust and efficient execution.

From Above and Beyond: Decoding Urban Aesthetics with the Visual Pollution Index

Advait Gupta, Manan Padsala, Devesh Jani, Tanmay Bisen, Aastha Shayla, Gargi Srivastava

From Above and Beyond is a framework for quantifying and visualizing urban aesthetics by creating a “Visual Pollution Index” (VPI). The system leverages drone-captured imagery and a fine-tuned YOLOv6 model to detect and categorize a wide spectrum of visual pollutants, from graffiti and potholes to cluttered sidewalks. Its core contribution is the VPI, a metric that holistically assesses an area’s aesthetic quality by multiplicatively combining the diversity of pollutants present (Counting Categories Ratio) with their perceived intensity (Severity-Weighted Score). This index is then used to generate intuitive, geospatial heatmaps, providing a data-driven and actionable tool for urban planners to identify and address areas of visual degradation. It is among the top papers to be accepted for publishing in Springer’s Studies in Computational Intelligence.

From Sky to Strategy: Construction Activity Index and Stage Estimation From Drone-Captured Imagery

Advait Gupta, Manan Padsala, Aastha Shayla, Tanmay Bisen, Susham Biswas, Abhemanyu Sarin

From Sky to Strategy introduces a framework for quantifying and visualizing construction site activity using drone-captured imagery. The system leverages a fine-tuned YOLOv6 model to accurately detect and classify various construction materials from a unique, high-resolution aerial dataset. Its core contribution is the Composite Construction Activity Index (CAI), a metric that provides a holistic view of site activity by multiplicatively combining the diversity of materials present (Material Diversity Ratio) with their spatial coverage (Percentage of Image Covered). The framework also maps detected materials to specific construction phases to estimate a project’s lifecycle stage, and culminates in the generation of geospatial heatmaps to provide an intuitive, data-driven tool for stakeholders in construction monitoring and urban planning.

Detection of pneumonia from chest X-ray images using transfer learning on deep CNN

Advait Gupta, Manan Padsala, Pallabi Saikia