A Computational Framework for Multi-Agent Behavior Analysis: Integrating Geometric Vision Extraction and Social Attention Inference

Overview

A Computational Framework for Multi-Agent Behavior Analysis: Integrating Geometric Vision Extraction and Social Attention Inference

This work presents a computational framework for analyzing interactions and structure in multi-agent dynamical systems. The framework integrates geometric vision extraction — recovering pose and motion information from video — with social attention inference, modeling how agents attend to and influence one another during interaction. The application domain is multi-animal behavior and pedestrian motion; the methodological contribution is learning structured, higher-order interactions and multimodal joint future prediction.

Key Components

• Higher-order graph interactions: modeling directed interactions between agents using graph structures beyond pairwise relations
• Structured representations: pose estimation and trajectory extraction as structured state estimation for multi-agent systems
• Multimodal forecasting and joint future prediction: predicting distributional futures for interacting agents rather than single point trajectories
• Novel geodesic loss: a loss function on the geometry of the trajectory manifold that improves prediction performance
• Representation learning: end-to-end pipelines for spatiotemporal interaction modeling

Resources

• Code: social-behavior (social attention model, srnn-pytorch)