Collective Perception ⠯

AI

Data Visualisation

Research

What does a machine see when it watches a shole of fish or a murmuration in the sky?

Collective Perception uses SAM3 in conjuction with Claude to track and analyse behaviour in wildlife footage. We build a layered reading of each collective: detections, trajectories, clusters, headings, and a complete behavioural analysis. Five ways of seeing the same moment.

What emerges isn't just data. The system begins to develop its own language for things it was never designed to understand - consensus, hesitation, the ones that don't follow. A portrait of collective intelligence, observed by a machine learning what that phrase might mean to itself.

‍

The Process

We started with footage of animals moving in groups - pigeons over a hillside, squirrels close to camera, pronghorns darting across a landscape. From there, we used Claude Code to build a custom tracking pipeline inside ComfyUI, combining Meta's SAM3 segmentation model with Hungarian algorithm tracking to identify and follow individual animals across every frame.

⠘

The pipeline runs in two phases. First, SAM3 segments and tracks each subject, assigning a persistent ID across the sequence. Second, Claude reads every tenth frame alongside the SAM3 tracking data and conducts a behavioural analysis of the collective - then breaks that reading into five separate lenses, each interpreting the scene through a different analytical frame: movement, hierarchy, hesitation, deviation, consensus.

⠯

All of it - the tracking data, the IDs, the behavioural labels, the lens groupings, is all exported to Houdini as geometry. Every animal carries its reading with it. The footage becomes a dataset that can be re-lit, re-rendered, and re-interpreted.

⠳