AFUN: Towards an Affordance Foundation Model for Functionality Understanding

AFUN: A Foundation Model for Predicting Where and How to Interact with Objects

In robotics, "affordance" is the bridge between seeing an object and knowing how to use it. A computer vision model might identify a "mug." An affordance model understands the handle is the part for grasping. It also knows the required motion is a lift, not a slide. Most current models fail to generalize. They work in labs but fail in unstructured real-world environments.

Researchers developed AFUN to solve this. It is a foundation model for functionality understanding. It identifies the "where" (the interaction region) and the "how" (the 3D motion path). AFUN uses a large-scale, standardized data pipeline. This allows it to work across diverse tasks and environments. It can be deployed without retraining for specific robot arms.

The Problem

Current affordance methods are fragmented. Most approaches fall into two camps. They either localize the interaction region without specifying motion. Or they predict motion but lack the scale to generalize. Static segmentation tells a robot to "touch the handle." However, it offers no guidance on whether to pull, rotate, or lift.

Conversely, motion-focused methods often rely on 2D trajectories. These become ambiguous when translated into 3D space. This gap creates a bottleneck for open-world robotics. A functional model must integrate task-conditional segmentation with 3D motion prediction .

Without this, robots cannot interpret commands like "open the microwave" as executable motor paths.

How It Works

AFUN's architecture performs joint prediction. It outputs a functional mask and a 3D motion curve in one pass .

The logic rests on three pillars:

1. MetaQuery Conditioning: The model uses a "MetaQuery" mechanism. This interfaces a frozen Vision-Language Model (VLM) with downstream decoders. Small, learnable "MetaQuery tokens" are appended to the VLM input. These tokens carry high-level reasoning from the task instruction into the segmentation and motion heads.
2. Unified Data Pipeline: The authors built a pipeline to ingest heterogeneous data. This includes robot teleoperation, human videos, and simulations .

The pipeline converts these into a shared schema. Crucially, they track actual object motion instead of hand or gripper trajectories .

This ensures the model learns the functional result of the interaction. 3. Bézier Spline Motion Representation: To make motion actionable, the model predicts parameters for an anchored 3D Bézier spline curve. The curve starts at the contact centroid (the center of the mask). It follows a smooth path defined by predicted control points. This provides a structured trajectory for robot controllers to follow.

Numbers

AFUN shows significant gains across several metrics. In segmentation, AFUN achieves a mean gIoU (geometric Intersection over Union, a measure of mask overlap) of 69.3%. This outperforms the strongest baseline by 23.9 points [Table 1]. This delta represents a much more precise identification of functional parts.

For contact-point accuracy, AFUN is much more precise. The authors report a hit-rate gain of 12.7% to 61.3% over baselines [Table 2]. This means the model is far more likely to point the robot to the exact spot it needs to touch.

On the AFUN test set, 3D motion prediction is highly accurate. The model achieves an Absolute Displacement Error (ADE, the average distance between predicted and true points) of 0.098m. It also achieves a Final Displacement Error (FDE, the error at the end of the trajectory) of 0.139m [Table 3]. These low error values suggest the predicted paths are physically reliable.

Finally, the model works in the real world. Using a Franka Research 3 arm, the authors achieved a 90% average success rate [Table 6]. This was demonstrated on tasks like opening microwaves and drawers.

What's Missing

There are gaps a production engineer should consider.

First, the "open-world" capability has limits. The model struggles with completely novel objects . For example, it fails on items like sun visors or certain spray bottles. These are classes not well-represented in the training data.

Second, the model depends on high-quality RGB-D input. The 3D motion is generated by back-projecting masks into 3D space. Therefore, depth sensor noise will affect the Bézier curve. Inaccurate depth can lead to imprecise trajectories.

Third, the research focuses on rigid-body manipulation. The pipeline filters out many non-rigid tasks. Moving to soft-body or deformable objects remains an unsolved challenge for this architecture.

Should You Prototype This

Yes, if you are working on articulated object manipulation.

If your roadmap involves doors, drawers, or lids, AFUN provides a path to skip manual motion planning. It can be deployed without embodiment-specific fine-tuning. This is a major benefit for rapid prototyping. The Bézier spline output provides a smooth, orientation-aware path. This is much easier to use than raw point clouds or 2D tracks.

However, if your use case involves highly deformable materials, wait for future iterations. You should also ensure your hardware provides reliable depth sensing.

Code is reportedly available; see the project page at https://www.zhaoningwang.com/AFUN for the canonical link.

Figures from the paper