Use Viam from an AI agent

An AI agent with a shell or an SDK can operate a Viam machine the same way a person or a program can. This page is written for the agent, or for the person setting one up. It covers what a machine is made of, how to reach it, what to do before moving anything, and what the platform does for you when things go wrong.

If you already have a key and a terminal, the shortest path is Drive a machine from the CLI.

The model in one screen

• A machine is a robot or device running viam-server. It has one or more parts; the main part is the one you talk to.
• A part serves resources: components (hardware such as an arm, a camera, a gripper, a motor) and services (software such as motion planning, vision, or data management). Every resource has an API (its type, for example rdk:component:arm), a model (the implementation, often from a module), and a name.
• Every API is defined in protobuf and served over gRPC. The machine can list its resources and describe its methods at runtime, so you can learn a machine without documentation for its hardware.
• The frame system is the machine’s kinematic tree: where each component is relative to its parent and to the world frame. The motion service plans against it.
• A key authenticates you. Keys are scoped to a machine, a location, or an organization. Today every key minted from the CLI has full write access to its scope, and a machine checks only that a key is valid, not what it may do.

Ways in

The two paths call the same API. Anything you can do with one you can do with the other; the difference is whether you want a process that stays connected.

Look before you move

The order that works, and the reason for each step:

1. List the resources (ResourceNames) and their state (GetMachineStatus). A resource that is still configuring or unhealthy will fail every call; the status says why.
2. Read the frame system (FrameSystemConfig) and each component’s kinematics and geometry. That tells you which frames exist, what is attached to what, and what the planner already knows about.
3. Observe before acting: an image, a point cloud, a sensor reading. A vision service can return an image, detections, and object point clouds in one call (CaptureAllFromCamera).
4. Plan motion through the motion service rather than commanding joints or poses directly. Move takes a goal pose in any frame, plus obstacles and constraints you declare, and returns collision-checked motion against the frame system’s geometry. The arm’s own MoveToPosition is unplanned.
5. Verify by observation, not by return value. A gripper’s Grab reporting success does not prove it lifted anything; look, or call IsHoldingSomething after the lift.

Two known limits of perception worth planning around: a 3D segmenter’s bounding-box center can include background points behind an object’s edges, so it reads farther from the camera axis than the object is; and depth is least reliable at the near edge of a camera’s range. Observing from directly above an object, from a moderate height, avoids both.

What the machine will not tell you

Most of what follows holds for any Viam machine, and none of it is discoverable from the machine’s own configuration or method listing. A few points are specific to a cell like this one (an arm with a wrist-mounted camera next to a gripper) and are marked where they appear. Read this before you plan a motion.

Moving with the motion service

The motion service plans a path for a component and executes it on the arm. Give it the name of the component you want moved, the destination as a pose in a named frame, and any constraints:

from viam.proto.common import Pose, PoseInFrame
from viam.proto.service.motion import Constraints, LinearConstraint
from viam.services.motion import MotionClient

motion = MotionClient.from_robot(machine, "builtin")

# Move the gripper's frame to a point 100 mm above the block, pointing straight down.
above_block = PoseInFrame(
    reference_frame="world",
    pose=Pose(x=600, y=100, z=880, o_x=0, o_y=0, o_z=-1, theta=0),
)
await motion.move(component_name="pick-grip", destination=above_block)

# Descend on a straight line. Use a linear constraint for the last 100 mm of any
# approach, and for every move while holding something.
straight = Constraints(
    linear_constraint=[LinearConstraint(line_tolerance_mm=5, orientation_tolerance_degs=5)]
)
at_grasp = PoseInFrame(reference_frame="world", pose=Pose(x=600, y=100, z=784, o_z=-1))
await motion.move(component_name="pick-grip", destination=at_grasp, constraints=straight)

Plan the long part of the approach as a free move to the standoff pose, then one linear move straight down, and one linear move back up. Keep the linear constraint for that short final segment. A descent built from many small linear steps tends to fail: with a linear constraint the motion service solves for a direct straight line and will not fall back to a curved path, so a single segment with no direct solution returns linear with cbirrt not allowed and no direct solutions found. Read that as this exact straight line being infeasible, not the goal being unreachable, and either widen the tolerance or plan that segment as a free move.

component_name is the component’s name as a string. Earlier versions of the SDKs took a ResourceName message here. If you pass one now, the call fails with bad argument type for built-in operation, which is the protobuf library rejecting the message where it expects a string. The same applies to get_pose.

get_pose reports where a component’s frame is, in any other frame:

gripper_in_world = await motion.get_pose(component_name="pick-grip", destination_frame="world")

Units and frames

Poses are in millimeters and degrees. Point clouds are in meters. The orientation of a pose is an orientation vector: o_x, o_y, o_z give the direction the frame’s z axis points, and theta is the roll about it. A gripper pointing straight down at the table has o_z=-1.

A vision service returns objects in the frame of the camera it read. To use them as motion targets, convert them with transform_pose on the machine client, which applies the frame system:

from viam.proto.common import PoseInFrame

camera_frame_center = PoseInFrame(reference_frame="wrist-cam", pose=obj.geometries.geometries[0].center)
world_center = await machine.transform_pose(camera_frame_center, "world")

The frame system holds fixed things: the arm’s base, the cameras, the table, a place pad. Read it with get_frame_system_config. Things that move are not in it. Read those from vision services, and pass them to move as obstacles in a WorldState when a path must avoid them.

Waiting

Pass a timeout on every call. A camera or vision read that takes more than a few seconds is stuck, not slow, and a call without a deadline waits forever. If a call does not return, get_operations on the machine client lists it, and cancel_operation ends it.

Some drivers report is_moving as true while the arm is still. Judge motion by change in joint positions over a short window, not by that flag.

Grasping

Grasp at or above the object’s center height. A grasp near the surface it rests on stalls the jaw on that surface, not on the object.

grab returns whether the gripper holds something. is_holding_something is the check to make after any move while carrying. When a client’s session ends, viam-server stops every actuator that session commanded. Whether a held object survives that stop depends on the gripper model: some models keep holding it, and on others the stop releases the drive and drops it, so check is_holding_something and keep a carry sequence in one session. A gripper that closed on nothing stops where it is, with no force, until the next command.

Do not read a held object with a wrist-mounted camera: a held object is well under the camera’s safe minimum range, so it fills most of the frame and a segmenter returns a biased center that reads like the grasp shifted when it did not. If you need to check the scene before releasing, read it from a fixed camera the arm does not block, not from the wrist camera.

Looking

With a wrist-mounted camera, read from 300 mm or more above an object. Closer than that, the gripper’s own fingertips enter the frame and color detectors find them. A fixed overhead camera, or a machine with no gripper, has no such limit. Check a vision result once against an independent depth read, then trust it; the check costs one call, rebuilding perception costs ten minutes. The first frame after a machine boots can be stale. If a reading is far from where the scene should be, read again.

What the platform does for you

• Stops actuators when your session ends. Each client connection carries a heartbeat. If it lapses because your process exits or crashes, the machine stops the actuators that session commanded, within about two seconds. Consequences: from the CLI each command is its own session, so a motion started by one command stops when that command exits; and on some gripper models the stop releases the drive, dropping what it holds. Sequences that must hold an object belong in one process with one client.
• Names every operation and lets you cancel it. GetOperations lists what is running; CancelOperation stops one; StopAll stops everything in one call.
• Refuses motion it cannot plan. A goal inside the table, or inside an obstacle you declared, fails with an error that names the constraint.
• Converges configuration. When you add a component or service, the machine applies it within about ten seconds; GetMachineStatus reports the revision it reached and each resource’s state.

Two things the heartbeat does not cover today: motion-service moves continue after the client that started them disappears, and there is no per-resource permission or approval step. A key that can read a camera can also move an arm. Use StopAll when a move must end, and give an agent a machine-scoped key rather than a location or organization key when it only needs one machine.

Remembering across sessions

Machines, parts, locations, and organizations each carry a metadata document you can read and write through the app API: a place for notes such as calibration offsets or what worked last time. It is a single JSON object replaced whole on every write, with no revision check, so two writers can overwrite each other; keep notes small and read before you write.

Growing the machine

A procedure that works can become a permanent capability of the machine. viam module generate scaffolds a module, viam module reload hot-loads it onto the running part, and the new resource then appears in ResourceNames like any built-in. See Build and deploy modules.

Reading errors

Next

• Drive a machine from the CLI: every command above, with the exact method names and request shapes.
• Connect to a machine and Control components for the SDK path.
• Motion service for planning, obstacles, and constraints in depth.