This is the Mux Robots documentation for managed AI video workflows, prerequisites, jobs, and Directives. # Use the Mux Robots API The Mux Robots API runs AI workflows on your videos with one API request. Use it to summarize, moderate, caption, or translate videos without your own AI pipeline. The Mux Robots API uses AI to extract structured data from your videos. Instead of integrating third-party AI services, managing async job queues, handling retries and failures, and writing custom glue code to tie it all together, you can send a single API request with an asset ID and get results back directly. The Robots API is our recommended, batteries-included approach for adding AI to your video workflows — it handles model selection, prompting, retries, and webhook delivery so you can focus on using the results. Robots is priced per Mux AI unit consumed by each job. The first 100,000 units each month are free. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for rates. Workflows that analyze, generate, or translate language share a set of [verified languages and language tags](/docs/guides/robots-supported-languages). Languages outside that set are processed on a best-effort basis. To run several workflows against the same asset in the right order, or to trigger them automatically when an asset is ingested, use [Directives](/docs/guides/robots-directives). Directives are the orchestration layer that dispatches Robots jobs on your behalf. If you want to build your own AI workflows from lower-level primitives (transcripts, embeddings, custom prompts, your choice of LLM provider), check out [`@mux/ai`](https://github.com/muxinc/ai) — our open-source TypeScript SDK. It's a great fit when you need to customize the pipeline beyond what the Robots API exposes. ## Choose an AI approach Start with the result you need and how much of the pipeline you want to own: | Approach | Choose it when | Examples | What you manage | | :-- | :-- | :-- | :-- | | **Mux Robots** | A [workflow from the full catalog](#what-you-can-do) fits and you want Mux to run it as a managed job. | Moderate uploads, generate chapters, summarize a library, or translate captions. Use Directives to coordinate workflows on new uploads. | Mux credentials, workflow parameters, prerequisites, and consuming the results. Mux manages models, provider credentials, execution, and job status. | | **[`@mux/ai`](https://github.com/muxinc/ai)** | You need custom prompts, model/provider choices, or a pipeline beyond the Robots controls. | Customize a video-analysis workflow or compose transcript and frame primitives with your own logic. | Your application runtime, chosen provider credentials and costs, orchestration, persistence, and failure handling. | | **Your own transcript + LLM call** | A narrow text task only needs an existing transcript and you are comfortable owning the integration. | Classify one transcript using an existing application prompt. | Transcript access, provider credentials and costs, context limits, retries, and result validation. Add a video/frame pipeline if the task needs visual evidence. | For production apps, prefer Robots when a built-in workflow meets the need. Use its documented parameters and `output_steering` controls before building a custom pipeline. Frame analysis is available in both Robots and `@mux/ai`; visual input alone does not require choosing one over the other. A transcript-only call cannot establish what is visible in the video. All three approaches may involve Mux asset/storage costs. Robots jobs consume Mux AI units; custom pipelines use your own AI-provider billing. See [pricing](/docs/pricing/overview) and the chosen workflow's prerequisites before running a batch. ## What you can do | Workflow | Category | Description | | :-- | :-- | :-- | | [Generate premium captions](/docs/guides/robots-generate-premium-captions) | Accessibility | Generate high-accuracy captions from a video's audio, with optional speaker labels and word-level timestamps | | [Edit a video's captions](/docs/guides/robots-edit-captions) | Accessibility | Edit an existing caption track with static replacements and profanity censoring | | [Translate captions](/docs/guides/robots-translate-captions) | Accessibility | Translate existing captions to another language | | [Translate audio](/docs/guides/robots-translate-audio) | Accessibility | Translate (dub) a video's spoken audio into another language | | [Summarize](/docs/guides/robots-summarize) | Insights | Generate a title, description, and tags for a video | | [Ask questions](/docs/guides/robots-ask-questions) | Insights | Ask questions about a video and get structured answers | | [Find key moments](/docs/guides/robots-find-key-moments) | Insights | Identify the most compelling moments in a video | | [Find best thumbnails](/docs/guides/robots-find-best-thumbnails) | Insights | Sample, score, and rank the best thumbnail frames from a video | | [Generate engagement insights](/docs/guides/robots-generate-engagement-insights) | Insights | Turn viewer engagement data into plain-language insights for a video | | [Generate contextual text](/docs/guides/robots-generate-contextual-text) | Insights | Create social posts, blog posts, and other written artifacts from video or audio | | [Analyze structure](/docs/guides/robots-analyze-structure) | Structure | Generate a title, summary, tags, chapters, scenes, and key moments together | | [Generate chapters](/docs/guides/robots-generate-chapters) | Structure | Create timestamped chapters from video content | | [Moderate](/docs/guides/robots-moderate) | Trust & Safety | Analyze a video for inappropriate content | Most workflows accept `output_steering` to guide style, audience, terminology, and which part of the video is analyzed. See each workflow's guide for its options. ## Workflow prerequisites Use a ready asset in the same environment as your token. It must no longer be live and must have a public or signed playback ID; DRM-only playback IDs are not supported. Pass the asset ID to the job, even when the workflow reads media through a playback ID. **Required** resources must be ready before a standalone job is created. **Recommended** resources improve output, but are not mandatory unless the table identifies a specific input or mode that requires them. | Workflow | Caption or transcript requirement | Other inputs and readiness | | :-- | :-- | :-- | | [Generate premium captions](/docs/guides/robots-generate-premium-captions) | No existing captions needed. | Requires audio. Reuses or temporarily creates an audio-only static rendition. Keep `upload_to_mux: true` when another workflow needs the generated track. | | [Edit captions](/docs/guides/robots-edit-captions) | **Required:** a specific ready source track, selected by `track_id`. | The track must belong to the asset. Check the replacement policy before writing edited captions back. | | [Translate captions](/docs/guides/robots-translate-captions) | **Required:** a specific ready source track, selected by `track_id`. | Choose `output_language_code`; check existing target-language tracks and the replacement policy. | | [Translate audio](/docs/guides/robots-translate-audio) | No caption prerequisite. | Requires audio. Reuses or temporarily creates an audio-only static rendition. For an audio-only asset, set `upload_to_mux: false` to receive downloadable audio. | | [Summarize](/docs/guides/robots-summarize) | **Recommended** for video; **required** for audio-only assets. | Video can fall back to sampled visual content. | | [Ask questions](/docs/guides/robots-ask-questions) | **Recommended** for video; **required** for audio-only assets. | Provide questions that the available transcript or visual evidence can answer. | | [Find key moments](/docs/guides/robots-find-key-moments) | **Required** for transcript-based selection. Optional for video with `use_shots: true`. | Shot-based selection generates or reuses shots and is not supported for audio-only assets. | | [Find best thumbnails](/docs/guides/robots-find-best-thumbnails) | No caption prerequisite. | Requires video; audio-only assets are rejected. Shot detection can improve sampling but is not a resource you must prepare first. | | [Generate engagement insights](/docs/guides/robots-generate-engagement-insights) | No caption prerequisite. | Requires accumulated Mux Data engagement metrics; a fresh asset without viewing data has nothing to analyze. | | [Generate contextual text](/docs/guides/robots-generate-contextual-text) | **Required:** ready captions or a supported audio-only transcript. | A storyboard adds context for video. `use_shots: true` generates or reuses shots and is rejected for audio-only assets. | | [Analyze structure](/docs/guides/robots-analyze-structure) | **Recommended** for video; **required** for audio-only assets. | Visual evidence can be used for video without captions. See the guide for shot-analysis controls. | | [Generate chapters](/docs/guides/robots-generate-chapters) | **Required:** ready captions. | The asset must also contain audio. | | [Moderate](/docs/guides/robots-moderate) | No caption prerequisite for video; **required** for audio-only assets. | Video moderation samples frames; audio-only moderation uses the transcript. | This table covers the public v1 workflows. Check each workflow's guide and API reference for duration limits, supported languages, and parameter-specific restrictions. ### Prepare and wait for captions For new uploads, [enable auto-generated captions](/docs/guides/add-autogenerated-captions-and-use-transcripts) in the asset inputs. For an existing asset, [request captions or upload subtitles](/docs/guides/add-subtitles-to-your-videos), or run [Generate premium captions](/docs/guides/robots-generate-premium-captions). Retrieve the asset and wait for the source text track to have `status: "ready"`. In v1, caption sources are subtitle tracks (`type: "text"`, `text_type: "subtitles"`). An audio-only asset may also use its single ready legacy text transcript without subtitle metadata. A chapter track is not a subtitle source. Asset readiness and caption readiness are separate: `video.asset.ready` does not guarantee the track is ready. Use `video.asset.track.ready` or bounded polling of the asset's tracks. When `language_code` selects a caption source, it must match the ready track's language exactly, including case and region: `en` does not match `en-US`. Without it, automatic selection uses the first ready subtitle in asset track order. Edit and translate captions instead identify the source with `track_id`. Retrieve that ID from the asset; do not use a job ID or a playback ID. If a source track is still preparing, wait. If it failed, is missing, or is in the wrong language, repair or choose the source before retrying the same job. A successful premium-caption job with `upload_to_mux: false` supplies a temporary subtitle URL, not a track that downstream workflows can select. ### Let a Directive prepare dependencies For recurring work, declare prerequisites as resources and reference them in each binding's `inputs`. This v1 configuration generates subtitles when needed and waits for them before chapter generation: ```json { "name": "Prepare subtitles and generate chapters", "subject": { "type": "video.asset" }, "resources": [ { "reference_id": "captions", "type": "video.asset.track", "kind": "subtitles", "source": { "via": "mux_api", "action": "generate_subtitles" } } ], "bindings": [ { "reference_id": "chapters", "workflow": "generate-chapters", "inputs": ["captions"], "parameters": {} } ] } ``` Create the Directive with `POST /robots/v1/directives`, then attach it to new assets or start a run for an existing asset as described in the [Directives guide](/docs/guides/robots-directives). A subtitle-generating action still needs an audio source. Inspect the run's node states if a resource or producer fails; declaring the dependency does not guarantee it can be produced. Shot generation may incur separate [Mux Video charges](/docs/pricing/overview#shots). ## How it works The Robots API is organized around **jobs**. A job represents a single unit of work — one row from the table above. Each type of work is called a **workflow**. The general pattern is: 1. **Create a job** by sending a POST request to the workflow endpoint with your Mux asset ID and any configuration parameters. 2. **Wait for the job to complete.** Jobs move through statuses: `pending` → `processing` → `completed` (or `errored`). A job can also be `cancelled` via the cancel endpoint. We strongly encourage using [webhooks](/docs/core/listen-for-webhooks) for status changes, but jobs can also be retrieved through the `GET /robots/v1/jobs/{workflow}/{JOB_ID}` endpoint. 3. **Read the results** from the job's `outputs` field once the status is `completed`. Jobs are automatically deleted after 30 days. ### Webhooks Every status change fires a [webhook event](/docs/core/listen-for-webhooks). This lets you react to completed jobs without polling. The event type follows the pattern `robots.v1.job.{workflow}.{status}`: | Event type | Description | | :-- | :-- | | `robots.v1.job.{workflow}.pending` | Job was created and is waiting to be picked up. | | `robots.v1.job.{workflow}.processing` | Job is actively running. | | `robots.v1.job.{workflow}.completed` | Job finished successfully. The full job object, including `outputs`, is in `event.data`. | | `robots.v1.job.{workflow}.errored` | Job failed. Check `event.data.errors` for details. | | `robots.v1.job.{workflow}.cancelled` | Job was cancelled before it completed. | For example, a completed moderation job sends a `robots.v1.job.moderate.completed` event. A completed summarization job sends `robots.v1.job.summarize.completed`. Note that multi-word workflow names use underscores in event types (e.g. `robots.v1.job.ask_questions.completed`, `robots.v1.job.find_key_moments.completed`), even though the API endpoints use hyphens. The webhook payload's `data` field contains the full job object — the same shape you'd get from polling the job endpoint — so you can read `outputs` directly from the event without making an additional API call. All Robots API endpoints live under `/robots/v1/`. ### Authentication The Robots API uses the same Mux API token authentication as the rest of the Mux API. Include your token ID and secret using HTTP Basic Auth. Select **Mux Robots** when creating an access token in the [Mux Dashboard](https://dashboard.mux.com/settings/access-tokens). Use **Read** (`robots:read`) to read workflows, jobs, and Directives, and **Write** (`robots:write`) to create jobs or manage Directives. Select both for a create-and-retrieve workflow. Also select Mux Video permissions for asset operations your integration performs. Existing tokens cannot gain permissions: create a replacement if yours lacks Robots access, then update your application credentials. Tokens must belong to the same environment as the assets and jobs. If the organization has not enabled Robots, an admin must complete the enablement step linked in the API error. ```bash curl https://api.mux.com/robots/v1/jobs/summarize \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` ### Job lifecycle Every job includes the following common fields: | Field | Description | | :-- | :-- | | `id` | Unique job identifier | | `status` | Current status: `pending`, `processing`, `completed`, `errored`, or `cancelled` | | `workflow` | The workflow type (e.g. `summarize`, `moderate`) | | `created_at` | Unix timestamp when the job was created | | `updated_at` | Unix timestamp when the job was last updated | | `passthrough` | An arbitrary string you can include when creating the job, returned as-is in responses | | `parameters` | The input parameters for this job. Varies by workflow — see each workflow's guide for details. | | `outputs` | Workflow results, present when status is `completed` | | `units_consumed` | Number of Mux AI units consumed by this job | | `errors` | Error details, present when status is `errored`. See the Robots API reference for the full error structure. | | `resources` | Related Mux resources linked to this job | | `directive` | The Directive and run that dispatched this job, with `id` and `run_id`. Absent for jobs created with a direct API request. | ### Managing jobs You can retrieve, list, and cancel jobs across all workflows. **Retrieve a job** by workflow and job ID: ```bash curl https://api.mux.com/robots/v1/jobs/{workflow}/{JOB_ID} \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` Replace `{workflow}` with the workflow name used when the job was created (e.g. `summarize`, `moderate`, `generate-chapters`). The response includes the full job object with `outputs` when the job is `completed`. **List jobs** with optional filters: ```bash curl "https://api.mux.com/robots/v1/jobs?workflow=summarize&status=completed&limit=10" \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` You can filter by `workflow`, `status`, `asset_id`, and paginate with `limit` (default 25, max 100) and `page` (default 1). Each entry is a summary with `id`, `workflow`, `status`, `created_at`, `updated_at`, and `_links.self`; follow `_links.self.href` to retrieve the full job. **Cancel a job** that is `pending` or `processing`. The job's status will transition to `cancelled`: ```bash curl -X POST https://api.mux.com/robots/v1/jobs/{JOB_ID}/cancel \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` # Run Robots workflows automatically with Directives Directives run multiple Mux Robots workflows on an asset in the correct order. Use them when you want workflows to start automatically, with no glue code or webhook listeners. Directives are the orchestration layer for [Mux Robots](/docs/guides/robots). A single Robots job runs one workflow on one asset. A Directive runs *several* workflows in the right order, and can trigger them automatically the moment an asset is ready. Reach for a Directive when you want to: * Run a few Robots workflows on the same asset (say, generate chapters **and** a summary **and** translate the captions). * Guarantee the prerequisites exist before a workflow runs (auto-generate captions before chapter detection, for example). * Trigger everything automatically when an asset is created, with no glue code, no webhook listeners, and no manual dispatch. You could wire this up yourself: subscribe to the Mux Video and Robots webhooks, dispatch each workflow, encode the ordering between them, and babysit the retries. A Directive trades all of that for a single declaration. You say what you want once, attach it to your assets, and Mux handles the rest. See the Directives API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Directives themselves are free; you pay only for the workflow runs they trigger. ## What you can do A Directive can run any Robots workflow, and each one takes the same parameters you'd set for a one-off Robots job. For the full list of workflows, see [What you can do](/docs/guides/robots#what-you-can-do) in the Robots guide. ## How it works A Directive has two parts: * **Bindings**: the workflows to run. * **Resources**: what those workflows depend on. You define both once, when you create the Directive. Every run after that uses the same configuration, whether you trigger it by hand or it fires automatically. When a run starts, the engine checks that each workflow has the resources it needs, resolves anything that's missing according to the resource's **source policy**, and then runs the workflows in the correct order. A Directive's configuration is locked in once you create it. You can delete a Directive, but to change what it runs, duplicate an existing Directive or create a new one and re-attach your assets. Deleting a Directive doesn't cancel runs already in progress; it just prevents new ones from starting. ## Create a Directive You can create a Directive from the [Mux Dashboard](https://dashboard.mux.com) or via the API. ### From the Dashboard In the **Directives** section, click **Create Directive**. You can start from scratch, or pick one of the prebuilt templates for common setups (like generating captions and then translating them) and customize it from there. Either way, give it a name, pick the workflows you want to run, set each workflow's parameters, and declare any resource dependencies. The Directives section of the Mux Dashboard with the “Create Directive” button highlighted. When you add a workflow that needs a resource you haven't declared yet, the editor adds that resource for you and picks a sensible source policy. Change it in the resource's inspector if you want different behavior. The Directive editor showing the Workflows and Resources panels side by side. ### From the API `POST /robots/v1/directives` with a Directive configuration: ```bash curl https://api.mux.com/robots/v1/directives \ -H "Content-Type: application/json" \ -X POST \ -d '{ "name": "Generate captions, then translate", "subject": { "type": "video.asset" }, "resources": [ { "reference_id": "captions_en", "type": "video.asset.track", "kind": "subtitles", "language_code": "en", "source": { "via": "workflow", "binding": "premium_captions" } } ], "bindings": [ { "reference_id": "premium_captions", "workflow": "generate-premium-captions", "parameters": { "language_code": "en" } }, { "reference_id": "translate_es", "workflow": "translate-captions", "inputs": ["captions_en"], "parameters": { "output_language_code": "es" } } ] }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` The response is a 201 with the new Directive, including its `id`: ```json { "data": { "id": "drv_abc123", "name": "Generate captions, then translate", "subject": { "type": "video.asset" }, "resources": [ /* … */ ], "bindings": [ /* … */ ], "_links": { "self": { "href": "/robots/v1/directives/drv_abc123" } }, "created_at": 1746403200, "updated_at": 1746403200 } } ``` Validation errors return a 422 with every problem collected into one message, so you can fix the config in a single pass. ## Configure resources and dependencies Some workflows need a specific resource on the asset before they can run. **Generate chapters**, for instance, reads from a caption text track. No captions, no chapter markers. A resource declaration has three parts: a `reference_id` (a stable handle you choose), a `type`, and a `source` (how the resource comes to exist on the asset at run start). ### Resource types | `type` | Describes | Selector fields | | :-- | :-- | :-- | | `video.asset.track` | A caption (subtitles) or audio track on the asset. | `kind`: `"subtitles"` or `"audio"` (required). `language_code`: BCP 47 code (optional; when omitted, any language of that kind matches). | | `video.asset.shots` | The asset's shot-detection output. | *None. Singleton per asset.* | ### Source policies The `source` field is a discriminated union on `via` that tells the engine how to satisfy the resource when it isn't already ready on the asset. `source` defaults to `{ "via": "external" }` when omitted. | `source` | What it does | | :-- | :-- | | `{ "via": "external" }` | Wait for the resource to appear and become ready on the asset. Use when you upload the resource out-of-band (e.g. captions uploaded directly to the asset) and want dependent workflows to hold until it lands. Fixed 24-hour wait cap. | | `{ "via": "required" }` | The resource must already be ready when the run starts. If it's absent, the dependent workflows fail immediately. Use this when "no resource, no run" is the behavior you want. | | `{ "via": "workflow", "binding": "" }` | Produced by another workflow binding in this Directive. Dependent workflows wait for that binding to complete and the resource to appear. See [Chain workflows together](#chain-workflows-together). | | `{ "via": "mux_api", "action": "", "parameters"?: {…} }` | The engine calls Mux Video to create the resource on demand, then waits for it. Supported actions: `generate_subtitles`, `create_track`, `generate_shots`. | Engine-initiated (`mux_api`) actions use the same Mux Video APIs you'd call yourself, so the same rules apply. Generating captions, for instance, needs an audio track on the asset. If the asset can't satisfy the underlying call (a silent video, say), the dependent workflows fail and the reason is reported in the run output. ### Workflow inputs Each workflow binding can declare `inputs`: a flat array of resource `reference_id`s the workflow depends on. Every referenced resource must be declared in the same Directive. ```json { "reference_id": "chapters", "workflow": "generate-chapters", "inputs": ["captions_en"] } ``` A few workflows also accept **recommended** inputs (captions for **Summarize**, for example). Recommended inputs sharpen the output but don't block the run if they're missing. The Dashboard surfaces them so you can opt in. ### Chain workflows together To chain workflows, declare a resource whose `source` is another binding in the same Directive. In the earlier example, `translate_es` waits for `premium_captions` because it lists `captions_en` in its `inputs`, and `captions_en` is `source: { "via": "workflow", "binding": "premium_captions" }`. The engine dispatches `premium_captions` first, waits for it to complete *and* for the captions to become ready on the asset (Mux tracks land asynchronously), then dispatches `translate_es`. If `premium_captions` errors, `translate_es` fast-fails with a diagnostic reason instead of waiting out the 24-hour cap. ### Replace existing tracks Set `replace_existing_tracks` in a binding's `parameters` like any other parameter. For example, `replace_generated` on `generate-premium-captions` replaces Mux Video's auto-generated captions as each asset is processed. A conflict found before dispatch shows as a `failed` binding with the reason. One found later errors the dispatched job. See [Managing existing tracks](/docs/guides/robots-generate-premium-captions#managing-existing-tracks). ```json { "reference_id": "premium_captions", "workflow": "generate-premium-captions", "parameters": { "replace_existing_tracks": "replace_generated" } } ``` A binding that replaces tracks can delete a track another binding reads. Saving the Directive returns a `422` when that can happen, unless the reader's resource has the deleting binding as its `source`, so the reader waits for the new track. In the example at the top of this guide, `translate_es` reads `captions_en`, which is sourced from `premium_captions`, so it's safe. [Edit captions](/docs/guides/robots-edit-captions#managing-existing-tracks) replaces its input track by default. If another binding reads the same caption resource, give that binding its own resource with `source: { "via": "workflow", "binding": "" }` so it reads the edited track, or set the edit binding's `replace_existing_tracks` to `fail`. ## Run a Directive There are three ways to trigger a run. ### By hand, from the Dashboard Open a saved Directive and click **Run on asset**. Pick an asset and the run starts immediately. This is the quickest way to do a one-off run, or to test a Directive before attaching it to assets automatically. The Directive detail page with the “Run on asset” dialog open and an asset selected. ### By hand, from the API `POST /robots/v1/directives/{DIRECTIVE_ID}/runs` with the target asset ID: ```bash curl https://api.mux.com/robots/v1/directives/YOUR_DIRECTIVE_ID/runs \ -H "Content-Type: application/json" \ -X POST \ -d '{ "asset_id": "YOUR_ASSET_ID" }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` The response is a 202 with the new run, including its `id`: ```json { "data": { "id": "drvrun_xyz789", "directive_id": "drv_abc123", "subject_id": "YOUR_ASSET_ID", "status": "pending", "node_states": [], "started_at": 1746403300, "completed_at": null } } ``` If a run for the same Directive and asset is already in progress, the API returns a 409 Conflict. Only one active run per pair. ### Automatically, by attaching to an asset When you create an asset, set the **Directives** field in the **Settings** step of the asset creation dialog. Any Directives you attach run automatically as soon as the asset is ready. The Settings step of the asset creation dialog with the Directives field set. You can also attach a Directive through the [Mux Video API](/docs/api-reference/video/assets#operation/create-asset) by including its ID in the `directives` field: ```bash curl https://api.mux.com/video/v1/assets \ -H "Content-Type: application/json" \ -X POST \ -d '{ "inputs": [{ "url": "https://example.com/my-video.mp4" }], "playback_policies": ["public"], "directives": [{ "id": "YOUR_DIRECTIVE_ID" }] }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` Attached Directives run once per asset. Attach the same Directive to the same asset twice and nothing doubles up; the operation is idempotent, so only one run starts. You can attach several *different* Directives to one asset, though. Each runs on its own, and the same workflow can run more than once if two Directives both include it. ## Track a run ### Webhooks Every directive-run lifecycle transition fires a [webhook event](/docs/core/listen-for-webhooks). Subscribing is the recommended way to react to runs without polling. | Event type | When it fires | | :-- | :-- | | `robots.v1.directive_run.created` | A run is triggered (manually or automatically). | | `robots.v1.directive_run.completed` | Every workflow in the run completed successfully. | | `robots.v1.directive_run.partial` | Some workflows completed; others errored. | | `robots.v1.directive_run.errored` | Every workflow errored, or the run could not start. | The event's `data` is the run object, in the same shape the run endpoints return. For example, an excerpt of a completed-run event: ```json { "type": "robots.v1.directive_run.completed", "data": { "id": "drvrun_xyz789", "directive_id": "drv_abc123", "subject_id": "YOUR_ASSET_ID", "status": "completed", "node_states": [ { "reference_id": "premium_captions", "status": "dispatched", "workflow": "generate-premium-captions", "job_id": "rjob_..." }, { "reference_id": "translate_es", "status": "dispatched", "workflow": "translate-captions", "job_id": "rjob_..." } ], "started_at": 1746403300, "completed_at": 1746403420 } } ``` The individual workflow jobs the Directive dispatched also fire their normal Robots job webhooks (`robots.v1.job..completed`, etc.), so you can react to either the run or the individual jobs. ### Retrieve a run If webhooks aren't an option, poll the run endpoint: ```bash curl https://api.mux.com/robots/v1/directives/YOUR_DIRECTIVE_ID/runs/YOUR_RUN_ID \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` Runs are retained for 30 days after they reach a terminal state. The response includes the current `status` and a `node_states` array with one entry per workflow binding, in the order the bindings appear in the Directive's `bindings`. Every entry carries a `reference_id` identifying the binding it describes, plus one of the shapes below, discriminated on its own `status`: | Node status | Additional fields | | :-- | :-- | | `dispatched` | `workflow`, `job_id`: the Robots job that runs the workflow. Look it up via `GET /robots/v1/jobs/{workflow}/{job_id}` for its outputs. | | `failed` | `reason`: a diagnostic string explaining why the binding didn't dispatch. | | `waiting_for_resources` | `workflow`: blocked waiting on an external Mux resource. | | `waiting_for_source_bindings` | `workflow`, `source_bindings`: blocked on producer bindings in the same Directive. `source_bindings` lists their `reference_id`s. | The run's own top-level `status` is one of `pending`, `dispatching`, `running`, `waiting`, `completed`, `partial`, or `errored`. ### On the Dashboard Each Directive keeps a list of its recent runs (retained for 30 days). Open a run to see the status of each workflow, the order it ran in, and anything that failed. A Directive run detail page listing each workflow, its status, and the order it ran in. The individual workflow jobs a Directive triggers sit alongside the ones you run by hand, under **Robots → Jobs** or on the asset details page. ## Managing Directives **Retrieve a Directive:** ```bash curl https://api.mux.com/robots/v1/directives/YOUR_DIRECTIVE_ID \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` **List Directives** in the current environment, newest first. Paginate with `limit` (default 25, max 100) and `page` (default 1). Each entry is a summary with `id`, `name`, `created_at`, `updated_at`, and `_links.self`; follow `_links.self.href` to retrieve a Directive's full configuration: ```bash curl "https://api.mux.com/robots/v1/directives?limit=25&page=1" \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` **List runs** for a Directive, newest first, within the 30-day retention window: ```bash curl https://api.mux.com/robots/v1/directives/YOUR_DIRECTIVE_ID/runs \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` **Delete a Directive** (soft-delete; the row stops appearing in GETs and can't start new runs, but any in-flight runs continue to completion): ```bash curl -X DELETE https://api.mux.com/robots/v1/directives/YOUR_DIRECTIVE_ID \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` ## Frequently asked questions ### Who can use Directives? Directives are available to all Mux Video customers, including accounts on the free plan, with [Mux Robots](/docs/guides/robots) enabled. ### How much do Directives cost? Directives themselves are free. You only pay for the workflow runs they trigger, at the same per-workflow rates as one-off Robots jobs. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for the numbers. ### What happens if a workflow errors partway through a run? Other workflows continue if they don't depend on the failed binding. Anything transitively downstream of the failure is marked `failed` with a diagnostic `reason`. The run's own top-level `status` becomes `partial` if some workflows completed and others errored, or `errored` if none succeeded. Either terminal state fires the corresponding webhook. # Language support for Mux Robots See the verified languages and BCP 47 language tags for Mux Robots workflows that analyze, generate, or translate language. Mux Robots workflows that analyze, generate, or translate language have been verified with the 86 languages below. Use the listed BCP 47 language tag when a workflow accepts an input or output language parameter. Languages not listed here may still work, but are processed on a best-effort basis. Output quality may vary. ## Verified languages | Language | Language tag | | :-- | :-- | | Afrikaans | `af` | | Arabic | `ar` | | Armenian | `hy` | | Assamese | `as` | | Asturian | `ast` | | Azerbaijani | `az` | | Belarusian | `be` | | Bengali | `bn` | | Bosnian | `bs` | | Bulgarian | `bg` | | Burmese | `my` | | Cantonese | `yue` | | Catalan | `ca` | | Cebuano | `ceb` | | Chichewa | `ny` | | Mandarin Chinese | `zh` | | Croatian | `hr` | | Czech | `cs` | | Danish | `da` | | Dutch | `nl` | | English | `en` | | Estonian | `et` | | Filipino | `fil` | | Finnish | `fi` | | French | `fr` | | Galician | `gl` | | Georgian | `ka` | | German | `de` | | Greek | `el` | | Gujarati | `gu` | | Hausa | `ha` | | Hebrew | `he` | | Hindi | `hi` | | Hungarian | `hu` | | Icelandic | `is` | | Indonesian | `id` | | Irish | `ga` | | Italian | `it` | | Japanese | `ja` | | Javanese | `jv` | | Kannada | `kn` | | Kazakh | `kk` | | Korean | `ko` | | Kyrgyz | `ky` | | Latvian | `lv` | | Lingala | `ln` | | Lithuanian | `lt` | | Luxembourgish | `lb` | | Macedonian | `mk` | | Malay | `ms` | | Malayalam | `ml` | | Maltese | `mt` | | Māori | `mi` | | Marathi | `mr` | | Mongolian | `mn` | | Nepali | `ne` | | Norwegian | `no` | | Occitan | `oc` | | Odia | `or` | | Pashto | `ps` | | Persian | `fa` | | Polish | `pl` | | Portuguese | `pt` | | Punjabi | `pa` | | Romanian | `ro` | | Russian | `ru` | | Serbian | `sr` | | Sindhi | `sd` | | Slovak | `sk` | | Slovenian | `sl` | | Somali | `so` | | Spanish | `es` | | Swahili | `sw` | | Swedish | `sv` | | Tagalog | `tl` | | Tajik | `tg` | | Tamil | `ta` | | Telugu | `te` | | Thai | `th` | | Turkish | `tr` | | Ukrainian | `uk` | | Urdu | `ur` | | Uzbek | `uz` | | Vietnamese | `vi` | | Welsh | `cy` | | Yoruba | `yo` | # Generate premium captions Generate high-accuracy captions for a video and attach them as a text track with the Mux Robots API. Use premium captions when accuracy matters, such as for accessibility or compliance. Generate captions for a Mux asset and automatically attach them as a text track. Premium captions use a higher-accuracy speech model than Mux's standard [auto-generated captions](/docs/guides/add-autogenerated-captions-and-use-transcripts), and add optional speaker labels, word-level timestamps, and custom phrase hints for proper nouns and jargon. Captions are generated directly from the asset's audio, so no existing track is required. See the Generate Premium Captions API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. If the asset already has a text track in the same language as the new captions, or with the same name, the request is rejected by default. Set `replace_existing_tracks` to delete conflicting tracks first. See [Managing existing tracks](#managing-existing-tracks). To produce captions, this workflow needs an `audio-only` [static rendition](/docs/guides/enable-static-mp4-renditions) of the asset. If the asset already has one, it's reused. If not, the workflow creates one to process the audio and deletes it once the job completes, so you aren't charged for extra storage. ## Create a `generate-premium-captions` job ```bash curl https://api.mux.com/robots/v1/jobs/generate-premium-captions \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "language_code": "en", "phrases": ["Mux", "API"] } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.generate_premium_captions.completed` webhook](/docs/guides/robots#webhooks):** the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/generate-premium-captions/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to caption. | | `language_code` | string | BCP 47 language code of the audio (e.g. `en`, `es`). Auto-detected when omitted. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `replace_existing_tracks` | string | What to do when the asset already has a conflicting text track: `fail` (the default), `replace_all`, or `replace_generated`. See [Managing existing tracks](#managing-existing-tracks). | | `track_name` | string | Custom name for the uploaded Mux text track. Defaults to `"{Language} (Generated)"` using the resolved language code. | | `include_speakers` | boolean | When `true`, speaker labels are identified and added to each caption cue. Useful for interviews, podcasts, and multi-speaker content. Defaults to `false`. | | `include_words` | boolean | When `true`, word-level timestamps are exported as a JSON file accessible via `temporary_words_url` in the output. Billed at a higher unit rate. Defaults to `false`. | | `upload_to_mux` | boolean | Whether to upload the generated captions as a new text track on the asset. Defaults to `true`. When `false`, no track is created (and `replace_existing_tracks` must be `fail`); the captions remain available via `temporary_srt_url`. | | `phrases` | array of strings | Best-effort list of words or short phrases (proper nouns, product names, jargon) likely to appear in the audio, used to bias recognition toward correct spellings. Up to 1,000 phrases, each up to 49 characters and 5 words. Phrases can't contain `<`, `>`, `{`, `}`, `[`, `]`, or `\`. Does not guarantee exact output. | ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.generate_premium_captions.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/generate-premium-captions/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `track_id` | string | Mux text track ID of the newly uploaded caption track. Omitted when `upload_to_mux` is `false`. | | `language_code` | string | Resolved language code of the generated captions (may differ from the requested code when auto-detected). | | `detected_language_code` | string | Language code reported by the transcription model. Only a genuine detection when `language_code` is omitted from the request; otherwise it echoes the requested language. | | `auto_language_confidence` | number | The transcription model's confidence in its automatic language detection, from 0 to 1. Always `1` when `language_code` is set in the request, since detection is skipped. | | `temporary_srt_url` | string | Temporary pre-signed URL to download the generated SRT file. Expires 7 days after the job completes. | | `temporary_words_url` | string | Temporary pre-signed URL to download the word-level timestamp JSON. Present when `include_words` is `true`. Expires 7 days after the job completes, so download and store it for long-term access. | | `replaced_track_ids` | array of strings | IDs of the tracks deleted before the new track was created. Absent when nothing was deleted. See [Managing existing tracks](#managing-existing-tracks). | ## Example completed job This example uses a 6-minute asset with `include_words: false`: at 500 units per minute, the job consumes 3,000 units. It is the job delivered in the [`robots.v1.job.generate_premium_captions.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/generate-premium-captions/{JOB_ID}`: ```json { "data": { "id": "rjob_yza567", "workflow": "generate-premium-captions", "status": "completed", "units_consumed": 3000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "language_code": "en", "replace_existing_tracks": "fail", "include_speakers": false, "include_words": false, "upload_to_mux": true, "phrases": ["Mux", "API"] }, "outputs": { "track_id": "track_en_abc123", "language_code": "en", "temporary_srt_url": "https://storage.googleapis.com/..." } } } ``` When `upload_to_mux` is `true` (the default), the caption track is automatically attached to your asset, and viewers will see the new language option in the player's caption menu. ## Managing existing tracks Use `replace_existing_tracks` to control what happens when the asset already has a text track in the same language (ignoring region, so `en` matches `en-US`) or with the same name as the new track. | Value | Behavior | | :-- | :-- | | `fail` (default) | Nothing is deleted, and the new track isn't added. | | `replace_all` | Conflicting tracks are deleted, then the new track is added. | | `replace_generated` | Conflicting tracks auto-generated by Mux Video are deleted. If any other track conflicts, nothing is deleted and the new track isn't added. | Use `replace_generated` to upgrade Mux's [auto-generated captions](/docs/guides/add-autogenerated-captions-and-use-transcripts) without touching tracks you uploaded. Tracks created by an earlier Robots job count as uploaded, so replacing one needs `replace_all`. * The IDs of deleted tracks are listed in `outputs.replaced_track_ids`. * A conflict is usually caught when you create the job, which returns a `422`. If it only shows up later, the job errors instead. That happens when a conflicting track is added while the job runs, or when you omit `language_code`, since conflicts can't be checked until the language is detected. Neither case is billed. * If you omit `language_code`, tracks are only deleted when the language is detected with high confidence. Otherwise the job errors and suggests setting `language_code`. ## Word-level timestamps When `include_words` is `true`, download the file at `temporary_words_url` to get word-level timestamps. It's a JSON array of token objects in playback order. The array interleaves `spacing` tokens between words so you can reconstruct the exact text, and `audio_event` tokens capture non-speech sounds. | Field | Type | Description | | :-- | :-- | :-- | | `text` | string | The token's text. For `word` tokens this includes any trailing punctuation (e.g. `"Matt,"`, `"is."`). For `spacing` tokens it's a single space (`" "`). For `audio_event` tokens it's a bracketed non-speech cue (e.g. `"[laughs]"`). | | `start` | number | Start time of the token, in seconds from the start of the media (fractional, e.g. `26.38`). | | `end` | number | End time of the token, in seconds. | | `type` | string | One of `word`, `spacing`, or `audio_event`. | | `speaker_id` | string | The speaker the token is attributed to, formatted `speaker_N` (zero-indexed). Set only when `include_speakers` is `true`. | ```json [ { "text": "Hey,", "start": 0.2, "end": 0.42, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 0.42, "end": 0.45, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "I'm", "start": 0.45, "end": 0.6, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 0.6, "end": 0.63, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "a", "start": 0.63, "end": 0.7, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 0.7, "end": 0.73, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "Mux", "start": 0.73, "end": 0.98, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 0.98, "end": 1.01, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "Robot,", "start": 1.01, "end": 1.4, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 1.4, "end": 1.43, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "beep", "start": 1.43, "end": 1.7, "type": "word", "speaker_id": "speaker_0" }, { "text": " ", "start": 1.7, "end": 1.73, "type": "spacing", "speaker_id": "speaker_0" }, { "text": "boop.", "start": 1.73, "end": 2.05, "type": "word", "speaker_id": "speaker_0" }, { "text": "[beeps]", "start": 2.1, "end": 2.4, "type": "audio_event", "speaker_id": "speaker_1" }, { "text": "Beep", "start": 2.5, "end": 2.72, "type": "word", "speaker_id": "speaker_1" }, { "text": " ", "start": 2.72, "end": 2.75, "type": "spacing", "speaker_id": "speaker_1" }, { "text": "boop", "start": 2.75, "end": 2.98, "type": "word", "speaker_id": "speaker_1" }, { "text": " ", "start": 2.98, "end": 3.01, "type": "spacing", "speaker_id": "speaker_1" }, { "text": "yourself!", "start": 3.01, "end": 3.5, "type": "word", "speaker_id": "speaker_1" } ] ``` # Edit a video's captions Edit an existing caption track on a Mux asset with the Mux Robots API. Use find-and-replace rules to fix terms, or censor profanity with an LLM. Edit an existing text track on a Mux asset. The workflow supports three kinds of edits — static word/phrase `replacements`, speaker label renames via `speaker_replacements`, and LLM-driven profanity censoring via `auto_censor_profanity` — and you can combine them in a single job. By default the edited captions replace the original track, keeping its name. See [Managing existing tracks](#managing-existing-tracks). See the Edit Captions API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Caption editing requires an existing, ready text track on the asset. Add captions first, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before creating an edit-captions job. You must provide at least one of `replacements`, `speaker_replacements`, or `auto_censor_profanity`. ## Create an `edit-captions` job ```bash curl https://api.mux.com/robots/v1/jobs/edit-captions \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "track_id": "YOUR_TRACK_ID", "replacements": [ { "find": "Mucks", "replace": "Mux", "case_sensitive": true }, { "find": "gonna", "replace": "going to" } ], "auto_censor_profanity": { "mode": "blank" } } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.edit_captions.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/edit-captions/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID whose text track will be edited. | | `track_id` | string | **Required.** The existing, ready Mux text track ID to edit. | | `replacements` | array | Static find/replace rules applied to cue text. See [Replacements](#replacements) below. | | `speaker_replacements` | array | Renames the bracketed speaker labels at the start of cues. See [Speaker replacements](#speaker-replacements) below. | | `auto_censor_profanity` | object | LLM-driven profanity detection and censorship rules. See [Auto-censor profanity](#auto-censor-profanity) below. | | `upload_to_mux` | boolean | Whether to upload the edited VTT back to the asset as a text track. Defaults to `true`. | | `replace_existing_tracks` | string | What to do with the source track: `replace` (the default) or `fail`. See [Managing existing tracks](#managing-existing-tracks). | | `track_name` | string | Name for the edited text track. Defaults to the source track's name. Required when `replace_existing_tracks` is `fail`. | ### Replacements Each entry in `replacements` is applied to every cue in the source track. | Field | Type | Description | | :-- | :-- | :-- | | `find` | string | **Required.** Exact word or phrase to match in cue text. | | `replace` | string | **Required.** Text to insert when a match is found. Use an empty string to delete the match. | | `case_sensitive` | boolean | When `true`, `find` only matches with exact case. Defaults to `false`, so `gonna` also matches `Gonna` and `GONNA`. | ### Speaker replacements Use `speaker_replacements` to rename speakers, for example in captions generated by [generate premium captions](/docs/guides/robots-generate-premium-captions) with `include_speakers`. Each entry renames a bracketed label at the start of a cue. Spoken text that contains the same words isn't changed. | Field | Type | Description | | :-- | :-- | :-- | | `find` | string | **Required.** The current speaker label, without the square brackets. Must match the label in the track exactly. | | `replace` | string | **Required.** The new speaker label, without the square brackets. | Labels can't contain square brackets or line breaks. ```json { "speaker_replacements": [ { "find": "speaker_0", "replace": "Alice" }, { "find": "speaker_1", "replace": "Bob" } ] } ``` ### Auto-censor profanity Use `auto_censor_profanity` to detect and replace profane terms automatically. | Field | Type | Description | | :-- | :-- | :-- | | `detection_method` | string | How profanity is detected. Only `llm` is supported, which uses an LLM to identify profanity in cue text. Defaults to `llm`. | | `mode` | string | Replacement strategy: `blank` (bracketed underscores, e.g. `[____]`), `remove` (drop the match entirely), or `mask` (replace characters with question marks). Defaults to `blank`. | | `always_censor` | array of strings | Words or short phrases that should always be censored, even if the model doesn't flag them. | | `never_censor` | array of strings | Words or short phrases that should never be censored, even if the model flags them. | ## Output The `outputs` object is included once the job's status is `completed`. You'll receive it on the [`robots.v1.job.edit_captions.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/edit-captions/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `total_replacement_count` | integer | Total number of cue text replacements applied across all edit operations. | | `track_id` | string | Mux text track ID of the newly created edited captions, distinct from the source `track_id` parameter. Present when `upload_to_mux` is `true` and the upload succeeds. | | `temporary_vtt_url` | string | Temporary pre-signed URL to download the edited VTT file. The URL expires 7 days after the job completes. | | `replaced_track_ids` | array of strings | IDs of the tracks deleted before the edited track was added. Absent when nothing was deleted. See [Managing existing tracks](#managing-existing-tracks). | ## Example completed job This is the job delivered in the [`robots.v1.job.edit_captions.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/edit-captions/{JOB_ID}`: ```json { "data": { "id": "rjob_example123", "workflow": "edit-captions", "status": "completed", "units_consumed": 500, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "mux_asset_123abc", "track_id": "text_track_456def", "replacements": [ { "find": "Mucks", "replace": "Mux", "case_sensitive": true }, { "find": "gonna", "replace": "going to" } ], "upload_to_mux": true, "replace_existing_tracks": "replace" }, "outputs": { "total_replacement_count": 5, "track_id": "text_track_789ghi", "temporary_vtt_url": "https://s3.example.com/edited.vtt?sig=abc", "replaced_track_ids": ["text_track_456def"] } } } ``` When `upload_to_mux` is `true` (the default), the edited caption track is attached to your asset. By default it replaces the original, so viewers see the corrected captions in the player's caption menu. ## Managing existing tracks Use `replace_existing_tracks` to choose what happens to the source track when the edit is uploaded. | Value | Behavior | | :-- | :-- | | `replace` (default) | The source track is deleted and the edited track takes its place, with the same name, language, and closed-captions setting. | | `fail` | Nothing is deleted. The edited track is added alongside the source, so it needs a `track_name` different from the source's. | * Track names are unique on an asset. Under either value, the request is rejected if a track other than the source already has the edited track's name. * `replace` requires `upload_to_mux` to be `true`. * With `replace`, if the edited track can't be added, Mux attempts to restore the source under a new track ID and the job errors. Restoration is best-effort and can also fail. * The deleted source's ID is listed in `outputs.replaced_track_ids`. * A conflict is usually caught when you create the job, which returns a `422`. If a conflicting track is added while the job runs, the job errors instead. Neither case is billed. # Translate captions Translate existing captions into another language with the Mux Robots API. Use translated captions when your videos reach viewers in more than one language. Translate an existing caption track on a Mux asset from one language to another. The translated captions can be automatically attached to the asset as a new text track, making multilingual video accessible with a single API call. See the Translate Captions API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Caption translation requires an existing caption track on the asset. Make sure your asset has captions, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before creating a translate-captions job. By default, the request is also rejected if the asset already has a text track in the target language, or with the same name as the new track. See [Managing existing tracks](#managing-existing-tracks). ## Create a `translate-captions` job ```bash curl https://api.mux.com/robots/v1/jobs/translate-captions \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "track_id": "YOUR_TRACK_ID", "output_language_code": "es" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.translate_captions.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/translate-captions/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID whose captions will be translated. | | `track_id` | string | **Required.** The text track ID of the source caption track to translate. | | `output_language_code` | string | **Required.** BCP 47 target language code (e.g. `es`, `ja`). See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `upload_to_mux` | boolean | Whether to upload the translated VTT and attach it as a text track on the asset. Defaults to `true`. | | `replace_existing_tracks` | string | What to do when the asset already has a conflicting text track: `fail` (the default), `replace_all`, or `replace_generated`. See [Managing existing tracks](#managing-existing-tracks). | | `track_name` | string | Name for the translated text track. Defaults to `"{Language} (Auto-translated)"`, e.g. `"Spanish (Auto-translated)"`. An existing track with this name is handled by `replace_existing_tracks`. | | `never_translate` | array of strings | Best-effort list of terms (brand names, proper nouns) to keep verbatim in the translation. Up to 100 terms, each up to 100 characters. Terms can't contain `<`, `>`, or invisible characters. Does not guarantee exact output. | ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.translate_captions.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/translate-captions/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `track_id` | string | Mux text track ID of the newly created translated captions, distinct from the source `track_id` parameter. Present when `upload_to_mux` is `true`. | | `temporary_vtt_url` | string | Temporary pre-signed URL to download the translated VTT file, including when `upload_to_mux` is `false`. Expires 7 days after the job completes. | | `never_translate_terms_preserved` | boolean | Present when `never_translate` was set. `false` when at least one term wasn't kept verbatim. | | `replaced_track_ids` | array of strings | IDs of the tracks deleted before the new track was created. Absent when nothing was deleted. See [Managing existing tracks](#managing-existing-tracks). | ## Example completed job This example uses a five-minute asset, consuming 2,600 units. It is the job delivered in the [`robots.v1.job.translate_captions.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/translate-captions/{JOB_ID}`: ```json { "data": { "id": "rjob_pqr678", "workflow": "translate-captions", "status": "completed", "units_consumed": 2600, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "track_id": "YOUR_TRACK_ID", "output_language_code": "es", "upload_to_mux": true }, "outputs": { "track_id": "track_abc123", "temporary_vtt_url": "https://storage.googleapis.com/..." } } } ``` When `upload_to_mux` is `true` (the default), the translated caption track is automatically attached to your asset. Your viewers will see the new language option in the player's caption menu without any additional work. ## Managing existing tracks Use `replace_existing_tracks` to control what happens when the asset already has a text track in the target language (ignoring region, so `es` matches `es-MX`) or with the same name as the new track. | Value | Behavior | | :-- | :-- | | `fail` (default) | Nothing is deleted, and the new track isn't added. | | `replace_all` | Conflicting tracks are deleted, then the new track is added. | | `replace_generated` | Conflicting tracks auto-generated by Mux Video are deleted. If any other track conflicts, nothing is deleted and the new track isn't added. | Tracks created by an earlier Robots job count as uploaded, so replacing one needs `replace_all`. * A job that would delete the source track you're translating is rejected. Choose a source track in a different language. * The IDs of deleted tracks are listed in `outputs.replaced_track_ids`. * A conflict is usually caught when you create the job, which returns a `422`. If a conflicting track is added while the job runs, the job errors instead. Neither case is billed. # Translate audio Translate and dub the spoken audio of a video into another language with the Mux Robots API. Use dubbed audio to reach viewers in more languages. Translate the spoken audio of a Mux asset into another language and automatically attach the result as a new audio track, so viewers can switch languages from the player's audio menu. Where [translate captions](/docs/guides/robots-translate-captions) produces a text track in another language, `translate-audio` produces a dubbed audio track. See the Translate Audio API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Audio translation requires an asset with an audio track. A `translate-audio` job is rejected if the asset has no audio. By default, it's also rejected if the asset already has an audio track in the target language, or with the same name as the new track. See [Managing existing tracks](#managing-existing-tracks). ## Create a `translate-audio` job ```bash curl https://api.mux.com/robots/v1/jobs/translate-audio \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "es" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.translate_audio.completed` webhook](/docs/guides/robots#webhooks):** the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/translate-audio/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID whose audio will be translated. | | `output_language_code` | string | **Required.** BCP 47 target language code (e.g. `es`, `fr`, `ja`). See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `upload_to_mux` | boolean | Whether to upload the dubbed audio and attach it as a new audio track on the asset. Defaults to `true`. | | `replace_existing_tracks` | string | What to do when the asset already has a conflicting audio track: `fail` (the default), `replace_all`, or `replace_generated` (which behaves as `fail` for audio). See [Managing existing tracks](#managing-existing-tracks). | | `track_name` | string | Name for the dubbed audio track. Defaults to `"{Language} (Auto-dubbed)"`, e.g. `"Spanish (Auto-dubbed)"`. An existing track with this name is handled by `replace_existing_tracks`. | ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.translate_audio.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/translate-audio/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `track_id` | string | Mux audio track ID of the newly created dubbed track. Present when `upload_to_mux` is `true`. | | `temporary_audio_url` | string | Temporary pre-signed URL to download the dubbed audio file. Expires 7 days after the job completes. | | `replaced_track_ids` | array of strings | IDs of the tracks deleted before the new track was created. Absent when nothing was deleted. See [Managing existing tracks](#managing-existing-tracks). | ## Example completed job This example uses a five-minute asset, consuming 350,000 units. It is the job delivered in the [`robots.v1.job.translate_audio.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/translate-audio/{JOB_ID}`: ```json { "data": { "id": "rjob_vwx234", "workflow": "translate-audio", "status": "completed", "units_consumed": 350000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "es", "upload_to_mux": true }, "outputs": { "track_id": "track_es_abc123", "temporary_audio_url": "https://storage.googleapis.com/..." } } } ``` When `upload_to_mux` is `true` (the default), the dubbed audio is attached to your asset as a new audio track. Viewers can select the new language from the player's audio menu. See [add alternate audio tracks](/docs/guides/add-alternate-audio-tracks-to-your-videos) for more on multi-language audio. ## Managing existing tracks Use `replace_existing_tracks` to control what happens when the asset already has an audio track in the target language (ignoring region, so `es` matches `es-MX`) or with the same name as the new track. | Value | Behavior | | :-- | :-- | | `fail` (default) | Nothing is deleted, and the new track isn't added. | | `replace_all` | Conflicting tracks are deleted, then the new track is added. | Audio tracks are never auto-generated by Mux Video, so `replace_generated` behaves as `fail`. * The asset's original audio track is never deleted. If it conflicts, the new track isn't added. * The IDs of deleted tracks are listed in `outputs.replaced_track_ids`. * A conflict is usually caught when you create the job, which returns a `422`. If a conflicting track is added while the job runs, the job errors instead. Neither case is billed. # Analyze the structure of a video Generate a title, summary, tags, chapters, scenes, and key moments from a video or audio asset with a single call to the Mux Robots API, and steer the time range, scene and chapter density, audience, and topic vocabulary. Generate a title, summary, tags, chapters, scenes, and key moments together from one Mux asset or a selected time range. Use one job to fill metadata in your CMS, add chapter navigation to a player, and find highlights to review for clips. All outputs are built from shared transcript and visual evidence. See the Analyze Structure API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. If you previously used [Find Scenes](/docs/guides/robots-find-scenes) for segmentation, read `outputs.scenes` to get scene data alongside the other metadata. Use [Directives](/docs/guides/robots-directives) to run analysis automatically as assets are ingested. The asset must be ready, no longer live, and have a public or signed playback ID. Video assets can be analyzed without captions, but captions provide spoken context. Audio-only assets require a ready caption track. You can [generate captions](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [add subtitles](/docs/guides/add-subtitles-to-your-videos) before running the workflow. For video, the workflow requests or waits for [Mux Shots](/docs/guides/find-different-shots-in-your-video); shot generation can incur separate [Mux Video charges](/docs/pricing/overview#shots). ## Create an `analyze-structure` job ```bash curl https://api.mux.com/robots/v1/jobs/analyze-structure \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "en" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.analyze_structure.completed` webhook](/docs/guides/robots#webhooks)**. The payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, poll `GET /robots/v1/jobs/analyze-structure/{JOB_ID}` with the `id` from the response until the status is `completed`. Jobs are automatically deleted after 30 days. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux video or audio asset ID to analyze. | | `language_code` | string | Preferred caption language as a BCP 47 tag. An explicit value must match a ready source exactly, including case: `en` does not match `en-US`. When omitted, the first ready subtitle track is selected. Audio-only assets can also use a single ready legacy text transcript. | | `output_language_code` | string | BCP 47 language code for generated text. Auto-detected from available evidence when omitted. Transcript cues and quotes remain in their source language. | | `output_steering` | object | Controls for the analyzed time range, audience, terminology, topics, and scene and chapter density. See [Output steering](#output-steering). | | `update_asset_meta` | boolean | When `true`, writes the generated title to `asset.meta.title`. Defaults to `false`. See [Update the asset](#update-the-asset). | | `update_asset_chapters` | boolean | When `true`, replaces or creates a chapters track. Defaults to `false`. See [Update the asset](#update-the-asset). | See [language support for Mux Robots](/docs/guides/robots-supported-languages) for verified languages and language tags. ### Output steering Scope bounds which media is analyzed. The other controls provide best-effort guidance and do not guarantee exact output. | Field | Type | Description | | :-- | :-- | :-- | | `scope` | object | Execution window in seconds on the original asset timeline. Supply `start_time`, `end_time`, or both. An omitted start means the beginning; an omitted end means the asset's end. The start must be nonnegative, the end must be positive, and the start must precede the end. An end past the asset's duration is clamped to the asset's end, and a range that starts at or after the asset's end is rejected. The start is inclusive and the end is exclusive. Returned timestamps remain relative to the original asset's start, rather than restarting at zero for the selected range. | | `granularity` | string | Moves scene and chapter density together. Supported values: `coarse`, `balanced`, and `fine`. | | `audience` | string | Intended audience, up to 160 characters. | | `brand_terms` | array of strings | Preferred terms to use when supported by the source. Supply 1–10 terms, up to 40 characters each and 240 characters combined. | | `topic_taxonomy` | object | Controlled vocabulary for topics. Optional `name`; required `values` array with 1–50 entries, each with a `label` and optional `description` and `aliases`. `allow_other` controls whether topics outside the vocabulary may be used. | For example, analyze only the product demonstration between 30 and 150 seconds: ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "output_steering": { "scope": { "start_time": 30, "end_time": 150 }, "granularity": "fine", "audience": "developers learning to upload videos", "brand_terms": ["Mux Video"] } } } ``` ### Update the asset Analysis returns metadata without writing it to the asset by default. Set `update_asset_meta` to `true` to write the generated title to `asset.meta.title`. Set `update_asset_chapters` to `true` to create a chapters text track, replacing existing chapters tracks. ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "update_asset_meta": true, "update_asset_chapters": true } } ``` Each update reports its own outcome in `outputs.asset_updates`. An update failure does not fail successful analysis. If no chapters are generated, the chapters update is skipped, its outcome is omitted, and any existing chapters track is preserved. Chapter replacement deletes existing chapters tracks before creating the new track. If deletion succeeds but creation fails, the asset can be left without chapters. Check `asset_updates.chapters.status` and `replaced_track_ids` even when the job is `completed`. ## Output The `outputs` object is included once the job is `completed`, in the [`robots.v1.job.analyze_structure.completed`](/docs/guides/robots#webhooks) webhook or the job's `GET` response. All timestamps are **seconds on the original asset timeline**, including fractional seconds. | Field | Type | Description | | :-- | :-- | :-- | | `title` | string | Generated title. | | `summary` | string | Generated summary. | | `tags` | array of strings | Generated tags. Empty when no tag qualifies. | | `chapters` | array | Ordered chapters with `start_time`, `end_time`, and `title`. Chapters are eligible when the analyzed range is at least 60 seconds and has at least two coverage segments, but may still be empty. | | `scenes` | array | Ordered video scenes with time boundaries, a title, and overlapping transcript cues. Always empty for audio-only assets. | | `key_moments` | array | Selected moments in timeline order, with time boundaries, a title, a quality score, and overlapping transcript cues. Empty when none qualify. | | `asset_updates` | object | Outcomes of requested asset updates. Title status is `updated` or `failed`; chapters status is `created` or `failed`, with `track_id` for the new track and `replaced_track_ids` for deleted tracks when present. | Scenes and key moments can include audible and visual narratives and notable concepts. Scenes can also include a shot breakdown; key moments can include a verbatim `quotable_segment` with `time` and `text`. Transcript cues have `start_time`, `end_time`, and `text`, and remain in their source language. See the API reference for the full nested output fields. ## Example completed job This example analyzes a two-minute video, consuming 4,000 units. It is the job delivered in the [`robots.v1.job.analyze_structure.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/analyze-structure/{JOB_ID}`: ```json { "data": { "id": "rjob_abc123", "workflow": "analyze-structure", "status": "completed", "units_consumed": 4000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "en" }, "outputs": { "title": "Uploading a video with Mux", "summary": "A developer demonstrates creating a direct upload and playing the resulting video.", "tags": ["video uploads", "video playback"], "chapters": [ { "start_time": 0, "end_time": 60, "title": "Create a direct upload" }, { "start_time": 60, "end_time": 120, "title": "Play the uploaded video" } ], "scenes": [ { "start_time": 0, "end_time": 60, "title": "Create a direct upload", "cues": [], "visual_narrative": "A developer creates a direct upload in the dashboard.", "shot_count": 2 }, { "start_time": 60, "end_time": 120, "title": "Play the uploaded video", "cues": [{ "start_time": 62, "end_time": 67, "text": "The video is ready to play." }], "audible_narrative": "The developer explains that the upload is ready for playback.", "visual_narrative": "A player loads the uploaded video.", "shot_count": 2 } ], "key_moments": [ { "start_time": 60, "end_time": 75, "title": "First playback", "overall_score": 0.85, "cues": [{ "start_time": 62, "end_time": 67, "text": "The video is ready to play." }], "quotable_segment": { "time": 62, "text": "The video is ready to play." } } ] } } } ``` # Generate chapters Create timestamped chapters for a video with the Mux Robots API, and steer the chapter style, granularity, and audience. Use chapters when viewers must jump to sections of long content, such as lectures or podcasts. Generate timestamped chapters from a video's caption track. This is useful for creating a table of contents for long-form video content, improving navigation, and making video content easier to scan. See the Generate Chapters API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Chapter generation uses caption tracks on the asset to understand the video content. Make sure your asset has captions, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before creating a chapter generation job. ## Create a `generate-chapters` job ```bash curl https://api.mux.com/robots/v1/jobs/generate-chapters \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.generate_chapters.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/generate-chapters/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video. | | `language_code` | string | BCP 47 language code of the caption track to analyze (e.g. `en`, `fr`). When omitted, the first ready subtitle track in asset track order is used. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `output_language_code` | string | BCP 47 language code for the output chapter titles. Auto-detected from the transcript if omitted. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `output_steering` | object | Curated controls that guide chapter style, granularity, audience, and brand terminology without changing the output schema. See [Output steering](#output-steering). | | `update_asset_chapters` | boolean | When `true`, the chapters are written to the asset as a chapters text track once the job completes, replacing any chapters track already on the asset. Defaults to `false`. See [Add chapters to the asset](#add-chapters-to-the-asset). | ### Output steering Use `output_steering` when you want best-effort control over how chapters are generated. These fields guide the workflow but do not guarantee exact output. All `output_steering` fields are optional; include only the fields you need. | Field | Type | Description | | :-- | :-- | :-- | | `chapter_style` | string | Preferred style for generated chapter titles. Supported values: `descriptive`, `punchy`, `educational`, `seo`, and `platform_neutral`. | | `chapter_granularity` | string | Preferred chapter granularity. Supported values: `coarse`, `balanced`, and `fine`. | | `audience` | string | Intended audience used to guide chapter titles and emphasis. | | `brand_terms` | array of strings | Preferred brand or domain terms to use when supported by the source content. | ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "output_steering": { "chapter_style": "educational", "chapter_granularity": "fine", "audience": "home cooks following along step by step", "brand_terms": ["Dutch oven", "mirepoix"] } } } ``` ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.generate_chapters.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/generate-chapters/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `chapters` | array | Generated chapters, ordered by start time. | | `chapters[].start_time` | number | Chapter start time in seconds. The first chapter always starts at 0. | | `chapters[].title` | string | Concise chapter title. | | `asset_updates.chapters` | object | What happened when the chapters were written to the asset. Present only when `update_asset_chapters` is `true`. See [Add chapters to the asset](#add-chapters-to-the-asset). | ## Example completed job This example uses a five-minute asset, consuming 550 units. It is the job delivered in the [`robots.v1.job.generate_chapters.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/generate-chapters/{JOB_ID}`: ```json { "data": { "id": "rjob_ghi789", "workflow": "generate-chapters", "status": "completed", "units_consumed": 550, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "output_steering": { "chapter_style": "educational", "chapter_granularity": "fine", "audience": "home cooks following along step by step", "brand_terms": ["Dutch oven", "mirepoix"] } }, "outputs": { "chapters": [ { "start_time": 0, "title": "Gather your ingredients" }, { "start_time": 18.5, "title": "Preheat the oven to 375°F" }, { "start_time": 42.1, "title": "Dice the onions and garlic" }, { "start_time": 95.3, "title": "Sear the chicken thighs" }, { "start_time": 158.7, "title": "Deglaze with white wine" }, { "start_time": 203.4, "title": "Simmer the sauce" }, { "start_time": 278.9, "title": "Plate and garnish" } ] } } } ``` ## Add chapters to the asset Set `update_asset_chapters` to `true` to have Mux write the generated chapters to the asset as a chapters text track, so they're delivered with the asset. An asset holds a single chapters track, so any chapters track already on the asset is deleted and replaced. That includes one in a different language, or one you added yourself. The new track uses the language of the chapter titles. The write is best-effort: if it fails, the job still completes with the generated chapters. Check `outputs.asset_updates.chapters` for the outcome. | Field | Type | Description | | :-- | :-- | :-- | | `status` | string | `created` when the asset now carries these chapters. `failed` when the write errored; the chapters in `outputs` are still valid and can be added manually. | | `track_id` | string | Mux text track ID of the new chapters track. Present when `status` is `created`. | | `replaced_track_ids` | array of strings | IDs of the tracks that were deleted. Absent when nothing was deleted. On a `failed` status with `replaced_track_ids` present, the old track was deleted but the new one wasn't added, so the asset has no chapters track. | ```json { "asset_updates": { "chapters": { "status": "created", "track_id": "track_chapters_abc123", "replaced_track_ids": ["track_chapters_old456"] } } } ``` # Moderate a video Analyze a video for inappropriate content with the Mux Robots API, and limit moderation to part of the video. Use this workflow when your platform accepts user-generated video and you must review uploads. Analyze a video for inappropriate content. Mux samples thumbnails from the video and scores each one for sexual and violent content. You can configure the thresholds that determine whether content gets flagged. See the Moderate API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. ## Create a `moderate` job ```bash curl https://api.mux.com/robots/v1/jobs/moderate \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "thresholds": { "sexual": 0.7, "violence": 0.8 } } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.moderate.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/moderate/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to moderate. | | `language_code` | string | Language code of the caption track used for transcript analysis on audio-only assets; ignored for video assets. An explicit value must match a ready track exactly, including case: `en` does not match `en-US`. When omitted, the first ready subtitle track in asset track order is used. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `thresholds` | object | Score thresholds that determine whether content is flagged. | | `thresholds.sexual` | number | Score threshold (0.0-1.0) for sexual content. Defaults to 0.7. Lower the value to be more strict (e.g. `0.5` to flag borderline content). | | `thresholds.violence` | number | Score threshold (0.0-1.0) for violent content. Defaults to 0.8. Lower the value to be more strict. | | `sampling_interval` | integer | Interval in seconds between sampled thumbnails. Minimum 5. For example, `10` samples a frame every 10 seconds. Good when you want consistent coverage regardless of video length. | | `max_samples` | integer | Maximum number of thumbnails to sample. Samples are distributed evenly across the video with the first and last frames pinned. For example, `20` on a 10-minute video samples roughly every 30 seconds. Good when you want predictable cost per job. | | `output_steering` | object | Optional controls that restrict where moderation runs on the asset. | | `output_steering.scope` | object | Time window in seconds on the asset timeline, with `start_time` and `end_time`. Moderation is restricted to media within this window, while returned timestamps remain absolute asset timestamps. Omit `start_time` to begin at the asset start, and `end_time` to continue through the asset end. | | `on_flagged` | object | Optional action to take automatically when `exceeds_threshold` is `true`. When omitted, the job only reports scores and takes no action. | | `on_flagged.action` | string | The action to perform. Currently the only supported value is `delete_playback_ids`, which deletes every playback ID on the asset, making it unplayable while preserving the underlying asset so it can be re-published or reviewed. | ## Sampling behavior By default, the workflow samples a thumbnail every 10 seconds. Use `sampling_interval` to override this; the minimum is 5 seconds. Smaller intervals give denser coverage at higher cost; larger intervals are cheaper but may miss brief moments of flagged content. For example, with `sampling_interval: 10`, a 60-second video produces 6 samples — at 0, 10, 20, 30, 40, and 50 seconds. If you set `max_samples` alongside `sampling_interval`, the cap takes over once the interval would produce more samples than allowed. See the parameter descriptions above. ## Choose a sampling interval and thresholds The defaults (`sexual: 0.7`, `violence: 0.8`, one sample every 10 seconds) are a reasonable starting point. The right configuration depends on your platform's risk model: who can upload, how quickly content becomes public, who your audience is, and the impact to your business of a missed detection compared to an incorrect takedown. ### Establishing a risk model Open UGC platforms where anyone can sign up and publish immediately carry the most risk and warrant the strictest settings. Platforms with vetted uploaders or a publishing delay can lean on the defaults. Closed platforms (internal video, course content, fitness apps) are mostly guarding against account compromise and mistakes, and can prioritize cost instead. | Platform profile | Suggested starting point | | :-- | :-- | | Open UGC, instant publishing | `sexual: 0.6`, `violence: 0.6`, `sampling_interval: 5`, and a human review queue for scores near the threshold | | Vetted uploaders, or a review delay before publishing | The defaults: `sexual: 0.7`, `violence: 0.8`, `sampling_interval: 10` | | Closed or internal platform | `sexual: 0.8`, `violence: 0.9`, `max_samples: 20` | For zero-tolerance audiences, such as platforms for children, go stricter still, like the thresholds in the [strict moderation example](#example-strict-moderation) below. Whatever you pick, compare your first few hundred jobs against the calls your human reviewers actually make, then adjust. ### Understanding scores returned by the moderation model A thumbnail's score is how confident the model is that the category is present in that frame, not how extreme the content is. Clearly visible but mild nudity can score higher than graphic violence in a dark, partially obscured shot. That distinction matters when you pick thresholds: * **A threshold is a confidence bar, not a severity dial.** Lowering `sexual` from `0.7` to `0.5` doesn't redefine what counts as sexual content; it flags frames the model is less certain about. You'll catch more borderline content, and more false positives. * **Don't map score bands to punishment tiers.** A `0.9` isn't "worse" than a `0.75`; the model is just more sure. Use the score to decide whether a human looks at the video, and let the human judge severity against your content policy. * **Reserve automated actions for high-confidence flags.** If you use [`on_flagged`](#take-action-on-flagged-content) to unpublish content automatically, pair it with thresholds high enough that you can tolerate acting without review. ### Using sampling behavior to manage moderation cost Each sample costs [100 units ($0.001)](/docs/pricing/overview#mux-robots-pricing); a job's total cost is just its sample count. The two controls described in [Sampling behavior](#sampling-behavior) determine that count in opposite ways: * **`sampling_interval` fixes coverage; cost scales with duration.** One sample every N seconds means no stretch of video longer than N seconds goes unchecked, whether the upload is two minutes or two hours. Use it when you want a coverage guarantee and can let spend vary per video. * **`max_samples` fixes cost; coverage thins as videos get longer.** The same N samples are spread evenly across the video, with the first and last frames pinned, so a long upload costs the same as a short one but leaves wider gaps between samples. Use it when you need a predictable worst-case cost per video, especially when upload durations vary widely. | Configuration | 10-minute video | 1-hour video | | :-- | :-- | :-- | | `sampling_interval: 5` | 120 samples ($0.12) | 720 samples ($0.72) | | `sampling_interval: 10` (the default) | 60 samples ($0.06) | 360 samples ($0.36) | | `max_samples: 20` | 20 samples ($0.02), one every ~30 seconds | 20 samples ($0.02), one every ~3 minutes | You can also combine them: set `sampling_interval` for the coverage you want and `max_samples` as a spending cap that takes over on long videos. For example, `sampling_interval: 10` with `max_samples: 60` gives full 10-second coverage on videos up to 10 minutes, and caps any longer video at $0.06. How dense you need to go depends on what you're screening for: * **Sexual content is rarely hidden.** Uploaders generally aren't burying a few seconds of nudity inside an otherwise innocent video. When it's present, it tends to run through much of the content, so sparse sampling (a `max_samples` cap, or a 30-60 second interval) still hits it with high probability. * **Violence is more often brief.** A fight in a live stream recording or a single violent scene can fit between sparse samples. If violence is your primary risk, spend on a denser interval instead. ### Using multiple passes to re-inspect borderline content Rather than paying for dense sampling on every upload, one option is to run a cheap first pass on everything and spend more only where the results are ambiguous: 1. **First pass:** run every upload with sparse sampling (say `max_samples: 20`) and your normal thresholds. 2. **Clear results:** publish videos that score well below your thresholds; flag or action videos above them. 3. **Ambiguous results:** when `max_scores` lands just under a threshold (within `0.2` of it, for example), run a second `moderate` job with `sampling_interval: 5`. Use `output_steering.scope` to restrict the second pass to the region around the suspicious samples in `thumbnail_scores` instead of re-scanning the whole video. ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "sampling_interval": 5, "output_steering": { "scope": { "start_time": 120, "end_time": 300 } } } } ``` Most uploads on most platforms are innocuous, so the bulk of your library clears at the first-pass price of $0.02 per video, and only the ambiguous few need a second job. ### Maintaining a human reporting path No sampling interval catches everything, and no threshold gets every judgment right. Treat automated moderation as triage, not a verdict: route borderline scores to a human review queue, and give users a way to report content that slips through. At minimum, that's in-product reporting and a dedicated abuse address. See [contact mechanisms](/docs/examples/moderate-video-content#contact-mechanisms) and the other operational strategies in the [Moderate video content guide](/docs/examples/moderate-video-content). ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.moderate.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/moderate/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `thumbnail_scores` | array | Per-thumbnail moderation scores, each with `sexual` and `violence` fields (0.0-1.0). Also includes a `time` field (seconds) for video assets; absent for transcript moderation. | | `max_scores` | object | Highest scores across all thumbnails, with `sexual` and `violence` fields. | | `exceeds_threshold` | boolean | `true` if any category's max score exceeds its configured threshold. | | `flagged_action` | object | Present only when `on_flagged` was set and `exceeds_threshold` was `true`. Records the action taken, with `action` (the action performed) and `deleted_playback_ids` (the playback IDs removed from the asset). | ## Example completed job This is the job delivered in the [`robots.v1.job.moderate.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/moderate/{JOB_ID}`: ```json { "data": { "id": "rjob_def456", "workflow": "moderate", "status": "completed", "units_consumed": 200, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "thresholds": { "sexual": 0.7, "violence": 0.8 } }, "outputs": { "thumbnail_scores": [ { "time": 0.0, "sexual": 0.01, "violence": 0.02 }, { "time": 5.0, "sexual": 0.03, "violence": 0.05 } ], "max_scores": { "sexual": 0.03, "violence": 0.05 }, "exceeds_threshold": false } } } ``` Moderation works by sampling thumbnail frames from your video. You can control how many frames are analyzed with `sampling_interval` or `max_samples`. More samples give better coverage but increase processing time and cost. ## Example: strict moderation If content safety is critical for your platform, lower the thresholds and increase the sample density: ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "thresholds": { "sexual": 0.3, "violence": 0.4 }, "max_samples": 50 } } ``` With lower thresholds, even mildly suggestive or mildly violent content will cause `exceeds_threshold` to return `true`, giving you a signal to flag the video for human review before publishing. ## Take action on flagged content By default, a `moderate` job only reports scores — it never changes your asset. Set `on_flagged` to have the workflow act automatically when `exceeds_threshold` is `true`: ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "thresholds": { "sexual": 0.3, "violence": 0.4 }, "on_flagged": { "action": "delete_playback_ids" } } } ``` When the content exceeds a threshold, `delete_playback_ids` removes **every** playback ID on the asset, making it immediately unplayable while preserving the underlying master asset so you can review it and re-publish later by creating a new playback ID. The action is best-effort and non-fatal: if a deletion fails, the job still completes with its moderation scores rather than erroring. The outcome is reported back in the job output as `flagged_action`: ```json { "outputs": { "thumbnail_scores": [ { "time": 40.0, "sexual": 0.82, "violence": 0.10 } ], "max_scores": { "sexual": 0.82, "violence": 0.10 }, "exceeds_threshold": true, "flagged_action": { "action": "delete_playback_ids", "deleted_playback_ids": ["abc123...", "def456..."] } } } ``` Deleting playback IDs is irreversible — the IDs themselves cannot be restored, and any cached or shared URLs using them stop working. If this `moderate` job runs as part of a Directive that targets the same asset with other workflows, deleting playback IDs can break sibling workflows that need a playback ID. # Summarize a video Generate a title, description, and keyword tags for a video with the Mux Robots API, and steer the style, audience, and tag vocabulary. Use this workflow to fill metadata for user uploads or a large video library. Generate a title, description, and keyword tags for a video. This is useful for auto-populating metadata in a CMS, improving search and discovery, or creating social media descriptions from video content. See the Summarize API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Summarization works best when your asset has a text track. Without one, the workflow relies on sampled stills from the video, which can reduce accuracy. For the best results, make sure your asset has captions, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before running a summarize job. ## Create a `summarize` job ```bash curl https://api.mux.com/robots/v1/jobs/summarize \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "tone": "neutral" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.summarize.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/summarize/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to summarize. | | `tone` | string | Tone for the generated summary. `neutral` for straightforward, factual analysis. `playful` for witty, conversational copy (good for social media). `professional` for polished, executive-level language (good for reports or investor updates). | | `title_length` | integer | Maximum title length in words. | | `description_length` | integer | Maximum description length in words. | | `tag_count` | integer | Number of tags to generate. Defaults to 10. | | `language_code` | string | BCP 47 language code of the caption track to analyze (e.g. `en`, `fr`). An explicit value must match a ready track exactly, including case: `en` does not match `en-US`. When omitted, the first ready subtitle track in asset track order is used. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `output_language_code` | string | BCP 47 language code for the generated summary output (e.g. `en`, `fr`, `ja`). Auto-detected from the transcript if omitted. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `output_steering` | object | Curated controls that guide summary style, audience, brand terminology, and tags without changing the output schema. See [Output steering](#output-steering). | | `update_asset_meta` | boolean | When `true`, the generated title is written to the asset's `meta.title` once the job completes. Best effort: a failed write does not fail the job. The outcome is reported in `outputs.asset_updates`. | ### Output steering Use `output_steering` when you want best-effort control over how the summary is generated. These fields guide the workflow but do not guarantee exact output. All `output_steering` fields are optional; include only the fields you need. | Field | Type | Description | | :-- | :-- | :-- | | `scope` | object | Time window in seconds on the asset timeline, with `start_time` and `end_time`. Unlike the other fields, scope is enforced: the summary is generated only from media within this window. Omit `start_time` to begin at the asset start, and `end_time` to continue through the asset end. | | `summary_style` | string | Preferred style for the generated title and description. Supported values: `concise`, `detailed`, and `editorial`. | | `audience` | string | Intended audience used to guide the generated title and description. | | `brand_terms` | array of strings | Preferred brand or domain terms to use when supported by the source content. | | `tag_taxonomy` | object | Controlled vocabulary for generated tags. Tags may be deterministically filtered to this vocabulary after generation. | | `tag_taxonomy.name` | string | Optional customer-facing name for the taxonomy. | | `tag_taxonomy.values` | array | Controlled vocabulary values. Each value has a required `label` and optional `description` and `aliases`. | | `tag_taxonomy.allow_other` | boolean | When `false`, generated tags are filtered to taxonomy labels and aliases. When `true`, unmatched tags may remain. | ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "tone": "professional", "output_steering": { "summary_style": "concise", "audience": "online shoppers comparing kitchen gadgets", "brand_terms": ["Ember", "Ember Ceramic Mug 2"], "tag_taxonomy": { "name": "Product categories", "values": [ { "label": "Smart home", "description": "App-connected or temperature-controlled devices", "aliases": ["connected device", "smart gadget"] }, { "label": "Kitchen accessories" } ], "allow_other": true } } } } ``` ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.summarize.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/summarize/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `title` | string | Generated title capturing the essence of the video. | | `description` | string | Generated description of the video content (typically 2-4 sentences). | | `tags` | array | Generated keyword tags for the video. | | `asset_updates` | object | Outcome of the asset title update when `update_asset_meta` is `true`, as `asset_updates.title.status` (`updated` or `failed`). Omitted when no update outcome was recorded. | ## Example completed job This example uses a five-minute asset, consuming 650 units. It is the job delivered in the [`robots.v1.job.summarize.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/summarize/{JOB_ID}`: ```json { "data": { "id": "rjob_abc123", "workflow": "summarize", "status": "completed", "units_consumed": 650, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "tone": "professional", "output_steering": { "summary_style": "concise", "audience": "online shoppers comparing kitchen gadgets", "brand_terms": ["Ember", "Ember Ceramic Mug 2"], "tag_taxonomy": { "name": "Product categories", "values": [ { "label": "Smart home", "description": "App-connected or temperature-controlled devices", "aliases": ["connected device", "smart gadget"] }, { "label": "Kitchen accessories" } ], "allow_other": true } } }, "outputs": { "title": "Ember Ceramic Mug 2 — Temperature Control Demo", "description": "A hands-on look at the Ember Ceramic Mug 2, demonstrating its app-controlled temperature settings, battery life, and auto-sleep feature on the charging coaster.", "tags": ["ceramic mug", "temperature control", "smart mug", "kitchen gadget", "gift idea", "coffee accessories", "app-controlled", "rechargeable", "white", "14oz"] } } } ``` # Ask questions about a video Ask questions about a video and get structured answers with the Mux Robots API, and limit the answers to part of the video. Use this workflow to extract information, classify videos, or build decision logic on video content. Ask one or more questions about a video and get structured answers. This is a flexible workflow that lets you extract specific information from video content, classify videos, or build decision-making logic around video analysis. See the Ask Questions API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. ## Create an `ask-questions` job ```bash curl https://api.mux.com/robots/v1/jobs/ask-questions \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "questions": [ { "question": "Does this video contain a product demonstration?", "answer_options": ["yes", "no"] }, { "question": "Is the speaker using a presentation?", "answer_options": ["yes", "no"] }, { "question": "What programming language is shown?", "answer_options": ["python", "javascript", "go", "other"] } ] } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.ask_questions.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/ask-questions/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to analyze. | | `questions` | array | **Required.** One or more questions to ask about the video. All questions are evaluated in a single AI call. | | `questions[].question` | string | The question to ask about the video content. | | `questions[].answer_options` | array | Allowed answer values the AI must choose from. Defaults to `["yes", "no"]`. Can be customized to any set of options, e.g. `["high", "medium", "low"]`. Mutually exclusive with `free_form_reply`. | | `questions[].free_form_reply` | boolean | When `true`, the AI replies with free-form prose instead of choosing from `answer_options`. Mutually exclusive with `answer_options`. Treat the answer as untrusted model output. | | `language_code` | string | BCP 47 language code of the caption track to analyze (e.g. `en`, `fr`). An explicit value must match a ready track exactly, including case: `en` does not match `en-US`. When omitted, the first ready subtitle track in asset track order is used. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `max_free_form_answer_length` | integer | Maximum character length for free-form answers, between 1 and 1000. Defaults to 500. Ignored unless at least one question sets `free_form_reply` to `true`. | | `output_steering` | object | Optional controls that restrict question answering to part of the asset. | | `output_steering.scope` | object | Time window in seconds on the asset timeline, with `start_time` and `end_time`. Omit `start_time` to begin at the asset start, and `end_time` to continue through the asset end. | ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.ask_questions.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/ask-questions/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `answers` | array | One answer per question, in the same order as the input questions. | | `answers[].question` | string | The original question that was asked. | | `answers[].answer` | string or null | The answer, constrained to one of the provided `answer_options`. For `free_form_reply` questions, a prose answer up to `max_free_form_answer_length` characters. Null when the question was skipped. | | `answers[].skipped` | boolean | `true` if the question could not be answered from the video content. | | `answers[].confidence` | number | Confidence score from 0.0 to 1.0. | | `answers[].reasoning` | string | Explanation citing specific visual or audio evidence from the video. | Confidence scores give you a signal for how certain the AI is: values above 0.9 indicate clear, unambiguous evidence; 0.7-0.9 strong evidence with minor ambiguity; 0.5-0.7 moderate evidence; below 0.5 weak or uncertain evidence. Questions can be skipped if the AI determines the question is not relevant to the video content. When a question is skipped, `answer` is `null`, `skipped` is `true`, and `confidence` is `0`. Your code should handle skipped answers — see the example response below. ## Example completed job This example uses a five-minute asset and three questions, consuming 1,050 units. It is the job delivered in the [`robots.v1.job.ask_questions.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/ask-questions/{JOB_ID}`: ```json { "data": { "id": "rjob_jkl012", "workflow": "ask-questions", "status": "completed", "units_consumed": 1050, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "questions": [ { "question": "Does this video contain a product demonstration?", "answer_options": ["yes", "no"] }, { "question": "Is the speaker using a presentation?", "answer_options": ["yes", "no"] }, { "question": "What programming language is shown?", "answer_options": ["python", "javascript", "go", "other"] } ] }, "outputs": { "answers": [ { "question": "Does this video contain a product demonstration?", "answer": "yes", "skipped": false, "confidence": 0.95, "reasoning": "The video shows a screen recording of a product interface with the speaker walking through features and clicking through different screens." }, { "question": "Is the speaker using a presentation?", "answer": "no", "skipped": false, "confidence": 0.88, "reasoning": "The video shows a live product demo and code editor rather than a slide deck." }, { "question": "What programming language is shown?", "answer": null, "skipped": true, "confidence": 0, "reasoning": "No code or programming language is visible in the video." } ] } } } ``` ## Custom answer options The `answer_options` field lets you constrain answers to any set of values, making it easy to build classification and routing logic. The default is `["yes", "no"]`, but you can use any set of strings. Since `answer_options` is specified per question, you can use different options for different questions in the same job. **Classify production quality:** ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "questions": [ { "question": "What is the production quality of this video?", "answer_options": ["professional", "semi-professional", "amateur"] } ] } } ``` **Auto-categorize uploads for a video library:** ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "questions": [ { "question": "What type of content is this video?", "answer_options": ["tutorial", "interview", "product-demo", "vlog", "other"] } ] } } ``` # Find key moments Find the most compelling moments in a video with the Mux Robots API, and steer which moments are picked and how they're titled. Use the results to create highlight clips, previews, or social media teasers. Identify the most compelling moments in a video. This workflow analyzes the transcript, and optionally the video's shots, to find segments that stand out for their hook strength, clarity, emotional intensity, novelty, or soundbite quality. It's useful for generating highlight reels, social media clips, or preview content. See the Find Key Moments API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. For a complete agent workflow from episode to downloadable clips, use the [podcast clipping prompt](/docs/prompts/podcast-clips). It covers caption preparation, reviewing candidate moments, and creating the selected clips. ## Create a `find-key-moments` job ```bash curl https://api.mux.com/robots/v1/jobs/find-key-moments \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "max_moments": 5 } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.find_key_moments.completed` webhook](/docs/guides/robots#webhooks)** — the payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/find-key-moments/{JOB_ID}` with the `id` from the response until the status is `completed`. By default, key moment extraction uses transcript cues from the asset to identify compelling segments. Make sure your asset has captions, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before creating a find-key-moments job. For a video asset, set `use_shots: true` to let visual evidence drive selection; captions are then not required. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to analyze. | | `max_moments` | integer | Maximum number of key moments to extract (1-25). When omitted, defaults to 10 for assets up to one hour and 25 for longer assets. | | `target_duration` | object | Preferred highlight duration range in seconds. Both `min` and `max` are required when provided. | | `target_duration.min` | number | **Required.** Preferred minimum highlight duration in seconds, with millisecond precision. | | `target_duration.max` | number | **Required.** Preferred maximum highlight duration in seconds, with millisecond precision. | | `use_shots` | boolean | When `true`, shots are generated or reused and visual evidence drives moment selection. When `false` or omitted, selection uses the transcript and the asset must have a caption track. Not supported for audio-only assets. | | `output_steering` | object | Curated controls that guide moment selection, titles, audience, and concepts without changing the output schema. See [Output steering](#output-steering). | ### Output steering Use `output_steering` when you want best-effort control over which moments are selected and how they're described. These fields guide the workflow but do not guarantee exact output. All `output_steering` fields are optional; include only the fields you need. | Field | Type | Description | | :-- | :-- | :-- | | `selection_strategy` | string | Preferred definition of a strong standalone moment. Supported values: `standalone_hooks`, `educational_takeaways`, `story_beats`, `product_moments`, and `speaker_highlights`. | | `title_style` | string | Preferred style for generated moment titles. Supported values: `descriptive`, `punchy`, `educational`, and `social`. | | `audience` | string | Intended audience used to guide moment selection and titles. | | `brand_terms` | array of strings | Preferred brand or domain terms to use when supported by the source content. | | `rubric_priorities` | array of strings | Up to 4 rubric dimensions used as tie-breakers after applying the selection strategy. Supported values: `clarity_in_isolation`, `emotional_intensity`, `novelty`, and `soundbite_quality`. | | `topic_taxonomy` | object | Controlled vocabulary used to steer notable audible concepts without changing the response schema. | | `topic_taxonomy.name` | string | Optional customer-facing name for the taxonomy. | | `topic_taxonomy.values` | array | Controlled vocabulary values. Each value has a required `label` and optional `description` and `aliases`. | | `topic_taxonomy.allow_other` | boolean | When `true`, non-taxonomy values may be used when no taxonomy value applies. | | `scope` | object | Time window in seconds on the asset timeline, with `start_time` and `end_time`. Moments are selected only from media within this window, while returned timestamps remain absolute asset timestamps. Omit `start_time` to begin at the asset start, and `end_time` to continue through the asset end. Use it to skip an intro, ad reads, or a segment you have already clipped. | ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "max_moments": 5, "output_steering": { "selection_strategy": "standalone_hooks", "title_style": "social", "audience": "developers scrolling a social feed", "brand_terms": ["Mux Video", "Mux Data"], "rubric_priorities": ["soundbite_quality", "emotional_intensity"], "topic_taxonomy": { "name": "Themes", "values": [ { "label": "Video as data", "description": "Treating video content as structured, queryable information", "aliases": ["structured video", "queryable video"] }, { "label": "Developer experience" } ], "allow_other": true } } } } ``` ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.find_key_moments.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/find-key-moments/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `moments` | array | Extracted key moments, ordered by position in the video. | | `moments[].start_time` | number | Moment start time in seconds on the asset timeline. Fractional seconds are supported. | | `moments[].end_time` | number | Moment end time in seconds on the asset timeline. Fractional seconds are supported. | | `moments[].overall_score` | number | Weighted quality score (0.0-1.0) based on hook strength, clarity, emotional intensity, novelty, and soundbite quality. | | `moments[].title` | string | Short catchy title for the moment (3-8 words). | | `moments[].quotable_segment` | object | Best short, verbatim quote from within the moment, with `text` and `time` (quote start time in seconds). Omitted when the moment has no transcript quote. | | `moments[].audible_narrative` | string | One-sentence summary of what is being said. | | `moments[].notable_audible_concepts` | array | Key audible concepts (2-5 word phrases). | | `moments[].visual_narrative` | string | One-sentence summary of what is visually happening. Present for video assets only. | | `moments[].notable_visual_concepts` | array | Scored visual concepts extracted from sampled frames (video assets only). Each has `concept`, `score`, and `rationale`. | | `moments[].cues` | array | Contiguous transcript segments with `start_time`, `end_time` (in seconds), and `text`. | ## Example completed job This example uses a five-minute asset, consuming 4,000 units. It is the job delivered in the [`robots.v1.job.find_key_moments.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/find-key-moments/{JOB_ID}`: ```json { "data": { "id": "rjob_mno345", "workflow": "find-key-moments", "status": "completed", "units_consumed": 4000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "max_moments": 3 }, "outputs": { "moments": [ { "start_time": 12.4, "end_time": 28.9, "overall_score": 0.92, "title": "The Future of Video Data", "quotable_segment": { "time": 16.2, "text": "Every video you upload is a dataset waiting to be queried." }, "audible_narrative": "The speaker explains how AI transforms video from passive content into structured, queryable data.", "notable_audible_concepts": ["video as data", "AI transformation", "structured information"], "visual_narrative": "The speaker gestures at a diagram showing video processing pipeline stages.", "notable_visual_concepts": [ { "concept": "pipeline diagram", "score": 0.87, "rationale": "Directly illustrates the concept being discussed" } ], "cues": [ { "start_time": 12.4, "end_time": 16.2, "text": "What's exciting is that video isn't just content anymore." }, { "start_time": 16.2, "end_time": 22.1, "text": "Every video you upload is a dataset waiting to be queried." } ] } ] } } } ``` # Find best thumbnails Find and rank the best thumbnail frames in a video with the Mux Robots API, and steer which kinds of frames score highest. Use this workflow to pick poster images or thumbnails for your video gallery. Find the strongest thumbnail frames in a video. Mux samples frames across a Mux asset, scores each one for thumbnail quality, and returns the top candidates ranked best to worst with per-category subscores and a short description of each frame. It's useful for auto-selecting a poster image, building a thumbnail picker, or A/B testing thumbnail options. See the Find Best Thumbnails API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Find best thumbnails requires a video asset in the `ready` state. Audio-only assets are not supported, since the workflow scores sampled video frames. ## Create a `find-best-thumbnails` job ```bash curl https://api.mux.com/robots/v1/jobs/find-best-thumbnails \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "max_thumbnails": 3 } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.find_best_thumbnails.completed` webhook](/docs/guides/robots#webhooks)**. The payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/find-best-thumbnails/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video asset to pull thumbnail candidates from. | | `max_thumbnails` | integer | **Optional.** Maximum number of candidate thumbnails to return, from `1` to `5`. Defaults to `5`. | | `output_steering` | object | **Optional.** Curated controls that guide how candidates are scored without changing the response schema. See [Output steering](#output-steering). | | `update_asset_thumbnail` | boolean | **Optional.** When `true`, the highest-scoring thumbnail's timestamp is written to the asset's `thumbnail_time` once the job completes, making that frame the asset's default thumbnail. Defaults to `false`. See [Set the asset's default thumbnail](#set-the-assets-default-thumbnail). | ### Output steering Use `output_steering` when you want best-effort control over which frames score highest. These fields guide the workflow but do not guarantee exact output. All `output_steering` fields are optional; include only the fields you need. | Field | Type | Description | | :-- | :-- | :-- | | `selection_strategy` | string | Preferred kind of thumbnail candidate to favor. Supported values: `face_or_action`, `clean_composition`, `high_contrast`, `brand_safe`, and `campaign_thumbnail`. | | `looking_for` | string | Open-ended description of what you're explicitly looking for in candidate thumbnails. | | `audience` | string | Intended audience used to guide scoring. | | `campaign_style` | string | Short description of the campaign or channel thumbnail style to prefer when supported by the frame. | | `scoring_priorities` | array of strings | Rubric criteria to emphasize in close calls, matching the [subscores](#output). Supported values: `focus`, `face_or_action`, `composition`, `contrast_color`, and `brand_fit`. | | `scope` | object | Time window in seconds on the asset timeline, with `start_time` and `end_time`. Frames are sampled only from within this window, while returned timestamps remain absolute asset timestamps. Omit `start_time` to begin at the asset start, and `end_time` to continue through the asset end. The window must contain at least one video frame. | ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "max_thumbnails": 5, "output_steering": { "selection_strategy": "face_or_action", "looking_for": "a clear shot of the host mid-demo with the product visible", "audience": "developers evaluating a video API", "campaign_style": "clean, high-contrast tech explainer", "scoring_priorities": ["focus", "contrast_color"] } } } ``` ### Set the asset's default thumbnail By default, the workflow only returns candidates and leaves the asset untouched. Set `update_asset_thumbnail` to `true` to have Mux write the top-scoring frame's timestamp to the asset's `thumbnail_time` when the job completes, so requests to `https://image.mux.com/{PLAYBACK_ID}/thumbnail.jpg` return that frame. ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "max_thumbnails": 3, "update_asset_thumbnail": true } } ``` The write is best-effort. The outcome is reported in `outputs.asset_updates.thumbnail.status` as `updated` or `failed`. If it fails, the job still completes and returns its ranked candidates, and you can set `thumbnail_time` yourself from `outputs.best_thumbnails`. A new default thumbnail will appear for some clients sooner than others, depending on their local cache settings. See [Set a default thumbnail](/docs/guides/set-a-default-thumbnail#when-the-new-thumbnail-appears) for how the change propagates, and for how to set or clear `thumbnail_time` directly. ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.find_best_thumbnails.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/find-best-thumbnails/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `best_thumbnails` | array | Selected candidate thumbnails, ordered best to worst by overall quality score. | | `best_thumbnails[].time` | number | Frame timestamp in seconds on the asset timeline. Fractional seconds are supported. | | `best_thumbnails[].overall` | number | Overall quality score from `0` to `1`. | | `best_thumbnails[].description` | string | AI-generated description of the frame content. | | `best_thumbnails[].alt_text` | string | Suggested alternative text for the frame, for use with the thumbnail image in an accessible UI. | | `best_thumbnails[].subscores` | object | Per-category quality scores, each from `0` to `1`. | | `best_thumbnails[].subscores.focus` | number | Sharpness and focus quality. | | `best_thumbnails[].subscores.face_or_action` | number | Presence of faces or dynamic action. | | `best_thumbnails[].subscores.composition` | number | Visual composition and framing quality. | | `best_thumbnails[].subscores.contrast_color` | number | Color contrast and vibrancy. | | `best_thumbnails[].subscores.brand_fit` | number | Alignment with brand guidelines. | | `asset_updates` | object | Outcome of the asset write requested with `update_asset_thumbnail`. Omitted when no outcome was recorded. | | `asset_updates.thumbnail.status` | string | `updated` when the asset's `thumbnail_time` was written, or `failed` when the write did not succeed. | Each candidate is identified by its `time`, in seconds. To render the frame as an image, pass it to the [image API](/docs/guides/get-images-from-a-video): `https://image.mux.com/{PLAYBACK_ID}/thumbnail.jpg?time={TIME}`. ## Example completed job This example uses a five-minute asset, consuming 5,250 units. It is the job delivered in the [`robots.v1.job.find_best_thumbnails.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/find-best-thumbnails/{JOB_ID}`: ```json { "data": { "id": "rjob_thumb123", "workflow": "find-best-thumbnails", "status": "completed", "units_consumed": 5250, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "max_thumbnails": 3 }, "outputs": { "best_thumbnails": [ { "time": 42, "overall": 0.91, "description": "The host faces the camera mid-sentence with the product dashboard sharp behind them.", "alt_text": "Host speaking to the camera in front of a product dashboard.", "subscores": { "focus": 0.95, "face_or_action": 0.92, "composition": 0.88, "contrast_color": 0.86, "brand_fit": 0.9 } }, { "time": 118.5, "overall": 0.78, "description": "A wide shot of the upload screen with the cursor on the upload button.", "alt_text": "Upload screen with the cursor on the upload button.", "subscores": { "focus": 0.82, "face_or_action": 0.6, "composition": 0.85, "contrast_color": 0.8, "brand_fit": 0.83 } } ] } } } ``` # Generate engagement insights Turn viewer engagement data into plain-language insights for a Mux video with the Mux Robots API. Use the insights to learn where viewers drop off and why. Generate plain-language insights from how viewers watched a video. This workflow analyzes the engagement patterns Mux Data has collected for an asset (which moments held attention and which lost it) and returns per-moment insights for the most notable moments plus an overall summary and trends. It's useful for explaining retention to non-analysts, prioritizing edits, and surfacing what's working across a library. See the Generate Engagement Insights API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. Generate engagement insights builds on [Mux Data engagement metrics](/docs/guides/engagement-hotspots-and-heatmaps), so the asset needs accumulated viewing data before there's anything to analyze. You'll need: * Mux Data collecting views for the content. By using [Mux Player](/docs/guides/mux-player-web), this happens automatically with no extra setup. With a third-party or custom player, add a [Mux Data SDK](/docs/guides/track-your-video-performance) or [custom integration](/docs/guides/build-a-custom-data-integration) so views are tracked. * Enough completed views for Mux to aggregate engagement. Freshly published content or content without views won't have data to analyze. ## Create a `generate-engagement-insights` job ```bash curl https://api.mux.com/robots/v1/jobs/generate-engagement-insights \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID" } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.generate_engagement_insights.completed` webhook](/docs/guides/robots#webhooks)**. The payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v1/jobs/generate-engagement-insights/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video to analyze engagement for. | ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.v1.job.generate_engagement_insights.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v1/jobs/generate-engagement-insights/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `moment_insights` | array | Per-moment engagement insights for the most notable moments. | | `moment_insights[].start_time` | number | Moment start time in seconds on the asset timeline. Fractional seconds are supported. | | `moment_insights[].end_time` | number | Moment end time in seconds on the asset timeline. Fractional seconds are supported. | | `moment_insights[].engagement_score` | number | Normalized engagement score from `0` to `1`. | | `moment_insights[].insight` | string | Primary insight explaining the engagement pattern for the moment. | | `overall_insight` | object | Overall engagement analysis across the video. | | `overall_insight.summary` | string | Summary of overall engagement patterns. | | `overall_insight.trends` | array of strings | Key trends identified across the video. | ## Example completed job This is the job delivered in the [`robots.v1.job.generate_engagement_insights.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/generate-engagement-insights/{JOB_ID}`: ```json { "data": { "id": "rjob_eng123", "workflow": "generate-engagement-insights", "status": "completed", "units_consumed": 2000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID" }, "outputs": { "moment_insights": [ { "start_time": 30.0, "end_time": 60.0, "engagement_score": 0.89, "insight": "Viewers are highly engaged during the product demo, with minimal drop-off." } ], "overall_insight": { "summary": "Engagement peaks during hands-on demonstrations and drops during introductory segments.", "trends": [ "Product demos drive the highest retention", "Viewers skip past the first 15 seconds of intro" ] } } } } ``` # Generate contextual text Turn an asset into social posts, blog posts, or newsletter copy with the Mux Robots API, and steer the voice, audience, and call to action. Request named variants of a shared set of written artifacts. You can use the Generate Contextual Text workflow to turn an interview, tutorial, product demonstration, or any other video, into social posts, blog posts, and newsletter copy grounded in the source media. You can request all the outputs together, and use variants to get more than one version of the whole set: independent takes to choose between, or different angles you describe with `instructions`. See the Generate Contextual Text API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Check the shared [workflow prerequisites](/docs/guides/robots#workflow-prerequisites) before creating a job. The workflow builds a shared editorial brief from the asset's central idea, supporting points, and concrete details, then writes each artifact from that brief. The text addresses the subject directly: a tutorial can become an instructional article, and a product demonstration can become promotional copy. For a simple title, description, and tags workflow, use [Summarize](/docs/guides/robots-summarize). The asset must be ready, no longer live, and have a public or signed playback ID and a ready caption track. Audio-only assets can also use a single ready legacy text track that isn't marked as captions. [Generate captions](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [add your own captions](/docs/guides/add-subtitles-to-your-videos) before creating a job. Video assets include a storyboard as visual context; the transcript remains the authoritative source. ## Create a `generate-contextual-text` job An **artifact** is one deliverable. A **variant** is a named version of every requested artifact. Two variants and two artifacts produce four pieces of text. A variant's `key` only labels it in the results. Any difference in angle comes from its `instructions`. ```bash curl https://api.mux.com/robots/v1/jobs/generate-contextual-text \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "en", "output_steering": { "audience": "video developers", "voice": "conversational" }, "variants": [ { "key": "option_a", "instructions": "Playful voice for developers who are new to video." }, { "key": "option_b" } ], "artifacts": [ { "key": "x_post", "channel": "x", "max_length": { "unit": "characters", "value": 280 } }, { "key": "blog_post", "instructions": "Write a short blog post.", "max_length": { "unit": "words", "value": 1200 }, "format": "markdown" } ] } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.v1.job.generate_contextual_text.completed` webhook](/docs/guides/robots#webhooks)**. The payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, poll `GET /robots/v1/jobs/generate-contextual-text/{JOB_ID}` with the `id` from the response until the status is `completed`. Jobs are automatically deleted after 30 days. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux video or audio asset ID whose transcript grounds the text. | | `variants` | array | **Required.** 1–5 named versions of the complete artifact set. See [Variants and artifacts](#variants-and-artifacts). | | `artifacts` | array | **Required.** 1–5 deliverables to generate for each variant. See [Variants and artifacts](#variants-and-artifacts). | | `language_code` | string | BCP 47 language code of the caption source. An explicit value must match a ready source exactly, including case: `en` does not match `en-US`. When omitted, the first ready caption track is selected. Audio-only assets can also use a single ready legacy text track. | | `output_language_code` | string | BCP 47 language code for generated text. Defaults to the selected transcript track's language when available. | | `use_shots` | boolean | When `true`, generates or reuses Mux Shots and adds a bounded sample of shot frames as visual evidence. Omitted or `false` uses the storyboard without additional shot frames. Rejected for audio-only assets. Shot generation can incur separate [Mux Video charges](/docs/pricing/overview#shots). | | `output_steering` | object | Shared controls for the analyzed time range, audience, voice, call to action, and brand terminology. See [Output steering](#output-steering). | See [language support for Mux Robots](/docs/guides/robots-supported-languages) for verified languages and language tags. ### Variants and artifacts Each variant and artifact needs a `key`: a lowercase snake\_case identifier starting with a letter, up to 64 characters. Keys must be unique within each array. A key identifies the result; it does not guide generation. Use `instructions` to describe the angle or deliverable. Naming a variant `funny` does not make it funny. Omit `instructions` to get independent takes on the same brief, which is useful when you want several options to choose from or test. Add `instructions` when each variant should take a specific angle. [Output steering](#output-steering) applies to every variant. To give variants different voices or audiences, describe them in each variant's `instructions`, for example `"Playful voice for developers who are new to video"`. Each generated artifact in each variant costs 250 units, in addition to the job's base and duration charges. Two variants with two artifacts count as four artifacts. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing). | Field | Type | Description | | :-- | :-- | :-- | | `variants[].key` | string | **Required.** Identifier returned with this variant. | | `variants[].instructions` | string | Optional guidance for the variant's angle, 1–500 characters. Omit it to request an independent take on the shared brief. | | `artifacts[].key` | string | **Required.** Identifier returned with this artifact in every variant. | | `artifacts[].instructions` | string | Optional guidance for the deliverable, 1–500 characters. | | `artifacts[].channel` | string | Publishing conventions and default length. Supported values: `generic`, `x`, `linkedin`, `facebook`, `instagram`, `tiktok`, and `youtube`. Defaults to `generic`. | | `artifacts[].max_length` | object | Hard output cap with `unit` (`characters` or `words`) and integer `value`. Character caps range from 10 to 20,000; word caps range from 5 to 3,000. Defaults to the channel's cap. `x` always also enforces 280 characters, even when you specify a word cap. | | `artifacts[].format` | string | `plain` (default) or `markdown`. Use `markdown` when the result should be allowed to include Markdown formatting. | Default channel caps are 300 words for `generic` and `linkedin`, 250 words for `facebook` and `youtube`, 100 words for `tiktok`, 1,500 characters for `instagram`, and 280 characters for `x`. An explicit `max_length` overrides the default, subject to the `x` ceiling. Each artifact aims for about 90% of its cap, whatever the length of the source. A one-minute video and a 15-minute video get text of about the same length, so set `max_length` to the length you want rather than relying on the channel default. `generic` has no platform conventions; its style follows the length cap. Short caps produce a single focused piece. For a blog post, article, or newsletter entry, use `generic` with a `max_length` above 600 words and `format: "markdown"`, as in the `blog_post` artifact above. ### Output steering Scope limits which transcript cues and visual context ground the text. The other controls apply best-effort writing guidance to every artifact and do not guarantee exact wording. | Field | Type | Description | | :-- | :-- | :-- | | `scope` | object | Execution window in seconds on the original asset timeline. Supply `start_time`, `end_time`, or both. An omitted start means the beginning; an omitted end means the asset's end. The start must be nonnegative, the end must be positive, and the start must precede the end. An end past the asset's duration is clamped to the asset's end, and a range that starts at or after the asset's end is rejected. | | `audience` | string | Intended reader, up to 160 characters. | | `voice` | string | `conversational`, `editorial`, `playful`, or `professional`. | | `call_to_action` | string | `none`, `soft`, or `direct`, guiding whether and how text invites further engagement. | | `brand_terms` | array of strings | Preferred terms to use when the source supports them. Supply 1–10 terms, up to 40 characters each and 240 characters combined. | For example, generate copy from just the first two minutes: ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "variants": [{ "key": "main" }], "artifacts": [{ "key": "post", "channel": "linkedin" }], "output_steering": { "scope": { "start_time": 0, "end_time": 120 }, "audience": "video developers", "voice": "professional", "call_to_action": "soft", "brand_terms": ["Mux Video"] } } } ``` The selected range must contain usable transcript evidence. A range with no usable transcript cues cannot produce grounded text and the job can fail during processing. ## Output The `outputs` object is included once the job is `completed`, in the [`robots.v1.job.generate_contextual_text.completed`](/docs/guides/robots#webhooks) webhook or the job's `GET` response. | Field | Type | Description | | :-- | :-- | :-- | | `variants` | array | Generated versions of the complete artifact set. | | `variants[].key` | string | The requested variant key. | | `variants[].artifacts` | array | Generated deliverables for this variant. | | `variants[].artifacts[].key` | string | The requested artifact key. | | `variants[].artifacts[].content` | string | Generated text in the requested format, constrained by the artifact's length cap. | Match results to your request using the variant and artifact keys. Each returned artifact in each variant counts once toward unit consumption. ## Example completed job This example uses a two-minute asset with two variants and two artifacts, consuming 1,520 units. It is the job delivered in the [`robots.v1.job.generate_contextual_text.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v1/jobs/generate-contextual-text/{JOB_ID}`: ```json { "data": { "id": "rjob_abc123", "workflow": "generate-contextual-text", "status": "completed", "units_consumed": 1520, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "output_language_code": "en", "output_steering": { "audience": "video developers", "voice": "conversational" }, "variants": [ { "key": "option_a", "instructions": "Playful voice for developers who are new to video." }, { "key": "option_b" } ], "artifacts": [ { "key": "x_post", "channel": "x", "max_length": { "unit": "characters", "value": 280 }, "format": "plain" }, { "key": "blog_post", "instructions": "Write a short blog post.", "channel": "generic", "max_length": { "unit": "words", "value": 1200 }, "format": "markdown" } ] }, "outputs": { "variants": [ { "key": "option_a", "artifacts": [ { "key": "x_post", "content": "Never shipped video before? Make a direct upload, send your file, and grab the playback ID. That's the whole trip from your laptop to a working player with Mux Video." }, { "key": "blog_post", "content": "# Your first video, start to finish\n\nGood news: getting a video from your laptop into a player takes three steps. First, create a direct upload. Once the video is ready, its playback ID connects the uploaded asset to your player." } ] }, { "key": "option_b", "artifacts": [ { "key": "x_post", "content": "Uploading a video and playing it are separate steps. Follow the asset until it is ready, then use its playback ID to load the player." }, { "key": "blog_post", "content": "# Understand the upload lifecycle\n\nA direct upload creates a destination for your video. The walkthrough then follows the uploaded asset to readiness before loading it into a player." } ] } ] } } } ``` # Migrate Robots from Beta to GA Update Robots requests, response handling, and caption track selection for v1 while keeping existing v0 jobs accessible. Use this guide to move an existing integration from the Robots Beta API (`/robots/v0`) to the GA API (`/robots/v1`). V1 changes request fields, response shapes, caption track selection, and webhook event types. Changing the API URL alone is not enough. Paste this prompt into your agent to review your integration, apply the relevant changes, and test the migration.
The Robots v0 API is deprecated and remains available for six months after the v1 launch. Existing jobs and directives are tied to the API version used to create them. Creating v1 resources does not migrate existing v0 resources. If you also use Directives or Robots webhooks, follow [Migrate Robots Directives from Beta to GA](/docs/guides/robots-migrate-directives-to-v1) and [Migrate Robots webhooks from Beta to GA](/docs/guides/robots-migrate-webhooks-to-v1). See the Robots API reference for the full contracts. ## Plan your rollout During the migration, v0 and v1 run side by side. Jobs and Directives you already have stay on v0, while anything you create through v1 returns v1 response shapes and sends v1 webhook events. Follow these steps in order so your integration can handle both versions at every stage, and so you can switch new requests back to v0 until the cutover is complete. 1. Update and deploy webhook handlers that accept both versions **before** creating v1 jobs or Directives. 2. Update request construction and response parsing using the mappings below. Store the API version that created each job or Directive alongside its ID. 3. Test v1 with representative assets, including assets with several caption tracks and any audio-only content you process. 4. Switch new job creation to v1. Continue reading and acting on existing v0 jobs through v0. 5. Create v1 replacements for any v0 Directives, then update the integrations that reference their IDs. 6. Keep v0 webhook handling while v0 Directives are still running or v0 webhook deliveries are still arriving. V0 and v1 don't recognize each other's jobs as duplicates. Submitting the same job to both runs it twice, and both runs consume units. When comparing versions, use a test asset and review any options that write results back to the asset. ## Update request paths and parameters Change new job requests from `/robots/v0/jobs/{workflow}` to `/robots/v1/jobs/{workflow}`. Mux API authentication, access tokens, and the `parameters` request envelope stay the same. For example, key-moment duration preferences change from integer milliseconds to seconds. **V0 request:** ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "target_duration_ms": { "min": 15000, "max": 45000 } } } ``` **V1 request:** ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "target_duration": { "min": 15, "max": 45 } } } ``` Update these fields wherever you construct workflow requests, including [Directive binding parameters](/docs/guides/robots-migrate-directives-to-v1#convert-the-configuration). | Workflow | V0 field | V1 action | | :-- | :-- | :-- | | `find-key-moments` | `target_duration_ms.min` / `.max` | Use `target_duration.min` / `.max`, dividing values by 1,000. Fractional seconds with millisecond precision are supported. | | `translate-audio`, `translate-captions` | `to_language_code` | Rename to `output_language_code`. | | `summarize`, `generate-chapters` | `prompt_overrides` | Remove it and express supported preferences through `output_steering`. This is not a one-to-one prompt conversion. | | `generate-premium-captions` | `replace_existing` | Use `replace_existing_tracks`: `true` maps to `"replace_all"`; `false` maps to `"fail"`. | | `edit-captions` | `delete_original_track` | Choose `replace_existing_tracks`: `"replace"` replaces the source, and `"fail"` keeps it and creates a separately named track. | | `edit-captions` | `track_name_suffix` | Supply the complete desired `track_name` instead. | For edit-captions, preserving the source requires `replace_existing_tracks: "fail"` and a `track_name` different from the source track's name. The default v1 replacement policy is `"replace"`. Review this explicitly if your v0 integration used the legacy naming or deletion options. Unknown request fields are ignored, rather than treated as aliases. For example, sending only `to_language_code` to a v1 translation endpoint does not satisfy the required `output_language_code` field. Remove obsolete fields from your integration even when the request succeeds without them. ## Update response parsing These changes apply to both a job's `GET` response and the `data` object of its v1 webhook. ### Read media times as seconds Media offsets in v1 use seconds and can contain fractional values. These are positions on the original asset timeline, not Unix timestamps. A value of `23.456` represents the same position as `23456` milliseconds in v0. | Workflow / output | V0 | V1 | | :-- | :-- | :-- | | `find-best-thumbnails`, `outputs.best_thumbnails[]` | `timestamp_ms` | `time` | | `find-key-moments`, `outputs.moments[]` and nested `cues[]` | `start_ms`, `end_ms` | `start_time`, `end_time` | | `find-key-moments`, `quotable_segment` | `start_ms` | `time` | | `generate-engagement-insights`, `outputs.moment_insights[]` | `start_ms`, `end_ms` | `start_time`, `end_time` | Remove any existing millisecond-to-second conversion when reading v1 values, or playback positions will be scaled twice. Unix timestamps such as `created_at` and `updated_at` already use seconds and do not need this conversion. Chapter start times already use seconds too. ### Read track IDs and asset update outcomes | Workflow / output | V0 | V1 | | :-- | :-- | :-- | | `edit-captions`, `translate-captions`, `translate-audio` | `outputs.uploaded_track_id` | `outputs.track_id` | | Workflows reporting deleted tracks | `replaced_tracks: [{ "id": "TRACK_ID", ... }]` | `replaced_track_ids: ["TRACK_ID"]` | | `generate-premium-captions`, legacy alias | `outputs.replaced_track_id` | `outputs.replaced_track_ids[]` | | `generate-premium-captions` | `outputs.detected_language` | `outputs.detected_language_code` | | `generate-chapters`, legacy alias | `outputs.asset_update` | `outputs.asset_updates.chapters` | | `generate-chapters`, nested replacement outcome | `asset_updates.chapters.replaced_track_id` | `asset_updates.chapters.replaced_track_ids[]` | `replaced_track_ids` contains IDs of tracks that were deleted. It is omitted when no deletions were recorded; do not assume it is always present. V1 no longer includes the metadata objects previously returned in `replaced_tracks`. In caption editing and translation, `parameters.track_id` is the **source** track, while `outputs.track_id` identifies the **newly created** track. Writing results back to the asset is best effort. A job can complete successfully while an asset update reports `status: "failed"`, so check the corresponding `asset_updates` entry separately from the job status. Older jobs that recorded no update outcome omit it. For chapter replacement, a failed update with deleted track IDs means deletion succeeded but creation failed. ## Review caption track selection V1 uses the first ready caption track in asset track order, without preferring English. If your integration relied on the English preference, supply `language_code` explicitly when the workflow supports it. An explicit source `language_code` must match exactly, including case and region. For example, `en` does not select an `en-US` caption track. When checking for an existing track in a translation's target language, or a track to replace, `en` and `en-US` count as the same language. An explicit caption `track_id` must identify a ready caption track. Audio-only assets with exactly one ready text track that isn't marked as captions can use that track as a legacy transcript. Audio-only moderation no longer assumes `language_code: "en"` when you omit it. Before switching production requests, test with assets that have multiple languages, text tracks that aren't captions, or captions that are still preparing. [Jobs dispatched by a Directive](/docs/guides/robots-migrate-directives-to-v1#test-and-switch-automatic-triggers) follow the selection rules of the Directive's version. ## Replace Find Scenes integrations `find-scenes` is not included in the v1 release. Use [Analyze Structure](/docs/guides/robots-analyze-structure) (`POST /robots/v1/jobs/analyze-structure`) and read `outputs.scenes`. Analyze Structure also produces a shared title, summary, tags, chapters, and key moments. ```bash curl https://api.mux.com/robots/v1/jobs/analyze-structure \ -u "$MUX_TOKEN_ID:$MUX_TOKEN_SECRET" \ -H "Content-Type: application/json" \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID" } }' ``` Update your scene parser for `start_time` and `end_time` in seconds, including nested cue and shot intervals. Review the Analyze Structure contract rather than copying Find Scenes parameters unchanged: its output steering and result structure are different, and generated scene boundaries can differ. ## Keep existing jobs on the version that created them The v1 jobs list can discover jobs created through either version. Each summary includes `_links.self.href`, which points to the route for the version that created the job: ```json { "id": "rjob_example", "_links": { "self": { "href": "/robots/v0/jobs/summarize/rjob_example" } } } ``` This is an excerpt from a v1 list response. Follow the relative URL against `https://api.mux.com` rather than rebuilding it with `/v1`. List summaries do not contain full workflow parameters or outputs. Detail and cancel requests must use the version that created the job. A v0 job requested through a v1 detail route returns `404`; it is not converted into a v1 job on read. For cancellation, use `POST /robots/{VERSION}/jobs/{JOB_ID}/cancel` with that version. Store the version that created each new job, and keep it in polling queues, saved job links, and retry/cancel actions. A v1 webhook identifies a v1-created job; a v0 webhook identifies a v0-created job. ## Update error handling Read `error.messages[]` instead of the removed `error.message` field. For example: ```json { "error": { "type": "track_not_ready", "messages": ["No ready subtitle track available for this asset"] } } ``` Handle errors according to their type and status rather than matching message strings. | Situation | V1 response | Migration action | | :-- | :-- | :-- | | Malformed JSON | `400` | Correct the request encoding. | | Invalid recognized parameters or malformed IDs | `422 validation_error` | Correct the values or ID. | | Invalid Mux credentials | `401 unauthorized` | Check the credentials. | | Preparing asset or required caption track | `409 asset_not_ready` or `409 track_not_ready` | Wait for the resource to become ready before retrying. | | Duplicate active job | `409 conflict` | Track the existing job instead of creating another. | | Errored asset or unusable caption track | `422` | Correct the source; repeated readiness retries will not resolve it. | | Missing resource or wrong API version | `404` | Check the ID, environment, and the API version that created the resource. | These readiness and duplicate-job types describe job creation. Other operations can return `409` for operation-specific conflicts, such as cancelling a job already in a terminal state. ## Migration checklist * \[ ] New job requests use `/robots/v1` and the renamed parameters. * \[ ] Request builders no longer send removed fields. * \[ ] Response parsers read seconds, v1 track IDs, and `error.messages[]`. * \[ ] Caption track tests cover your actual languages and track order. * \[ ] Find Scenes integrations use Analyze Structure and its result schema. * \[ ] Job lists follow `_links.self`, and stored jobs keep the API version that created them. * \[ ] Webhook handlers that accept both versions are deployed before v1 creation starts. * \[ ] V0 Directives have v1 replacements. See [Migrate Robots Directives from Beta to GA](/docs/guides/robots-migrate-directives-to-v1). # Migrate Robots Directives from Beta to GA Create v1 Directives, convert binding and resource configuration, and switch asset creation to the new Directive without losing access to v0 runs. A Directive keeps the API version used to create it. To move a Directive from the Robots Beta API (v0) to the GA API (v1), create a new Directive with v1 configuration, test it, then replace the old Directive ID in your asset-creation integrations. The existing Directive does not change version. Nothing changes for your existing Directives today. They stay on v0 and keep running as normal: assets created with a v0 Directive ID still trigger it, and its runs, child jobs, and webhooks work as they do now. Migrate them to v1 before the v0 support period ends, six months after the v1 launch. Paste this prompt into your agent to review your integration, apply the relevant changes, and test the migration.
Start with the [Robots API migration guide](/docs/guides/robots-migrate-to-v1) for workflow parameter and output changes. Update and deploy [webhook handlers](/docs/guides/robots-migrate-webhooks-to-v1) before starting v1 runs. See the Directives API reference for the complete configuration. ## Choose how to create the v1 copy ### From the Dashboard Open an existing v0 Directive and choose **Create v1 copy**. Review the workflows, resources, and options, then create the new Directive. Legacy options that v1 no longer supports are omitted from the copy. Remove workflows unavailable in v1, such as Find Scenes, before creating it; replace scene analysis with Analyze Structure where appropriate. The copy has a new ID. The v0 Directive stays in place, and integrations that attach its ID keep triggering it. Creating a copy does not move asset references, modify external integrations, or delete the original. ### From the API 1. Fetch the existing configuration through `GET /robots/v0/directives/{DIRECTIVE_ID}`. 2. Convert it using the mappings below. Send configuration fields only, omitting response metadata such as `id`, timestamps, and `_links`. 3. Create the replacement with `POST /robots/v1/directives`. 4. Store the returned v1 Directive ID alongside its API version. ## Convert the configuration | Configuration field | V0 | V1 | | :-- | :-- | :-- | | Workflow bindings | `workflows[]` | `bindings[]` | | Binding options | `params` | `parameters` | | Mux API source options | `source.params` | `source.parameters` | | Caption resource selector | `kind: "caption"` | `kind: "subtitles"` | | Track resource language | `language` | `language_code` | | Caption generation source audio language | `source.params.source_audio_language` | `source.parameters.source_audio_language_code` | | Shot-generation action | `action: "request_shots"` | `action: "generate_shots"` | Binding `workflow`, binding/resource `reference_id`, `inputs[]`, and `source.binding` keep their names. Preserve reference IDs so dependencies still resolve. Don't put `asset_id` or the source `track_id` in binding parameters; the Directive supplies them. Apply the [workflow parameter changes](/docs/guides/robots-migrate-to-v1#update-request-paths-and-parameters) inside every binding too: translation targets become `output_language_code`, key-moment target durations become seconds, and deprecated fields are removed. Replace `find-scenes` bindings with an appropriately configured `analyze-structure` binding. For example, this v0 configuration generates a caption resource, then translates it: ```json { "name": "Generate and translate captions", "subject": { "type": "video.asset" }, "resources": [ { "reference_id": "captions_en", "type": "video.asset.track", "kind": "caption", "language": "en", "source": { "via": "mux_api", "action": "generate_subtitles", "params": { "language_code": "en", "source_audio_language": "en-US" } } } ], "workflows": [ { "reference_id": "translate_es", "workflow": "translate-captions", "inputs": ["captions_en"], "params": { "to_language_code": "es" } } ] } ``` Save this v1 configuration as `directive-v1.json`: ```json { "name": "Generate and translate captions v1", "subject": { "type": "video.asset" }, "resources": [ { "reference_id": "captions_en", "type": "video.asset.track", "kind": "subtitles", "language_code": "en", "source": { "via": "mux_api", "action": "generate_subtitles", "parameters": { "language_code": "en", "source_audio_language_code": "en-US" } } } ], "bindings": [ { "reference_id": "translate_es", "workflow": "translate-captions", "inputs": ["captions_en"], "parameters": { "output_language_code": "es" } } ] } ``` Create it through v1: ```bash curl https://api.mux.com/robots/v1/directives \ -u "$MUX_TOKEN_ID:$MUX_TOKEN_SECRET" \ -H "Content-Type: application/json" \ -d @directive-v1.json ``` Unknown fields are ignored, so a successful create request does not mean obsolete optional fields still take effect. Review the resulting configuration and its behavior on a test asset. ## Update list and detail handling `GET /robots/v1/directives` returns slim summaries of **both v0 and v1 Directives**, with `id`, `name`, `created_at`, `updated_at`, and `_links.self`. It no longer returns the full configuration in every list entry. Follow `_links.self.href` to get the full configuration. A summary with a `/robots/v0/directives/...` link still represents a v0 Directive even though you discovered it through the v1 list. The v0 list continues returning full v0 configurations. Individual detail, delete, and run routes require the version that created the Directive. A version mismatch returns `404`; the v1 detail endpoint only returns v1 Directives. Keep the version that created each Directive with any saved Directive ID. If an older saved link contains only an ID, try the v1 detail endpoint and fall back to v0 **only on 404**, not authentication or server errors. For a v0 Directive, keep using `/robots/v0/directives/{id}/runs` and its run-detail routes while it remains active. For a v1 Directive, use the corresponding v1 routes. ## Update run-response parsing V1 uses one run representation across start responses, lists, detail reads, and webhook `data`. It includes `id`, `directive_id`, `subject_id`, `status`, `node_states`, `started_at`, and `completed_at`. | Surface | V0 | V1 | | :-- | :-- | :-- | | HTTP run identifier | `run_id` | `id` | | Run relationship | V0 HTTP responses vary by operation | `directive_id` consistently identifies the Directive. | | Dispatched/waiting node workflow | `workflow_name` | `workflow` | | Source dependency status | `waiting_for_source_workflow` | `waiting_for_source_bindings` | | Source dependency references | `source_workflows[]` | `source_bindings[]` | | Directive-run webhook asset | `data.asset_id` | `data.subject_id` | `source_bindings[]` contains binding reference IDs, not workflow names. Node failure details remain in `reason`. Run timestamps are still Unix seconds. The manual start request still uses `{ "asset_id": "YOUR_ASSET_ID" }`; `subject_id` is a response field, not a renamed request parameter. Child jobs inherit the Directive's version. Fetch their details through that version too, rather than assuming that every job visible in your dashboard or list is v1. ## Test and switch automatic triggers A Directive triggers automatically when its ID is attached to an asset at creation. Test the new Directive, then switch those attachments to its ID. 1. Run the new Directive manually against a test asset: ```bash curl https://api.mux.com/robots/v1/directives/YOUR_V1_DIRECTIVE_ID/runs \ -u "$MUX_TOKEN_ID:$MUX_TOKEN_SECRET" \ -H "Content-Type: application/json" \ -d '{ "asset_id": "YOUR_TEST_ASSET_ID" }' ``` 2. Verify the v1 run, its child-job outputs, and the `robots.v1.directive_run.*` and `robots.v1.job.*` webhook handlers. V1 caption track selection rules also apply when a Directive dispatches jobs. 3. Replace the v0 Directive ID in asset-creation requests, upload settings, templates, and any stored integration configuration with the **new v1 ID**. The Video API route itself does not change: ```bash curl https://api.mux.com/video/v1/assets \ -u "$MUX_TOKEN_ID:$MUX_TOKEN_SECRET" \ -H "Content-Type: application/json" \ -d '{ "inputs": [{ "url": "https://example.com/my-video.mp4" }], "playback_policies": ["public"], "directives": [{ "id": "YOUR_V1_DIRECTIVE_ID" }] }' ``` 4. Confirm a newly created asset triggers the v1 Directive. A Directive runs on the version that created it; there is no version field to add to the Video request's `directives[]` entry. 5. Keep v0 history and webhook handling available until the old Directive has stopped running and outstanding deliveries have arrived. Delete the old Directive through its v0 route once you no longer need new runs from it. Attaching both the old and new Directive IDs can start both Directives. Their child jobs can both execute and consume units. Creating the v1 copy does not replace the old attachment automatically. To roll back during the v0 support period, restore the old v0 ID in the integrations that create assets. Keep the original Directive until you have verified the cutover; deleting it removes that rollback option. ## Migration checklist * \[ ] Every binding and source uses the v1 field names and units. * \[ ] Legacy options and workflows unavailable in v1 have been reviewed or replaced. * \[ ] The new Directive has been tested manually, including caption track selection and asset updates. * \[ ] List consumers fetch configuration through `_links.self`. * \[ ] Run parsers and child-job reads use the correct version. * \[ ] Webhook handlers accept both v0 and v1 events. * \[ ] Asset-creation integrations reference the new v1 Directive ID. * \[ ] A newly uploaded asset has successfully triggered the new Directive. # Migrate Robots webhooks from Beta to GA Handle versioned Robots job and Directive-run events before you start creating jobs and Directives through v1. The Robots Beta API is v0 and the GA API is v1, and each sends its own webhook events. Deploy support for v1 webhook events **before** you start creating jobs or Directives through `/robots/v1`. The API version that created a job or Directive determines both its event type and payload. Existing v0 resources continue sending v0 events, including events delivered after your application starts creating v1 resources. Paste this prompt into your agent to review your integration, apply the relevant changes, and test the migration.
Use [Migrate Robots from Beta to GA](/docs/guides/robots-migrate-to-v1) for request and output mappings, and [Migrate Robots Directives from Beta to GA](/docs/guides/robots-migrate-directives-to-v1) for switching asset creation to v1 Directives. ## Accept both event namespaces | Event family | V0 | V1 | | :-- | :-- | :-- | | Job lifecycle | `robots.job.{workflow}.{event}` | `robots.v1.job.{workflow}.{event}` | | Directive-run lifecycle | `robots.directive_run.{event}` | `robots.v1.directive_run.{event}` | Workflow names in event types use underscores: `find_key_moments`, `translate_audio`, or `generate_engagement_insights`. HTTP workflow paths continue using hyphens. For example, `robots.job.translate_audio.completed` becomes `robots.v1.job.translate_audio.completed`. Directive-run events use `created`, `completed`, `partial`, and `errored` in both namespaces. Update exact-match handlers, prefix checks, queues, analytics rules, and any event-type allowlists in your own infrastructure. A handler or downstream filter that only accepts `robots.job.*` will miss `robots.v1.job.*`. Keep the existing [webhook signature verification](/docs/core/verify-webhook-signatures), duplicate-delivery handling, and retry behavior. The new namespace does not replace those requirements. ## Route payloads by event type Use the event's `type` to choose the parser. Don't pick the parser based on which version your app currently creates jobs with: v0 and v1 events arrive side by side during the migration. The following example dispatches an **already verified** webhook event. Each handler then parses the payload for its version. ```javascript function handleRobotsEvent(event) { if ( event.type.startsWith('robots.v1.job.') || event.type.startsWith('robots.v1.directive_run.') ) { return handleV1RobotsEvent(event); } if ( event.type.startsWith('robots.job.') || event.type.startsWith('robots.directive_run.') ) { return handleV0RobotsEvent(event); } } ``` The envelope is unchanged: `data` is the job or run object itself, in both versions. Do not add an extra `job` or `directive_run` wrapper to your parser. For example, this is an excerpt from a v1 caption-translation event: ```json { "type": "robots.v1.job.translate_captions.completed", "data": { "id": "rjob_example", "workflow": "translate-captions", "status": "completed", "parameters": { "asset_id": "YOUR_ASSET_ID", "track_id": "SOURCE_TRACK_ID", "output_language_code": "es" }, "outputs": { "track_id": "NEW_TRACK_ID", "replaced_track_ids": ["DELETED_TRACK_ID"] } } } ``` This excerpt omits unchanged job and event fields. Download URLs remain available where the workflow supplies them. Replacement IDs are optional and appear only when deletions were recorded. ## Update job payload parsers V1 job webhook data uses the same v1 parameter and output field names as a v1 job detail response. Update these together: * Translation targets use `parameters.output_language_code`. * Key-moment duration preferences use `parameters.target_duration` in seconds. * Thumbnail times use `time`; moment and cue intervals use `start_time` and `end_time`. Values are seconds and can be fractional. * Caption editing, caption translation, and audio translation return `outputs.track_id`. * Track replacements use `replaced_track_ids[]` strings instead of `replaced_tracks[]` objects or singular aliases. * Premium captions report `detected_language_code`. * Chapter asset update outcomes use `asset_updates.chapters`. * Deprecated parameters and output aliases are absent. Use the [full response mapping](/docs/guides/robots-migrate-to-v1#update-response-parsing) to update each workflow you consume. Keep a v0 parser for old jobs instead of applying v1 renames or unit conversions to every incoming event. When fetching more details after an event, use `/robots/v1/jobs/{workflow}/{JOB_ID}` for v1-created jobs and `/robots/v0/jobs/{workflow}/{JOB_ID}` for v0-created jobs. A wrong-version detail request returns `404`. ## Update Directive-run payload parsers V1 events deliver the same run representation used by the v1 start, list, and detail endpoints. For example, a created event has the following `type` and `data`: ```json { "type": "robots.v1.directive_run.created", "data": { "id": "drvrun_example", "directive_id": "drv_example", "subject_id": "YOUR_ASSET_ID", "status": "pending", "node_states": [], "started_at": 1746403300, "completed_at": null } } ``` V0 Directive-run webhook data uses `asset_id`; v1 uses `subject_id`. Nodes use `workflow` instead of `workflow_name`. Nodes waiting on another binding report `waiting_for_source_bindings` and `source_bindings[]` instead of `waiting_for_source_workflow` and `source_workflows[]`. Failure details remain in `reason`. The webhook run identifier is `data.id` in both versions. The `run_id` → `id` rename applies to v0 **HTTP** run responses. See [run-response changes](/docs/guides/robots-migrate-directives-to-v1#update-run-response-parsing) for the complete mapping. Runs started by a Directive, and the jobs they dispatch, send events for the Directive's version. A v0 Directive keeps sending v0 events after you switch your own job requests to v1. [Create and attach a v1 Directive](/docs/guides/robots-migrate-directives-to-v1#test-and-switch-automatic-triggers) to move that Directive's events to v1. ## Test before switching to v1 1. Deploy handlers for both versions while you're still creating jobs through v0. 2. Create one v1 job for each workflow your integration uses. Verify event routing, times parsed as seconds, any track IDs, and follow-up GET requests. 3. If you use Directives, start a v1 run and verify both its run event and its child-job events. Check `partial` and `errored` handling too. 4. Confirm your consumer still processes a v0 delivery. Retries or outstanding runs can deliver v0 events after the cutover. 5. Switch new requests to v1, and keep monitoring both namespaces. If you switch new requests back to v0, keep both parsers deployed: jobs and Directives already created through v1 keep sending v1 events. Remove v0 handling only after you've stopped creating v0 jobs, your v0 Directives have stopped running, and remaining deliveries have been accounted for. # Migrate the Mux TypeScript SDK from Robots Beta to GA Upgrade @mux/ts to version 16, where client.robots calls the Robots GA API, and choose which calls move to v1 now and which stay on the Beta API for the moment. Version 16 of the [Mux TypeScript SDK](/docs/integrations/mux-typescript-sdk) (`@mux/ts`) makes the Robots GA API (v1) the default. `client.robots` now calls `/robots/v1`, and the Beta API (v0) has moved to `client.robotsV0`. This guide covers the SDK-specific steps. For field-by-field changes, see [Migrate Robots from Beta to GA](/docs/guides/robots-migrate-to-v1). Paste this prompt into your agent to review your integration, apply the relevant changes, and test the migration.
Many Robots calls written for version 15 still compile against version 16, but now send requests to v1. Ask Questions, for example, has the same request shape in both versions. Review every `client.robots` call after upgrading: either move it to `client.robotsV0` to stay on the Beta API for now, or update it for v1. ## Before you upgrade * **Switch from `@mux/mux-node` to `@mux/ts`.** `@mux/mux-node` was renamed, and 15.5.0 is its final release. Replace the dependency and update your imports: ```bash npm uninstall @mux/mux-node && npm install @mux/ts@^15 ``` ```diff - import Mux from '@mux/mux-node'; + import Mux from '@mux/ts'; ``` * **Accept the Robots terms in the Mux Dashboard.** Without them, v1 requests return `403`. * **Update and deploy your webhook handlers** so they accept both v0 and v1 events. See [Migrate Robots webhooks from Beta to GA](/docs/guides/robots-migrate-webhooks-to-v1). ## Upgrade to version 16 ```bash npm install @mux/ts@^16 ``` Version 16 has other breaking changes outside Robots, such as the removal of `client.data.realTime`. See the SDK's [migration guide](https://github.com/muxinc/mux-ts/blob/main/MIGRATION.md#migrating-from-v15-to-v16) for the full list. ## Choose v0 or v1 for each call For each Robots call, either keep it on the Beta API for now or move it to v1. **Stay on the Beta API:** switch to `client.robotsV0`. Its methods are deprecated, and each one points to its v1 replacement. Beta types keep their version 15 names under `Mux.RobotsV0`; for example, `Mux.Robots.WorkflowBinding` becomes `Mux.RobotsV0.WorkflowBinding`. ```diff - await mux.robots.jobs.summarize.create({ parameters: { asset_id } }); + await mux.robotsV0.jobs.summarize.create({ parameters: { asset_id } }); ``` **Move to v1:** keep `client.robots` and update the request and response fields. For example, Find Key Moments durations change from milliseconds to seconds: ```diff await mux.robots.jobs.findKeyMoments.create({ parameters: { asset_id, - target_duration_ms: { min: 15000, max: 45000 }, + target_duration: { min: 15, max: 45 }, }, }); ``` Renaming `target_duration_ms` to `target_duration` without converting the values asks for durations 1,000 times longer. Existing jobs and Directives keep the API version used to create them. Read and cancel v0 jobs through `client.robotsV0`, and v1 jobs through `client.robots`. ## Update code that moves to v1 The [API migration guide](/docs/guides/robots-migrate-to-v1) lists every field change. In summary: * **Times are in seconds:** every millisecond field on requests and responses becomes a seconds field, and fractional seconds are allowed. * **Renamed and removed fields:** for example, `to_language_code` becomes `output_language_code`, `uploaded_track_id` becomes `track_id`, and `replaced_tracks` becomes `replaced_track_ids`. * **Workflows:** Find Scenes has no v1 equivalent; use [Analyze Structure](/docs/guides/robots-analyze-structure) instead. * **Directives:** `workflows` becomes `bindings`, and each binding's `params` becomes `parameters`. `directives.list()` now returns summaries; retrieve a Directive for its full configuration. See [Migrate Robots Directives from Beta to GA](/docs/guides/robots-migrate-directives-to-v1). Some SDK types are renamed for v1 too: | Version 15 | Version 16 | | :-- | :-- | | `WorkflowBinding` | `DirectiveBinding` | | `TrackResourceDeclaration` | `DirectiveTrackResource` | | `ShotsResourceDeclaration` | `DirectiveShotsResource` | | `NodeState` | `DirectiveNodeState` | | `DirectiveRunDetail` | `DirectiveRun` | In version 15, `Mux.Robots.DirectiveRun` was the response from creating a run. In version 16, the same name is the full run, which creating a run now returns. ## Update webhook types v1 jobs and Directive runs send `robots.v1.job..` and `robots.v1.directive_run.` events, typed with payload types such as `WebhookSummarizeJob` and `WebhookDirectiveRun`. Beta events keep their names, and their payload types gain a `V0` suffix, such as `WebhookSummarizeJobV0`. If you stay on v0, the Directive-run event types now describe the run itself in `data`, which is what Mux delivers, instead of a `directive_run` wrapper. ## Migration checklist * \[ ] The dependency is `@mux/ts` version 16, with no `@mux/mux-node` imports left. * \[ ] Every `client.robots` call is either on `client.robotsV0` deliberately or updated for v1. * \[ ] v1 request builders and response parsers use seconds and the renamed fields. * \[ ] Imports of renamed Directive types are updated. * \[ ] Webhook handlers accept both v0 and v1 events and use the matching payload types. * \[ ] Non-Robots breaking changes in the SDK's migration guide are handled. # Find scenes Split a video into ordered scenes with the Mux Robots API. Use scene data to build chapter navigation, edit workflows, or search inside a video. Find Scenes is only available in the deprecated Robots Beta API (`/robots/v0`) and is not part of Robots GA (`/robots/v1`). Use [Analyze Structure](/docs/guides/robots-analyze-structure) instead: it returns scenes alongside a title, summary, tags, chapters, and key moments. To move an existing integration, see [Replace Find Scenes integrations](/docs/guides/robots-migrate-to-v1#replace-find-scenes-integrations). Find scenes in a video and get structured, timestamped scene metadata. This workflow segments a Mux asset into ordered scenes, using visual signals and transcript cues when available. It is useful for building scene indexes, searchable video catalogs, navigation aids, and review workflows. See the Find Scenes API reference for the full endpoint specification. See [Mux Robots pricing](/docs/pricing/overview#mux-robots-pricing) for unit costs. Find scenes requires a video asset with generated shots in the `ready` state. Audio-only assets are not supported. Segmentation works from visual shots alone, but a text track makes the output richer by adding audible context to each scene. For the best results, make sure your asset has captions, either [auto-generated](/docs/guides/add-autogenerated-captions-and-use-transcripts) or [manually added](/docs/guides/add-subtitles-to-your-videos), before running a find-scenes job. ## Create a `find-scenes` job ```bash curl https://api.mux.com/robots/v0/jobs/find-scenes \ -H "Content-Type: application/json" \ -X POST \ -d '{ "parameters": { "asset_id": "YOUR_ASSET_ID", "language_code": "en", "min_scenes": 4, "min_scene_duration_ms": 15000 } }' \ -u ${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET} ``` This request is **asynchronous**. The `POST` returns immediately with the job in `pending` status and does not include results. **We strongly recommend listening for the [`robots.job.find_scenes.completed` webhook](/docs/guides/robots#webhooks)**. The payload contains the full completed job, so no follow-up API call is needed. If webhooks aren't an option, you can poll `GET /robots/v0/jobs/find-scenes/{JOB_ID}` with the `id` from the response until the status is `completed`. ## Parameters | Parameter | Type | Description | | :-- | :-- | :-- | | `asset_id` | string | **Required.** The Mux asset ID of the video asset to segment into scenes. | | `language_code` | string | **Optional.** Preferred transcript language code to analyze when a matching ready transcript track is available. See [language support for Mux Robots](/docs/guides/robots-supported-languages). | | `min_scenes` | integer | **Optional.** Lower-bound hint for scene segmentation. Mux avoids collapsing clearly distinct beats below this count when the content supports more granular scene boundaries. | | `min_scene_duration_ms` | integer | **Optional.** Preferred minimum scene duration in milliseconds. Must be at least `1000`. Defaults to `15000`. | | `output_steering` | object | **Optional.** Curated controls that guide scene grouping, titles, narratives, and concepts without changing the response schema. See [Output steering](#output-steering). | ### Output steering Use `output_steering` when you want best-effort control over how scenes are grouped and described. These fields guide the workflow but do not guarantee exact output. All `output_steering` fields are optional; include only the fields you need. | Field | Type | Description | | :-- | :-- | :-- | | `segmentation_strategy` | string | Preferred segmentation strategy. Supported values: `editorial_beats`, `topic_changes`, `visual_transitions`, `action_progression`, and `instructional_steps`. | | `title_style` | string | Preferred scene title style. Supported values: `descriptive`, `editorial`, `search_optimized`, and `accessibility`. | | `narration_detail` | string | Preferred level of narrative detail. Supported values: `concise`, `balanced`, and `detailed`. | | `audience` | string | Intended audience used to guide scene grouping, titles, and narrative emphasis. | | `brand_terms` | array of strings | Preferred brand or domain terms to use when supported by the source content. | | `topic_taxonomy` | object | Controlled vocabulary used to steer scene concepts without changing the response schema. | | `topic_taxonomy.name` | string | Optional customer-facing name for the taxonomy. | | `topic_taxonomy.values` | array | Controlled vocabulary values. Each value has a required `label` and optional `description` and `aliases`. | | `topic_taxonomy.allow_other` | boolean | When `true`, non-taxonomy values may be used when no taxonomy value applies. | ```json { "parameters": { "asset_id": "YOUR_ASSET_ID", "output_steering": { "segmentation_strategy": "instructional_steps", "title_style": "search_optimized", "narration_detail": "detailed", "audience": "support agents reviewing product walkthroughs", "brand_terms": ["Mux Video", "Mux Data"], "topic_taxonomy": { "name": "Product areas", "values": [ { "label": "Video uploads", "description": "Uploading, ingesting, or configuring new video assets", "aliases": ["uploads", "direct uploads"] }, { "label": "Playback analytics", "description": "Viewer experience, engagement, or streaming performance" } ], "allow_other": true } } } } ``` ## Output The `outputs` object is included in the job once its status is `completed`. You'll receive it on the [`robots.job.find_scenes.completed`](/docs/guides/robots#webhooks) webhook (recommended), or you can fetch it with `GET /robots/v0/jobs/find-scenes/{JOB_ID}`. It contains: | Field | Type | Description | | :-- | :-- | :-- | | `scenes` | array | Ordered scene segments extracted from the asset. | | `scenes[].start_ms` | number | Scene start time in milliseconds. | | `scenes[].end_ms` | number | Scene end time in milliseconds. | | `scenes[].title` | string | Short human-readable title for the scene. | | `scenes[].cues` | array | Transcript cues that overlap the scene. Can be empty when no usable transcript is available. | | `scenes[].cues[].start_ms` | number | Cue start time in milliseconds. | | `scenes[].cues[].end_ms` | number | Cue end time in milliseconds. | | `scenes[].cues[].text` | string | Transcript text for this cue. | | `scenes[].audible_narrative` | string | One-sentence summary of what is being said during the scene. Present only when transcript cues are available. | | `scenes[].blended_narrative` | string | Combined audible and visual narrative for the scene when both signals are meaningfully available. | | `scenes[].notable_audible_concepts` | array | Multi-word descriptive phrases capturing key audible concepts in the scene. Present only when transcript cues are available. | | `scenes[].visual_narrative` | string | One-sentence summary of what is visually happening during the scene. | | `scenes[].notable_visual_concepts` | array | Scored visual concepts extracted from sampled scene frames. Each item has `concept`, `score`, and `rationale`. | | `scenes[].shots` | array | Optional ordered shot or beat breakdown within the scene, including timings and short visual descriptions. | | `scenes[].shots[].start_ms` | number | Shot or beat start time in milliseconds. | | `scenes[].shots[].end_ms` | number | Shot or beat end time in milliseconds. | | `scenes[].shots[].visual_description` | string | Short visual description of what happens during this shot or beat. | | `scenes[].shot_count` | integer | How many timed shot or beat entries make up the scene. When `shots` is present, this equals `shots.length`. | ## Example completed job This example uses a five-minute asset, consuming 3,000 units. It is the job delivered in the [`robots.job.find_scenes.completed`](/docs/guides/robots#webhooks) webhook's `data` field and returned by `GET /robots/v0/jobs/find-scenes/{JOB_ID}`: ```json { "data": { "id": "rjob_scn123", "workflow": "find-scenes", "status": "completed", "units_consumed": 3000, "created_at": 1746403200, "updated_at": 1746403260, "parameters": { "asset_id": "YOUR_ASSET_ID", "language_code": "en", "min_scenes": 4, "min_scene_duration_ms": 15000 }, "outputs": { "scenes": [ { "start_ms": 0, "end_ms": 18500, "title": "Opening Product Context", "cues": [ { "start_ms": 500, "end_ms": 4300, "text": "Today we are walking through the upload workflow." } ], "audible_narrative": "The speaker introduces the upload workflow and sets context for the walkthrough.", "blended_narrative": "A presenter introduces the product workflow while the interface is visible on screen.", "notable_audible_concepts": ["upload workflow walkthrough", "product context"], "visual_narrative": "A presenter stands beside a screen showing a video upload interface.", "notable_visual_concepts": [ { "concept": "upload interface", "score": 0.88, "rationale": "The visible UI anchors the scene around the upload workflow." } ], "shots": [ { "start_ms": 0, "end_ms": 8200, "visual_description": "The presenter appears next to the dashboard." }, { "start_ms": 8200, "end_ms": 18500, "visual_description": "The screen focuses on the upload controls." } ], "shot_count": 2 } ] } } } ```