iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

AWS Batch

By · Reviewed & last updated · Part of AWS Tutorial

AWS Batch runs containerised batch jobs at scale: scientific computing, video transcoding, ML inference, data pipeline steps. You define the job (a Docker image + CPU/RAM), submit it to a queue, and Batch provisions Spot or On-Demand EC2/Fargate capacity, runs it, and tears down. No cluster to manage, just bills for the seconds you actually compute.

Queue, environment, job definition, and submit

EXAMPLE
# 1) Compute environment — managed pool of capacity Batch will use
aws batch create-compute-environment \
  --compute-environment-name batch-spot \
  --type MANAGED \
  --state ENABLED \
  --service-role arn:aws:iam::123456789012:role/AWSBatchServiceRole \
  --compute-resources '{
    "type": "SPOT",
    "allocationStrategy": "SPOT_PRICE_CAPACITY_OPTIMIZED",
    "minvCpus": 0,
    "maxvCpus": 256,
    "desiredvCpus": 0,
    "instanceTypes": ["c7i","m7i"],
    "subnets": ["subnet-aaa111","subnet-bbb222"],
    "securityGroupIds": ["sg-0abc1234"],
    "instanceRole": "arn:aws:iam::123456789012:instance-profile/ecsInstanceRole",
    "spotIamFleetRole": "arn:aws:iam::123456789012:role/AmazonEC2SpotFleetRole",
    "bidPercentage": 70,
    "tags": { "project": "video-encode" }
  }'

# 2) Job queue — points at one or more compute environments by priority
aws batch create-job-queue \
  --job-queue-name encode-q \
  --state ENABLED \
  --priority 1 \
  --compute-environment-order order=1,computeEnvironment=batch-spot

# 3) Job definition — the container, command, and resource shape
aws batch register-job-definition \
  --job-definition-name encode-mp4 \
  --type container \
  --container-properties '{
    "image": "123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/encoder:1.4",
    "vcpus": 4,
    "memory": 8192,
    "jobRoleArn": "arn:aws:iam::123456789012:role/encoder-task-role",
    "command": ["/usr/local/bin/encode", "Ref::input_uri", "Ref::output_uri"],
    "environment": [
      { "name": "PRESET", "value": "fast" }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": { "awslogs-group": "/aws/batch/encode-mp4" }
    }
  }' \
  --retry-strategy attempts=2 \
  --timeout attemptDurationSeconds=3600

# 4) Submit a one-off job
aws batch submit-job \
  --job-name encode-2026-06-11-001 \
  --job-queue encode-q \
  --job-definition encode-mp4 \
  --parameters input_uri=s3://in/clip.mp4,output_uri=s3://out/clip.mp4

# 5) Submit a 1000-task array job — Batch fans it out across the pool
aws batch submit-job \
  --job-name nightly-array \
  --job-queue encode-q \
  --job-definition encode-mp4 \
  --array-properties size=1000

# 6) Watch progress
aws batch list-jobs --job-queue encode-q --job-status RUNNING --output table
aws batch describe-jobs --jobs <job-id> \
  --query 'jobs[0].{status:status, exit:container.exitCode, log:container.logStreamName}'

Why it matters

For embarrassingly-parallel work, array jobs are the unlock — you submit one job with size=N and pay for the compute, not for hand-rolled queue plumbing. Pair them with Spot capacity and a sensible retry strategy and large encoding/inference batches drop to a fraction of On-Demand price.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# Run long batch jobs — heavy compute, queued, retried.
Try it Yourself »

Discussion

Loading…