iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

TF Serving

By · Reviewed & last updated · Part of TensorFlow Tutorial

TensorFlow Serving is the production server for SavedModel artifacts. It speaks REST and gRPC, supports multiple model versions, can hot-swap a new model without downtime, and batches inflight requests for GPU throughput. Run it as a Docker container next to your app or behind a load balancer.

Export a SavedModel, run TF Serving, and call it

EXAMPLE
import tensorflow as tf

# 1) Train (or load) a Keras model
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,), name='features'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax', name='probs'),
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Iris-ish toy data
import numpy as np
X = np.random.rand(120, 4).astype('float32')
y = np.random.randint(0, 3, 120).astype('int32')
model.fit(X, y, epochs=2, verbose=0)

# 2) Export to a versioned directory: /models/iris/1/
export_dir = '/models/iris/1'
tf.saved_model.save(model, export_dir)

# 3) Run TF Serving in Docker, watching that directory
# docker run -p 8501:8501 -p 8500:8500 \
#   -v /models/iris:/models/iris \
#   -e MODEL_NAME=iris \
#   tensorflow/serving:2.16.1

# 4) Predict via REST (JSON)
# POST /v1/models/iris:predict
# {
#   "instances": [[5.1, 3.5, 1.4, 0.2], [6.7, 3.0, 5.2, 2.3]]
# }
import json, urllib.request
payload = json.dumps({'instances': X[:2].tolist()}).encode('utf-8')
req = urllib.request.Request('http://localhost:8501/v1/models/iris:predict',
                              data=payload, headers={'Content-Type': 'application/json'})
res = json.loads(urllib.request.urlopen(req).read())
print(res['predictions'])

# 5) Health and metadata
# GET /v1/models/iris               -> versions and state
# GET /v1/models/iris/metadata      -> input/output signatures

# 6) Hot-deploy a new version: write a SavedModel to /models/iris/2/
#    TF Serving picks it up and starts routing traffic to it automatically.

# 7) Batching config (a .conf file passed to the container) — important on GPU
# max_batch_size { value: 64 }
# batch_timeout_micros { value: 1000 }
# max_enqueued_batches { value: 8 }
# num_batch_threads { value: 4 }

# 8) gRPC client — lower latency, smaller payloads for production
# import grpc
# from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc
# channel = grpc.insecure_channel('localhost:8500')
# stub    = prediction_service_pb2_grpc.PredictionServiceStub(channel)
# request = predict_pb2.PredictRequest()
# request.model_spec.name = 'iris'
# request.inputs['features'].CopyFrom(tf.make_tensor_proto(X[:2]))
# response = stub.Predict(request, timeout=2.0)

Why it matters

Use TF Servings versioned directory layout (/models///) and let it manage rollouts. Replacing model.h5 in place is the easy way to ship a regression at 2am; dropping a new /models/iris/2/ alongside /1/ gives you instant rollback by deleting the bad version.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# Export as a SavedModel
model.export('saved_model/1')
# Run TF Serving in Docker
docker run -p 8501:8501 \
    --mount type=bind,source=$(pwd)/saved_model,target=/models/mymodel \
    -e MODEL_NAME=mymodel \
    tensorflow/serving
Try it Yourself »

Discussion

Loading…