The Great ML Serving Smackdown
You’ve trained a model. Now comes the part everybody ignores — getting it in front of something that needs predictions.
Every blog post says “use MLflow for deployment” and “BentoML is great” but nobody tells you when to pick which. Here’s the breakdown with real code.
The Cheat Sheet
| Scenario | Pick This |
|---|---|
| Production REST/gRPC API, 500+ req/sec, p95 < 50ms | BentoML |
| Research phase — 100s of hyperparam combos, plots, team sharing | MLflow |
| Simple CPU batch job, 1M users, speed doesn’t matter | MLflow |
| Hybrid ensemble — 3 CatBoost models + 1 LSTM on GPU | BentoML |
| Hackathon demo, quick REST endpoint, 5 minutes | MLflow |
| Enterprise platform, canary deploys, A/B testing, SLAs | BentoML + MLflow |
The answer is almost always both.
When MLflow Is Your Friend
Scenario 1: You’re Experimenting
You’re in a notebook, trying 47 different hyperparameter combos, and your results are scattered across your terminal history. MLflow Tracking fixes this:
import mlflow # Import MLflow for experiment tracking and model logging
from sklearn.ensemble import RandomForestClassifier # Scikit-learn classifier we'll train and log
from sklearn.metrics import accuracy_score # Metric to evaluate model quality on test data
data = load_wine() # Load the wine-quality dataset (assumed helper)
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target) # Split into train/test sets
with mlflow.start_run(): # Open an MLflow run context — all logs go into this run
model = RandomForestClassifier(n_estimators=200, max_depth=10) # Instantiate RF with 200 trees, depth 10
model.fit(X_train, y_train) # Train the classifier on the training features/labels
acc = accuracy_score(y_test, model.predict(X_test)) # Compute accuracy on held-out test set
mlflow.log_param("n_estimators", 200) # Log hyperparam — visible in MLflow UI for run comparison
mlflow.log_param("max_depth", 10) # Log second hyperparam so you can compare across runs
mlflow.log_metric("accuracy", acc) # Log the accuracy metric for this run
mlflow.sklearn.log_model(model, "model") # Serialize and store the trained model artifact as "model"
print(f"Logged run with accuracy {acc:.4f}") # Confirm the run was logged with its accuracy value
Spin up mlflow ui and you’ve got a dashboard. Compare runs, pick the winner, share a link with your team.
Scenario 2: Quick Batch Job
You’ve got a nightly batch job that scores 1M users with a random forest. No GPU, no latency drama:
import mlflow.sklearn # Import MLflow's sklearn module to load saved models
import pandas as pd # Pandas for reading/manipulating tabular batch data
model = mlflow.sklearn.load_model("models:/wine-classifier/Production") # Load the prod model from MLflow registry
df = pd.read_parquet("data/raw/today_users.parquet") # Read today's batch of user features from Parquet file
predictions = model.predict(df) # Run inference on the entire DataFrame — CPU-only, no batching
df["prediction"] = predictions # Append predictions as a new column to the DataFrame
df.to_parquet("data/predictions/results.parquet") # Write results back to Parquet for downstream consumption
Throw that in a cron job or an Airflow DAG and you’re done.
Scenario 3: Hackathon Demo
Your boss wants a demo by lunch. MLflow Model Serving is the fastest way to get a REST endpoint:
mlflow models serve -m models:/wine-classifier/Production -p 5001 --no-conda
# ^ Launch a REST server for the Production model from registry on port 5001, skip conda env creation for speed
Three seconds later, curl localhost:5001/invocations works. Is it production-grade? No. Does it get the demo done? Yes.
When BentoML Takes Over
MLflow serving is fine for low traffic. But the moment you need actual throughput, it starts sweating.
Scenario 1: High-Performance API
500 requests per second, each under 50 milliseconds. This is BentoML’s home turf:
import bentoml # Import BentoML for high-performance model serving
import numpy as np # NumPy for array manipulation in inference
from bentoml.io import JSON, NumpyNdarray # I/O descriptors: receive ndarray, return JSON
runner = bentoml.mlflow.get("wine-classifier:latest").to_runner() # Pull the latest MLflow model into a BentoML Runner
svc = bentoml.Service("wine-classifier", runners=[runner]) # Create a BentoML service that owns this Runner
@svc.api(input=NumpyNdarray(), output=JSON()) # Decorate endpoint: accepts raw ndarray, returns JSON dict
async def classify(input_data: np.ndarray) -> dict: # Async handler so the event loop can multiplex requests
result = await runner.predict.async_run(input_data) # Offload inference to Runner (adaptive batching, CPU/GPU)
return {"prediction": result.tolist(), "probabilities": "N/A"} # Convert result to list and return JSON response
Save as service.py and serve:
bentoml serve service:svc --reload
# ^ Start the BentoML HTTP server; --reload auto-restarts on file changes during development
BentoML handles batching internally, auto-scales workers, and gives you adaptive micro-batching out of the box.
Scenario 2: The Frankenstein Ensemble
Your model is a monster — 3 CatBoost models doing feature extraction, an LSTM on GPU for sequence data, and a final ensemble step in Python threads. BentoML’s Runner architecture handles the hybrid split:
import bentoml # Core BentoML import for service and runner orchestration
import numpy as np # NumPy for converting input dicts to model-ready arrays
from bentoml.io import JSON # JSON I/O descriptor for both input and output
catboost_runner = bentoml.xgboost.get("catboost-features:latest").to_runner() # Load CatBoost model as a Runner
lstm_runner = bentoml.pytorch.get("lstm-sequence:latest").to_runner() # Load PyTorch LSTM as a separate Runner
svc = bentoml.Service("ensemble-monster", runners=[catboost_runner, lstm_runner]) # Service owns both Runners
@svc.api(input=JSON(), output=JSON()) # Accept JSON dict, return JSON dict
async def predict(input_data: dict) -> dict: # Async handler lets both Runners run concurrently
cat_features = np.array(input_data["tabular"]) # Extract tabular features from the JSON payload
sequences = np.array(input_data["sequences"]) # Extract sequence data for the LSTM
cf = await catboost_runner.predict.async_run(cat_features) # Async inference on CatBoost Runner (CPU)
lstm_out = await lstm_runner.predict.async_run(sequences) # Async inference on LSTM Runner (GPU)
final = (cf * 0.6 + lstm_out * 0.4).tolist() # Weighted ensemble: 60% CatBoost, 40% LSTM
return {"ensemble_prediction": final} # Return combined prediction as JSON
With MLflow serving you’d need to hack around this.
Scenario 3: Enterprise With Canary Deploys
You need A/B testing between v2.3 and v2.4 of your model. BentoML supports multi-version deployment natively:
bentoml serve service:svc --deployment-version=v2.3
# ^ Start a BentoML server pinned to a specific deployment version for canary testing
And in your Kubernetes ingress, route 10% of traffic to v2.4:
apiVersion: networking.k8s.io/v1 # Use the stable v1 Kubernetes networking API
kind: Ingress # Declare this resource as an Ingress (HTTP router)
metadata:
annotations: # Ingress annotations configure nginx behaviour
nginx.ingress.kubernetes.io/canary: "true" # Mark this ingress as a canary rule
nginx.ingress.kubernetes.io/canary-weight: "10" # Route 10% of traffic to this ingress (v2.4)
spec:
... # (omitted) backend service pointing to the v2.4 BentoML deployment
MLflow serving doesn’t do canary.
The Real Answer: Use Both
MLflow is for the research side. BentoML is for the production side.
The actual 2026 stack looks like this:
Train & Log with MLflow
import mlflow # Import MLflow for experiment tracking and model registry
with mlflow.start_run(): # Start a new run — everything logged goes to this experiment
model = train_something() # (placeholder) train your actual model here
mlflow.log_metric("accuracy", 0.97) # Log a dummy accuracy metric for run comparison
mlflow.sklearn.log_model(model, "model") # Serialize the model artifact and register it in MLflow
Package with BentoML
import bentoml # Import BentoML to consume models from MLflow for serving
bentoml.mlflow.import_model("wine-classifier", # Import an MLflow model into BentoML's model store
model_uri="models:/wine-classifier/Production") # Source URI pointing to the Production stage in MLflow registry
bento = bentoml.Service("wine-classifier", runners=[...]) # Wrap the imported model in a BentoML Service (runners omitted)
Serve via BentoML
bentoml serve wine-classifier:latest --production
# ^ Deploy the latest bento in production mode with multi-worker, adaptive batching, and metrics endpoint
Monitor with Prometheus
Both expose /metrics.
When Each Fails
MLflow Serving can’t handle:
- More than ~50 concurrent requests
- GPU/CPU hybrid models
- Canary or A/B deployments
BentoML doesn’t help with:
- Experiment tracking
- Comparing runs and hyperparams
- Sharing results with non-engineers
That’s why you need both.
The Bottom Line
- Exploring? MLflow.
- Shipping to prod? BentoML.
- Building a real platform? Both.
The workflow: mess around in MLflow, find your best run, grab it with BentoML, serve it for real.
Comments
Leave a comment