Remove MLflow tracing from production deployment #118

New Issue

alvis · 2026-05-06T10:29:03Z

alvis commented

2026-05-06 10:29:03 +00:00

MLflow tracing was added to ml/serving/main.py for observability during M2/M3 development. It logs one run per /recommend call plus per-agent compute and infer runs.

Problem: In production this adds latency (synchronous REST calls to MLflow after each tip) and creates a hard dependency on the MLflow container being reachable. The current implementation is fire-and-forget (errors are swallowed), but the HTTP calls still add wall-clock time.

Action items:

Gate tracing behind an env var flag (e.g. MLFLOW_TRACING=true) rather than presence of MLFLOW_TRACKING_URI
Move MLflow logging to a background task / thread so it never blocks the response path
Or replace with async-only tracing (LangFuse, which is already in the Agap stack) and remove MLflow dependency from ml-serving entirely
Remove MLFLOW_TRACKING_URI from the full compose profile; only include it in the mlops profile
Ensure ml-serving starts and serves tips correctly with no MLflow configured (already the case — verify in CI)

Acceptance criteria:

Zero added latency to /recommend in production
No MLflow container dependency for the full compose profile

MLflow tracing was added to `ml/serving/main.py` for observability during M2/M3 development. It logs one run per `/recommend` call plus per-agent `compute` and `infer` runs. **Problem:** In production this adds latency (synchronous REST calls to MLflow after each tip) and creates a hard dependency on the MLflow container being reachable. The current implementation is fire-and-forget (errors are swallowed), but the HTTP calls still add wall-clock time. **Action items:** - Gate tracing behind an env var flag (e.g. `MLFLOW_TRACING=true`) rather than presence of `MLFLOW_TRACKING_URI` - Move MLflow logging to a background task / thread so it never blocks the response path - Or replace with async-only tracing (LangFuse, which is already in the Agap stack) and remove MLflow dependency from `ml-serving` entirely - Remove `MLFLOW_TRACKING_URI` from the `full` compose profile; only include it in the `mlops` profile - Ensure `ml-serving` starts and serves tips correctly with no MLflow configured (already the case — verify in CI) **Acceptance criteria:** - Zero added latency to `/recommend` in production - No MLflow container dependency for the `full` compose profile

alvis added this to the M4 — MLOps at scale milestone 2026-05-06 10:29:03 +00:00

alvis referenced this issue from a commit

2026-05-06 10:31:01 +00:00

feat(serving): add MLflow tracing to ml-serving for all agent calls

alvis referenced this issue from a commit

2026-05-06 10:41:57 +00:00

docs: update CLAUDE.md with session learnings (#118 tracing, compose gotchas)

alvis modified the milestone from M4 — MLOps at scale to M3 — Mobile & notifications

2026-05-14 10:50:08 +00:00

Sign in to join this conversation.