AI Systems Engineer - DevOps& Observability - Senior
$128,400 per year
Apply now (opens in a new tab)AI Systems Engineer - DevOps& Observability - Senior
L6
$128,400 per year
Occupations
Computer Systems Engineers/ArchitectsSoftware DevelopersNetwork and Computer Systems Administrators- Supports DevOps and delivery for AI workloads:build and operate the CI/CD/CV pipelines that ship AI services, agents, and runtime components, including automated build, test, continuous verification, release, and rollback, so AI workloads are delivered repeatably and safely into every environment.
- Own governance and discovery for AI assets, including service catalog/registry (Artifactory/Nexus, Harbor), experiment tracking and model metadata (MLflow), upstream registries/mirrors (HuggingFace/NGC), CVE/SBOM scanning (Trivy), lineage contracts (OpenLineage), and license management.
- Own resource and cost management, including quotas and rate limits, cost attribution and utilization (Apptio/OpenCost/Kubecost), so AI execution stays economically bounded and controllable per tenant and engagement.
- Own the full observability stack, including metrics (Prometheus/Mimir), logs (Loki), traces (Tempo/Jaeger), dashboards (Grafana), LLM debugging and evaluation (LangSmith/Langfuse), and SLA/alert notifications.
- Own the OpenTelemetry collection layer, including multi-tenant receiver, exporters and queues (Kafka sink), DCGM exporter for GPU telemetry, processor batching, and dynamic filtering, so every signal is captured and routed reliably.
- Automate GitOps-based delivery and continuous verification; embedding quality, integrity, and cost gates into pipelines so releases are policy-compliant by default rather than by manual review.
- Close the loop between delivery and observability by using telemetry, evaluation, and cost signals to drive deployment decisions, progressive rollout, and automated rollback of AI workloads.
- Ensure cost and telemetry are identity-stamped and per-tenant, so consumption and behavior are attributable end-to-end, keeping FinOps and observability tied to the workloads that generate the load.
- Skills and attributes for success
- Strong DevOps expertise: CI/CD/CV pipeline design, GitOps, continuous verification, and progressive/automated release and rollback for production workloads.
- Deep expertise operating model-serving and inference systems (Ray, vLLM/Triton/NIM) on GPUs at production scale.
- Deep observability skills: metrics, logs, traces, and OpenTelemetry.
- FinOps mindset: able to attribute, bound, and optimize AI consumption cost per tenant and workload.
- Familiarity with model/artifact governance, registries, CVE scanning, and license/lineage tracking.
- Comfortable operating across cloud, on-prem, edge, and air-gapped environments with consistent runtime and telemetry semantics.
- Strong communicator able to explain runtime, cost, and observability tradeoffs to engineers, architects, and leadership.
- To qualify you must have 8+ years in DevOps, MLOps, platform, or observability engineering, with hands-on production ownership of AI or high-throughput services.
- Strong hands-on DevOps experience, including CI/CD/CV pipelines and GitOps tooling (ArgoCD, Helm, GitHub Actions/GitLab CI, or equivalents) for automated build, test, release, and rollback.
- Hands-on expertise operating inference/model-serving frameworks (Ray Serve, vLLM, Triton, or NIM) on GPU infrastructure.
- Strong experience with observability stacks (Prometheus, Grafana, Loki, Tempo/Jaeger) and OpenTelemetry.
- Experience with API gateways and request routing (Envoy or equivalent), including streaming responses.
- Experience with cost management / FinOps tooling (OpenCost, Kubecost, or equivalent) and quota/rate-limit enforcement.
- Familiarity with model/artifact registries and supply-chain scanning (Harbor, MLflow, Trivy/SBOM).Proven track record operating AI or service infrastructure under compliance, security, or regulatory constraints.
- Ability to define clean ownership boundaries and consumption contracts with platform, trust, and data teams.
- Ideally, you'll also have
- Bachelor's or Master's degree in Computer Science or related technical field.
- Experience with LLM evaluation and debugging tooling (LangSmith, Langfuse) and prompt/response quality measurement.
- Experience with sandboxed/secure execution (gVisor, Firecracker, or microVM isolation) for untrusted or multi-tenant workloads.
- Familiarity with GPU telemetry (DCGM) and GPU utilization optimization.
- Experience with lineage and governance contracts (OpenLineage) and AI license management.
- Exposure to multi-tenant cost attribution and per-tenant SLA/alerting.
- Exposure to regulated delivery environments (financial services, tax, healthcare, risk).What we offer you
- At EY, we harness our collective strength to empower you to shape your future with confidence through professional growth, personal fulfillment and an inclusive culture. Learn more at ey.com/us/careers .The salary range for this job is:
Level
LeadL6
Salary
$128,400 per year
Location
Albany, NY
Occupation
Computer Systems Engineers/Architects
Industry
Computer Systems Design Services
Posted
yesterday
To get sharper similar jobs, create your profile using the link below.