Skip to main content
McCarthy Search

Senior ML Engineer

L5

sourcefuseBoise, IDtoday

Occupations

Data ScientistsComputer and Information Research ScientistsSoftware Developers

Industries

Custom Computer Programming ServicesComputer Systems Design ServicesOther Scientific and Technical Consulting Services
About the Role: We’re looking for hands-on ML engineers with deep learning-to-rank experience to work directly with a client running a two-sided healthcare marketplace that matches patients with clinicians. The client recently moved from a rule-based heuristic to a first-generation deep neural network ranker. Offline metrics looked strong, but online A/B results told a more complicated story, with mixed movement across funnel stages, appointment types, and devices. Your job is to diagnose why offline and online results diverged, then help the client build and launch the next version of their ranker with confidence. This is algorithm and modeling work: ranking, evaluation, and experimentation. It is not solutions architecture or systems integration.
The Challenge: The client’s search pipeline filters a large pool of candidate clinicians for each query and ranks a shortlist for display. Users move through a multi-stage funnel: impression, profile view, calendar reservation, optional registration, and completed appointment. Optimizing that funnel raises several hard, interconnected problems: Offline-to-online alignment: Offline ranking metrics don’t reliably predict online business outcomes, and the A/B methodology needs strengthening. Position and selection bias: Training data comes from truncated ranking lists, so the model learns only from what users were shown. Multi-objective labels: A single hybrid label currently blends several funnel objectives into one NDCG target, masking trade-offs between them. Session complexity: Users often run multiple searches within a single session. Delayed conversions: Conversions depend on clinician availability and can take a long time, and late conversions are currently dropped from training. Underused features: Signals such as device type likely matter a great deal but aren’t fully represented in the model.
What You’ll Do:
  • Diagnose the root causes behind divergent offline and online results across funnel stages, appointment types, and device segments.
  • Redesign evaluation so offline metrics better predict online impact, and improve A/B test design, power, and analysis.
  • Apply unbiased and counterfactual learning-to-rank techniques to correct for position and selection bias in logged data.
  • Rework label design and multi-objective optimization so the model balances competing funnel goals explicitly.
  • Develop approaches for delayed and censored conversion feedback rather than discarding late outcomes.
  • Model multi-search session behavior and bring in context features such as device type.
  • Train, tune, and evaluate ranking models at scale on AWS, and define a clear, low-risk launch plan for the next model version.
  • Mentor and upskill the client’s small in-house team, including an ML engineer new to ranking, so improvements outlast the engagement.
  • Technical Environment
  • Tens of millions of training samples with a few dozen query-side and provider-side features
  • Amazon S3 (Parquet) for data storage
  • Amazon SageMaker for model training and hosting
  • Python ML ecosystem (e.g., PyTorch or TensorFlow, LightGBM/XGBoost, TF-Ranking or similar)
  • What You Bring
  • RequiredUS-based, with authorization to work in the United States.
  • Hands-on, production experience building and shipping learning-to-rank systems for search, recommendations, or marketplaces.
  • Strong command of ranking objectives and metrics (pointwise, pairwise, and listwise approaches; NDCG, MRR, and related measures) and their limits.
Practical experience with position bias, selection bias, and counterfactual or unbiased LTR methods (e.g., inverse propensity weighting, click models, randomization-based propensity estimation).Deep experience with online experimentation: A/B test design, statistical power, variance reduction, segment analysis, and reconciling offline and online results. Experience with multi-objective ranking or funnel-based optimization. Proficiency in Python and modern ML frameworks, and comfort working with large-scale datasets. The ability to work directly with client stakeholders, explain findings clearly, and turn analysis into concrete modeling decisions.
Preferred: Experience in two-sided marketplaces (e.g., healthcare, gig economy, travel, real estate, or e-commerce).Experience modeling delayed feedback, censored labels, or survival-style conversion outcomes. Experience with session-aware or context-aware ranking, including device and channel effects. Hands-on experience with Amazon SageMaker and S3-based data pipelines. Healthcare industry experience. Prior consulting or client-facing engagements where you mentored in-house teams.
Apply now (opens in a new tab)
Apply now (opens in a new tab)

Level

SeniorL5

Location

Boise, ID

Occupation

Data Scientists

Industry

Custom Computer Programming Services

Posted

today

To get sharper similar jobs, create your profile using the link below.

Create profile