Lead Data Engineer

nazztecCary, NCtoday

Occupations

Data Warehousing Specialists Data Scientists Database Architects

Industries

Computer Systems Design Services Custom Computer Programming Services Software Publishers

🚨 READ CAREFULLY BEFORE APPLYING | URGENT HIRING🔹 ROLE: Lead Data Engineer – Hands-OnClient: MetLife

Location: Cary, NC — On-site / Hybrid

Experience: 12–18 Years

Employment Type: Full-Time

Eligibility: 🇺🇸 US CITIZENS ONLY🔥 ABOUT THE ENGAGEMENTThe engagement involves building a centralized, AI-first enterprise Data Hub for a global insurance and financial services organization using Azure Databricks. The platform ingests 150+ inbound data feeds and distributes data to 35+ downstream systems, leveraging a Medallion Architecture — Bronze / Silver / Gold. AI is embedded across ingestion, canonical mapping, data quality, reconciliation, and business-user access. This is a senior hands-on technical leadership role. The selected candidate will own the end-to-end technical design of the data and AI layers, build reference implementations, mentor engineering teams, and continue writing/reviewing production-grade Python, Scala, and PySpark code every week.⚠️ Candidates who have NOT written or reviewed production code within the past year will NOT be considered.🎯 WHAT YOU WILL OWNData Platform Architecture & Engineering Design and implement enterprise-scale Lakehouse architecture. Define Bronze / Silver / Gold data contracts. Design ADLS Gen2 zone layouts. Delta Lake table design, partitioning, schema evolution and retention. Build scalable and reusable data platform components. Metadata-Driven Ingestion Design parameterized ingestion frameworks for: Batch files Database extractsCDC feeds Streaming data Implement streaming pipelines using Azure Event Hubs / Kafka / Spark Structured Streaming. Build schema inference, data profiling, metadata and lineage capabilities.

Spark & Data Engineering:

Develop production-grade Python, Scala and PySpark solutions.
Establish coding and testing standards.
Conduct PR/code reviews.
Debug production incidents.
Optimize Spark workloads, clusters and pipelines.
Implement performance and cost optimization strategies.
Databricks & AzureAzure Databricks
Delta LakeUnity Catalog
Databricks Jobs & Workflows
Cluster and pool management
Model ServingADLS Gen2Azure Data Factory
Azure Event HubsAzure Monitor
Log AnalyticsCI/CD & Infrastructure as Code
Implement CI/CD for Databricks and ADF using Azure DevOps.
Work with Databricks Asset Bundles.
Implement Infrastructure as Code using Terraform.
Automate data pipeline testing and deployments.
Establish observability and operational monitoring.🤖 AI / LLM ENGINEERINGA major component of this role is designing and delivering production-grade AI/LLM solutions.
You will work on:
AI-augmented ingestion.
AI-assisted canonical mapping.
Auto-generated bridge documentation.
DML and canonical table generation.
Source-to-canonical mapping with human review gates.
AI-driven data quality.
Data drift and schema drift detection.
Volume anomaly detection.
Automated reconciliation.
Synthetic privacy-preserving test data generation.
Enterprise semantic layers.
Knowledge graphs.
GPT-powered conversational interfaces.
Text-to-SQL.Semantic-layer retrieval.
Row-level and column-level security.🧠 LLM / GENAI MUST-HAVESCandidates must have 3+ years of hands-on experience designing and shipping LLM-based systems in production, including:
RAG pipelines
Agentic AI workflows
Tool-calling workflows
Chunking strategies
Embedding strategies
Vector retrieval
Hybrid retrieval
Prompt engineeringLLM evaluation
Golden datasets
Regression test suites
Accuracy tracking
Hallucination monitoring
Human-in-the-loop feedback
Hands-on experience with at least one of:
LangChainLlamaIndexLangGraphProvider/platform experience:
Azure OpenAIOpenAIDatabricks Model Serving🔐 SECURITY, GOVERNANCE & DATA PROTECTIONStrong enterprise security and governance experience is required across:
Unity Catalog
Data lineage
Access controlPII standards
Entra IDManaged IdentitiesRBACPOSIX ACLsAzure Key Vault
Private Endpoints
Data privacy and protection
Enterprise data governance
The candidate will also participate in:
Architecture Review BoardsAI Governance Forums
Technical design reviews

Enterprise governance discussions👨 💻 MUST-HAVE TECHNICAL SKILLS✅ 12–18 years of total experience in Data Engineering / Data Platform delivery✅ Expert-level Python✅ Expert-level Scala✅ Expert-level PySpark / Spark✅ Strong SQL✅ Data modelling — dimensional & normalized✅ Data contracts & schema design✅ Azure Databricks✅ Delta Lake✅ Unity Catalog✅ Azure ADLS Gen2✅ Azure Data Factory✅ Azure Event Hubs✅ Kafka / Spark Structured Streaming✅ Databricks Jobs & Workflows✅ Spark performance tuning✅ Azure DevOps✅ Terraform✅ Databricks Asset Bundles✅ CI/CD✅ Metadata-driven frameworks✅ Data profiling & lineage✅ Enterprise Lakehouse / Medallion Architecture✅ Azure security & governance✅ 3+ years production LLM/GenAI engineering✅ RAG / Agentic AI / Tool Calling✅ LangChain / LlamaIndex / LangGraph✅ Azure OpenAI / OpenAI / Databricks Model Serving⭐ STRONGLY PREFERREDKnowledge graphs & ontologiesRDF / SPARQLNeo4jGraph modelling over Lakehouse Text-to-SQLEnterprise semantic layersML-based anomaly detection Time-series / transactional financial data Financial Services / Insurance domain Finance CloseGL / General Ledger SubledgerReconciliationActuarial dataLLMOps / MLOpsModel & prompt versioningLLM cost governanceAI observabilitydbt Great Expectations WorkdayWorkday Prism Accounting Center🏆 CERTIFICATIONS — PLUSDatabricks Data Engineer Professional Azure DP-203 Azure DP-700 Azure AZ-305🚨 IDEAL CANDIDATE PROFILEWe are looking for a hands-on Lead Data Engineer / Data Architect / Principal Data Engineer who can operate at both architecture and engineering levels. You should be comfortable designing the architecture, making technical decisions, writing production code, reviewing code, troubleshooting Spark workloads, mentoring engineers, and presenting architecture decisions to senior technical and business stakeholders.⚠️ This is NOT a pure Data Architect, Solution Architect, Engineering Manager, or AI Strategy role.⚠️ Recent hands-on production coding experience is mandatory.⚠️ US CITIZENS ONLY.#Nazztec #Nazztec Hiring #Join Nazztec #Hiring #UrgentHiring #LeadDataEngineer #Data Engineer #SeniorDataEngineer #Principal DataEngineer #Data Architect #Data Engineering #Data Platform #Data PlatformEngineer #BigData #BigDataEngineering #Azure #Microsoft Azure #Azure Databricks #Databricks #DeltaLake #Unity Catalog #AzureDataFactory #ADF #ADLS #ADLSGen2 #AzureEventHubs #Kafka #Spark #ApacheSpark #PySpark #Scala #Python #SQL #DataLake #Lakehouse #Lakehouse Architecture #Medallion Architecture #BronzeSilverGold #Data Architecture #Data Modeling #Data EngineeringJobs #AzureJobs #Databricks Jobs #AI #Artificial Intelligence #GenerativeAI #GenAI #LLM #LLMEngineering #LLMEngineer #LLMJobs #RAG #RAGPipeline #Retrieval AugmentedGeneration #AgenticAI #Agentic Workflows #Tool Calling #LangChain #LlamaIndex #LangGraph #AzureOpenAI #OpenAI #Databricks ModelServing #Prompt Engineering #Vector Database #VectorSearch #HybridSearch #Embeddings #Semantic Search #TextToSQL #Semantic Layer #Knowledge Graph #Neo4j #RDF #SPARQL #MLOps #LLMOps #AIEngineering #AIEngineer #Machine Learning #Anomaly Detection #Data Quality #Data Governance #Data Security #Data Lineage #Data Catalog #Metadata #Metadata Driven #Data Contracts #Schema Evolution #Data Profiling #Data Observability #DataOps #DevOps #AzureDevOps #Terraform #Infrastructure AsCode #IaC #CICD #Continuous Integration #Continuous Deployment #SparkTuning #Performance Tuning #CloudDataEngineering #Cloud Architecture #Enterprise Data #Enterprise Architecture #Financial Services #Insurance #Insurance Technology #FinTech #MetLife #CaryNC #North CarolinaJobs #USJobs #USAJobs #USCitizen #USCitizenJobs #TechJobs #ITJobs #Technology Jobs #Data Science #CloudJobs #Data Professionals #TechHiring #ITHiring #DataHiring #AIJobs #RemoteJobs #HybridJobs #OnsiteJobs

Apply now

Level

LeadL6

Location

Cary, NC

Occupation

Data Warehousing Specialists

Industry

Computer Systems Design Services

Posted

today

To get sharper similar jobs, create your profile using the link below.

Create profile