Agentic AI Databricks

Agentic AI Databricks

Raritan NJ Onsite

Experience 7 years Location Flexible Hybrid Function Data Engineering AIML Engineering Cloud Data Platforms

Role Summary

We are looking for an experienced Databricks Azure professional with strong expertise in cloudscale data engineering lakehouse architecture Agentic AI and Databricks Agent Bricks The ideal candidate will have 7 or more years of experience designing building and operationalizing enterprise data and AI platforms on Azure Databricks with handson exposure to generative AI solutions retrievalaugmented generation vector search AI agents model serving MLflow Unity Catalog and productiongrade governance practices

Roles Responsibilities

Design develop and maintain scalable data engineering and AIML solutions on Azure Databricks using Delta Lake Unity Catalog Databricks Workflows Databricks SQL and MLflow

Build and deploy Agentic AI applications using Databricks Agent Bricks Mosaic AI Agent Framework Knowledge Assistants Supervisor Agents toolcalling agents and multiagent orchestration patterns

Develop retrievalaugmented generation solutions using Databricks Vector Search embeddings model serving endpoints foundation models and governed enterprise knowledge sources

Architect lakehouse solutions across ingestion transformation orchestration governance security performance optimization and monitoring layers

Integrate Azurenative services such as Azure Data Lake Storage Azure Data Factory Azure Key Vault Azure DevOps Microsoft Entra ID Azure Functions and Azure OpenAI where applicable

Implement robust CICD pipelines for Databricks notebooks jobs bundles ML models and AI agent deployments using Azure DevOps or equivalent tools

Establish data and AI governance standards using Unity Catalog access controls lineage audit logging model registry and secure serving patterns

Optimize Spark workloads Delta tables SQL warehouses serverless compute cluster policies job performance and cost utilization

Collaborate with data engineers ML engineers architects product teams and business stakeholders to convert business requirements into reliable data and AI capabilities

Define best practices reusable frameworks coding standards deployment patterns and operational runbooks for Databricks and Agentic AI platforms

Support production operations incident resolution root cause analysis monitoring SLA management and continuous improvement for data and AI workloads

Mentor junior engineers and contribute to knowledge sharing technical design reviews proofofconcepts and platform modernization initiatives

MustHave Skills

7 years of overall experience in data engineering analytics engineering AIML engineering or cloud data platform engineering

Strong handson experience with Azure Databricks Apache Spark PySpark Spark SQL Delta Lake Delta Live Tables Databricks Workflows Databricks SQL and Lakehouse architecture

Practical experience with Databricks Agent Bricks Agentic AI patterns AI agents tool calling multiagent systems Knowledge Assistants Supervisor Agents and governed AI application development

Handson experience building generative AI solutions such as RAG applications LLM chains prompt engineering workflows vector search implementations embeddings pipelines and model serving solutions

Strong understanding of MLflow for tracking model lifecycle management tracing evaluation monitoring and deployment of ML and GenAI workloads

Experience with Unity Catalog data governance access control catalogschematable permissions lineage auditability and secure AIdata sharing patterns

Proficiency in Python SQL PySpark data modeling performance tuning API integration and productiongrade software engineering practices

Experience with Azure services including ADLS Gen2 Azure Data Factory Azure DevOps Azure Key Vault Microsoft Entra ID Azure Monitor and Azure OpenAI or comparable LLM services

Strong knowledge of CICD infrastructureascode concepts Gitbased development Databricks Asset Bundles environment promotion and release management

Ability to troubleshoot complex data pipelines Spark performance issues job failures security constraints and AI model or agent behavior in production environments

Good understanding of Responsible AI data privacy model evaluation hallucination mitigation guardrails prompt safety and enterprise AI governance

Excellent communication stakeholder management documentation and technical leadership skills

GoodtoHave Skills

Experience with LangChain LangGraph OpenAI Agents SDK LlamaIndex MCP servers AI Gateway or custom agent frameworks

Exposure to Databricks Apps Genie Foundation Model APIs Model Context Protocol integrations and enterprise chatbot or copilot implementations

Experience with streaming data pipelines using Structured Streaming Event Hu