Lead Software Engineer (Databricks Platform Engineer)

Summary:
 

In this role, you will lead the design, engineering, and evolution of the enterprise Databricks platform as a strategic capability that enables analytics, data products, machine learning, and AI solutions across the organization. You will develop reusable platform services, automation frameworks, reference architectures, and self-service capabilities that accelerate delivery while aligning with enterprise architecture, governance, and engineering standards. You will collaborate with cross-functional teams to enhance developer experience, drive platform adoption, evaluate emerging technologies, and deliver scalable, secure, and cost-effective healthcare data and AI solutions.

Responsibilities:

• Lead the design, implementation, and continuous evolution of the enterprise Databricks platform and supporting architecture.
• Develop reusable platform services, reference architectures, engineering standards, and implementation patterns for analytics, data products, streaming, and AI workloads.
• Build Infrastructure-as-Code, CI/CD pipelines, and automation frameworks to improve platform scalability, reliability, and operational efficiency.
• Create self-service capabilities, reusable templates, and engineering accelerators that simplify platform onboarding and deployment.
• Optimize platform performance, scalability, security, governance, and cost efficiency across cloud environments.
• Enable machine learning and AI platform capabilities by implementing patterns for MLflow, Model Registry, Feature Store, Vector Search, and other Databricks technologies.
• Partner with Enterprise Architecture, Data Engineering, Product Management, Cloud Engineering, Data Science, and client stakeholders to deliver enterprise platform capabilities.
• Evaluate emerging Databricks features and cloud technologies, lead proof-of-concepts, and recommend platform enhancements that support business objectives.
• Provide technical leadership, mentoring, and engineering guidance while promoting Platform-as-a-Product principles and engineering best practices.
• Drive continuous improvement through automation, innovation, and adoption of enterprise platform standards.

Qualifications:

• Relevant degree preferred. Advanced degree in Computer Science, Engineering, Information Systems, or a related field preferred.
• 7 or more years of relevant experience required.
• 3 or more years of hands-on experience designing, implementing, and supporting Databricks platform solutions required.
• Strong experience with Azure cloud services, cloud-native architectures, and enterprise platform engineering required.
• Experience with Terraform, Infrastructure-as-Code, and CI/CD pipelines, and infrastructure automation.
• Strong knowledge of Delta Lake, Unity Catalog, Apache Spark, and lakehouse architectures.
• Experience developing reusable platform services, automation frameworks, self-service capabilities, and engineering accelerators.
• Experience supporting machine learning and AI platform capabilities, including MLflow, Model Registry, Feature Store, Vector Search, or MLOps practices preferred.
• Databricks certification, experience with streaming and event-driven architectures, enterprise-scale data platforms, or healthcare industry experience preferred.