Senior Technical Manager

Job Summary

Mission:

To design, build, and maintain scalable, reliable, and secure data pipelines that enable seamless integration of diverse global data sources for Generative AI solutions. The Data Engineer ensures high-quality, timely, and accessible data for AI models by managing RAG indexes, monitoring pipeline health, and continuously onboarding new data sources. This mission supports the organisation’s goal of delivering intelligent, data-driven experiences across enterprise applications.

Key Responsibilities:

• Design, implement, and maintain robust data pipelines and ETL processes to support business and product requirements.
• Monitor the performance and reliability of data pipelines, proactively addressing issues and optimizing workflows.
• Connect and integrate diverse data sources (structured and unstructured) across global systems.
• Identify and integrate new potential data sources to enhance the breadth and quality of available data.
• Build and maintain Retrieval-Augmented Generation (RAG) indexes for efficient data retrieval and model performance.
• Optimise indexing strategies for large-scale, multi-domain datasets.
• Ensure data quality, consistency, and compliance with enterprise standards.
• Implement monitoring and alerting for data pipeline health and SLA adherence.
• Work closely with AI/ML engineers, data scientists, and product teams to deliver end-to-end solutions.
• Collaborate with cross-functional teams to align data engineering efforts with project roadmaps and timelines.

Key Responsibilities

Mission:

To design, build, and maintain scalable, reliable, and secure data pipelines that enable seamless integration of diverse global data sources for Generative AI solutions. The Data Engineer ensures high-quality, timely, and accessible data for AI models by managing RAG indexes, monitoring pipeline health, and continuously onboarding new data sources. This mission supports the organisation’s goal of delivering intelligent, data-driven experiences across enterprise applications.

Key Responsibilities:

• Design, implement, and maintain robust data pipelines and ETL processes to support business and product requirements.
• Monitor the performance and reliability of data pipelines, proactively addressing issues and optimizing workflows.
• Connect and integrate diverse data sources (structured and unstructured) across global systems.
• Identify and integrate new potential data sources to enhance the breadth and quality of available data.
• Build and maintain Retrieval-Augmented Generation (RAG) indexes for efficient data retrieval and model performance.
• Optimise indexing strategies for large-scale, multi-domain datasets.
• Ensure data quality, consistency, and compliance with enterprise standards.
• Implement monitoring and alerting for data pipeline health and SLA adherence.
• Work closely with AI/ML engineers, data scientists, and product teams to deliver end-to-end solutions.
• Collaborate with cross-functional teams to align data engineering efforts with project roadmaps and timelines.

Skill Requirements

Qualification and Experience

• Bachelor’s or Master’s degree in computer science engineering with 12+ years of experience
• Proficiency in programming languages: Python, PySpark, and SQL.
• Experience with data engineering tools, including Azure DataBricks, Azure DevOps, and VS Code.
• Strong understanding of working with REST APIs for data integration and automation.
• Hands-on experience with RAG architecture and vector databases
• Familiarity with Generative AI concepts and LLM integration.
• Familiarity with cloud-based data platforms and scalable data architecture.

Other Requirements

Personality requirement:

Ability to adapt to new working environments and technologies is crucial, as are your outstanding written and verbal communication skills. As a team player, you'll maintain transparent communication, providing daily updates within the team and towards stakeholders. Your customer-focused approach, coupled with an adaptable and flexible mindset, will be key in handling the varied work environment. A collaborative, good-natured, and helpful attitude, along with a transparent mindset in all situations, sets the tone for success. Your ability to handle critical and urgent situations effectively and calmly rounds out the qualities we value