Senior Data Engineer (Streaming + AI Systems)

We’re partnered with a capital-backed operator in the industrial / infrastructure space that’s investing heavily in building a modern data platform from the ground up. They’re hiring a Senior Data Engineer to lead the design and evolution of real-time data systems and scalable architecture supporting both operational intelligence and predictive modelling. This is a high-ownership role at the intersection of data engineering, applied statistics, and production AI. You’ll operate as a senior individual contributor with end-to-end responsibility across pipelines, platform design, and ML enablement. Why This Role Is Compelling True ownership of the data platform: architecture, pipelines, and production ML systems. Direct impact on operational performance and financial strategy, not just analytics. Senior-level autonomy with influence over technical direction and tooling choices. Lean, execution-focused team where senior engineers build and ship complete systems. Long-term stability paired with a builder environment and modern stack. Scope and Responsibilities Architect and operate real-time and batch data pipelines (Kafka, Spark, AWS ecosystem). Lead the evolution of a scalable lakehouse architecture for analytics and ML workloads. Design and productionize end-to-end ML pipelines, including feature engineering and real-time inference. Apply statistical methods (time-series, regression, anomaly detection) to high-impact business problems. Partner with stakeholders across operations, finance, and product to translate complex requirements into robust systems. Own infrastructure design using IaC (Terraform or equivalent) to ensure reproducibility and scale. Drive performance, reliability, and observability across data systems. Evaluate and introduce new tools and patterns across data engineering and applied AI. Requirements Tech Environment AWS (Glue, Lambda, ECS, S3, and related services) Apache Spark + lakehouse frameworks (Delta or equivalent) Kafka / streaming architectures Terraform (Infrastructure as Code) What They’re Looking For Proven experience building and scaling data platforms in production environments. Deep expertise in streaming systems and distributed data processing. Strong system design skills with the ability to operate at both architecture and implementation levels. Experience owning ML/data pipelines end-to-end, including production deployment. Solid grounding in statistical modelling and real-world applications. A bias toward ownership, pragmatism, and building systems that actually get used.