Data Engineer

Role: Senior Data Engineer Role Overview : Experienced Data Engineer to analyze, design and develop enterprise-scale data pipelines across legacy and strategic cloud data platforms. A key component of the position is analyzing existing data pipelines, particularly Omnia-based Spark pipelines, understanding the underlying business and transformation logic, and translating that logic into scalable data solutions on modern cloud platforms. The role requires strong hands-on experience with AWS, Snowflake, dbt, Airflow, PySpark, Python and advanced SQL.  Technical Expertise : Strong hands-on experience designing, developing and supporting scalable enterprise data pipelines. Strong Spark-etl experience, including the ability to analyze existing Omnia pipelines, understand their processing logic and derive new transformation and business rules. Strong working knowledge of AWS services, including S3, Glue, EMR, Lambda, Athena, Step Functions, SQS, IAM, Secrets Manager and CloudWatch, combined with hands-on experience in orchestrating data pipelines using Apache Airflow. Practical experience with Snowflake, covering data ingestion, data modelling, transformation and performance optimization. Experience using dbt to develop data models, incremental processing, reusable macros, automated tests, documentation and dependency management. Excellent SQL skills, with experience in complex data transformations, query optimization, reconciliation and understanding existing business logic. Familiarity with Git, automated testing, CI/CD, Jenkins and DevSecOps practices within a data engineering environment. Good Python programming skills for data processing, automation, and reusable framework components. Previous CBA experience, particularly with Omnia or CBA data platforms and engineering standards, is highly desirable. Key Responsibilities Conduct detailed analysis of existing end-to-end data pipelines across multiple platforms, with a particular focus on Omnia - PySpark-based processing. Review existing source code, SQL, transformation rules, mappings and data flows to understand the current business and technical logic. Derive, document, and implement new transformation logic based on the analysis of existing pipelines and business requirements. Design, develop and maintain reliable, scalable, and reusable data pipelines using AWS, Snowflake and dbt. Develop batch and, where required, event-driven data processing solutions using cloud-native and distributed data technologies. Translate existing Omnia data processing logic into strategic cloud-based data engineering solutions. Build ingestion, transformation, enrichment and data integration pipelines across data lakes, data warehouses and cloud lakehouse platforms. Develop complex and performance-optimized SQL for data transformation, reconciliation, validation and analytical processing. Analyze end-to-end data flows from source capture through transformation, enrichment and downstream consumption.