Data Engineer Pyspark

The Applications Development Intermediate Programmer Analyst is an intermediate level position responsible for participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team. The overall objective of this role is to contribute to applications systems analysis and programming activities. This is a hands-on development role responsible for building, testing, optimizing, and supporting batch and real-time data-processing solutions using technologies such as Apache Spark, Databricks, Kafka, Scala, Python, and SQL. The successful candidate will work within a data-engineering team to deliver reliable and maintainable data pipelines, troubleshoot performance and production issues, and follow established architecture, coding, testing, security, governance, and operational standards under the guidance of senior developers and technical leads. Responsibilities: Develop, test, enhance, and support batch and real-time data pipelines using Apache Spark, Databricks, Kafka, Scala, Python, and SQL. Write clean, reusable code and participate in code reviews, unit testing, integration testing, data validation, and deployment activities. Analyze Spark execution plans, DAGs, metrics, and logs to troubleshoot failures and improve processing performance. Build and maintain Iceberg or Delta Lake solutions using established Lakehouse and medallion architecture patterns. Collaborate with technical leads, developers, analysts, and support teams to deliver assigned work and resolve production issues while following established controls and standards. Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems Apply fundamental knowledge of programming languages for design specifications. Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging Serve as advisor or coach to new or lower-level analysts Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents Has the ability to operate with a limited level of direct supervision. Can exercise independence of judgement and autonomy. Acts as SME to senior stakeholders and /or other team members.