Data Engineer – PySpark, Hadoop, Hive & Big Data Pipeline Development

Job Summary Synechron is seeking a Data Engineer with strong expertise in PySpark and the Cloudera Data Platform (CDP) to design, develop, and maintain scalable data pipelines. The role is responsible for building reliable data ingestion and transformation processes, working with large-scale datasets, and supporting distributed data-processing environments. The Data Engineer will contribute to business objectives by improving data availability, quality, integrity, processing efficiency, and operational reliability. The successful candidate will collaborate with cross-functional teams to deliver data-driven solutions that support business and technology requirements. Software Requirements Required Software Skills PySpark: Strong hands-on experience developing, optimizing, and maintaining production-grade data pipelines. Cloudera Data Platform (CDP): Strong practical experience working with CDP environments and related data-processing capabilities. Apache Spark: Hands-on experience with Spark processing, job execution, performance tuning, and troubleshooting. Hadoop: Experience working with Hadoop-based distributed data ecosystems. Hive: Experience developing and optimizing Hive queries, tables, and data-processing workflows. HDFS: Experience managing and processing data stored in the Hadoop Distributed File System. ETL tools and processes: Practical experience designing data ingestion, transformation, validation, and loading workflows. Data warehousing technologies: Experience working with data warehouse concepts, structures, and processing patterns. Cloud and distributed data environments: Experience with cloud-based or distributed data-processing platforms. Version experience: Experience with the versions of PySpark, Spark, Hadoop, Hive, HDFS, CDP, and associated tools adopted by the assigned Synechron project; ability to work with current supported releases and understand version-related compatibility considerations. Preferred Software Skills Experience with cloud-native data platforms and managed distributed-processing services. Familiarity with workflow orchestration, data- pipeline monitoring, scheduling, and alerting tools. Experience with source-control, code-review, build, deployment, and automated testing tools. Familiarity with data-quality, metadata-management, lineage, and observability tools. Experience working with containerized or infrastructure-as-code environments. Exposure to streaming, real-time processing, or event-driven data platforms. Equivalent tools may be considered where they provide comparable capabilities. Overall Responsibilities Design and develop scalable data pipelines using PySpark and Cloudera Data Platform (CDP). Build, optimize, and maintain data ingestion and transformation processes for large-scale datasets. Develop reusable, maintainable, and testable data-processing components. Implement data validation and quality checks to support data accuracy, consistency, integrity, and availability.