Senior Data Engineer

What success looks like in this role: We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data platforms and data processing pipelines. The ideal candidate will have strong expertise in Python, PySpark, AWS, DynamoDB, and OpenSearch, along with exposure to Java. This role requires hands-on experience building cloud-native big data solutions, optimizing large-scale data processing workloads, and enabling data-driven decision-making across the organization. Key Responsibilities Design, develop, and maintain large-scale data pipelines using Python and PySpark. Build and optimize batch and near real-time data processing solutions. Design scalable data architectures on AWS. Develop ETL/ELT workflows for ingesting, transforming, and publishing datasets. Work with structured and semi-structured data from multiple sources. Design and maintain NoSQL database solutions using DynamoDB. Implement search and analytics solutions using OpenSearch. Optimize data processing jobs for performance, scalability, and cost efficiency. Collaborate with product teams, architects, and software engineers to define data requirements. Monitor, troubleshoot, and resolve data platform and pipeline issues. Implement data quality, governance, security, and observability best practices. Participate in code reviews and mentor junior engineers. Required Technical Skills Data Engineering Strong expertise in Python. Extensive experience with PySpark and distributed data processing. Hands-on experience building data lakes and modern data platforms. Strong understanding of ETL/ELT frameworks and data modeling. Experience processing large datasets efficiently. AWS Cloud Expert-level knowledge of AWS services such as: EMR Glue Lambda S3 Athena Redshift ECS/EKS Step Functions EventBridge IAM CloudWatch SNS/SQS Databases Strong SQL skills and query optimization experience. Experience with NoSQL databases, particularly DynamoDB. Understanding of partitioning, indexing, and data access patterns. Search & Analytics Hands-on experience with OpenSearch / Elasticsearch. Experience designing indexes and search schemas. Knowledge of query optimization and search performance tuning. Experience implementing analytics and monitoring dashboards. Programming Strong Python development skills. Working knowledge of Java. Understanding of object-oriented programming and software engineering best practices. Preferred Qualifications Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 8+ years of experience in Data Engineering. Experience with streaming platforms such as Kafka or Kinesis. Experience with Airflow or other workflow orchestration tools. Experience with Docker and Kubernetes. Exposure to Data Lake and Lakehouse architecture. Experience in Agile/Scrum development environments. Good to Have Apache Kafka Apache Airflow Terraform Databricks Delta Lake Snowflake CI/CD pipelines Machine Learning data pipelines Key Competencies