Senior Data Engineer
S&P Global · Hyderabad, Telangana, India
Free account: apply link, HR contact and an AI-written cold email
S&P Global · Hyderabad, Telangana, India
Free account: apply link, HR contact and an AI-written cold email
About the Role: Grade Level (for internal use): 11Key Responsibilities Data Pipeline Transition and Platform Delivery Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform. Provide hands-on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns. Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets. Ensure pipelines are designed and operated in a way that supports long-term platform consistency, reliability, observability, and ease of maintenance. Support the design and operation of cloud-native data pipelines using AWS services such as:Amazon S3 for durable data lake storage AWS Glue for data integration, cataloging, and processing AWS Lambda for event-driven processing Amazon Kinesis for streaming use cases AWS Lake Formation for governed data lake controls AWS IAM for secure access control Promote the use of encryption, environment-level controls, access policies, and platform guardrails to support secure access to platform resources and data products. Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns. AI-Assisted Engineering and Context Engineering Use AI-assisted engineering tools such as Claude, GitHub CoPilot, LLMs, and similar platforms to accelerate data engineering delivery while maintaining enterprise quality and security standards. Apply AI tooling to support common engineering activities, including:Code generation Code refactoring SQL and PySpark optimization Unit test generation Documentation creation Pipeline analysis Troubleshooting and debugging Data quality rule suggestions Runbook and operational support documentation Apply prompt engineering and context engineering practices to improve the accuracy, relevance, and reliability of AI-generated outputs. Structure technical context for AI-assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements. Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation. Review, validate, and refine AI-generated code, recommendations, documentation, and analysis before applying them to production-grade engineering work. Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review. Data Onboarding and Asset-Agnostic Enablement
Want to reach S&P Global's HR directly?
NextKnock finds the hiring contact and writes a personal cold email from your resume. You send it from your own Gmail.