Senior Lead Data Engineer, Data Platform

About the role

Data Platform Engineering team at S&P Global builds and operates the enterprise Databricks‑on‑AWS lakehouse that delivers governed data for analytics. The Senior Lead Data Engineer drives migration of pipelines to Databricks, oversees onboarding of new data assets, and leads a small offshore engineering team ensuring reliable, secure data workflows across S3, Glue, and Lake Formation. Based in Hyderabad or Chennai with on‑site work model; relocation assistance available.

What you’ll do

• Migrate existing pipelines to Databricks following established patterns
• Support and troubleshoot production data pipelines post‑migration
• Lead and mentor a small offshore engineering team
• Guide data asset onboarding across Bronze, Silver, Gold layers
• Implement data quality, metadata, and lineage controls
• Develop CI/CD pipelines and automation for deployments
• Monitor performance using CloudWatch and optimize workloads
• Document processes and communicate status to stakeholders

What you’ll bring

• 8+ years data engineering experience
• Proficiency with Spark, SQL, and Python
• Hands‑on Databricks on AWS
• Experience with AWS services: S3, Glue, IAM, Lake Formation
• Knowledge of data governance and metadata management
• Familiarity with CI/CD and infrastructure as code
• Strong problem‑solving and communication skills
• Experience leading engineering teams

Nice to have

• Enterprise data onboarding experience
• Familiarity with Delta Lake and Apache Iceberg
• Experience with AWS Lake Formation and Glue Data Catalog
• Knowledge of data mastering and governance frameworks
• Proficiency in PySpark
• Experience with IaC tools like Terraform or CloudFormation
• Databricks certification
• AWS certification

Skills

Benefits

• Health & wellness benefits
• Generous paid time off
• Continuous learning programs
• Retirement and financial planning options
• Family friendly perks