Experience: 5–8 Years

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, SQL, and Delta Lake.
  
• Build and optimize batch and real-time data pipelines on AWS using services such as S3, Lambda, Glue, EMR, Redshift, and related cloud-native components.
  
• Develop, implement, and support Delta Live Tables (DLT) pipelines for reliable, high-quality, and auditable data processing.
  
• Work with Unity Catalog to implement data governance, access control, lineage, and security best practices.
  
• Develop and optimize ETL/ELT pipelines with a focus on performance tuning, scalability, reliability, and cost efficiency.
  
• Implement infrastructure and platform automation using Terraform and Infrastructure as Code (IaC) principles.
  
• Build and maintain CI/CD pipelines using Jenkins, Git, and related DevOps tools to enable automated deployments and testing.
  
• Support deployment, monitoring, and maintenance of Databricks workloads across development, testing, and production environments.
  
• Collaborate with architects, product owners, data scientists, and business stakeholders to understand requirements and deliver high-quality solutions.
  
• Participate in cloud migration, modernization, and data platform transformation initiatives on AWS.
  
• Contribute to data quality, observability, logging, monitoring, and alerting frameworks to ensure platform reliability.
  
• Work with APIs, data integration frameworks, and external systems to enable seamless data exchange across platforms.
  
• Troubleshoot production issues, perform root cause analysis, and drive continuous improvement initiatives.
  
• Follow software engineering best practices, participate in code reviews, and ensure adherence to coding, security, and operational standards.
  
• Work effectively within Agile teams, contributing to sprint planning, estimation, development, and release activities.
  
• Collaborate with global teams and support business-critical deliveries across different time zones when required.
  

Minimum Qualifications:

• Deep hands‑on experience with Databricks (PySpark, SQL, Delta Lake) in large‑scale production environments.
  
• Proven experience designing and leading end‑to‑end data pipelines on Databricks.
  
• Strong hands‑on experience with Delta Live Tables (DLT) for governed data processing.
  
• Advanced experience with Unity Catalog for data governance, access control, lineage, and metadata.
  
• Expertise in Spark performance tuning, scalability, and cost optimization.
  
• Experience managing Databricks workflows, monitoring, and production operations.
  
• Strong knowledge of cloud‑based data platforms (AWS preferred).
  
• Proven ability to mentor engineers and lead technical delivery.
  

Preferred Qualifications:

• Experience building and operating AI agents for workflow automation and decision support.
  
• Strong understanding of data observability, platform reliability, and FinOps practices in Databricks.
  
• Experience working in regulated or financial services domains.
  
• Exposure to enterprise‑scale AI governance, compliance, and audit requirements.
  
• Ability to influence architecture decisions and align stakeholders.
  

• Excellent communication skills with both technical and non‑technical audiences.