Job Summary
We are seeking an experienced Data Engineer with 7–8 years of hands-on experience in Python, PySpark, SQL, and Databricks. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines, building robust ETL/ELT processes, and managing large-scale data processing solutions to support business intelligence, analytics, and data-driven decision-making.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Python, PySpark, SQL, and Databricks.
- Build and optimize ETL/ELT workflows for batch and large-scale data processing.
- Develop data transformation logic using PySpark and advanced SQL.
- Work with Databricks notebooks, jobs, clusters, and Delta Lake tables.
- Ingest, cleanse, transform, validate, and integrate data from multiple sources.
- Optimize data processing performance, query execution, and resource utilization.
- Implement data quality checks, error handling, logging, and monitoring.
- Collaborate with data architects, analysts, data scientists, and business stakeholders to understand data requirements.
- Develop and maintain data models and curated datasets for reporting and analytics.
- Implement data governance, security, access controls, and data integrity standards.
- Troubleshoot pipeline failures and resolve data processing issues.
- Participate in code reviews, technical design discussions, testing, and Agile ceremonies.
Required Qualifications
- 7–8 years of experience in data engineering, data development, or related roles.
- Strong hands-on programming experience with Python and PySpark.
- Advanced SQL skills, including complex queries, joins, aggregations, CTEs, and performance tuning.
- Hands-on experience with Databricks for data engineering and distributed data processing.
- Strong understanding of ETL/ELT concepts and data pipeline architecture.
- Experience working with large datasets and distributed computing frameworks.
- Knowledge of data warehousing, dimensional modeling, and data transformation techniques.
- Experience with Delta Lake, Spark SQL, and Databricks Jobs.
- Familiarity with Git, version control, testing, and deployment practices.
- Strong analytical, troubleshooting, and problem-solving skills.