Sr. Databricks Engineer – Job Summary
Role Overview:
- Senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities.
- Reports to the Sr. Director of Data Platform Transformations & Operations.
- Serves as a technical lead for major initiatives and accountable for production support and operational excellence.
- Focus: ~70% new development, ~30% operations/support/platform reliability.
- Primary technologies: Azure Databricks, PySpark, SQL, Delta Lake, cloud data engineering.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines, ingestion frameworks, and transformation processes.
- Implement Medallion (Bronze/Silver/Gold) architecture for reporting, analytics, AI, and self-service data.
- Develop and operationalize AI agent solutions on Databricks, integrating models, tools, and APIs.
- Build reusable frameworks, utilities, and platform components for productivity and quality.
- Develop and modernize batch, near-real-time, and streaming integration solutions.
- Serve as technical lead—solution design, technical reviews, tool recommendations, and implementation oversight.
- Partner with architects to ensure scalable, secure, governed, and cost-effective solutions.
- Mentor other engineers and promote coding, testing, and documentation standards.
- Implement data quality, validation, reconciliation, monitoring, and security controls.
- Build and maintain CI/CD pipelines, automated testing, and release processes (Azure DevOps/Git).
- Contribute to platform observability, alerting, monitoring, performance tuning, and cost optimization.
- Participate in production support, incident response, troubleshooting, and root cause analysis.
- Create and maintain runbooks, support documentation, and knowledge-sharing resources.
- Collaborate with cross-functional teams to translate requirements into platform solutions.
- Support technical discovery, estimation, planning, delivery, and evaluation of new cloud/data/AI capabilities.
Required Skills & Experience:
- Bachelor's in Computer Science, Information Systems, Engineering, Data Science, or related field; Master's preferred.
- 10+ years in data engineering, warehousing, or platform engineering.
- 5+ years designing/implementing cloud-based data platforms.
- 3+ years hands-on Azure Databricks in enterprise environments.
- Advanced SQL, Python, PySpark, and Spark optimization.
- Experience with Databricks agentic AI, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
- Deep understanding of lakehouse concepts, Medallion architecture, data modeling, and analytics data structures.
- Experience with Azure DevOps, Git, CI/CD, automated testing/deployment; Databricks Asset Bundles preferred.
- Implementation of data quality, validation, governance, and security controls.
- Strong leadership, solution design, troubleshooting, and communication skills.
- Experience in regulated environments (financial services/banking/audit-sensitive) preferred.
- Databricks Data Engineer certification preferred.
Nice to Have:
- Financial services industry experience.
- GenAI project deployment experience.
- Data governance expertise.
- Familiarity with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development.
Location/Logistics:
- Onsite in Austin, TX; local candidates only.
- Must attend onsite interview; confirm candidate's willingness and availability.
Role Overview:
- Senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities.
- Reports to the Sr. Director of Data Platform Transformations & Operations.
- Serves as a technical lead for major initiatives and accountable for production support and operational excellence.
- Focus: ~70% new development, ~30% operations/support/platform reliability.
- Primary technologies: Azure Databricks, PySpark, SQL, Delta Lake, cloud data engineering.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines, ingestion frameworks, and transformation processes.
- Implement Medallion (Bronze/Silver/Gold) architecture for reporting, analytics, AI, and self-service data.
- Develop and operationalize AI agent solutions on Databricks, integrating models, tools, and APIs.
- Build reusable frameworks, utilities, and platform components for productivity and quality.
- Develop and modernize batch, near-real-time, and streaming integration solutions.
- Serve as technical lead—solution design, technical reviews, tool recommendations, and implementation oversight.
- Partner with architects to ensure scalable, secure, governed, and cost-effective solutions.
- Mentor other engineers and promote coding, testing, and documentation standards.
- Implement data quality, validation, reconciliation, monitoring, and security controls.
- Build and maintain CI/CD pipelines, automated testing, and release processes (Azure DevOps/Git).
- Contribute to platform observability, alerting, monitoring, performance tuning, and cost optimization.
- Participate in production support, incident response, troubleshooting, and root cause analysis.
- Create and maintain runbooks, support documentation, and knowledge-sharing resources.
- Collaborate with cross-functional teams to translate requirements into platform solutions.
- Support technical discovery, estimation, planning, delivery, and evaluation of new cloud/data/AI capabilities.
Required Skills & Experience:
- Bachelor's in Computer Science, Information Systems, Engineering, Data Science, or related field; Master's preferred.
- 10+ years in data engineering, warehousing, or platform engineering.
- 5+ years designing/implementing cloud-based data platforms.
- 3+ years hands-on Azure Databricks in enterprise environments.
- Advanced SQL, Python, PySpark, and Spark optimization.
- Experience with Databricks agentic AI, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
- Deep understanding of lakehouse concepts, Medallion architecture, data modeling, and analytics data structures.
- Experience with Azure DevOps, Git, CI/CD, automated testing/deployment; Databricks Asset Bundles preferred.
- Implementation of data quality, validation, governance, and security controls.
- Strong leadership, solution design, troubleshooting, and communication skills.
- Experience in regulated environments (financial services/banking/audit-sensitive) preferred.
- Databricks Data Engineer certification preferred.
Nice to Have:
- Financial services industry experience.
- GenAI project deployment experience.
- Data governance expertise.
- Familiarity with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development.
Location/Logistics:
- Onsite in Austin, TX; local candidates only.
- Must attend onsite interview; confirm candidate's willingness and availability.