Lead Data Architect - Azure Databricks & Advanced

Lead Data Architect – Azure Databricks & Advanced Analytics Role Overview

We are seeking an experienced Lead Data Architect with deep expertise in Azure Databricks and advanced analytics to design and deliver scalable, cloud-native data solutions. This role requires strong hands-on technical skills combined with architectural leadership to drive end-to-end analytics platforms aligned with business objectives.

The ideal candidate will bring extensive experience in big data technologies, data engineering, and Azure cloud services, with a strong focus on Databricks, SQL, and modern data lakehouse architectures.

Key Responsibilities

Lead the architecture, design, and implementation of advanced analytics solutions using Azure Databricks and Microsoft Fabric

Collaborate closely with business stakeholders and cross-functional IT teams to gather requirements and translate them into effective data solutions

Own end-to-end delivery of data platforms, ensuring alignment with business needs, architectural standards, and best practices

Design and develop scalable data pipelines and ETL / ELT processes using Azure Databricks, PySpark, and related tools

Integrate Azure Databricks with other Azure services, including Azure Data Lake, Azure Synapse, Azure Data Factory, and on-premise systems

Provide technical leadership and mentorship to data engineering teams, fostering continuous learning and innovation

Ensure comprehensive documentation of data architectures, pipelines, and data flows

Enforce best practices for code quality, data security, governance, and scalability

Stay current with evolving Databricks and Azure technologies to drive continuous improvement and innovation

Essential Technical Skills

Strong hands-on experience with Azure Databricks, including cluster management, notebook development, and Delta Lake

Proficiency in big data technologies such as Spark, Hadoop, and data processing frameworks like PySpark

Deep understanding of Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Synapse, and Azure Data Factory

Experience designing and implementing ETL / ELT pipelines, data lakes, and data warehouses

Advanced SQL skills and familiarity with NoSQL databases

Experience with CI / CD pipelines and version control systems such as Git

Strong knowledge of cloud security and governance best practices

Soft Skills

Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders

Strong problem-solving mindset with a proactive approach to identifying and resolving issues

Proven leadership and mentoring skills, with experience leading and guiding data engineering teams

Nice to Have Skills

Experience with Power BI for dashboarding and reporting

Exposure to Microsoft Fabric for analytics and integration workloads

Knowledge of Spark Streaming for real-time data processing

Familiarity with Azure Resource Manager (ARM) templates and Infrastructure as Code (IaC) practices

Experience Requirements

12+ years of demonstrated experience in developing data ingestion and transformation pipelines using Databricks, Azure Synapse notebooks, and Azure Data Factory

Strong hands-on experience with Delta Tables, Delta Lake, and Azure Data Lake Storage Gen2

Proven expertise using Auto Loader and Delta Live Tables for efficient data ingestion and transformation

Experience building and optimizing query layers using Databricks SQL

Demonstrated experience integrating Databricks with Azure Synapse, ADLS Gen2, and Power BI for end-to-end analytics solutions

Prior experience developing, optimizing, and deploying Power BI reports

Solid understanding of modern CI / CD practices, particularly for Databricks and cloud-native architectures