Azure Data Engineer Lead

Job Summary:

We are seeking a skilled and motivated Senior Azure Data Engineer with 8-12 years of relevant experience. Candidate will play a pivotal role in designing, implementing, and optimizing large-scale data solutions on the Microsoft Azure platform. Candidate will work closely with other cross-functional teams to ensure data solutions are scalable, secure, and high-performance. The ideal candidate will have a strong background in data engineering, deep expertise in Azure Databricks and proven leadership skills to guide and mentor the team.

Key Responsibilities:

• Architect and Design Data Solutions: Lead the design and implementation of scalable, secure and high-performing data solutions on Azure, ensuring alignment with business goals and technical requirements.
• Data Pipeline Development: Build and optimize end-to-end data pipelines to collect, process and analyze data from various sources using Azure Databricks, Azure Data Factory and other Azure data services.
• Cloud Infrastructure Management: Architect and maintain cloud data infrastructure using Azure Data Lake, Azure Synapse Analytics, Azure SQL Data Warehouse and Azure Blob Storage.
• Collaborate with Cross-functional Teams: Work closely with data architects, business analysts and developers to gather requirements, define data models and ensure smooth integration of data across systems.
• Performance Tuning and Optimization: Continuously monitor and optimize the performance of data workflows and pipelines to ensure efficient data processing.
• Data Governance & Security: Implement best practices for data governance, security and compliance, ensuring that data solutions adhere to company and regulatory standards.
• Mentor and Lead Teams: Provide technical leadership and mentorship to junior data engineers, guiding them in the best practices of data engineering and cloud technologies.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
• 8-12 years of experience in data engineering, with at least 5+ years of hands-on experience with Microsoft Azure cloud technologies.
• Strong experience with Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, Azure Data Lake and other related Azure services.
• Hands-on experience using these services: PySpark, Notebooks, Structured Streaming, Unity Catalog, Delta Live Tables, Workflows and SQL Warehouse
• In-depth knowledge of data modeling, ETL/ELT processes, data warehousing and data pipeline architecture.
• Proven experience in building and optimizing large-scale data pipelines in a cloud environment.
• Strong experience with SQL, NoSQL databases and big data processing technologies (e.g., Spark, Hadoop, Kafka).
• Hands-on experience with data transformation, data integration and data quality frameworks.
• Expertise in data security and compliance standards in a cloud-based environment.
• Ability to design and implement data governance strategies for large datasets.
• Strong leadership, communication and collaboration skills, with a proven ability to guide teams and manage complex projects.

Good to have:

• Microsoft Azure Data Engineer Certification
• Databricks Certified Data Engineer Professional
• Familiarity with GenAI Concepts
• Familiar with DevOps Practices