Azure Databricks Engineer - Assistant Manager

Job Title : Azure Databricks, Scala, Snowflake & Azure DevOps Engineer Who We Are Looking For We are seeking an experienced Azure Data Engineer/Developer with strong hands-on expertise in Azure Databricks, Apache Spark, Scala, Snowflake, SQL, and Azure DevOps. The successful candidate will bring strong data-engineering expertise, experience with automated CI/CD pipelines and cloud migration, and the ability to deliver effectively within Agile teams. Why This Role Is Important to Us This role is central to building and modernizing scalable cloud-based data solutions. The engineer will design, develop, test, deploy, and support data platforms that improve processing efficiency, reliability, automation, and enterprise data delivery. What You Will Be Responsible For Data Engineering and Platform Development Design and develop scalable, secure, and maintainable data-engineering solutions on Microsoft Azure. Build and support high-volume data-processing workloads using Azure Databricks, Apache Spark, and Scala. Develop reliable ETL/ELT pipelines for data ingestion, transformation, processing, and delivery. Design and implement Snowflake data-processing and data-warehouse solutions. Develop and optimize SQL queries, stored procedures, functions, and transformation logic. Integrate Databricks with Snowflake and other enterprise data sources. Create reusable, modular, and maintainable Scala and Spark components. Develop batch and streaming data pipelines using Lakeflow Connect, Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Structured Streaming. Apply Unity Catalog capabilities for data governance, access control, lineage, catalog and schema management, and secure data sharing. Design and optimize Delta Lake solutions using ACID transactions, schema enforcement and evolution, time travel, change data feed, and table-maintenance practices. Build incremental and event-driven processing solutions using streaming tables, materialized views, Azure Event Hubs, or Kafka. Develop and tune Databricks SQL workloads using SQL warehouses, query profiling, Photon acceleration, and cost-optimization practices. Configure serverless computing, cluster policies, Databricks Runtime versions, workload isolation, and autoscaling based on workload requirements. Automate Databricks deployments using Declarative Automation Bundles, the Databricks CLI, REST APIs, service principals, and OAuth-based authentication. Monitor pipeline health, data quality, lineage, streaming backlogs, and resource usage using event logs, system tables, query history, and operational alerts. Implement data-quality controls, manage schema drift and quarantined records, and build resilient pipelines with checkpoint recovery, idempotent processing, and restart capabilities. Apply enterprise security practices, including managed identities, private networking, secure secret management, and Azure Key Vault integration.