Data Engineer – Azure Databricks
Location: Seattle, WA (Preferred)
Also Open To: St. Louis, MO | Dallas/Plano, TX | Charleston, SC | Ridley Park, PA
Duration: 6+ Months (Extension Possible)
Employment Type: W2 Only
Work Authorization: US Citizen or Green Card Holders Only
Position Overview
We are seeking an experienced Data Engineer with strong Azure Databricks expertise to support a large-scale enterprise data migration and modernization initiative. This role requires a well-rounded engineer with hands-on experience across the full software development lifecycle, including architecture, development, migration, deployment, production support, and ongoing optimization of cloud-based data platforms.
The ideal candidate will possess deep expertise in Python, PySpark, Azure Databricks, SQL, Azure services, CI/CD, Infrastructure as Code, and production operations. This position involves close collaboration with business stakeholders, application teams, infrastructure teams, and data consumers to deliver scalable, secure, and high-performing data solutions.
Work Breakdown
- 60% Development & Engineering
- 20% Business & Stakeholder Collaboration
- 20% Production Support & Operations
Required Technical SkillsData Engineering & Development
- Strong programming experience in Python and PySpark
- Advanced SQL skills for analytics, ETL, and data transformation workloads
- Experience designing, developing, and optimizing large-scale ETL/ELT pipelines
- Strong understanding of data modeling, data integration, and data governance principles
- Experience processing large volumes of structured and unstructured data
Azure Databricks
- Hands-on experience with Azure Databricks
- Cluster configuration, optimization, and management
- Job orchestration and scheduling
- Workspace administration and governance
- Performance tuning and workload optimization
- Monitoring and troubleshooting production workloads
Azure Cloud Services
Strong experience with:
- Azure Storage Accounts
- Azure Synapse Analytics
- Azure Key Vault
- Virtual Machine Scale Sets (VMSS)
- Azure Function Apps
- Azure Web Apps
- Azure Log Analytics Workspace
- Service Principals
- Managed Identities
- Azure Container Apps (ACA)
- Azure Container Instances
Infrastructure & DevOps
- Experience creating and managing Azure infrastructure using ARM Templates
- Strong experience with GitLab and Azure DevOps
- CI/CD pipeline development and release management
- Source control and branching strategies
- Infrastructure as Code (IaC) best practices
Security & Networking
- Experience implementing secure access using:
- Azure Active Directory
- Managed Identities
- Service Principals
- Understanding of Azure networking concepts
- Experience integrating secure networking solutions within cloud data platforms
Soft Skills
- Excellent communication and stakeholder management skills
- Strong analytical and problem-solving abilities
- Ability to work effectively in Agile/Scrum environments
- Experience collaborating with cross-functional teams and business users
Key Responsibilities
- Design, develop, and maintain end-to-end data pipelines and ETL/ELT solutions using Python and PySpark.
- Build, optimize, and support large-scale data processing workloads within Azure Databricks.
- Configure and manage Databricks clusters, job scheduling, governance controls, and monitoring frameworks.
- Develop scalable and secure cloud-based data integration solutions using Azure services.
- Collaborate with business stakeholders and application teams to gather requirements and translate them into technical solutions.
- Implement secure authentication and authorization mechanisms using Azure Active Directory, Managed Identities, and Service Principals.
- Create and maintain Infrastructure as Code (IaC) solutions using ARM Templates.
- Configure and support Azure services including Storage Accounts, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, and Container Services.
- Partner with networking and security teams to design secure cloud-based data architectures.
- Implement monitoring, alerting, logging, and cost optimization strategies across data workloads.
- Build and maintain CI/CD pipelines using GitLab and Azure DevOps.
- Support production deployments, incident management, troubleshooting, and operational activities.
- Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog grooming, and retrospectives.
- Ensure compliance with enterprise data governance, lineage, security, and regulatory requirements.
- Provide on-call and production support for critical business applications and data platforms.
- Work closely with both legacy and modernization teams to support enterprise transformation initiatives.
Ideal Candidate Profile
The ideal candidate will have:
- 8+ years of Data Engineering experience
- Strong Azure Databricks implementation experience
- Proven experience in large-scale migration and modernization projects
- Experience owning solutions from development through production deployment and support
- Strong cloud engineering, automation, and operational support background
- Ability to thrive in a highly collaborative enterprise environment
Pay: $55.00 - $60.00 per hour
Work Location: In person
