Company: Tech Scalerz
Location: Remote (United States)
Job Type: Full-Time
Work Authorization: Applicants must be authorized to work in the United States. No visa sponsorship is available for this position now or in the future.
About the Role
Our client is seeking an experienced Senior Data Engineer to design, build, and optimize large-scale cloud data platforms using Azure Databricks and Apache Spark. You will work on modern Lakehouse architectures, high-volume data pipelines, and real-time data processing solutions supporting enterprise analytics and AI initiatives.
This is an excellent opportunity for an experienced Data Engineer who enjoys solving complex data engineering challenges in a cloud-first environment.
Required Qualifications
- 8+ years of experience in Data Engineering
- Strong hands-on experience with one or more cloud platforms:
- Microsoft Azure (preferred)
- AWS
- Google Cloud Platform (GCP)
- Expert knowledge of Azure Databricks
- Strong experience with Apache Spark (PySpark and/or Scala)
- Advanced SQL skills
- Experience designing and developing ETL/ELT pipelines
- Experience working with large-scale distributed data processing
- Experience with Delta Lake and Lakehouse architecture
- Experience with Unity Catalog
- Experience with Auto Loader
- Experience implementing Medallion Architecture
- Experience with Spark Structured Streaming
- Strong understanding of Data Warehousing and Lakehouse concepts
- Experience tuning Databricks clusters and Spark jobs for performance
- Experience troubleshooting production data pipelines
- Experience with Azure Kubernetes Service (AKS)
- Experience supporting Azure DevOps pipelines
Preferred Qualifications
- Experience with Serverless Databricks
- Experience with Databricks Job Workflows
- Experience working in Agile environments
- Strong problem-solving and debugging skills
- Excellent communication skills
Responsibilities
- Design, develop, and maintain enterprise-scale ETL and ELT pipelines
- Build scalable batch and streaming data pipelines using Azure Databricks
- Develop high-performance Spark applications using PySpark and Scala
- Design and implement Lakehouse solutions using Delta Lake
- Optimize Spark SQL, clusters, notebooks, and workloads
- Configure and manage Unity Catalog and data governance
- Monitor, troubleshoot, and optimize production data pipelines
- Work closely with architects, data scientists, and business stakeholders
- Improve performance, scalability, reliability, and cost efficiency of data platforms
- Support CI/CD and deployment processes using Azure DevOps
- Participate in architecture discussions and technical design reviews
Technologies
- Azure
- Azure Databricks
- Apache Spark
- PySpark
- Scala
- SQL
- Delta Lake
- Unity Catalog
- Auto Loader
- Structured Streaming
- Azure DevOps
- AKS
- ETL
- ELT
- Data Warehousing
- Lakehouse Architecture
- Medallion Architecture
Benefits
- 100% Remote
- Competitive salary
- Opportunity to work on modern cloud data platforms
- Health Benefits
- Collaborative engineering team
- Long-term career growth
Apply
Please apply through Indeed or send your resume to:
[email protected]
Pay: $125,000.00 - $250,000.00 per year
Work Location: Remote
