Position Title: Senior Databricks Migration Engineer (Software Developer III)
Location: Rosslyn, VA (Remote)| Preference for candidates local to Virginia or the East Coast)
Clearance Requirements: Must be a U.S. Citizen and eligible to obtain a Public Trust Clearance.
Position Status: Contract
Position Description:
Seneca Resources is seeking an experienced Senior Databricks Migration Engineer to support a large-scale enterprise data modernization initiative. This is an excellent opportunity for a senior-level Data Engineer or Databricks Architect who is passionate about cloud data platforms, scalable analytics, and modern Lakehouse architecture. In this role, you will lead the migration of legacy SQL Server and Azure Data Factory (ADF) solutions to a modern Databricks Lakehouse platform. You will design high-performance data pipelines, optimize enterprise-scale analytics, establish governance and security standards, and mentor engineering teams through the transition from traditional SQL development to distributed Spark-based data engineering. This position offers the opportunity to influence enterprise architecture, drive cloud modernization efforts, and implement best practices that will shape the organization's future data platform.
Key Responsibilities:
- Lead migration of legacy SQL Server stored procedures and Azure Data Factory (ADF) pipelines to Databricks Lakehouse using Delta Lake architecture.
- Design scalable Lakehouse solutions utilizing Bronze, Silver, and Gold data layers.
- Build reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
- Develop optimized dimensional models, star schemas, and Gold Layer datasets to maximize Microsoft Power BI performance.
- Optimize Databricks SQL Warehouses for DirectQuery and Import mode workloads.
- Implement advanced Databricks performance optimization techniques including Z-Ordering, Liquid Clustering, Data Skipping, Materialized Views, and file optimization.
- Establish governance standards for cluster sizing, auto-scaling, serverless SQL compute, and cost optimization.
- Create monitoring dashboards to analyze Databricks Unit (DBU) consumption and improve platform efficiency.
- Design secure enterprise data governance using Unity Catalog, Microsoft Entra ID (Azure Active Directory), RBAC, row-level security, and column-level security.
- Define Delta Lake partitioning strategies and storage optimization best practices.
- Lead technical design sessions, architecture reviews, pair programming, mentoring, and code reviews.
- Produce technical documentation, architecture diagrams, migration playbooks, and operational standards.
- Drive knowledge transfer initiatives to ensure long-term operational success for internal engineering teams.
- Collaborate with business stakeholders, architects, developers, and analytics teams to deliver scalable data solutions.
- Communicate effectively with both technical and non-technical audiences while delivering projects on schedule.
Required Experience
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline.
- 5+ years of Data Engineering, Data Platform Engineering, or Big Data development experience.
- 5+ years of experience working with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform (GCP).
- Hands-on experience designing and implementing Databricks Lakehouse solutions in enterprise environments.
- Strong expertise with Apache Spark, PySpark, Delta Lake, Databricks SQL, Delta Live Tables (DLT), and Databricks Workflows.
- Experience migrating enterprise data warehouses from SQL Server or similar relational platforms to cloud-native analytics platforms.
- Experience building scalable ETL/ELT pipelines and metadata-driven ingestion frameworks.
- Strong knowledge of dimensional modeling, star schema design, enterprise data warehousing, and incremental data loading.
- Experience optimizing Power BI datasets and Databricks SQL Warehouses for enterprise reporting.
- Experience implementing Unity Catalog, Microsoft Entra ID (Azure AD), RBAC, row-level security, and column-level security.
- Strong SQL and Python programming skills.
- Experience leading technical projects, mentoring engineers, and conducting architecture reviews.
- Excellent analytical, troubleshooting, documentation, and communication skills.
Preferred Qualifications
- Experience with Azure Data Lake Storage (ADLS).
- Experience designing enterprise-scale data governance and security architectures.
- Knowledge of distributed computing and cloud-native analytics platforms.
- Experience monitoring and optimizing Databricks cost and performance (DBU optimization).
- Familiarity with enterprise data quality frameworks and metadata-driven architecture.
- Microsoft Azure and/or Databricks certifications are highly desirable.
About Seneca Resources
At Seneca Resources, we are more than just a staffing and consulting firm—we are a trusted career partner. With offices across the U.S. and clients ranging from Fortune 500 companies to government organizations, we provide opportunities that help professionals grow their careers while making an impact. When you work with Seneca, you're choosing a company that invests in your success, celebrates your achievements, and connects you to meaningful work with leading organizations nationwide. We take the time to understand your goals and match you with roles that align with your skills and career path. Our consultants and contractors enjoy competitive pay, comprehensive health, dental, and vision coverage, 401(k) retirement plans, and the support of a dedicated team who will advocate for you every step of the way. Seneca Resources is proud to be an Equal Opportunity Employer, committed to fostering a diverse and inclusive workplace where all qualified individuals are encouraged to apply.
