Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual’s skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company’s 401(k) retirement plan. Additional disclaimer: Unless otherwise noted in the job description, the position Vaco/Highspring is filing for is occupied. Please note, however, that Vaco/Highspring is regularly asked to provide talent to other organizations. By submitting to this position, you are agreeing to be included in our talent pool for future hiring for similarly qualified positions. Submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. Further assessment of candidates beyond this initial phase within Vaco/Highspring will be otherwise assessed by recruiters and hiring managers. Vaco/Highspring does not have knowledge of the tools used by its clients in making final hiring decisions and cannot opine on their use of AI products.
Reports to: Vice President, Enterprise Data Platforms
Direct Reports: Approximately 10 Data Engineers
Peer Leaders: Director, Data Products; Director, Data Architecture
Core Platform: Databricks
Delivery Environment: Azure DevOps (ADO), Agile/Scrum
The Company is seeking a Director of Data Engineering & Platform to serve as the engineering and platform leader for the enterprise data organization. Reporting to the Vice President of Enterprise Data Platforms, this leader will own the teams, engineering practices, platform operations, and delivery processes responsible for turning the Company’s enterprise data strategy and architecture into scalable, production-ready solutions.
The Director will lead a team of approximately 10 data engineers operating across multiple Databricks environments and will be accountable for the reliability, scalability, performance, and disciplined delivery of the Company’s enterprise data platform.
This is a highly technical leadership role requiring deep Databricks experience combined with strong people leadership and delivery management capabilities. The Director will remain close enough to the technology to guide engineering decisions, challenge designs, establish standards, and troubleshoot complex platform issues while developing and leading a high-performing engineering organization.
The role will operate as a key member of the VP’s data leadership team alongside the Director of Data Products and Director of Data Architecture. Together, these leaders will connect business priorities, enterprise architecture, and engineering execution.
-
Lead, develop, and manage a team of approximately 10 data engineers responsible for building and operating the Company’s enterprise data platform.
-
Establish a strong engineering culture focused on quality, accountability, scalability, automation, and predictable delivery.
-
Set engineering standards for data pipelines, transformations, testing, deployment, monitoring, documentation, and production support.
-
Provide technical guidance and mentorship to engineers while developing future technical and team leaders.
-
Maintain sufficient technical depth to participate in design reviews, challenge engineering approaches, and help resolve complex technical issues.
-
Establish clear performance expectations and continually improve the productivity and capabilities of the engineering organization.
-
Serve as the primary engineering leader for the Company’s Databricks ecosystem.
-
Lead engineering and platform execution across multiple Databricks environments and workspaces.
-
Establish consistent standards for development, testing, promotion, security, monitoring, and production operations across environments.
-
Drive platform performance, scalability, reliability, observability, and cost optimization.
-
Partner with Data Architecture on the implementation and evolution of the Company’s lakehouse architecture.
-
Establish reusable frameworks and engineering patterns leveraging Databricks, Spark, Delta Lake, Python, and SQL.
-
Support adoption and optimization of capabilities including Unity Catalog, Delta Lake, Databricks Workflows, MLflow, and broader Databricks tooling.
Deep, production-level Databricks experience is a must-have for this position.
A major component of this role is creating a disciplined and highly effective engineering delivery organization.
-
Own the engineering team's Agile/Scrum delivery process.
-
Serve as a strong Scrum leader, driving sprint planning, backlog refinement, daily stand-ups, retrospectives, capacity planning, and delivery commitments.
-
Establish clear sprint goals, engineering priorities, and measurable delivery expectations.
-
Partner with Data Products to translate product priorities and business requirements into executable engineering work.
-
Manage engineering capacity across platform initiatives, new development, technical debt, production support, and strategic projects.
-
Identify and remove blockers that impact engineering velocity.
-
Create visibility into engineering progress, dependencies, risks, and delivery commitments.
-
Drive continuous improvement through sprint metrics, retrospectives, and engineering performance data.
-
Own and mature the engineering team's use of Azure DevOps (ADO) for planning, execution, and delivery management.
-
Establish standards for Epics, Features, User Stories, Tasks, Bugs, and engineering backlogs.
-
Ensure engineering work is properly defined, estimated, prioritized, assigned, and tracked.
-
Build visibility into sprint velocity, capacity, backlog health, delivery risks, and dependencies.
-
Leverage ADO to improve transparency and communication across Engineering, Data Products, Data Architecture, and executive leadership.
-
Partner with engineering teams on CI/CD and automated deployment practices.
Strong hands-on Azure DevOps and Agile/Scrum leadership experience is required.
-
Lead the design, development, and operation of scalable pipelines ingesting and transforming clinical, financial, operational, and enterprise data.
-
Establish modern engineering standards for ETL/ELT, batch and near-real-time processing, orchestration, testing, observability, and code promotion.
-
Ensure pipelines meet established standards for performance, reliability, data quality, and availability.
-
Build reusable engineering frameworks that accelerate the integration of new data sources.
-
Ensure engineering capabilities can scale to support the Company’s continued organizational growth and acquisition activity.
-
Establish production support and incident-management practices for critical data pipelines and platform services.
Partner closely with the Director of Data Products to translate business and clinical priorities into an executable engineering roadmap.
Data Products defines what needs to be delivered and why; Data Engineering determines how it gets built and delivered reliably at scale.
Partner closely with the Director of Data Architecture to implement the Company’s enterprise data architecture, governance, modeling, security, and data quality standards.
Data Architecture establishes the architectural direction and standards; Data Engineering turns those standards into production technology and engineering practices.
Provide the VP with clear visibility into engineering capacity, platform health, delivery performance, technical risks, resource needs, and major dependencies.
The Director will be the VP’s primary leader for engineering execution and Databricks platform delivery.
-
Establish clear SLAs and operational standards for critical data pipelines and platform services.
-
Implement monitoring, alerting, observability, and incident-response practices.
-
Drive automated testing and data validation throughout the engineering lifecycle.
-
Establish CI/CD and disciplined code-promotion processes across Databricks environments.
-
Ensure appropriate source control, peer review, documentation, and release-management practices.
-
Continuously identify opportunities to improve platform performance and engineering productivity.
-
Balance new feature delivery with technical debt, platform maintenance, and reliability.
-
Partner with Data Architecture, Security, and Governance teams to ensure engineering solutions meet enterprise security and data governance standards.
-
Ensure appropriate access controls, auditability, encryption, and secure development practices across Databricks environments.
-
Ensure data engineering practices support HIPAA and applicable healthcare regulatory requirements.
-
Implement data lineage, quality, classification, and governance standards within engineering workflows.
-
8+ years of progressive data engineering experience, including significant experience building enterprise-scale data platforms.
-
3+ years of engineering leadership experience, ideally managing teams of 8–15+ engineers.
-
Deep hands-on Databricks experience is required, including experience operating complex production environments.
-
Demonstrated experience working across multiple Databricks environments/workspaces.
-
Strong expertise with Python, SQL, PySpark/Spark, and Delta Lake.
-
Strong experience building production-grade ETL/ELT pipelines and modern lakehouse architectures.
-
Experience with Databricks Workflows and/or orchestration technologies such as Azure Data Factory or Airflow.
-
Strong understanding of CI/CD, automated testing, observability, source control, and modern engineering practices.
-
Heavy Azure DevOps (ADO) experience, including backlog management, sprint planning, work-item management, capacity planning, and delivery reporting.
-
Demonstrated experience leading Agile/Scrum engineering teams and driving disciplined sprint execution.
-
Strong experience managing technical teams while remaining meaningfully engaged with architecture and engineering decisions.
-
Experience operating in Azure or another major cloud environment.
-
Healthcare or other highly regulated industry experience.
-
Experience working with clinical, EHR, claims, financial, or healthcare operational data.
-
Experience with Unity Catalog and enterprise Databricks governance.
-
Experience with Azure Data Factory, Azure Data Lake, Azure Functions, dbt, and related Azure data technologies.
-
Experience supporting analytics, data science, machine learning, or AI workloads.
-
Experience supporting enterprise growth, acquisitions, or integration of new business units and data sources.
The ideal candidate is an engineering leader first.
This person should be equally comfortable leading a team of 10 engineers, reviewing a complex Databricks design, managing competing priorities in ADO, running a disciplined Scrum process, and explaining platform risks or delivery timelines to senior leadership.
This is not primarily a BI, analytics, data governance, or data strategy position. Those capabilities will sit alongside this role within the broader data organization.
The Director of Data Engineering & Platform owns the critical middle layer between strategy and execution: the people, platform, processes, and engineering discipline required to reliably turn the Company’s data strategy into production capabilities.
-
Databricks: Deep, recent, production-level Databricks experience.
-
Engineering Leadership: Proven success leading and developing a data engineering team of comparable scale.
-
Platform Leadership: Experience owning engineering execution across multiple environments and complex enterprise data ecosystems.
-
Azure DevOps: Heavy hands-on ADO experience managing engineering backlogs, sprints, capacity, dependencies, and delivery.
-
Scrum: Strong Scrum leadership with the ability to create structure, accountability, and predictable engineering delivery.
-
Technical Depth: Strong Python, SQL, Spark/PySpark, pipeline, orchestration, CI/CD, and modern data engineering knowledge.
-
Execution: Ability to translate architecture and product priorities into scalable, production-ready engineering solutions.
Vaco by Highspring is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment because of race (including but not limited to traits historically associated with race such as hair texture and hair style), color, sex (includes pregnancy or related conditions), religion or creed, national origin, citizenship, age, disability, status as a veteran, union membership, ethnicity, gender, gender identity, gender expression, sexual orientation, marital status, political affiliation, or any other protected characteristics as required by federal, state or local law.
Vaco by Highspring and its parents, affiliates, and subsidiaries are committed to the full inclusion of all qualified individuals. As part of this commitment, Vaco by Highspring and its parents, affiliates, and subsidiaries will ensure that persons with disabilities are provided reasonable accommodations. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please contact [email protected].
Vaco by Highspring also wants all applicants to know their rights that workplace discrimination is illegal.
By submitting to this position, you agree that you will be giving Vaco by Highspring the exclusive right to present your as a candidate for the foregoing employment opportunity. You further agree that you have represented information about yourself accurately and have not affirmatively misrepresented your qualifications. You also agree to maintain as confidential, to the fullest extent permitted by law, any information you learn from Vaco by Highspring about the position and you will limit disclosure of information about the position only to the extent necessary to perform any obligations in furtherance of your application. In exchange, Vaco by Highspring agrees to exercise reasonable efforts to represent you through all solicitation, job screening and resume dispersal.
For residents of Ontario, Canada: Based on Highspring’s discussions with its Client, Highspring’s understanding is that this position for employment is a current vacancy (either through Highspring as a contractor or with the client directly).
Vaco by Highspring and its parents, affiliates, and subsidiaries (“we,” “our,” or “Vaco by Highspring”) respects your privacy and are committed to providing transparent notice of our policies.
- California residents may access Vaco by Highspring HR Notice at Collection for California Applicants and Employees here.
- Virginia residents may access our state specific policies here.
- Residents of all other states may access our policies here.
- Canadian residents may access our policies in English here and in French here.
- Residents of countries governed by GDPR may access our policies here.
Additionally, submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. More details about Vaco by Highspring’s use of AI can be found here (https://www.highspring.com/ai-use-notices/). Further assessment of candidates beyond this initial phase will be conducted by recruiters and hiring managers. Vaco by Highspring does not know and cannot opine on if its client’s use of AI products in hiring.
Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to:
- the individual’s skill sets, experience and training;
- licensure and certification requirements;
- office location and other geographic considerations;
- other business and organizational needs.
With that said, as required by local law, Vaco by Highspring believes that the following salary range referenced above reasonably estimates the base compensation for an individual hired into this position in geographies that require salary range disclosure. The individual may also be eligible for discretionary bonuses.
