CEI is seeking a
Databricks Architect to join our growing organization!
Client/Industry: CEI – AI / IT Solutions/Consulting and Professional Services
Job Title: Databricks Technical Architect
Location:
Hybrid in Pittsburgh, PA 15205 (
3 Days On-Site / 2 Days Remote) |
OR |
Remote (1st pref-EST; 2nd pref-CST)
Work Schedule/Shift: Monday to Friday, Standard business hours |
Minimum expectation of ~40 work hours per week.
Duration/Length of Assignment: W2 Direct Hire / Full Time
Compensation: Competitive pay (Salary) | Benefits available (
optional) [Medical, Dental, Vision, and 401(k) match]
Position Overview:
This role supports CEI’s continued evolution into an AI-first engineering and transformation company by building and scaling enterprise-grade data platforms that enable analytics, reporting, and AI/ML initiatives. The position sits within CEI’s Solutions and Engineering organization, working alongside data engineering, analytics, governance, and business teams to deliver a modern Lakehouse platform. The individual will act as a senior technical leader, partnering across multiple stakeholders while guiding a team of data engineers and contributing directly to architectural decisions and implementation. The role operates within a collaborative, cross-functional environment where engineering standards, governance, and platform scalability are core priorities. At a high level, this individual will design and implement end-to-end data solutions, establish platform standards, and ensure the data ecosystem supports both analytical and AI-driven use cases. The expectation is to balance architecture ownership with hands-on development, contributing directly to platform buildout, optimization, and governance. This role requires consistent collaboration with internal teams, active participation in architecture discussions, and ongoing mentorship of engineers while ensuring delivery of reliable, scalable, and cost-efficient data solutions.
Required Skills/Experience/Qualifications:
- 8–12+ years of experience in data engineering, data architecture, or related roles
- 3+ years of hands-on production experience with Databricks
- Strong expertise in Apache Spark using PySpark and Spark SQL
- Experience designing and implementing Lakehouse architectures using Delta Lake and medallion (Bronze/Silver/Gold) patterns, with a strong grasp of partitioning, Z-ordering, liquid clustering, and Delta optimization techniques
- Experience with Unity Catalog including governance design, access control, and data lineage
- Implementation experience on at least one major cloud platform: AWS, Azure, or GCP
- Experience implementing infrastructure-as-code using Terraform and/or Databricks Asset Bundles
- Experience building and managing CI/CD pipelines for data engineering workflows (Git-based, with environment promotion)
- Strong understanding of data modeling approaches including dimensional modeling and Data Vault
- Experience with ETL/ELT patterns and distributed data systems
- Experience with cost optimization and performance tuning within Databricks environments at scale
- Understanding of data governance, security, and compliance principles and ability to implement them within platform design
Preferred Skills (Not Required):
- Scala is a plus but not required
- Databricks certifications such as Data Engineer Professional or Solution Architect
- Experience with Structured Streaming and event-driven architectures (Kafka, Event Hubs, Kinesis, Pub/Sub)
- Experience with ingestion frameworks such as Lakeflow Connect
- Familiarity with MLflow, feature stores, and ML platform integrations
- Experience supporting BI tools such as Power BI or Tableau at scale, including semantic modeling
- Exposure to open table formats such as Iceberg or Delta UniForm and Delta Sharing.
- Experience working within regulated industries such as healthcare, financial services, or public sector
Day to Day/Responsibilities:
- Architect end-to-end data solutions within Databricks, spanning ingestion, transformation, governance, and serving layers using the medallion (Bronze/Silver/Gold) architecture pattern on Delta Lake
- Lead hands-on development of production data pipelines using PySpark and Spark SQL, with Scala leveraged where appropriate, ensuring code quality, scalability, and performance across workloads
- Perform code reviews and establish engineering standards through reusable frameworks and reference implementations to ensure consistency across the platform
- Own and design Unity Catalog structures, including catalog and schema organization, access controls, row- and column-level security, data classification, lineage tracking, and audit capabilities
- Partner directly with data governance teams to translate business policies into enforceable platform-level controls within Databricks
- Define and implement the data serving layer strategy using Databricks SQL, including materialized views, performance tuning, and enabling consumption through BI tools such as Power BI and Tableau, including semantic modeling
- Establish platform-as-code practices using Terraform and/or Databricks Asset Bundles, ensuring consistent infrastructure deployment and environment standardization
- Design and operate CI/CD pipelines for notebooks, jobs, and infrastructure using Git-based workflows with environment promotion, supporting controlled releases across development, testing, and production environments
- Define orchestration strategies using Databricks Workflows and Lakeflow Declarative Pipelines, integrating with tools such as Airflow when cross-platform orchestration is required
- Own cost and performance governance across the platform, including cluster policies, serverless versus classic compute decisions, Photon usage, job versus all-purpose cluster strategy, and DBU budgeting/chargeback models
- Collaborate with AI and ML teams to support platform integration with MLflow, feature engineering on Delta Lake, and emerging Databricks AI capabilities such as vector search and Mosaic, ensuring the platform supports model development and deployment
- Provide technical leadership and mentorship to data engineers, acting as the primary Databricks subject matter expert across architecture discussions and cross-functional forums
About Us: CEI is an AI-first engineering and transformation company that designs, builds, and operationalizes production-grade AI and data solutions across enterprise environments. Originally established as an IT consulting and staffing firm, CEI has evolved into a technology partner focused on delivering enterprise-scale AI transformation, modern data platforms, and application development solutions. We work across industries including healthcare, financial services, utilities, retail, and manufacturing, supporting both mid-market and Fortune 1000 clients. CEI combines consulting, engineering, and delivery capabilities to not only define strategies but also build and implement solutions at scale. With a focus on integrating AI into core business operations, CEI enables organizations to modernize their technology landscape, improve decision-making, and generate measurable outcomes through advanced data and AI capabilities.
#INDGEN #ZR
