Key Responsibilities Design & Development: Lead the design and development of custom AWS utilities that allow analysts to query disparate data sources (S3 and Redshift) with minimal effort and high productivity. Pipeline Engineering: Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL to ingest and transform data across platforms. Resiliency & Scalability: Apply an "enterprise-grade" development mindset. Build solutions that are resilient, scalable, and capable of quick automated recovery in the event of failure. Utility Creation: Create tools to streamline the user experience for analysts, such as building automated processes to convert SQL outputs for multi-source tables. Infrastructure Ops: Utilize CloudFormation to manage and deploy infrastructure components. Required Technical Skills AWS Core: Deep proficiency in AWS Glue, S3, Lambda, IAM, and Step Functions. Data Engineering: Expert-level knowledge of PySpark and Spark SQL. Querying & SQL: Extensive experience with Athena and Presto; strong SQL skills for complex data manipulation. Infrastructure as Code: Proven experience with CloudFormation for operational excellence. Development Mindset: Ability to write clean, maintainable code for automation and custom utility development. Dremio: Experience with Dremio implementation (reflections, views, using system tables). Required Qualifications 6-8+ years of Data Engineering or IT experience. 4+ years of AWS cloud development experience. Expert Python programming skills. Advanced SQL development experience. Strong hands-on experience with Apache Spark / PySpark. Experience building cloud-native ETL pipelines. Strong understanding of distributed computing. Experience designing scalable data models. Experience with Git and CI/CD pipelines. Strong analytical and problem-solving skills. Build scalable and auto recoverable pipelines. Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline. Nice to Have Tableau: Familiarity with Tableau for end-user data consumption.
