This role focuses on designing, implementing, and maintaining scalable enterprise ETL processes and robust data pipelines for a global client base. The Data Engineer will leverage big data frameworks such as Apache Spark and Hadoop, along with SQL and Python, to optimize data processing and ensure high data quality.
The role involves collaborating with cross-functional teams to automate routine tasks and deliver accurate, high-value data solutions across various industries.
Required Education
- Bachelor’s degree in a quantitative discipline such as Engineering, Mathematics, Finance, Business, or a related field.
- Equivalent practical experience may also be considered.
Required Qualifications / Skills / Experience
- Experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and methodologies.
- Strong knowledge of writing and optimizing SQL queries to retrieve, manipulate, and analyze data efficiently.
- Hands-on experience with big data technologies, including:
- Apache Spark:
- PySpark
- Spark SQL
- Spark Streaming
- Hadoop ecosystem:
- HDFS / Ozone
- Hive
- YARN
- Understanding of data modeling concepts and database design to support scalable data solutions.
- Familiarity with Python.
- Ability to analyze and troubleshoot data issues and provide solutions with minimal supervision.
- Basic knowledge of testing and validating data to ensure accuracy and consistency in data pipelines.
- Excellent verbal and written communication skills, with the ability to explain complex ideas clearly to both technical and non-technical stakeholders.
Preferred Qualifications / Skills / Experience
- Experience supporting the design, implementation, and maintenance of enterprise ETL processes for global data platforms.
- Ability to develop scalable and efficient code for data processing while ensuring timely availability and accessibility.
- Experience leveraging big data processing frameworks such as Apache Spark and Hadoop to build and optimize data pipelines.
- Ability to collaborate with senior engineers to resolve data challenges and maintain high data quality.
- Experience supporting data delivery processes alongside Data Engineers and Analysts to provide accurate, high-value data solutions across different clients and industries.
- Ability to build strong working relationships with team members and clients while contributing to local and global projects.
Job Duties / Responsibilities
- Learn and apply industry best practices, including:
- Version control
- Code reviews
- Data validation
- Quality assurance processes
- Use SQL and database technologies to optimize data processing and reduce processing time for large datasets.
- Design, implement, and maintain data pipelines using:
- ETL frameworks
- Orchestration tools
- Distributed data processing engines
- Participate in automation initiatives to streamline routine data tasks and improve operational efficiency.
- Ensure compliance with client internal policies and applicable external regulations.
Key Technical Skills
- Programming: Python, SQL
- Big Data Technologies: Apache Spark, PySpark, Spark SQL, Spark Streaming, Hadoop, HDFS/Ozone, Hive, YARN
- Data Engineering: ETL, Data Pipelines, Data Modeling, Database Design, Data Validation
- Engineering Practices: Version Control, Code Reviews, Automation, Troubleshooting
Employment Details
- Estimated Duration: 3 months
- Shift: Standard Work Hours
Pay: $61.00 - $67.00 per hour
Application Question(s):
- Do you have a Bachelor’s degree in Engineering, Mathematics, Finance, Business, Computer Science, or another quantitative discipline (or equivalent practical experience)?
- Do you have professional experience working as a Data Engineer or in a similar data-focused role?
- How would you rate your SQL skills, particularly in writing and optimizing queries for large datasets?
- Do you have hands-on experience developing or maintaining ETL/ELT processes and data pipelines?
- Have you worked with Apache Spark technologies (such as PySpark, Spark SQL, or Spark Streaming)?
- Are you proficient in Python for data processing, automation, or data engineering tasks?
- Do you have experience working with Hadoop ecosystem technologies (such as HDFS/Ozone, Hive, or YARN)?
- Do you have experience with data modeling, database design, and building scalable data solutions?
- Have you performed data validation, testing, or troubleshooting to ensure data accuracy and consistency in pipelines?
- Are you comfortable collaborating with technical and non-technical stakeholders to communicate data solutions and resolve issues?
Work Location: In person
