Statement of Purpose:
The Azure Data Engineer will be responsible for implementing data ingestion and transformation pipelines. We are seeking someone with deep technical skills in a variety of technologies to play an important role in developing and delivering early proofs of concept and production implementation.
· Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Databricks
· Delivering and presenting proofs of concept to of key technology components to project stakeholders.
· Developing scalable and re-usable frameworks for ingesting of assorted data sets
· Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained
· Working with event based / streaming technologies to ingest and process data
· Working with other members of the project team to support delivery of additional project components (API interfaces, Search)
· Working with Azure CI/CD pipelines
· Working with Azure Dev Ops to support and maintain source control
· Evaluating the performance and applicability of multiple tools against customer requirements
· Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
· Strong knowledge of Data Management principles
· Experience in building ETL / data warehouse transformation processes
· Direct experience of building data pipelines using Azure Data Factory and Apache Spark (preferably Databricks).
· Experience using geospatial frameworks on Apache Spark and associated design and development patterns
· Microsoft Azure Big Data Architecture certification.
· Hands on experience designing and delivering solutions using the Azure Data Analytics platform (Cortana Intelligence Platform) including Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, Azure Stream Analytics
· Ability to work seamlessly within a team, react quickly to a fast paced, and rapidly changing environment.
· Strong capacity for logical thinking with the ability to understand complex applications.
· Excellent written and verbal communication skills.
· Bachelor’s degree and or combination of equivalent education and experience in related fields.
· Minimum of 8 years hands on experience providing solution design with BI systems and BI reports.
· 5+ years hands on experience using databases.
Job Type: Full-time
Pay: From $70,000.00 per year
- 401(k) matching
- Dental insurance
- Employee assistance program
- Employee discount
- Flexible schedule
- Flexible spending account
- Health insurance
- Health savings account
- Life insurance
- Paid time off
- Retirement plan
- Vision insurance
Work Location: One location