Position: Hadoop Administrator
Duration: Full Time/Long Term
Location: San Francisco, CA
High level requirements for the project is to upgrade and setup a 10,000 node cluster and perform the following tasks:
Uinux Ubuntu Upgrade
AWS Placement Group group setup
The project will have 2 phases. As part of phase 1, customer would like to upgrade Linux Ubuntu operating system across the 10,000 node cluster and also setup the AWS Placement Group. After the phase 1 completed successfully and the environment is stabilized, customer would like to upgrade the HBase version.
Implementing, managing and administering the overall Hadoop infrastructure
Manage Hadoop cluster environment including deployment, service allocation and configuration for the cluster, capacity planning, performance tuning, and ongoing monitoring.
As the need is to work with open source Apache Distribution then the Hadoop admins will to have to manually setup all the configurations- Core-Site, HDFS-Site, YARN-Site and Map Red-Site.
Troubleshooting application errors and ensuring that they do not occur again.
The Hadoop Admin will have responsibility to upgrade across large size cluster of about 10,000 nodes so, it will be vital that admin has experience in automating such upgrades using Terraform AWS tool to enable "Infrastructure as code" for the upgrades.
Detailed understanding and hands on experience of Terraform to setup automated processes to automatically determine resource dependencies, build/upgrade servers in cloud.
The Hadoop Admin will have to upgrade Linux Ubuntu and setup AWS Place Groups in phase 1 and upgrade HBase in phase 2
Experience deploying and maintaining Hadoop clusters
Excellent knowledge of Linux Ubuntu.
Deep understanding of Hadoop cluster security, networking connectivity and IO throughput along with other factors that affect distributed system performance
Experience with engineering high availability and performance tuning
Working knowledge of automation tools (e.g., Puppet, Chef, Ansible)
Working knowledge of hardening Hadoop with Kerberos, TLS, etc.
Ability to quickly perform critical analysis and use creative approaches for solving complex problems
Excellent written and verbal communication skills
Required Skills/Mandatory skills
Hadoop ecosystem and HDFS (mandatory)
Spark Core (mandatory)
Spark SQL, DataFrames and DataSets (mandatory)
Spark Streaming (mandatory)
Setting up Hadoop Cluster on AWS (mandatory)
AWS Terraform (mandatory)
Good to have skills/ Optional skills
Sqoop (Good to have)
Pig (Good to have)
NoSQL DataBases: Cassandra, MongoDB (optional)
Business Intelligence Tools: Jasper Soft, Tableau, Microstrategy, SAP BI (optional)
ETL Tools (optional).
Please send me your updated resumes to murali.channa AT Bitwsieglobal.com
Job Types: Full-time, Part-time, Temporary, Contract
Salary: $80.00 to $100.00 /hour
- AWS: 3 years (Preferred)
- Hadoop: 5 years (Preferred)
- Uinux Ubuntu Upgrade: 5 years (Preferred)
- HBase: 1 year (Preferred)
- Temporarily due to COVID-19