Bigdata Engineer

PRUDENS - New York, NY

30+ days ago
Our Client is looking for a bigdata developer . This is a Full-Time position and only GC holders or US Citizens is considered.

The developer will build data engineering pipelines and has the ability to ETL/ELT data coming from disparate sources (RDBMS, Stream, File, Web, etc.) leveraging big data technologies. Experience in Architecture and development of large scale production environment.Developer will provide engineering support for cyber security products developed and also experience in System, network and application troubleshooting

  • Strong experience with hadoop ecosystem: Hadoop, Spark, Map/Reduce, Hive/Pig, Impala/Drill, etc.
  • Expert in Apache Spark 2.X framework (pyspark & Java)
  • Experience with messaging and data transport tools: Kafka, Sqoop, NiFi, LogStash, Syslog-ng, rsyslog, etc.
  • Strong interpersonal and communication skills; capable of writing documentation, training users in complex topics, making presentations to junior and very senior audience
  • Ability to work under pressure in a fast-paced environment while remaining productive and professional; exercise patience and ability to multi task
  • Comfortable with most aspect of operating system administration such as tweaking, hardening and configuring services
  • A solid understanding of Unix-based operating systems, including paging/swapping, IPC, drivers and filesystem (inode, partitions, etc.)
  • provided by Dice Hadoop, Spark, Map Reduce, Hive, Pig, Impala, Drill, Kafka, AWS, Python etc.