Nifi Developer - Data Ingestion

• Company Excellence and Eminence LLP
• No. of Openings 8
• Salary Not Disclosed
• Work Type on-site

Job Description :

Key Skills :

- Knowledge on Azure, Nifi Registry, Nifi Templates, Nifi Performance Tuning, Building Data ingestion pipeline, coding skills in Pyspark or Spark with Scala

- XML, XPATH, XSLT, JSON, JPATH, CSV, AVRO and HTML

Skills and experience :

- Collaborating within Agile Scrum teams as a key contributor responsible for the design, test and implementation of interfaces required.

- Developing and performing automated builds, testing, and deployments in support of NiFi development.

- Experience creating custom NiFi processors.

- Experience with high-velocity high-volume stream processing: Apache Kafka, Apache NiFi, Stream Sets, etc.

- Experience configuring and administering NiFi installations.

- Experience in building data ingestion workflowspipeline flows using NiFi, NiFi registry

- Experience with NiFi performance tuning.

- Knowledge of security principles and methods.

- Experience with Agile methodologies, such as Scrum, ideally in a DevOps environment.

- Ability to optimize workflows based in NiFi.

- Knowledge of IP networking including TCP, UDP, DNS, DHCP, firewalls, IP routing, etc.

- Experience with NiFi-supported scripting languages (Python, Groovy).

- Full Software Development Life Cycle (SDLC) experience.

- Experience with performance tuning of NiFi processing.

- Excellent verbal and written communication

Roles and responsibilities :

- Implemented solutions utilizing Advanced AWS Components: EMR, EC2, etc integrated with Big DataHadoop Distribution Frameworks: Zookeeper, Yarn, Spark, Scala, NiFi etc.

- Designed and Implemented Spark Jobs to be deployed and run on existing Active clusters.

- Configured Postgres Database on EC2 instances and made sure application that was created is up and running, Trouble Shooted issues to meet the desired application state.

- Worked on creating and configuring secure VPC, Subnets, and Security Groups through private and public networks.

- Created alarms, alerts, notifications for Spark Jobs to email and slack group message job status and log in CloudWatch.

- Worked on NiFi data Pipeline to process large set of data and configured Lookup's for Data Validation and Integrity.

- Worked on generation large set of test data with data integrity using java which used in Development and QA Phase.

- Worked in Spark Scala, improving the performance and optimized of the existing applications running on EMR cluster.

- Worked on a Spark Job to Convert CSV data to Custom HL7FHIR objects using FHIR API's.

- Deployed SNS, SQS, Lambda function, IAM Roles, Custom Policies, EMR with Spark and Hadoop setup and bootstrap scripts to setup additional software's needed to perform the job in QA and Production Environment using Terraform Scripts.

- Worked on Spark Job to perform Change Data Capture (CDC) on Postgres Tables and updated target tables using JDBC properties.