Nifi Samples

Exercise 1: Read data from REST URL and write to HDFS

Processor1:

Use 'InvokeHTTP' processor and Provide URI. 

• HTTP Method: POST
• URL: http://samples.openweathermap.org/data/2.5/weather?q=London&appid=14e8adf775bccd703001ef35eefdfe3c

Processor2:

use the processor 'putHDFS' and specify the directory

You need to copy the core-site.xml and hdfs-site.xml from one of your hadoop nodes to the machine where NiFi is running. Then configure PutHDFS so that the configuration resources are "/path/to/core-site.xml,/path/to/hdfs-site.xml". That is all that is required from the NiFi perspective, those files contain all of the information it needs to connect to the Hadoop cluster.

You'll also need to ensure that the machine where NiFi is running has network access to all of the machines in your Hadoop cluster.

If required give yarn-site.xml path as well.

Exercise 2: Read data from Kafka topic in AVRO Format and insert into Hive table using Nifi

Read data in text format from Kafka and write to Hive table. https://www.youtube.com/watch?v=4yBc7hHvaQU

Kafka Producer and consumer in Java.

http://aseigneurin.github.io/2016/03/04/kafka-spark-avro-producing-and-consuming-avro-messages.html

https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries

Points:

1. If the Processor is the last one and needs to be ended without further flow then in Settings select Auto-terminate option for the appropriate status.