Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster

I’ve created 2 Ansible Roles (chilcano.apache-nifi and chilcano.apache-nifi-toolkit) to automate the creation of a multi-node and secure NiFi cluster.

At the moment, the chilcano.apache-nifi Ansible Role doesn’t implement Cluster State coordination through Apache ZooKeeper. It will be implemented in the next version of this Ansible Role.

For other side, I’ve implemented only TLS Toolkit Standalone mode in the chilcano.apache-nifi-toolkit Ansible Role.

Further details and samples about both Ansible Roles can be found at Ansible Galaxy:
chilcano.apache-nifi-toolkit
chilcano.apache-nifi

Once presented both Ansible Roles, I’m going to explain how to automate the creation of several instances of Apache NiFi, secure and not secure. See image below:

Automated provisioning Apache NiFi multi-node cluster with Ansible and Vagrant

Automated provisioning Apache NiFi multi-node cluster with Ansible and Vagrant

We are going to create 5 NiFi instances, the first NiFi instance nf1 will be a standalone instance running over HTTP.
The second instance will will be a customized instance running over HTTPS with Client Certificate authentication.
The third, fourth and fifth instances will run over HTTPS with Client Certificate authentication with configuration provided for NiFi TLS Toolkit. This configuration, key-pair, Java key stores and certificates will be generated in other VM instance provided for chilcano.apache-nifi-toolkit Ansible Role. See the image below:

Apache NiFi Toolkit - folder structure and files generated

Apache NiFi Toolkit – folder structure and files generated

I’ve created an Ansible Playbook for you, you can download from this Git repository: https://github.com/chilcano/ansible-apache-nifi-multi-nodes

How to use it – steps

1. Clone the Ansible playbooks

$ git clone https://github.com/chilcano/ansible-apache-nifi-multi-nodes

2. Install chilcano.apache-nifiand chilcano.apache-nifi-toolkit Ansible Roles

$ cd ansible-apache-nifi-multi-nodes
$ ansible-galaxy install -r playbooks/requirements.yml

3. Create all VMs with Vagrant

Create all 6 VMs by using Vagrant.

$ cd infra/Vagrant
$ vagrant up

4. Ansible provisioning with Vagrant

Now, I’m going to provision (run Ansible Playbooks) through Vagrant. This step will install Apache NiFi TLS Toolkit in the nftk1 VM, once provisioned, Vagrant will provision 5 VMs following the Ansible Playbook playbooks/main.yml.
It is very important to start the provision of all NiFi instances after provisioning nftk1.
In the playbooks/main.yml you will see the nftk1 is declared on top and after nf1, nf2, nf3, nf4 and nf5.

$ vagrant provision

$ vagrant status
Current machine states:

nftk1                     running (virtualbox)
nf5                       running (virtualbox)
nf4                       running (virtualbox)
nf3                       running (virtualbox)
nf2                       running (virtualbox)
nf1                       running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

Now we can verify if all instances are running as expected, before we have to install the Client Certificate (CN=chilcano_OU=INTIX.p12) generated in our browser.

Install the Client Certificate

Install the Client Certificate

The Client Certificate only is required when connecting to nf2, nf3, nf4 and nf5 because these instances are running over HTTPS with Mutual SSL/TLS Authentication (based on Client Certificate).

Select the Client Certificate

Select the Client Certificate

Open the URL (http://nf1:8080/nifi, http://nf2:9443/nifi, http://nf3:9443/nifi, http://nf4:9443/nifi and http://nf5:9443/nifi) from Firefox. Instead of hostname you can use the IP address (see inventory file).

Open NiFi from Firefox

Open NiFi from Firefox

ToDo

  1. Improve the Ansible Role chilcano.apache-nifi to implement Cluster Status coordination through Apache ZooKeeper.
  2. Improve the Ansible Role chilcano.apache-nifi-toolkit to implement Client/Server mode.
  3. Deploy a sample DataFlow in NiFi.

End.

@Chilcano

Tagged with: , , , , ,
Posted in Big Data, DevOps, Security
One comment on “Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster
  1. […] Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster by Roger Carhuatocto […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
%d bloggers like this: