Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster

I’ve created 2 Ansible Roles (chilcano.apache-nifi and chilcano.apache-nifi-toolkit) to automate the creation of a multi-node and secure NiFi cluster.

At the moment, the chilcano.apache-nifi Ansible Role doesn’t implement Cluster State coordination through Apache ZooKeeper. It will be implemented in the next version of this Ansible Role.

For other side, I’ve implemented only TLS Toolkit Standalone mode in the chilcano.apache-nifi-toolkit Ansible Role.

Further details and samples about both Ansible Roles can be found at Ansible Galaxy:
chilcano.apache-nifi-toolkit
chilcano.apache-nifi

Once presented both Ansible Roles, I’m going to explain how to automate the creation of several instances of Apache NiFi, secure and not secure. See image below:

Automated provisioning Apache NiFi multi-node cluster with Ansible and Vagrant

Automated provisioning Apache NiFi multi-node cluster with Ansible and Vagrant

We are going to create 5 NiFi instances, the first NiFi instance nf1 will be a standalone instance running over HTTP.
The second instance will will be a customized instance running over HTTPS with Client Certificate authentication.
The third, fourth and fifth instances will run over HTTPS with Client Certificate authentication with configuration provided for NiFi TLS Toolkit. This configuration, key-pair, Java key stores and certificates will be generated in other VM instance provided for chilcano.apache-nifi-toolkit Ansible Role. See the image below:

Apache NiFi Toolkit - folder structure and files generated

Apache NiFi Toolkit – folder structure and files generated

I’ve created an Ansible Playbook for you, you can download from this Git repository: https://github.com/chilcano/ansible-apache-nifi-multi-nodes

How to use it – steps

1. Clone the Ansible playbooks

$ git clone https://github.com/chilcano/ansible-apache-nifi-multi-nodes

2. Install chilcano.apache-nifiand chilcano.apache-nifi-toolkit Ansible Roles

$ cd ansible-apache-nifi-multi-nodes
$ ansible-galaxy install -r playbooks/requirements.yml

3. Create all VMs with Vagrant

Create all 6 VMs by using Vagrant.

$ cd infra/Vagrant
$ vagrant up

4. Ansible provisioning with Vagrant

Now, I’m going to provision (run Ansible Playbooks) through Vagrant. This step will install Apache NiFi TLS Toolkit in the nftk1 VM, once provisioned, Vagrant will provision 5 VMs following the Ansible Playbook playbooks/main.yml.
It is very important to start the provision of all NiFi instances after provisioning nftk1.
In the playbooks/main.yml you will see the nftk1 is declared on top and after nf1, nf2, nf3, nf4 and nf5.

$ vagrant provision

$ vagrant status
Current machine states:

nftk1                     running (virtualbox)
nf5                       running (virtualbox)
nf4                       running (virtualbox)
nf3                       running (virtualbox)
nf2                       running (virtualbox)
nf1                       running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

Now we can verify if all instances are running as expected, before we have to install the Client Certificate (CN=chilcano_OU=INTIX.p12) generated in our browser.

Install the Client Certificate

Install the Client Certificate

The Client Certificate only is required when connecting to nf2, nf3, nf4 and nf5 because these instances are running over HTTPS with Mutual SSL/TLS Authentication (based on Client Certificate).

Select the Client Certificate

Select the Client Certificate

Open the URL (http://nf1:8080/nifi, http://nf2:9443/nifi, http://nf3:9443/nifi, http://nf4:9443/nifi and http://nf5:9443/nifi) from Firefox. Instead of hostname you can use the IP address (see inventory file).

Open NiFi from Firefox

Open NiFi from Firefox

ToDo

  1. Improve the Ansible Role chilcano.apache-nifi to implement Cluster Status coordination through Apache ZooKeeper.
  2. Improve the Ansible Role chilcano.apache-nifi-toolkit to implement Client/Server mode.
  3. Deploy a sample DataFlow in NiFi.

End.

Tagged with: , , , , ,
Posted in Big Data, DevOps, Security

My 3 Ansible Roles to mass provision Kismet and Apache MiNiFi for wardriving at scale

This blog post is implementing the same scenario used in the previous post Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible. The unique difference is this new blog post I’m using Ansible Roles instead of Ansible Playbooks where all automated operations as install, configure, run, etc. were implemented in Ansible Tasks.

mass-provisioning-kismet-minifi-raspberrypi-ansible-2-pkg

Then, I’ve refactored all Ansible Tasks and I’ve created 3 Ansible Roles. They are in Ansible Galaxy and are:

Ansible Role Kismet RPi Build (https://galaxy.ansible.com/chilcano/kismet-rpi-build)

An Ansible Role that builds / compiles from scratch and packs (Debian/Raspbian binary) Kismet on a Raspberry Pi. This Role provides the following features:

  • Download the Kismet source code.
  • Compile the source code in a Raspberry Pi.
  • Generate a Kismet Debian/Raspbian package suitable for Raspberry Pi (ARMv7).

Ansible Role Kismet RPi Wardriving (https://galaxy.ansible.com/chilcano/kismet-rpi-wardriving)

An Ansible Role that installs, configures and runs Kismet on a Raspberry Pi. This Role provides the following features:

  • Install Kismet and dependencies.
  • Configure Kismet by deploying an customized kismet.conf.
  • Download MAC Addresses Manufacturer List.
  • Enable monitor mode in the Raspberry Pi before starting Kismet.
  • Run Kismet as a systemd service.

Ansible Role Apache MiNiFi (https://galaxy.ansible.com/chilcano/apache-minifi)

An Ansible Role that installs, configures and runs Apache MiNiFi in tiny devices like a Raspberry Pi, although you can use it on any distro. This Role provides the following features:

  • Install Apache MiNiFi and Java SDK.
  • Configure Apache MiNiFi.
  • Run Apache MiNiFi as a systemd service.

How to use everything

Well, I’ve updated and deleted Ansible Tasks not used in the existing Ansible Raspberry Pi Wardriving Github repo because now I’m using 3 Ansible Roles. Just download the same Github repo with Tag 2.0.0 and execute the new Ansible Playbook with the same inventory file.

Install all Ansible Roles needed as below or using requirements.yml file:

$ sudo ansible-galaxy install geerlingguy.apache
$ sudo ansible-galaxy install chilcano.kismet-rpi-build
$ sudo ansible-galaxy install chilcano.kismet-rpi-wardriving
$ sudo ansible-galaxy install chilcano.apache-minifi

$ ansible-galaxy list

- chilcano.apache-minifi, 1.0.1
- chilcano.kismet-rpi-build, 1.0.4
- chilcano.kismet-rpi-wardriving, 1.1.1
- geerlingguy.apache, 2.0.1
- knopki.locale, a1232f836b5514c58da381d9479640e40d874906
- Stouts.hostname, 1.1.0
- Stouts.timezone, 2.2.0

…and continue with this:

$ git clone https://github.com/chilcano/ansible-raspberrypi-wardriving --branch 2.0.0 --single-branch

$ cd ansible-raspberrypi-wardriving

$ git tag -l
1.0.0
2.0.0

$ git checkout tags/2.0.0 -b 2.0.0

$ ansible-playbook -i inventory main_all.yml -k

And that’s it !.
I hope It helps.
Bye.

Tagged with: , , , , ,
Posted in Big Data, DevOps, IoT, Security

Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible

Lately I’m focusing on Automation in Big-Data Projects, and with my experience in Cyber Security I can bring new approaches and ideas to those Big-Data Projects are related to IT Security aspects (Threat Analisys, Privacy, Intrusion Detection, etc.).
With this post I will start a serie of articles explaining how to do “Data Ingestion/Capture” in the Edge by using:

  • Ansible to provisioning.
  • Multiple and remote devices in the Edge (Raspberry Pi).
  • Kismet to capture WIFI Anonymous traffic in the Edge.
  • Apache MiNiFi to enrich and route the captured traffic in the Edge.
  • Apache NiFi to collect the enriched traffic and forward to Solr / ElasticSearch.

…everything, near to real-time ;)

And if you want to explore in deep other use cases, this article is a must read: (EDGE INTELLIGENCE FOR IOT WITH APACHE MINIFI)[https://hortonworks.com/blog/edge-intelligence-iot-apache-minifi/]

In short, I will explain in this post the following:
– Manage several devices in the Edge by using Ansible.
– Mass provision of Kismet and Apache MiNiFi.
– Performing operational tasks in several devices.

Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible - Architecture

Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible – Architecture

1. Preparing your Raspberry Pi.

1.1. Prepare your MicroSD cards with the latest Raspbian image.

I’m going to use Raspbian Jessie Lite 2017-01-11 (http://director.downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2017-01-10/2017-01-11-raspbian-jessie-lite.zip).

$ diskutil list

$ diskutil unmountDisk /dev/disk3

$ sudo dd bs=1m if=2017-01-11-raspbian-jessie-lite.img of=/dev/rdisk3

$ touch /Volumes/boot/ssh

$ diskutil unmountDisk /dev/disk3

… repeat this process as times as Raspberry Pis you have.

1.2. Connect all Raspberry Pis to the Network.

Connect all Raspberry Pi to your Router, now your PC and all Pis are connected to same network, and if you have DHCP enabled your Raspberry Pi will have an IP address automatically.
Now, we have to get all IP addresses assigned to each Raspberry Pi. You could use Fing App in Android or install Fing in your PC.
I’m going to use Fing on Mac OSX.

$ sudo fing
Password:

19:58:18 > Discovery profile: Default discovery profile
19:58:18 > Discovery class:   data-link (data-link layer)
19:58:18 > Discovery on:      192.168.0.0/24
...
19:58:18 > Host is up:   192.168.0.17
           HW Address:   B8:27:EB:1E:12:63 (Raspberry Pi Foundation)

19:58:18 > Host is up:   192.168.0.18
           HW Address:   B8:27:EB:9C:C2:E3 (Raspberry Pi Foundation)

19:58:18 > Host is up:   192.168.0.19
           HW Address:   B8:27:EB:F0:F3:EA (Raspberry Pi Foundation)

19:58:18 > Host is up:   192.168.0.20
           HW Address:   B8:27:EB:5A:B5:5D (Raspberry Pi Foundation)
...

You could try this too from your Mac OSX:

$ arp -a -i en0

? (192.168.0.1) at 90:21:6:89:82:a9 on en0 ifscope [ethernet]
? (192.168.0.5) at c4:57:6e:94:91:c2 on en0 ifscope [ethernet]
? (192.168.0.10) at 44:65:d:9d:4e:3e on en0 ifscope [ethernet]
? (192.168.0.17) at b8:27:eb:1e:12:63 on en0 ifscope [ethernet]
? (192.168.0.18) at b8:27:eb:9c:c2:e3 on en0 ifscope [ethernet]
? (192.168.0.19) at b8:27:eb:f0:f3:ea on en0 ifscope [ethernet]
? (192.168.0.20) at b8:27:eb:5a:b5:5d on en0 ifscope [ethernet]
? (192.168.0.255) at (incomplete) on en0 ifscope [ethernet]
? (224.0.0.251) at 1:0:5e:0:0:fb on en0 ifscope permanent [ethernet]
? (239.255.255.250) at 1:0:5e:7f:ff:fa on en0 ifscope permanent [ethernet]
broadcasthost (255.255.255.255) at (incomplete) on en0 ifscope [ethernet]

Now, open a Terminal and connect through SSH to each Raspberry Pi just to verify the connection with your devices.

$ ssh pi@192.168.0.17

With this information (MAC and IP addresses) I’m ready to start with Ansible to do automation on my Raspberry Pi Cluster.

1.3. Initial configuration and provision for all Raspberry Pi through Ansible.

To do the initial configuration in all Raspberry Pis I’ve used the next:
– The Raspberry Pi Dramble Ansible Git repository. Although these Ansible Playbooks are to provision an Apache HTTPd, Drupal, MySQL Cluster, our intention is to use it initially to manage the Raspberry Pi cluster at infrastructure level.
– The mikolak.raspi-config Ansible Role to configure each Raspberry Pi like if used raspi-config tool.

That means:

  • Resize SD, update Raspbian and manage shutdown and reboot operations.
  • Set a proper Hostname, configure Network interfaces (eth0 and wlan0), DNS, etc.

I’ve updated and extended the Raspberry Pi Dramble Ansible Git repository to reset the Network configuration and to build from source code, to install and to start Kismet (http://www.kismetwireless.net).

You can download my Ansible Playbooks from here: https://github.com/chilcano/ansible-raspberrypi-wardriving

$ git clone https://github.com/chilcano/ansible-raspberrypi-wardriving

$ cd ansible-raspberrypi-wardriving

Now update networking/inventory. You will need to use all MAC Addresses and IP assigned to each Raspberry Pi collected in previous step. The final file look like:

$ nano setup/networking/inventory

[pis]
192.168.0.17
192.168.0.18
192.168.0.19
192.168.0.20

[pis:vars]
ansible_ssh_user=pi
ansible_ssh_user_new=picuy
path_to_ssh_key=/Users/Chilcano/.ssh/id_rsa.pub

And update networking/vars.yml.

$ nano setup/networking/vars.yml

---
# Mapping of what hardware MAC addresses should be configured with specific IPs.
mac_address_mapping:
  "b8:27:eb:1e:12:63":
    name: rpi17.intix.info
    ip: "192.168.0.17"
  "b8:27:eb:9c:c2:e3":
    name: rpi18.intix.info
    ip: "192.168.0.18"
  "b8:27:eb:f0:f3:ea":
    name: rpi19.intix.info
    ip: "192.168.0.19"
  "b8:27:eb:5a:b5:5d":
    name: rpi20.intix.info
    ip: "192.168.0.20"

# Nameservers to use in resolv.conf.
dns_nameservers:
  - "8.8.8.8"
  - "8.8.4.4"

gateway:
  eth0: "192.168.0.1"
  wlan0: "192.168.0.1"

rpi_nic_static:
  eth0: false
  wlan0: false

Finally, It is time to run the Ansible Playbook to provision the initial configuration.

$ cd setup/networking

$ ansible-playbook -i inventory main.yml -k

SSH password:

PLAY ***************************************************************************

TASK [setup] *******************************************************************

paramiko: The authenticity of host '192.168.0.17' can't be established.
The ssh-rsa key fingerprint is 66cf86cb30b2ca92658808f26d453c31.
Are you sure you want to continue connecting (yes/no)?
yes

paramiko: The authenticity of host '192.168.0.20' can't be established.
The ssh-rsa key fingerprint is 7d259fe2fcf0b36164441bd4dbee321e.
Are you sure you want to continue connecting (yes/no)?
yes

paramiko: The authenticity of host '192.168.0.18' can't be established.
The ssh-rsa key fingerprint is de91abe013a8af491d0c8aa5b5d6a9aa.
Are you sure you want to continue connecting (yes/no)?
yes

paramiko: The authenticity of host '192.168.0.19' can't be established.
The ssh-rsa key fingerprint is 447a195e33bbee9d4479ed46110dd3d4.
Are you sure you want to continue connecting (yes/no)?
yes
ok: [192.168.0.17]
ok: [192.168.0.18]
ok: [192.168.0.20]
ok: [192.168.0.19]

TASK [Set the current MAC address for eth0.] ***********************************
ok: [192.168.0.17]
ok: [192.168.0.18]
ok: [192.168.0.19]
ok: [192.168.0.20]

TASK [Set variables based on eth0 MAC address.] ********************************
ok: [192.168.0.17]
ok: [192.168.0.18]
ok: [192.168.0.19]
ok: [192.168.0.20]

TASK [Set up networking-related files.] ****************************************
changed: [192.168.0.18] => (item={u'dest': u'/etc/hostname', u'template': u'hostname.j2'})
changed: [192.168.0.18] => (item={u'dest': u'/etc/hosts', u'template': u'hosts.j2'})
changed: [192.168.0.18] => (item={u'dest': u'/etc/network/interfaces', u'template': u'interfaces.j2'})
changed: [192.168.0.18] => (item={u'dest': u'/etc/resolv.conf', u'template': u'resolv.conf.j2'})
changed: [192.168.0.18] => (item={u'dest': u'/etc/dhcpcd.conf', u'template': u'dhcpcd.conf.j2'})
changed: [192.168.0.17] => (item={u'dest': u'/etc/hostname', u'template': u'hostname.j2'})
changed: [192.168.0.17] => (item={u'dest': u'/etc/hosts', u'template': u'hosts.j2'})
changed: [192.168.0.17] => (item={u'dest': u'/etc/network/interfaces', u'template': u'interfaces.j2'})
changed: [192.168.0.17] => (item={u'dest': u'/etc/resolv.conf', u'template': u'resolv.conf.j2'})
changed: [192.168.0.17] => (item={u'dest': u'/etc/dhcpcd.conf', u'template': u'dhcpcd.conf.j2'})
changed: [192.168.0.19] => (item={u'dest': u'/etc/hostname', u'template': u'hostname.j2'})
changed: [192.168.0.19] => (item={u'dest': u'/etc/hosts', u'template': u'hosts.j2'})
changed: [192.168.0.19] => (item={u'dest': u'/etc/network/interfaces', u'template': u'interfaces.j2'})
changed: [192.168.0.19] => (item={u'dest': u'/etc/resolv.conf', u'template': u'resolv.conf.j2'})
changed: [192.168.0.19] => (item={u'dest': u'/etc/dhcpcd.conf', u'template': u'dhcpcd.conf.j2'})
changed: [192.168.0.20] => (item={u'dest': u'/etc/hostname', u'template': u'hostname.j2'})
changed: [192.168.0.20] => (item={u'dest': u'/etc/hosts', u'template': u'hosts.j2'})
changed: [192.168.0.20] => (item={u'dest': u'/etc/network/interfaces', u'template': u'interfaces.j2'})
changed: [192.168.0.20] => (item={u'dest': u'/etc/resolv.conf', u'template': u'resolv.conf.j2'})
changed: [192.168.0.20] => (item={u'dest': u'/etc/dhcpcd.conf', u'template': u'dhcpcd.conf.j2'})

RUNNING HANDLER [update hostname] **********************************************
changed: [192.168.0.19]
changed: [192.168.0.18]
changed: [192.168.0.17]
changed: [192.168.0.20]

RUNNING HANDLER [delete dhcp leases] *******************************************
changed: [192.168.0.19] => (item=/var/lib/dhcp/dhclient.leases)
ok: [192.168.0.19] => (item=/var/lib/dhcpcd5/dhcpcd-eth0.lease)
changed: [192.168.0.17] => (item=/var/lib/dhcp/dhclient.leases)
ok: [192.168.0.17] => (item=/var/lib/dhcpcd5/dhcpcd-eth0.lease)
changed: [192.168.0.18] => (item=/var/lib/dhcp/dhclient.leases)
ok: [192.168.0.18] => (item=/var/lib/dhcpcd5/dhcpcd-eth0.lease)
changed: [192.168.0.20] => (item=/var/lib/dhcp/dhclient.leases)
ok: [192.168.0.20] => (item=/var/lib/dhcpcd5/dhcpcd-eth0.lease)

PLAY RECAP *********************************************************************
192.168.0.17               : ok=6    changed=3    unreachable=0    failed=0
192.168.0.18               : ok=6    changed=3    unreachable=0    failed=0
192.168.0.19               : ok=6    changed=3    unreachable=0    failed=0
192.168.0.20               : ok=6    changed=3    unreachable=0    failed=0

The above result means you have connected to each Raspberry Pi and a proper Hostname based on each MAC Address have been assigned successfully.
First of all, I will check that using Ansible.

$ cd ansible-raspberrypi-wardriving/setup/networking

$ ansible pis -i inventory -m ping -k

SSH password:
192.168.0.19 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.0.18 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.0.17 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.0.20 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Now, you can use SSH to connect to all Raspberry Pi and check the hostname assigned.

$ ssh picuy@192.168.0.17
picuy@192.168.0.17's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Feb 14 22:02:13 2017 from 192.168.0.3

SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.

picuy@rpi17:~ $ exit

In this point, you will be able to continue configuring your Raspberry Pis through Ansible Playbook performing Linux commands remotely.

1.4. Performing Linux commands remotely through Ansible.

Get free memory available in each Raspberry Pi.

$ ansible pis -i inventory -a "free -m" -k
SSH password:
192.168.0.18 | SUCCESS | rc=0 >>
             total       used       free     shared    buffers     cached
Mem:           925         87        837          6          9         43
-/+ buffers/cache:         35        890
Swap:           99          0         99

192.168.0.17 | SUCCESS | rc=0 >>
             total       used       free     shared    buffers     cached
Mem:           925         87        838          6          9         43
-/+ buffers/cache:         34        890
Swap:           99          0         99

192.168.0.19 | SUCCESS | rc=0 >>
             total       used       free     shared    buffers     cached
Mem:           925         89        835          6          9         43
-/+ buffers/cache:         37        888
Swap:           99          0         99

192.168.0.20 | SUCCESS | rc=0 >>
             total       used       free     shared    buffers     cached
Mem:           434         74        359          4          9         37
-/+ buffers/cache:         27        406
Swap:           99          0         99

Performing different Linux commands.

$ ansible pis -i inventory -a "cat /etc/hostname" -k

$ ansible pis -i inventory -a "cat /etc/hosts" -k

$ ansible pis -i inventory -a "ping -c 2 pi17.intix.info" -k

$ ansible pis -i inventory -a "ping -c 3 holisticsecurity.io" -k

$ ansible pis -i inventory -a "df -h" -k

But, if some command require sudo you can provide it using -s flag, although it is deprecated, it is still valid.

$ ansible pis -i inventory -a "ifconfig wlan0" -s -k

…or rebooting all Raspberry Pi.

$ ansible pis -i inventory -a "shutdown -r now" -s -k

SSH password:
192.168.0.18 | SUCCESS | rc=0 >>

192.168.0.17 | SUCCESS | rc=0 >>

192.168.0.19 | SUCCESS | rc=0 >>

No handlers could be found for logger "paramiko.transport"
192.168.0.20 | SUCCESS | rc=0 >>

1.5. Restoring the network configuration.

If you try to connect to one Raspberry Pi and the SSH connection is taking a few seconds, or if you are running ping http://www.google.com and Raspberry Pi is not reaching that. Then, you probably are facing issues with gateway and networking configuration in your Raspberry Pi and need restore or enable default configuration.
Then, let’s go to check the network configuration, basically you have to check these files in each Raspberry Pi:

  • /etc/dhcpcd.conf
  • /etc/hosts
  • /etc/network/interfaces

I’ve updated above cloned Ansible scripts for you with the right configuration. Basically I have updated the Ansible templates (Jinja2) to do:

  • Restoring etc/network/interfaces to get IP address automatically through DHCP.
  • Configuring /etc/dhcpcd.conf with our default gateway on eth0.

Also I have updated the setup/networking/main.yml Ansible Playbook and the setup/networking/vars.yml to restore default network configuration.

Again, run the updated Ansible Playbook and verify if the changes with the right gateway, DNS, etc. were applied.

$ ansible-playbook -i inventory main.yml -k

1.6. Set IP address automatically through DHCP on eth0.

In previous step I updated the Ansible Playbooks and Jinja2 templates to restore default assignation of IP addresses through DHCP.
Well, now I’m going to use it. Just update setup\networking\vars.yml setting rpi_nic_static.eth0 to false.

Run fing to get all IP addresses re-assigned to all Raspberry Pis and update the setup\networking\inventory and provision your new configuration.

$ ansible-playbook -i inventory main.yml -k

2. Massive provisioning of Kismet through Ansible on multiples Raspberry Pi.

I’m going to create an Ansible Playbooks to:

  • Enable WIFI interface in monitor mode on each Raspberry Pi.
  • Download Kismet source code and build it for Raspberry Pi (ARM chipset).
  • Install and configure Kismet on each Raspberry Pi.

To do that I will follow my previous blog posts (Capturing WIFI anonymous traffic using Raspberry Pi and WSO2 BAM – Part I) where I explained step by step all commands to be performed in order to run Kismet on Raspberry Pi.
The result final is a set of Ansible Playbooks located under ansible-raspberrypi-wardriving/playbooks/kismet in the Git repo (https://github.com/chilcano/ansible-raspberrypi-wardriving), and they are:

$ tree ansible-raspberrypi-wardriving/playbooks/kismet

ansible-raspberrypi-wardriving/playbooks/kismet
├── main_build.yml
├── main_clean.yml
├── main_install.yml
├── tasks
│   ├── build.yml
│   ├── clean_installation.yml
│   ├── install_from_deb.yml
│   ├── kill_process.yml
│   ├── localmirror.yml
│   └── run_as_systemd.yml
├── templates
│   ├── description-pak.j2
│   ├── kismet.conf.j2
│   ├── kismet_drone.conf.j2
│   ├── warpi.service.j2
│   └── warpi.sh.j2
└── vars.yml

2 directories, 15 files

Now, to run these Ansible Playbooks I have to follow the same steps above explained:
– Get all IP addresses and MAC addresses.
– Automatic assignation of IP addresses (DHCP).
– Assignation of a proper hostname.
– Change default username and password.

To do that, just follow the steps in section 1.3. Initial configuration and provision for all Raspberry Pi through Ansible of this post.
After that, You have to update your ansible-raspberrypi-wardriving/inventory and ansible-raspberrypi-wardriving/playbooks/kismet/vars.yml files.

Now, You are ready to run these Kismet Ansible Playbooks, then let’s do it:

$ cd ansible-raspberrypi-wardriving
$ ansible-playbook -i inventory main_kismet_install.yml -k

And if you get the below message, then you have already provisioned successfully Kismet in all Raspberry Pi configured in your ansible-raspberrypi-wardriving/inventory.

...
...
TASK [Copying 'warpi.sh.j2' template.] *****************************************
changed: [192.168.0.19]
changed: [192.168.0.18]

TASK [Copying 'warpi.service.j2' template.] ************************************
changed: [192.168.0.18]
changed: [192.168.0.19]

TASK [Reloading 'warpi' systemd service.] **************************************
changed: [192.168.0.18]
changed: [192.168.0.19]

TASK [Starting 'warpi' systemd service.] ***************************************
changed: [192.168.0.19]
changed: [192.168.0.18]

TASK [debug] *******************************************************************
ok: [192.168.0.18] => {
    "msg": "Service 'warpi' started successfully. Now Kismet is sending events on 2501 port ( nc -vz localhost 2501 )"
}
ok: [192.168.0.19] => {
    "msg": "Service 'warpi' started successfully. Now Kismet is sending events on 2501 port ( nc -vz localhost 2501 )"
}

PLAY RECAP *********************************************************************
192.168.0.18               : ok=34   changed=17   unreachable=0    failed=0
192.168.0.19               : ok=37   changed=14   unreachable=0    failed=0

And if you connect to your Raspberry Pi through SSH you can the status of Kismet there:

$ ssh picuy@192.168.0.19

...

picuy@rpi19:~ $ sudo systemctl status warpi
● warpi.service - Enable monitor mode and manage Kismet Server as service
   Loaded: loaded (/etc/systemd/system/warpi.service; enabled)
   Active: active (running) since Fri 2017-03-17 11:25:17 UTC; 3min 39s ago
 Main PID: 9023 (warpi.sh)
   CGroup: /system.slice/warpi.service
           ├─9023 /bin/sh /home/picuy/warpi.sh
           └─9028 kismet_server

Mar 17 11:25:22 rpi19.intix.info warpi.sh[9023]: INFO: Detected new managed network "BTWifi-with-FON", BSSID 02:03:D8:4D:17:
Mar 17 11:25:22 rpi19.intix.info warpi.sh[9023]: 72, encryption no, channel 11, 144.40 mbit
Mar 17 11:25:22 rpi19.intix.info warpi.sh[9023]: INFO: Detected new managed network "EE-ej96p2", BSSID E4:3E:D7:DB:9E:53,
Mar 17 11:25:22 rpi19.intix.info warpi.sh[9023]: encryption yes, channel 11, 144.40 mbit
Mar 17 11:25:41 rpi19.intix.info warpi.sh[9023]: INFO: Detected new probe network "", BSSID 88:32:9B:69:6F:3A,
Mar 17 11:25:41 rpi19.intix.info warpi.sh[9023]: encryption no, channel 0, 72.20 mbit
Mar 17 11:28:21 rpi19.intix.info warpi.sh[9023]: INFO: Detected new managed network "BTHomeHub2-SS4C", BSSID 00:26:44:19:7B:
Mar 17 11:28:21 rpi19.intix.info warpi.sh[9023]: CD, encryption yes, channel 7, 130.00 mbit
Mar 17 11:28:28 rpi19.intix.info warpi.sh[9023]: INFO: Detected new managed network "BTWiFi-with-FON", BSSID 02:26:44:19:7B:
Mar 17 11:28:28 rpi19.intix.info warpi.sh[9023]: CF, encryption no, channel 7, 130.00 mbit

3. Massive provisioning of Apache MiNiFi through Ansible on multiples Raspberry Pi.

If you have not read my previous post about Apache NiFi, well I can say that is a Data Mediator Engine and ETL with steroids suitable for BigData Projects and Apache MiNiFi is the perfect complement to it for IoT Projects.
Well, the scenario where I want to use Apache NiFi and Apache MiNiFi is in IoT, Security/Privacy space and the best way to validate this approach is using Ansible to do automation massively (Raspberry Pi in the edge) without pain.
I’ve created Ansible Playbooks to manage the installation and configuration of Apache MiNiFi in Raspberry Pi, they are located under ansible-raspberrypi-wardriving/playbooks/minifi in the Git repo (https://github.com/chilcano/ansible-raspberrypi-wardriving), and they are:

$ tree ansible-raspberrypi-wardriving/playbooks/minifi

ansible-raspberrypi-wardriving/playbooks/minifi
├── main_clean.yml
├── main_install.yml
├── tasks
│   ├── clean.yml
│   ├── install.yml
│   └── run.yml
├── templates
│   └── minifipi.service.j2
└── vars.yml

2 directories, 7 files

The same Ansible Playbooks should work in other devices too.
I’m going to repeat the same previous steps before running MiNiFi Ansible Playbooks. Check the section 2. Massive provisioning of Kismet through Ansible on multiples Raspberry Pi. for further details.
Now, You are ready to run these MiNiFi Ansible Playbooks, then let’s do it:

$ cd ansible-raspberrypi-wardriving

$ ansible-playbook -i inventory main_minifi_install.yml -k

And if you get the below message, then you have already provisioned successfully Apache MiNiFi in all Raspberry Pi configured in your ansible-raspberrypi-wardriving/inventory.

...
...
TASK [debug] *******************************************************************
ok: [192.168.0.18] => {
    "msg": "The MiNiFi service 'minifipi.service' has started successfully."
}

PLAY RECAP *********************************************************************
192.168.0.18               : ok=22   changed=8    unreachable=0    failed=0

Remember that you can execute the command remotely via Ansible, commands like ping, shutdown, free:

$ cd ansible-raspberrypi-wardriving/setup/networking

$ ansible pis -i inventory -m ping -k

$ ansible pis -i inventory -a "free -m" -k

…and/or check the current status of MiNiFi and/or Kismet as below I explain:

$ ansible pis -i inventory -a "systemctl -p MainPID,ControlGroup,Id,Description,FragmentPath show *pi" -k

192.168.0.17 | SUCCESS | rc=0 >>


192.168.0.44 | SUCCESS | rc=0 >>
MainPID=685
ControlGroup=/system.slice/warpi.service
Id=warpi.service
Description=Enable monitor mode and manage Kismet Server as service
FragmentPath=/etc/systemd/system/warpi.service

MainPID=5490
ControlGroup=/system.slice/minifipi.service
Id=minifipi.service
Description=Apache MiNiFi as service
FragmentPath=/etc/systemd/system/minifipi.service

This results means:

  • In RPi 192.168.0.17 the warpi (Kismet) and minifipi (MiNiFi) services are not running.
  • In RPi 192.168.0.18 the warpi (Kismet) and minifipi (MiNiFi) services are running.

And if you want further details about the Apache MiNiFi running in RPi 192.168.0.18, just execute this command:

$ ansible 192.168.0.18 -i inventory -a "systemctl status minifipi" -k

SSH password:
192.168.0.18 | SUCCESS | rc=0 >>
● minifipi.service - Apache MiNiFi as service
   Loaded: loaded (/etc/systemd/system/minifipi.service; enabled)
   Active: active (running) since Mon 2017-03-20 16:56:59 UTC; 42min ago
  Process: 5474 ExecStart=/home/picuy/minifi-0.1.0/bin/minifi.sh start (code=exited, status=0/SUCCESS)
 Main PID: 5490 (minifi.sh)
   CGroup: /system.slice/minifipi.service
           ├─5490 /bin/sh /home/picuy/minifi-0.1.0/bin/minifi.sh start
           ├─5491 /usr/bin/java -cp /home/picuy/minifi-0.1.0/conf:/home/picuy/minifi-0.1.0/lib/bootstrap/*:/home/picuy/minifi-0.1.0/lib/* -Xms12m -Xmx24m -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/home/picuy/minifi-0.1.0/logs -Dorg.apache.nifi.minifi.bootstrap.config.pid.dir=/home/picuy/minifi-0.1.0/run -Dorg.apache.nifi.minifi.bootstrap.config.file=/home/picuy/minifi-0.1.0/conf/bootstrap.conf org.apache.nifi.minifi.bootstrap.RunMiNiFi start
           └─5510 java -classpath /home/picuy/minifi-0.1.0/./conf:/home/picuy/minifi-0.1.0/./lib/jetty-util-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/httpcore-nio-4.4.5.jar:/home/picuy/minifi-0.1.0/./lib/jsr311-api-1.1.1.jar:/home/picuy/minifi-0.1.0/./lib/commons-codec-1.10.jar:/home/picuy/minifi-0.1.0/./lib/httpasyncclient-4.1.1.jar:/home/picuy/minifi-0.1.0/./lib/jcl-over-slf4j-1.7.12.jar:/home/picuy/minifi-0.1.0/./lib/nifi-framework-core-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-server-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-security-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-schemas-3.1.jar:/home/picuy/minifi-0.1.0/./lib/curator-recipes-2.11.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-security-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-framework-authorization-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-api-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-properties-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/spring-core-4.2.4.RELEASE.jar:/home/picuy/minifi-0.1.0/./lib/nifi-properties-loader-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/antlr-runtime-3.5.2.jar:/home/picuy/minifi-0.1.0/./lib/json-path-2.0.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-web-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/commons-lang3-3.4.jar:/home/picuy/minifi-0.1.0/./lib/nifi-security-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-site-to-site-reporting-task-1.0.0.jar:/home/picuy/minifi-0.1.0/./lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/home/picuy/minifi-0.1.0/./lib/jersey-client-1.19.jar:/home/picuy/minifi-0.1.0/./lib/jul-to-slf4j-1.7.12.jar:/home/picuy/minifi-0.1.0/./lib/ecj-4.4.2.jar:/home/picuy/minifi-0.1.0/./lib/guava-18.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-webapp-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/lucene-core-4.10.4.jar:/home/picuy/minifi-0.1.0/./lib/commons-collections4-4.0.jar:/home/picuy/minifi-0.1.0/./lib/apache-jsp-8.0.33.jar:/home/picuy/minifi-0.1.0/./lib/nifi-site-to-site-client-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-site-to-site-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/httpclient-4.4.1.jar:/home/picuy/minifi-0.1.0/./lib/jackson-jaxrs-1.9.2.jar:/home/picuy/minifi-0.1.0/./lib/jetty-xml-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-data-provenance-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/curator-framework-2.11.0.jar:/home/picuy/minifi-0.1.0/./lib/curator-client-2.11.0.jar:/home/picuy/minifi-0.1.0/./lib/javax.servlet-api-3.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jersey-core-1.19.jar:/home/picuy/minifi-0.1.0/./lib/jetty-servlet-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/jetty-servlets-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-runtime-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jackson-annotations-2.6.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-framework-api-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/apache-jsp-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-persistent-provenance-repository-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/javax.el-api-3.0.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-jsp-jdt-2.3.3.jar:/home/picuy/minifi-0.1.0/./lib/minifi-framework-core-0.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-socket-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/javax.json-1.0.4.jar:/home/picuy/minifi-0.1.0/./lib/nifi-ssl-context-service-api-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-user-actions-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/logback-core-1.1.7.jar:/home/picuy/minifi-0.1.0/./lib/jasypt-1.9.2.jar:/home/picuy/minifi-0.1.0/./lib/nifi-processor-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-logging-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-administration-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/httpcore-4.4.1.jar:/home/picuy/minifi-0.1.0/./lib/minifi-runtime-0.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jackson-databind-2.6.1.jar:/home/picuy/minifi-0.1.0/./lib/quartz-2.2.1.jar:/home/picuy/minifi-0.1.0/./lib/apache-el-8.0.33.jar:/home/picuy/minifi-0.1.0/./lib/jetty-continuation-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/commons-logging-1.2.jar:/home/picuy/minifi-0.1.0/./lib/nifi-framework-cluster-protocol-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/javax.servlet.jsp-api-2.3.1.jar:/home/picuy/minifi-0.1.0/./lib/javax.json-api-1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-client-dto-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/nifi-framework-core-api-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/asm-1.0.2.jar:/home/picuy/minifi-0.1.0/./lib/json-smart-2.1.1.jar:/home/picuy/minifi-0.1.0/./lib/javax.servlet.jsp-2.3.2.jar:/home/picuy/minifi-0.1.0/./lib/asm-3.3.1.jar:/home/picuy/minifi-0.1.0/./lib/javax.el-3.0.1-b08.jar:/home/picuy/minifi-0.1.0/./lib/jackson-xc-1.9.2.jar:/home/picuy/minifi-0.1.0/./lib/slf4j-api-1.7.12.jar:/home/picuy/minifi-0.1.0/./lib/minifi-nar-utils-0.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jersey-json-1.19.jar:/home/picuy/minifi-0.1.0/./lib/nifi-schema-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jackson-mapper-asl-1.9.13.jar:/home/picuy/minifi-0.1.0/./lib/logback-classic-1.1.7.jar:/home/picuy/minifi-0.1.0/./lib/minifi-utils-0.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jettison-1.1.jar:/home/picuy/minifi-0.1.0/./lib/bcprov-jdk15on-1.54.jar:/home/picuy/minifi-0.1.0/./lib/javax.servlet.jsp.jstl-api-1.2.1.jar:/home/picuy/minifi-0.1.0/./lib/jetty-http-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/nifi-expression-language-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jetty-io-9.3.9.v20160517.jar:/home/picuy/minifi-0.1.0/./lib/jackson-core-2.6.1.jar:/home/picuy/minifi-0.1.0/./lib/nifi-utils-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/bcpkix-jdk15on-1.54.jar:/home/picuy/minifi-0.1.0/./lib/nifi-write-ahead-log-1.1.0.jar:/home/picuy/minifi-0.1.0/./lib/jackson-core-asl-1.9.13.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx256m -Xms256m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/home/picuy/minifi-0.1.0/./conf/nifi.properties -Dnifi.bootstrap.listen.port=38084 -Dapp=MiNiFi -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/home/picuy/minifi-0.1.0/logs org.apache.nifi.minifi.MiNiFi

4. Conclusions.

  • You can see the really value of using Ansible when you are provisioning multiple devices. Just execute Ansible command to start performing Linux command and create your own Ansible Playbooks to provision software like Kismet and MiNiFi.
  • Remember I never provisioned / installed an Ansible agent in the Device side, just download my Playbooks in my PC and ready !.
  • For other side you can tweak your Playbooks in order to deploy your services with restricted Linux privileged users. That is required when you are doing Automation in a constrained / restricted devices or VMs.
  • The Kismet and MiNiFi Ansible Playbooks are ready to be used in a PoC, but I don’t recommend to use it in PROD because they need to be improved. For example I have to:
    • Implement them as Ansible Roles.
    • Implement Ansible Tasks to start Kismet and MiNiFi as systemd services with restricted Linux user, no root.
    • Implement Ansible Tasks to read and send in batch the logs or event files for Kismet and MiNiFi to external system as Syslog Server or Solr or Elasticsearch.

In the next blog post I will explain how to integrate/connect each Raspberry Pi (Kismet and MiNiFi) to a centralized Apache NiFi by using Ansible, of course!.

5. References.

Using Ansible with Raspberry Pi cluster.
http://www.pidramble.com

Setup a Headless Raspberry Pi with Raspbian Jessie on OS X.
http://blog.smalleycreative.com/linux/setup-a-headless-raspberry-pi-with-raspbian-jessie-on-os-x/

macOS Sierra SSH “Permission Denied”.
https://rolfje.wordpress.com/2016/11/12/macos-sierra-ssh-permission-denied/

Tagged with: , , , , , , , ,
Posted in Big Data, DevOps, IoT, Security

Data routing, transformation, and system mediation in Big Data & IoT scenarios with Apache NiFi

So a few months ago I published a serie of post explaining how to capture WIFI traffic and process it near to real time by using WSO2 BAM, CEP Siddhi, Apache Cassandra, Apache Thrift, Kismet running on a Raspberry Pi and Docker.

01-wifi-traffic-capture-wso2-bam

Now, after several Big Data and Security projects, I can add to previous solution, fresh air and improve the technological approach.

Using Elasticsearch, Logstash and Kibana

Well, the first approach I considered was starting with ELK stack (Elasticsearch, Logstash and Kibana), that is the natural way to follow.

02-wifi-traffic-capture-elasticsearch-logstash-kibana

 

But, there are still some issues to face:

  • Deal with the resilience.
    • Several times Logstash stops because it was processing a malformed incoming message.
  • Portability.
    • Logstash uses Java, Ruby and should be compiled and tuned for ARM architectures (Raspberry Pi). Yes, there are some instructions to do that, but I don’t want to spent time to do that and I would like to focus on data analysis.
  • Large scaling.
    • I would like to avoid to deploy Logstash in each Raspberry Pi just to transform in JSON the captured 802.11 (WIFI) traffic and send it to Elasticsearch. Other approach what I want to avoid is to deploy Logstash with the UDP/TCP Input Plugin in the Elasticsearch side, because both choices need parse/transform/filter the captured traffic by using GROK and Elasticsearch Index Templates for each Logstash instance deployed. What if I have 100 or more Raspberry Pi distributed in different locations?.
  • Security.
    • I’m using Kismet installed in each Raspberry Pi to capture 802.11 traffic, by default Kismet sends that traffic over UDP, UDP is faster but not secure. The big problem with Logstash listening UDP traffic over a port  is that Logstash is susceptible to DoS attacks and the traffic to be spoofed. I have to update UDP to the “secure UDP”, UDP over SSL/TLS for example.
  • Monitoring/Tracking.
    • How to monitor if Kismet is running in the Raspberry Pi?, How to know if Raspberry Pi is healthy ?.
  • Administrable remotely.
    • Definitely I can’t do that in a massively distributed Raspberry Pi’s.

Then, what can I do ?….

Apache NiFi to the rescue!

03-apache-nifi-logo

I was involved in several Integration Project where I frequently used WSO2 ESB.

WSO2 ESB is based on Apache Synapse, it is a lightweight and high-performance Enterprise Service Bus (ESB). Powered by a fast and asynchronous mediation engine, It provides support for XML, SOAP and REST. It supports HTTP/S, Mail (POP3, IMAP, SMTP), JMS, TCP, UDP, VFS, SMS, XMPP and FIX through “mediators”.

Other opensource and popular choice is Apache Camel. Also We can consider ETL tools such as Pentaho Data Integration (a.k.a Pentaho Kettle), but all them are too heavy to use with/in a Raspberry Pi. Until I found the Apache NiFi.

Taken from Apache NiFi webpage:

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:

Web-based user interface.
– Seamless experience between design, control, feedback, and monitoring
Highly configurable.
– Loss tolerant vs guaranteed delivery
– Low latency vs high throughput
– Dynamic prioritization
– Flow can be modified at runtime
– Back pressure
Data Provenance.
– Track dataflow from beginning to end
Designed for extension
– Build your own processors and more
– Enables rapid development and effective testing
Secure.
– SSL, SSH, HTTPS, encrypted content, etc…
– Multi-tenant authorization and internal authorization/policy management.

What do you think about that? Do you think that Apache NiFi can help me ?. Yes, It does. The new approach would be as follows:

04-wifi-traffic-capture-apache-nifi-minifi

The above choice covers basically all gaps above explained. In the side of Raspberry Pi we could use Apache MiNiFi, a subproject of NiFi suitable for constrained resources. The specific goals comprise:

  • small and lightweight footprint
  • central management of agents
  • generation of data provenanceFor other side, the below choice is also a valid alternative. Even as PoC that demonstrates the ease and the power of using Apache NiFi, this approach is enough.

05-wifi-traffic-capture-apache-nifi

In the next post I will share technical details and code to implement the above approach. Meanwhile I share four great resources:

Conclusions

  • Apache NiFi as system mediator (data routing, transformation, etc.) to does data routing, data streaming, move big data chunks, pull, push and put from/to different sources of data, is the perfect companion for Big Data projects.
  • Apache NiFi speaks different languages through Processors. I can replace Logstash with all Input and Output Plugins easily. I can connect Apache NiFi to Elasticsearch (Put/Fetch Elasticsearch), Apache Hadoop (PutHDFS, FetchHDFS), Twitter, Kafka, etc.

 

Tagged with: , , ,
Posted in BAM, Big Data, DevOps, IoT

Provisioning massively cross-compiled binaries to Raspberry Pi (arm) using Vagrant, VirtualBox, Ansible and Python

If you are involved in an IoT or Mobile Application provisioning Project you probably need build a mechanism to spread your application binaries to all Devices on stock and to all the rolled out Devices.

With this Proof-of-concept I will shown you how to build the app binary provisioning system for your custom platform, in this case I’m going to use Raspberry Pi (ARM processor) quickly avoiding perform unnecessary tasks and providing also an ARM cross-compiling platform.

To implement this I will use Vagrant to create an Ubuntu VM mounts the Raspbian OS image internally ready to be used for ARM cross-compiling. There is a special part in this blog post where explains how to NFS mount to provide remote booting for all Raspberry Pi’s connected to same network.

I provide a new Github repository with all the updated scripts required for this PoC. You can download from here:
I would like to mention that this work is based on https://github.com/twobitcircus/rpi-build-and-boot where I’ve created a Vagrantfile for VirtualBox, tweaked the Ansible Playbook and I have documented the process I’ve followed to make it work successfully in my environment (VirtualBox instead of Parallels and booting from NFS).

Requirements:

I’m using a Mac OS X (El Capitan – Version 10.11.3) with the next tools:

  • VirtualBox 5.0.16
  • Vagrant 1.8.1
  • Ansible 2.0.1.0 (installed via Pip)
  • Python 2.7.11
  • Raspberry Pi 2 Model B
  • Raspbian OS (2015-09-24-raspbian-jessie.img)
  • OpenFramework for cross-compiling (http://openframeworks.cc)

Why Ansible instead of other configuration management tools ?

Why Ansible (http://docs.ansible.com/ansible/intro_installation.html) instead of other configuration management tools as Puppet, Chef, …?. Because, Ansible is simple and agentless; you can use it with just with a simple SSH terminal, nothing special is required to be installed in the Host, also because it is written in Python and as you have seen in my previous post, I’m using intensively Python and it is becoming my favorite programming language. You can install Ansible using the same Python installation tools and obviously, you can import ansible from your Python scripts.
To install Ansible on Mac OS X (El Capitan – Version 10.11.3) is easy, just follow these steps:

$ sudo easy_install pip
$ sudo pip install ansible --quiet

// upgrading Ansible and Pip
$ sudo pip install ansible --upgrade
$ sudo pip install --upgrade pip

Preparing the Raspberry Pi

1. Copy RPi image to SD

Identify the disk (not partition) of your SD card, unmount and copy the image there:

$ diskutil list
$ diskutil unmountDisk /dev/disk2
$ cd /Users/Chilcano/Downloads/@isos_vms/raspberrypi-imgs
$ sudo dd bs=1m if=2015-09-24-raspbian-jessie.img of=/dev/rdisk2

2. Connect the Raspberry Pi directly to your Host (MAC OS X)

Using an ethernet cable, connect your Raspberry Pi to your Host, in my case I’ve a MAC OS X and I’m going to share my WIFI Network connection.
Then, enabling Internet Sharing and the “Thunderbolt Ethernet” an IP address will be assigned to the Raspberry Pi, also Raspberry Pi will have Internet access/Network access and the MAC OS X can connect via SSH to the Raspberry Pi.
All that will be possible without a hub, switch, router, screen or keyboard, etc. This will be useful, because we are going to install new software in Raspberry Pi.

After connect your Raspberry Pi to your MAC OS X, turn on by connecting an USB cable, in your MAC OS X open a Terminal and issue a SSH command, before re-generate the SSH keys.

Note that the default hostname of any Raspberry Pi is raspberrypi.local.

// cleaning existing keys
$ ssh-keygen -R raspberrypi.local

// connect to RPi using `raspberry` as default password
$ ssh pi@raspberrypi.local

After connecting, you will check the assigned IP address and the shared Internet Connection. Now, check out your connection.

pi@raspberrypi:~ $ ping www.docker.com
PING www.docker.com (104.239.220.248) 56(84) bytes of data.
64 bytes from 104.239.220.248: icmp_seq=1 ttl=49 time=212 ms
64 bytes from 104.239.220.248: icmp_seq=2 ttl=49 time=214 ms
^C
--- www.docker.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 6970ms
rtt min/avg/max/mdev = 207.205/213.294/217.893/3.513 ms

3. Configure your RPi

Boot your RPi and open a shell. Then enter:

pi@raspberrypi:~ $ sudo raspi-config

In the raspi-config menu, select Option 1 Expand Filesystem, change Keyboard layout, etc. and reboot.

Just if mirrordirector.raspbian.org mirror is not available, remove http://mirrordirector.raspbian.org/raspbian/ repository and add a newest.

pi@raspberrypi ~ $ sudo nano /etc/apt/sources.list

#deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb http://ftp.cica.es/mirrors/Linux/raspbian/raspbian/ jessie main contrib non-free rpi

# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi

4. Install OpenFrameworks tools and dependencies into Raspberry Pi

Download and unzip OpenFrameworks into RPi under /opt.

pi@raspberrypi:~ $ cd /opt
pi@raspberrypi:/opt $ sudo wget http://openframeworks.cc/versions/v0.9.0/of_v0.9.0_linuxarmv7l_release.tar.gz
pi@raspberrypi:/opt $ sudo tar -zxf of_v0.9.0_linuxarmv7l_release.tar.gz
pi@raspberrypi:/opt $ sudo rm of_v0.9.0_linuxarmv7l_release.tar.gz

Now, update the dependencies required when cross-compiling by running install_dependencies.sh.

pi@raspberrypi:~ $ sudo /opt/of_v0.9.0_linuxarmv7l_release/scripts/linux/debian/install_dependencies.sh

Now, compile oF, compile and execute an oF example.

// compiling oF
pi@raspberrypi:~ $ sudo make Release -C /opt/of_v0.9.0_linuxarmv7l_release/libs/openFrameworksCompiled/project
...
se/libs/openFrameworksCompiled/lib/linuxarmv7l/obj/Release/libs/openFrameworks/math/ofMatrix4x4.o /opt/of_v0.9.0_linuxarmv7l_release/libs/openFrameworksCompiled/lib/linuxarmv7l/obj/Release/libs/openFrameworks/math/ofQuaternion.o /opt/of_v0.9.0_linuxarmv7l_release/libs/openFrameworksCompiled/lib/linuxarmv7l/obj/Release/libs/openFrameworks/math/ofVec2f.o
HOST_OS=Linux
HOST_ARCH=armv7l
checking pkg-config libraries:   cairo zlib gstreamer-app-1.0 gstreamer-1.0 gstreamer-video-1.0 gstreamer-base-1.0 libudev freetype2 fontconfig sndfile openal openssl libpulse-simple alsa gtk+-3.0
Done!
make: Leaving directory '/opt/of_v0.9.0_linuxarmv7l_release/libs/openFrameworksCompiled/project'

// executing an example
pi@raspberrypi:~ $ sudo make -C /opt/of_v0.9.0_linuxarmv7l_release/apps/myApps/emptyExample
pi@raspberrypi:~ $ cd /opt/of_v0.9.0_linuxarmv7l_release/apps/myApps/emptyExample
pi@raspberrypi /opt/of_v0.9.0_linuxarmv7l_release/apps/myApps/emptyExample $ bin/emptyExample

5. Make an new image file from the existing and updated Raspberry Pi

Remove the SD card from the Raspberry Pi, insert the SD card in your Host (in my case is MAC OS X) and use dd to make an new image file.

$ diskutil list
$ diskutil unmountDisk /dev/disk2
$ sudo dd bs=1m if=/dev/rdisk2 of=2015-09-24-raspbian-jessie-of2.img

15279+0 records in
15279+0 records out
16021192704 bytes transferred in 381.968084 secs (41943799 bytes/sec)

Very important:

  • The 2015-09-24-raspbian-jessie-of.img will be shared and after mounted from the guest VM, for that, set the user and permissions to 2015-09-24-raspbian-jessie-of.img as shown below:
$ sudo chmod +x 2015-09-24-raspbian-jessie-of2.img
$ sudo chown Chilcano 2015-09-24-raspbian-jessie-of2.img

$ ls -la
total 110439056
drwxr-xr-x  33 Chilcano  staff         1122 Apr 11 19:12 ./
drwxr-xr-x  35 Chilcano  staff         1190 Mar 23 19:26 ../
-rwxr-xr-x   1 Chilcano  staff  16021192704 Apr 11 17:29 2015-09-24-raspbian-jessie-of1.img*
-rwxr-xr-x   1 Chilcano  staff  16021192704 Apr 11 19:19 2015-09-24-raspbian-jessie-of2.img*
-rwxr-xr-x   1 Chilcano  staff   4325376000 Apr 11 17:02 2015-09-24-raspbian-jessie.img*
-rwxr-xr-x   1 Chilcano  staff  16021192704 Mar 31 12:31 2016-03-18-raspbian-jessie-of1.img*
-rwxr-xr-x   1 Chilcano  staff   4033871872 Apr  5 16:31 2016-03-18-raspbian-jessie.img*
...

Building the Vagrant box

1. In your MAC OS X, to clone the rpi-build-and-boot github repository

$ git clone https://github.com/twobitcircus/rpi-build-and-boot
$ cd rpi-build-and-boot

Copy/Move the newest RPi image created above into rpi-build-and-boot folder.

$ mv /Users/Chilcano/Downloads/@isos_vms/raspberrypi-imgs/2015-09-24-raspbian-jessie-of2.img .

2. Install Vagrant and vbguest plugin into MAC OS X

$ wget https://releases.hashicorp.com/vagrant/1.8.1/vagrant_1.8.1.dmg
$ vagrant plugin install vagrant-vbguest

3. Create a new Vagrantfile with VirtualBox as provider in the same folder rpi-build-and-boot

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  # https://atlas.hashicorp.com/ubuntu/boxes/trusty64 [Official Ubuntu Server 14.04 LTS (Trusty Tahr) builds]
  config.vm.box = "ubuntu/trusty64"
  config.vm.provider "virtualbox" do |vb|
    config.vbguest.auto_update = true
    vb.customize ["modifyvm", :id, "--memory", "6144"]
    vb.customize ["modifyvm", :id, "--cpus", "4"]    
  end
  # If you want to use this system to netboot Raspberry Pi, then uncomment this line
  #config.vm.network "public_network", bridge: "en4: mac-eth0", ip: "10.0.0.1"
  config.vm.network "public_network", bridge: "ask", ip: "10.0.0.1"
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"
  end
end

4. Getting boot and root partitions offsets to do loop mounting in Vagrant

Using ./tool.py offsets I will get the offsets of the boot and root partitions, after getting offset, copy the output of this tool to the top of playbook.yml.
To run tool.py in MAC OS X, you will need Python configured.

$ ./tool.py offsets 2015-09-24-raspbian-jessie-of2.img

    image: 2015-09-24-raspbian-jessie-of2.img
    offset_boot: 4194304
    offset_root: 62914560

The idea to loop-mount the RPi image is to create a full structure of directories and files of a Raspberry Pi distribution under a mounting-point in a Vagrant box. This structure is required to do cross-compiling and move/copy new binaries and ARM cross-compiled binaries.

5. Mounting Raspberry Pi image and booting from Vagrant using NFS

Using ./tool.py netboot image.img /dev/rdiskX [--ip=10.0.0.Y] you will copy just the boot partition in a new and tiny SD card.
This new SD card with a fresh boot partition will be useful to boot from the network/remotely. The RPi will download the root partition from Vagrant, in fact, Vagrant will be sharing the custom RPi image (2015-09-24-raspbian-jessie-of2.img) via NFS to any Raspberry Pi connected to same network and having a pre-loaded boot partition.

The idea behind is to provision a custom RPi image massively avoiding to waste time copying and creating SD card for each Raspberry Pi. Also, this method is useful to provision software, configuration, packages, or in my case, provide cross-compiled software for ARM architectures massively.

$ diskutil list

// a new SD on disk3 will be used
$ diskutil unmountDisk /dev/disk3

$ ./tool.py netboot 2015-09-24-raspbian-jessie-of2.img /dev/rdisk3

2015-09-24-raspbian-jessie-of2.img /dev/rdisk3 10.0.0.101
The following partitions will be destroyed
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *4.0 GB     disk3
   1:             Windows_FAT_32 boot                    58.7 MB    disk3s1
   2:                      Linux                         3.9 GB     disk3s2

are you sure? y
OK
Unmount of all volumes on disk3 was successful
sudo dd if=2015-09-24-raspbian-jessie-of2.img of=/dev/rdisk3 bs=62914560 count=1
Password:
1+0 records in
1+0 records out
62914560 bytes transferred in 6.846875 secs (9188799 bytes/sec)
Disk /dev/rdisk3 ejected

Note that tool.py netboot automatically will assigns to RPi the 10.0.0.101 as IP address and 8.8.8.8 and 8.8.4.4 as DNS servers to eth0.
You can check or modify previously these values by editing the cmdline.txt file placed in the boot RPi partition. You can edit it from a running Raspberry Pi or from a mounted partition.

6. Download and unzip oF (OpenFramework) into rpi-build-and-boot folder

If you forgot copy OpenFramework in your RPi, you can do now. Using the Ansible playbook.yml, the oF will be copied to your RPi.

$ cd rpi-build-and-boot
$ wget http://openframeworks.cc/versions/v0.9.0/of_v0.9.0_linuxarmv7l_release.tar.gz
$ tar -zxf of_v0.9.0_linuxarmv7l_release.tar.gz

7. Update the Ansible playbook.yml

I’ve had to tweak the playbook.yml to avoid warnings, add DNS to cmdline.txt and add iptables filters to get Internet access on RPi using Host shared NIC. Here the updated Ansible playbook.yml:

---
- hosts: all
  remote_user: vagrant
  become: yes
  become_method: sudo
  vars:
    of_version: of_v0.9.0_linuxarmv7l_release
    raspbian_image: 2015-09-24-raspbian-jessie-of2.img
    offset_boot: 4194304
    offset_root: 62914560
  tasks:
    - apt: upgrade=dist update_cache=yes
    - file: path=/opt/raspberrypi state=directory

    - apt: name=nfs-kernel-server
    - lineinfile: dest=/etc/exports line="/opt/raspberrypi/root 10.0.0.0/24(rw,sync,no_root_squash,no_subtree_check)"

    - lineinfile: dest=/etc/cron.d/opt_raspberrypi_root line="* * * * * root /bin/mount /opt/raspberrypi/root" create=yes

    - service: name=nfs-kernel-server state=restarted

    - apt: name=build-essential
    - apt: name=pkg-config
    - apt: name=git
    - apt: name=python-pip
    - apt: name=python-dev
    - apt: name=unzip
    - apt: name=gawk
    - apt: name=libudev-dev

    - apt: name=sshpass

    - pip: name=ansible
    - pip: name=paramiko
    - pip: name=PyYAML
    - pip: name=jinja2
    - pip: name=httplib2

    - apt: name=tinyproxy
    - lineinfile: dest="/etc/tinyproxy.conf" line="Allow 10.0.0.0/8"
    - service: name=tinyproxy state=restarted

    - file: path=/opt/raspberrypi/boot state=directory
    - file: path=/opt/raspberrypi/root state=directory

    - mount: src="/vagrant/{{raspbian_image}}" name="/opt/raspberrypi/boot" fstype="auto"  opts="loop,offset={{offset_boot}},noauto" state="mounted"
    - mount: src="/vagrant/{{raspbian_image}}" name="/opt/raspberrypi/root" fstype="auto"  opts="loop,offset={{offset_root}},noauto" state="mounted"
    - lineinfile: dest=/etc/rc.local line="mount /opt/raspberrypi/root" insertbefore="exit 0"
    - lineinfile: dest=/etc/rc.local line="mount /opt/raspberrypi/boot" insertbefore="exit 0"

    # the rpi is unbootable unless it is told not to mount the root filesystem from the card!. also added dns to cmdline.txt and iptables filter. 
    - lineinfile: dest=/opt/raspberrypi/root/etc/fstab regexp="^\/dev\/mmcblk0p2" state="absent"
    - replace: dest=/opt/raspberrypi/boot/cmdline.txt regexp="rootwait$" replace="dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=deadline root=/dev/nfs rootfstype=nfs nfsroot=10.0.0.1:/opt/raspberrypi/root,udp,vers=3 rw fsck.repair=no rootwait ip=10.0.0.101:10.0.0.1:10.0.0.1:255.255.255.0:rpi:eth0:off:8.8.4.4:8.8.8.8 smsc95xx.turbo_mode=N" backup="no"

    # build helpies
    - file: path=/opt/RPI_BUILD_ROOT state=directory
    - file: src=/opt/raspberrypi/root/etc dest=/opt/RPI_BUILD_ROOT/etc state=link
    - file: src=/opt/raspberrypi/root/lib dest=/opt/RPI_BUILD_ROOT/lib state=link
    - file: src=/opt/raspberrypi/root/opt dest=/opt/RPI_BUILD_ROOT/opt state=link
    - command: rsync -avz /opt/raspberrypi/root/usr/ /opt/RPI_BUILD_ROOT/usr

    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libanl.so.1           dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libanl.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libBrokenLocale.so.1  dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libBrokenLocale.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libcidn.so.1          dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libcidn.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libcrypt.so.1         dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libcrypt.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libdbus-1.so.3.8.13   dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libdbus-1.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libdl.so.2            dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libdl.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libexpat.so.1.6.0     dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libexpat.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libglib-2.0.so.0      dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libglib-2.0.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/liblzma.so.5.0.0      dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/liblzma.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libm.so.6             dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libm.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnsl.so.1           dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnsl.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_compat.so.2    dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_compat.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_dns.so.2       dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_dns.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_files.so.2     dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_files.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_hesiod.so.2    dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_hesiod.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_nisplus.so.2   dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_nisplus.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libnss_nis.so.2       dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libnss_nis.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libpcre.so.3          dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libpcre.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libpng12.so.0         dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libpng12.so.0 state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libresolv.so.2        dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libresolv.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libthread_db.so.1     dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libthread_db.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libusb-0.1.so.4       dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libusb-0.1.so.4 state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libusb-1.0.so.0.1.0   dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libusb-1.0.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libutil.so.1          dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libutil.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libz.so.1.2.8         dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libz.so state=link
    - file: src=/opt/raspberrypi/root/lib/arm-linux-gnueabihf/libudev.so.1.5.0      dest=/opt/raspberrypi/root/usr/lib/arm-linux-gnueabihf/libudev.so state=link

    - file: path=/tmp/CROSS_BUILD_TOOLS state=directory
    - copy: src=build_cross_gcc.sh dest=/tmp/CROSS_BUILD_TOOLS/build_cross_gcc.sh mode=0744
    - shell: /tmp/CROSS_BUILD_TOOLS/build_cross_gcc.sh chdir=/tmp/CROSS_BUILD_TOOLS creates=/opt/cross/bin/arm-linux-gnueabihf-g++

    - lineinfile: dest="/home/vagrant/.profile" line="export GST_VERSION=1.0"
    - lineinfile: dest="/home/vagrant/.profile" line="export RPI_ROOT=/opt/raspberrypi/root"
    #######- lineinfile: dest="/home/vagrant/.profile" line="export RPI_BUILD_ROOT=/opt/RPI_BUILD_ROOT"
    - lineinfile: dest="/home/vagrant/.profile" line="export TOOLCHAIN_ROOT=/opt/cross/bin"
    - lineinfile: dest="/home/vagrant/.profile" line="export PLATFORM_OS=Linux"
    - lineinfile: dest="/home/vagrant/.profile" line="export PLATFORM_ARCH=armv7l"
    - lineinfile: dest="/home/vagrant/.profile" line="export PKG_CONFIG_PATH=$RPI_ROOT/usr/lib/arm-linux-gnueabihf/pkgconfig:$RPI_ROOT/usr/share/pkgconfig:$RPI_ROOT/usr/lib/pkgconfig"

    - unarchive: src={{of_version}}.tar.gz dest=/opt/raspberrypi/root/opt creates=/opt/raspberrypi/root/opt/{{of_version}}
    - file: src={{of_version}} dest=/opt/raspberrypi/root/opt/openframeworks state=link
    - file: src=/opt/raspberrypi/root/opt/openframeworks dest=/opt/openframeworks state=link
    - command: chown -R vagrant /opt/raspberrypi/root/opt/{{of_version}}

    # forwarding traffic from eth0 (internet) to eth1 (rpi connection) with iptables
    - replace: dest=/etc/sysctl.conf regexp="^#net.ipv4.ip_forward=1$" replace="net.ipv4.ip_forward=1"
    - shell: /bin/echo 1 > /proc/sys/net/ipv4/ip_forward
    - command: iptables -A FORWARD -o eth0 -i eth1 -m conntrack --ctstate NEW -j ACCEPT
    - command: iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    - command: iptables -A POSTROUTING -t nat -j MASQUERADE
    - shell: /sbin/iptables-save | /usr/bin/tee /etc/iptables.backup
    - service: name=ufw state=restarted
  handlers:

8. Create the Vagrant box

$ cd rpi-build-and-boot
$ vagrant up --provider virtualbox

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: A newer version of the box 'ubuntu/trusty64' is available! You currently
==> default: have version '20160311.0.0'. The latest is version '20160406.0.0'. Run
==> default: `vagrant box update` to update.
==> default: Setting the name of the VM: rpi-build-and-boot_default_1460455393206_79951
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: bridged
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
...
TASK [service] *****************************************************************
changed: [default]

PLAY RECAP *********************************************************************
default                    : ok=82   changed=76   unreachable=0    failed=0

… let's have coffee ;)

After that, restart the Vagrant box recently created.

$ vagrant halt
==> default: Attempting graceful shutdown of VM...

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: A newer version of the box 'ubuntu/trusty64' is available! You currently
==> default: have version '20160311.0.0'. The latest is version '20160406.0.0'. Run
==> default: `vagrant box update` to update.
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Specific bridge 'ask' not found. You may be asked to specify
==> default: which network to bridge to.
==> default: Available bridged network interfaces:
1) en0: Wi-Fi (AirPort)
2) en1: Thunderbolt 1
3) en2: Thunderbolt 2
4) p2p0
5) awdl0
6) en4: mac-eth0
==> default: When choosing an interface, it is usually the one that is
==> default: being used to connect to the internet.
    default: Which interface should the network bridge to? 6
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: bridged
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
GuestAdditions 5.0.16 running --- OK.
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /vagrant => /Users/Chilcano/1github-repo/rpi-build-and-boot
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

Connect your Raspberry Pi -with the SD card and boot partition copied- using ethernet clable to your Host PC (in my case is a Mac OS X), wait some seconds and check if Raspberry Pi has started from the root partition shared by NFS from the Vagrant box.

$ ping raspberrypi.local
PING raspberrypi.local (10.0.0.101): 56 data bytes
64 bytes from 10.0.0.101: icmp_seq=0 ttl=64 time=0.386 ms
64 bytes from 10.0.0.101: icmp_seq=1 ttl=64 time=0.471 ms
^C
--- raspberrypi.local ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.386/0.428/0.471/0.042 ms

Chilcano@Pisc0 : ~/1github-repo/rpi-build-and-boot
$ ping 10.0.0.101
PING 10.0.0.101 (10.0.0.101): 56 data bytes
64 bytes from 10.0.0.101: icmp_seq=0 ttl=64 time=0.450 ms
64 bytes from 10.0.0.101: icmp_seq=1 ttl=64 time=0.591 ms
^C
--- 10.0.0.101 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.450/0.520/0.591/0.071 ms 

And check if Raspberry Pi is running but from Vagrant box.

$ vagrant ssh
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Tue Apr 12 10:55:29 UTC 2016

  System load:  0.07               Processes:           129
  Usage of /:   11.8% of 39.34GB   Users logged in:     0
  Memory usage: 2%                 IP address for eth0: 10.0.2.15
  Swap usage:   0%                 IP address for eth1: 10.0.0.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud


Last login: Tue Apr 12 10:27:47 2016 from 10.0.2.2

vagrant@vagrant-ubuntu-trusty-64:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:c9:24:d6
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fec9:24d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:665 errors:0 dropped:0 overruns:0 frame:0
          TX packets:427 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:67162 (67.1 KB)  TX bytes:54225 (54.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:b3:e9:a4
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb3:e9a4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29474 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60947 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5247033 (5.2 MB)  TX bytes:70887820 (70.8 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vagrant@vagrant-ubuntu-trusty-64:~$ ping 10.0.0.101
PING 10.0.0.101 (10.0.0.101) 56(84) bytes of data.
64 bytes from 10.0.0.101: icmp_seq=1 ttl=64 time=0.536 ms
64 bytes from 10.0.0.101: icmp_seq=2 ttl=64 time=0.745 ms
64 bytes from 10.0.0.101: icmp_seq=3 ttl=64 time=0.910 ms
^C
--- 10.0.0.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.536/0.730/0.910/0.154 ms
vagrant@vagrant-ubuntu-trusty-64:~$ ping google.com
PING google.com (216.58.211.206) 56(84) bytes of data.
64 bytes from mad01s25-in-f14.1e100.net (216.58.211.206): icmp_seq=1 ttl=63 time=14.1 ms
64 bytes from mad01s25-in-f14.1e100.net (216.58.211.206): icmp_seq=2 ttl=63 time=13.5 ms
64 bytes from mad01s25-in-f14.1e100.net (216.58.211.206): icmp_seq=3 ttl=63 time=13.9 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2008ms
rtt min/avg/max/mdev = 13.521/13.883/14.137/0.296 ms

9. Check if ARM cross-compiling works in the VirtualBox guest

Check if the cross-compiling Variables have been defined.

vagrant@vagrant-ubuntu-trusty-64:~$ cat /home/vagrant/.profile

...
export GST_VERSION=1.0
export RPI_ROOT=/opt/raspberrypi/root
export TOOLCHAIN_ROOT=/opt/cross/bin
export PLATFORM_OS=Linux
export PLATFORM_ARCH=armv7l
export PKG_CONFIG_PATH=$RPI_ROOT/usr/lib/arm-linux-gnueabihf/pkgconfig:$RPI_ROOT/usr/share/pkgconfig:$RPI_ROOT/usr/lib/pkgconfig

Check if RPi has been mounted.

vagrant@vagrant-ubuntu-trusty-64:~$ ll /opt/raspberrypi/boot/
vagrant@vagrant-ubuntu-trusty-64:~$ ll /opt/raspberrypi/root/

And check if oF works by compiling an example.

$ make -C /opt/openframeworks/apps/myApps/emptyExample

Conclusions

  • As you have seen above, using Vagrant, Ansible and Python you can build easily a Provisioning system for massive delivery of binaries/packages for Raspberry Pi or Mobile Devices.
  • Also, you could replace OpenFramework tool (http://openframeworks.cc) used for ARM cross-compiling for other similar Tool if you have different target Device, to do that, just modify the part related to that in the Ansible Playbook.

Finally, in the next blog post, I will explain how to cross-compile the Kismet tool (https://www.kismetwireless.net/download.shtml) from source for Raspberry Pi (ARM).

I hope you have enjoyed.
See you soon.

References:

  1. Loop-mounting partitions from a disk image:
  2. Ansible documentation:
  3. TCPDump cross-compiling for Android:
  4. ARM Cross Compiling with Mac OS X:
  5. Pre-built environment for Raspberry Pi cross-compiling and NFS booting:
  6. How to Build a GCC Cross-Compiler:
  7. A Vagrant plugin to keep your VirtualBox Guest Additions up to date:
  8. openFrameworks – an open source C++ toolkit:
  9. Vboxvfs lacks support for symbolic / hard links
  10. Cross compiler for OF 0.9.0/Jessie/arm6/RPi1
  11. How to cross compile an application for OpenWRT
  12. Cross-Compiling or Building Android tcpdump?
Tagged with: , , , , ,
Posted in DevOps, IoT, Linux

PaaS or micro-PaaS for Microservices? – a simple technology review

“How do you eat an elephant? One bite at a time” – This phrase makes sense, everybody understands It but sometimes is forgotten.
Happily some technology companies have managed to internalize this phrase in its processes and products.

PaaS for Microservices

Below some examples:

Many people and many companies make big mistake when they are entirely focused on big goals. If you have a big goal, you probably spend a lot of time and effort on achieving it.

Well, I will explain on this blog post How this simple concept is being applied to PaaS (Platform as a Service) today and will mention some opensource!.

1. Key concepts

  • Agile
    • You aren’t avoiding the big goal, you are solving the problem step-by-step. And to do so, you need to be organized, forget obsolete methodologies and not waste time.
    • Pre-shaved Yaks (https://www.flickr.com/photos/zimki/243779431/in/photostream).
    • In other words, automates everything you can, organize small teams to create small and independent products, etc.
  • K-I-S-S (Keep It simple, stupid)
    • The road is long and difficult (the learning curve is steep), then it makes it easy and enjoyable, and if the stretch is unavoidable, then try to automate it.
  • Don’t reinvent the wheel
    • Scrum, Kanban, TDD, Unix, Linux, etc. all these are `things that worked before and work now. Please, use them.
  • Free as in beer
    • Free and open source

2. What is PaaS (Platform as a Service)?

PaaS definition - Wikipedia

Wikipedia mentions that Zimki was the first PaaS and was released in the year 2006. Zimki was an end-to-end JavaScript web application development and utility computing platform that removed all the repetitive tasks encountered when creating web applications and web services. After of Zimki born other:

  • Google App Engine
  • WSO2 Stratos (Apache Stratos)
  • Redhat’s OpenShift
  • Saleforce’s Heroku
  • Jelastic
  • Etc.

I ask myself, Are they really suitable for creating Microservices today?. In my opinion, Yes, they are suitable but after a heavy lifting and re-designing.
There are good news about that because the main Software Companies are working on that, making them lighter, more agile and versatile, someone are focused to Cloud, to on-Premise, to Containers or to RAD (rapid application development). Just check out Openshift, CloudFoundry, etc.

3. What’s out there?

Well, after searching the internet, the result is a first version of the PaaS list.

PaaS

  1. Zato
    • https://zato.io
    • Open-source ESB, SOA, REST, APIs and cloud integrations in Python.
  2. Flynn
    • https://flynn.io
    • Runs anything that can run on Linux, not just stateless web apps. Includes built-in database appliances (just PostgreSQL right now) and handles TCP traffic as well as HTTP and HTTPS.
    • Supports Java, Go, Node.js, PHP, Python and Ruby as languages.
  3. Deis
    • http://deis.io
    • Open source PaaS that makes it easy to deploy and manage applications on your own servers. Deis builds upon Docker and CoreOS to provide a lightweight PaaS with a Heroku-inspired workflow.
    • Deis can deploy any language or framework using Docker. If you don’t use Docker, Deis also includes Heroku buildpacks for Ruby, Python, Node.js, Java, Clojure, Scala, Play, PHP, Perl, Dart and Go.
  4. Tsuru
  5. Nanobox
    • https://desktop.nanobox.io
    • https://github.com/nanobox-io/nanobox
    • Nanobox allows you to stop configuring environments and just code. It guarantees that any project you start will work the same for anyone else collaborating on the project. When it’s time to launch the project, you’ll know that your production app will work, because it already works on nanobox.
    • Nanobox detects your app type and automatically configures the environment and installs everything your app needs to run (more of 15 programming languages and frameworks)
  6. Otto
    • https://ottoproject.io
    • Development and Deployment Made Easy (successor to Vagrant).
    • Otto knows how to develop and deploy any application on any cloud platform, all controlled with a single consistent workflow to maximize the productivity of you and your team.
  7. Rack
    • https://convox.com
    • https://github.com/convox/rack
    • A Convox Rack is a private Platform-as-a-Service (PaaS). It gives you a place to deploy your web applications and mobile backends without having to worry about managing servers, writing complex deployment recipes, or monitoring process uptime. We call this platform a “Rack”.
  8. Empire
    • http://engineering.remind.com
    • https://github.com/remind101/empire
    • Empire is a control layer on top of Amazon EC2 Container Service (ECS) that provides a Heroku like workflow. It conforms to a subset of the Heroku Platform API, which means you can use the same tools and processes that you use with Heroku, but with all the power of EC2 and Docker.
  9. Dokku
    • http://dokku.viewdocs.io/dokku
    • The smallest PaaS implementation you’ve ever seen.
    • Powered by Docker, you can install Dokku on any hardware. Use it on inexpensive cloud providers.
    • Dokku by default does not provide any datastores such as MySQL or PostgreSQL. You will need to install plugins to handle that.
  10. Gondor
    • https://gondor.io
    • Managed Python hosting with command-line deployment and support for PostgreSQL, Redis, Celery, Elasticsearch and more.
  11. AppFog
    • https://www.ctl.io/appfog
    • AppFog, CenturyLink’s Platform-as-a-Service (PaaS) based on Cloud Foundry, enables developers to focus on writing great cloud-based applications without having to worry about managing the underlying infrastructure.

Microservices frameworks

  1. Dropwizard
    • http://www.dropwizard.io
    • Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.
    • Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.
  2. Ratpack
    • https://ratpack.io
    • Ratpack is a set of Java libraries for building modern HTTP applications. It provides just enough for writing practical, high performance, apps. It is built on Java 8, Netty and reactive principles.
  3. Spark
    • http://sparkjava.com
    • Spark – A micro framework for creating web applications in Java 8 with minimal effort
  4. Vertx
    • http://vertx.io
    • Vert.x is event driven and non blocking. This means your app can handle a lot of concurrency using a small number of kernel threads. Vert.x lets your app scale with minimal hardware.
    • You can use Vert.x with multiple languages including Java, JavaScript, Groovy, Ruby, and Ceylon.
  5. Seneca
    • http://senecajs.org
    • Seneca is a microservices toolkit for Node.js. It helps you write clean, organized code that you can scale and deploy at any time.
  6. Kong
    • https://getkong.org
    • Kong is a scalable, open source API Layer (also known as an API Gateway, or API Middleware). Kong runs in front of any RESTful API and is extended through Plugins, which provide extra functionalities and services beyond the core platform.
    • Kong is built on top of reliable technologies like NGINX and Apache Cassandra, and provides you with an easy to use RESTful API to operate and configure the system.
  7. Unirest
    • http://unirest.io
    • Unirest is a set of lightweight HTTP libraries available in multiple languages (Node.js, Ruby, PHP, Java, Python, Objective-C, .Net).

4. Conclusions

  • As you can see, the trend is to provide a set of tools to do more easy the application development on-premise or/and on-cloud. The idea behind is to remove all the repetitive tasks encountered when creating web applications and web services (aspects related to infrastructure and operations from setting up servers, scaling, configuration, security and backups), the Pre-Shaved Yaks concept was introduced.
  • In other side, there are custom PaaS created from existing lightweight frameworks using Docker Containers, below some references. This confirms, right now, that there are mature tools and frameworks ready to be used in the construction of these platforms.

5. References

Tagged with: , , , , ,
Posted in Cloud, Microservices, PaaS

Running multi-container (WSO2 BAM & MAC Address Lookup) Docker Application using Docker Compose

In my 4 previous blog post I explained each part of this Proof-of-concept, they are:

  1. Analysing Wireless traffic in real time with WSO2 BAM, Apache Cassandra, Complex Event Processor (CEP Siddhi), Apache Thrift and Python:
  2. A Python Microservice in a Docker Container (MAC Address Manufacturer Lookup):

Now, in this blog post I’m going to explain how to run two Docker Containers, the WSO2 BAM and the MAC Address Manufacturer Lookup containers, by using Docker Compose.

// clone 2 repositories
$ git clone https://github.com/chilcano/docker-wso2bam-kismet-poc.git
$ cd docker-wso2bam-kismet-poc
$ git clone https://github.com/chilcano/wso2bam-wifi-thrift-cassandra-poc.git

// run docker compose
$ docker-compose up -d

Starting dockerwso2bamkismetpoc_mac-manuf_1
Starting dockerwso2bamkismetpoc_wso2bam-dashboard-kismet_1

Below a diagram explaining this.

802.11 traffic capture PoC - Docker Compose

Now, if you want to run all together in a few minutes, just runs the Docker Compose Yaml file.
For a deeply explanation, follow the instructions on README.md (https://github.com/chilcano/docker-wso2bam-kismet-poc).

If everything is OK, you will get a huge amount of data (WIFI traffic) stored in Apache Cassandra and a simple Dashboard showing captured MAC Addresses and Manufacturer of the Wireless Devices (PC, Mobiles, WIFI Access Points, Tablets, etc..) around of your Raspberry Pi.

Visualising 802.11 captured traffic with the MAC Address Manufacturer

I hope you will find this blog posts useful.
Bye.

Tagged with: , , , , , , ,
Posted in BAM, IoT, Linux, Microservices, Security, SOA
Archives