Everything generates data: Capturing WIFI anonymous traffic using Raspberry Pi and WSO2 BAM (Part II)
After configuring the Raspberry Pi to capture WIFI/802.11 traffic (first blog post), we have to store this traffic in a Database (NoSQL and RDBMS). The idea is to process in real-time and/or batch the stored data.
To capture the
WIFI/802.11traffic is so difficult and in this second blog post I will explain how to address next difficults:
- Kismet captures 802.11 layer-2 wireless network traffic (Network IP blocks such as TCP, UDP, ARP, and DHCP packets) what should be decoded.
- The traffic should be captured and stored in real-time, we have to use a protocol optimized to capture quickly and low latency.
- The library that implements that protocol should have low memory footprint, because Kismet will run in a Raspberry Pi.
- The protocol to be used should be developer-friendly in both sides (Raspberry Pi side and WSO2 BAM - Apache Cassandra side).
Everything generates data: Capturing WIFI anonymous traffic using Raspberry Pi and WSO2 BAM (Part I)
Yes, in this digital world, everything generates data, but before to do
BigData, you have to follow these steps:
- Capture: Acquires, Integrates data.
- Store: Classification, Consolidate, Transformation, Storage Design, etc.
- Analysis: Exploration, visualization, modeling, prediction, etc.
Trailing and checking the performance and the health of (micro)services are important tasks to be accomplished. The logging is a time consuming process and we have to prepare before in order to be more productive. There are many tools out there, opensource, commercial, on-cloud, such as log.io, ELK, Clarity, rTail, Tailon, frontail, etc. In my opinion, for a VM used to development the most simple, fresh and lightweight tool is rTail.
rTail - Viewing WSO2 and Wiremock raw log events
The log event management is a task very important when working with (Micro)services. If you collect-store-index all logs, then you will be able to create your business metrics (KPI). You only must understand what by collecting logs you have to use special tools with technical special requirements:
- Huge amout of logs could transform in your
- You have to collect and query your logs in real time if you think your Application is critical.
- You have to
managein agile way the life cycle of your data (your logs), they are an asset very important for your Organization.
Kibana - Viewing WSO2 and Wiremock raw log events
- Huge amout of logs could transform in your
The last year I was working in a big UK’s project where I had to help in creation and preparation of the WSO2 Infrastructure to expose a bunch of functionalities as (Micro)services. While we solve everyday problems in infrastructure such as Continuous Delivery, Security, Mediation, Logging, Testing, etc. I had to code many APIs and to explain how to do It to the other team members (Teaching by doing). I will explain how to create an API layer from scratch using WSO2 but avoiding the common errors and bad practices.
If you want some lightweight tool and easy to use to collect, track and visualise all type of events of your logs in real time and from an unique and standard way as a browser, then rTail is your tool. From rTail Github page https://github.com/kilianc/rtail:
rtailis a command line utility that grabs every line in
stdinand broadcasts it over UDP. That’s it. Nothing fancy. Nothing complicated. Tail log files, app output, or whatever you wish, using
rtailbroadcasting to an
rtail-server– See multiple streams in the browser, in realtime.
This VM is suitable to develop with WSO2 products and puts focus only in the server side (WSO2 servers, mock server and different tools to host our microservices) and not in desktop side (Eclipse, SoapUI, Maven, etc.).
Vagrant box - WSO2 Development Server Map
In this post I evaluated the level of integration of WSO2 ESB with different opensource message brokers such as Qpid, RabbitMQ, ActiveMQ and WSO2 Message Broker. At the end, we got the conclusion that RabbitMQ is / was the most used message broker, poorly integrated and not easily integrable with WSO2 ESB.
The OASIS WS-SecurityPolicy language is built on top of W3C WS-Policy TR’s framework and defines a set of policy assertions that can be used in a granular way to apply or to secure at message-level, in part of message, at operation-level or even at transport-level or service-level.
Those individual policy assertions can be combined using policy operators defined in the WS-Policy framework to create security policies that can be used to secure our exposed services in WSO2 ESB and/or WSO2 DSS.
In this post entry I will explain how to apply custom
WS-SecurityPolicyto exposed services of WSO2 ESB 4.8.1 and WSO2 DSS 3.2.1 and will explore some common use cases. Since WSO2 ESB 4.8.1 and WSO2 DSS 3.2.1 have the same security module, any example will work on both.
The Sample 100: Using WS-Security for Outgoing Messages is a great example to start learning with WS-Security, WS-Policy and how to deal with the security of SOAP webservices ….. but the problem is that this example does not work, also that this example is not very well explained. You could get run successfully this example and not understand what happen behind of it.