NFStream : A Flexible Network Data Analysis Framework

NFStream is a Python package providing fast, flexible, and expressive data structures designed to make working with online or offline network data both easy and intuitive.

It aims to be the fundamental high-level building block for doing practical, real world network data analysis in Python. Additionally, it has the broader goal of becoming a common network data processing framework for researchers providing data reproducibility across experiments.

Main Features

Performance: nfstream is designed to be fast (x10 faster with pypy3 support) with a small CPU and memory footprint.
Layer-7 visibility: nfstream deep packet inspection engine is based on nDPI. It allows nfstream to perform reliable encrypted applications identification and metadata extraction (e.g. TLS, QUIC, TOR, HTTP, SSH, DNS).
Flexibility: add a flow feature in 2 lines as an NFPlugin.
Machine Learning oriented: add your trained model as an NFPlugin.

How to use it?

Dealing with a big pcap file and just want to aggregate it as network flows? nfstream make this path easier in few lines:

from nfstream import NFStreamer
my_awesome_streamer = NFStreamer(source=”facebook.pcap”) # or network interface (source=”eth0″)
for flow in my_awesome_streamer:
print(flow) # print it, append to pandas Dataframe or whatever you want :)!

NFEntry(
id=0,
first_seen=1472393122365,
last_seen=1472393123665,
version=4,
src_port=52066,
dst_port=443,
protocol=6,
vlan_id=0,
src_ip=’192.168.43.18′,
dst_ip=’66.220.156.68′,
total_packets=19,
total_bytes=5745,
duration=1300,
src2dst_packets=9,
src2dst_bytes=1345,
dst2src_packets=10,
dst2src_bytes=4400,
expiration_id=0,
master_protocol=91,
app_protocol=119,
application_name=’TLS.Facebook’,
category_name=’SocialNetwork’,
client_info=’facebook.com’,
server_info=’*.facebook.com’,
j3a_client=’bfcc1a3891601edb4f137ab7ab25b840′,
j3a_server=’2d1eb5817ece335c24904f516ad5da12′
)

From pcap to Pandas DataFrame?

import pandas as pd
streamer_awesome = NFStreamer(source=’devil.pcap’)
data = []
for flow in streamer_awesome:
data.append(flow.to_namedtuple())
my_df = pd.DataFrame(data=data)
my_df.head(5) # Enjoy!

Didn’t find a specific flow feature? add a plugin to nfstream in few lines:

from nfstream import NFPlugin

class my_awesome_plugin(NFPlugin):
def on_update(self, obs, entry):
if obs.length >= 666:
entry.my_awesome_plugin += 1

streamer_awesome = NFStreamer(source=’devil.pcap’, plugins=[my_awesome_plugin()])
for flow in streamer_awesome:
print(flow.my_awesome_plugin) # see your dynamically created metric in generated flows

Also Read – Blinder : A Python Library To Automate Time-Based Blind SQL Injection

Prerequisites

apt-get install libpcap-dev

Installation

Using pip

Binary installers for the latest released version are available:

pip3 install nfstream

Build from source

If you want to build nfstream on your local machine:

git clone https://github.com/aouinizied/nfstream.git
cd nfstream
python3 setup.py install

Download