TSharkVM : TShark + ELK Analytics Virtual Machine

tsharkVM, this project builds virtual machine which can be used for analytics of tshark -T ek (ndjson) output. The virtual appliance is built using vagrant, which builds Debian 10 with pre-installed and pre-configured ELK stack.

After the VM is up, the process is simple:

  • decoded pcaps (tshark -T ek output / ndjson) are sent over TCP/17570 to the VM
  • ELK stack in VM will process and index the data
  • Kibana is running in VM and can be accessed on http://127.0.0.1:15601/app/kibana#/dashboards

Instructions To Build VM From Ubuntu Desktop

Clone source code

git clone https://github.com/H21lab/tsharkVM.git

Build tshark VM

sudo apt update
sudo apt install tshark virtualbox vagrant
bash ./build.sh

Upload pcaps to VM

#copy your pcaps into ./Trace
#run following script
bash upload_pcaps.sh
#or use tshark directly towards 127.0.0.1 17570/tcp
tshark -r trace.pcapng -x -T ek > /dev/tcp/localhost/17570

Open Kibana with browser

firefox http://127.0.0.1:15601/app/kibana#/dashboards

Open Main Dashboard and increase time window to e.g. last 100 years to see there the sample pcaps.

SSH to VM

cd ./VM
vagrant ssh

Delete VM

cd ./VM
vagrant destroy default

Start VM

cd ./VM
vagrant up

Stop VM

cd ./VM
vagrant halt

SSH into VM and check if ELK is running correctly

cd ./VM
vagrant ssh
sudo systemctl status kibana.service
sudo systemctl status elasticsearch.service
sudo systemctl status logstash.service

Elasticsearch Mapping Template

In the project is included simple Elasticseacrh mapping template generated for the frame,eth,ip,udp,tcp,dhcp protocols. To handle additional protocols efficiently it can be required to update the mapping template in the following way:

  • Create custom mapping, by selecting required protocols
    tshark -G elastic-mapping –elastic-mapping-filter frame,eth,ip,udp,tcp,dns > ./Kibana/custom_tshark_mapping.json
  • Deduplicate and post-process the mapping to fit current Elasticsearch version
    ruby ./Public/process_tshark_mapping_json.rb
  • Upload file to vagrant VM
    cd VM
    vagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.json
    cd ..
  • Connect to VM and upload template in the Elasticsearch
    cd VM
    vagrant ssh
    cd tsharkVM/Kibana
    curl -X PUT “localhost:9200/_index_template/packets_template” -H ‘Content-Type: application/json’ -d@custom_tshark_mapping_deduplicated.json

Alternative can be using the dynamic mapping. See template ./Kibana/template_tshark_mapping_dynamic.json. And consider setting the numeric_detection parameter true/false depending on the mapping requirements and pcaps used. Upload the template into Elasticsearch in similar way as described above.

Limitations

tshark -G elastic-mapping –elastic-mapping-filter mapping could be outdated, it is not following properly the Elasticsearch changes and the output can be duplicated. The manual configuration and post-processing of the mapping template is required.

Program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.

R K

Recent Posts

WID_LoadLibrary : The Intricacies Of DLL Management In Windows

WID_LoadLibrary is a custom implementation inspired by the Windows API function LoadLibrary, which is used…

15 hours ago

Locksmith : A Tool For Securing Active Directory Certificate Services

Locksmith is a specialized tool designed to identify and remediate vulnerabilities in Active Directory Certificate…

15 hours ago

Uscrapper Vanta : A Cutting-Edge OSINT Tool For Advanced Data Extraction

Uscrapper Vanta is a powerful open-source intelligence (OSINT) tool designed to revolutionize web scraping and…

15 hours ago

Pake : Transforming Webpages Into Desktop Applications

Pake is an innovative tool designed to convert any webpage into a desktop application with…

19 hours ago

Bevy : Exploring The Frontier Of Game Development With Rust

Bevy is an open-source, data-driven game engine built in Rust, designed to simplify game development…

19 hours ago

AppFlowy Cloud : Enhancing Collaboration With Secure Cloud Infrastructure

AppFlowy Cloud is a robust component of the AppFlowy ecosystem, designed to provide secure user…

2 days ago