Kali Intelligence Suite : Shall Aid In The Fast, Autonomous, Central, And Comprehensive Collection Of Intelligence By Executing Standard Penetration Testing Tools

0
144

Kali Intelligence Suite (KIS) is an intelligence gathering and data mining tool for penetration testers. It shall aid in the fast, autonomous, central, and comprehensive collection of intelligence by automatically:

  • executing Kali Linux tools (e.g., dnsrecon, gobuster, hydra, nmap, etc.)
  • querying publicly available APIs (e.g., Censys.io, Haveibeenpwned.com, Hunter.io, Securitytrails.com, DNSdumpster.com, Shodan.io, etc.)
  • storing the collected data in a central PostgreSQL database (see next section)
  • providing an interface to query and analyze the gathered intelligence

After the execution of each Kali Linux tool or querying APIs, KIS analyses the collected information and extracts as well as reports interesting information like newly identified user credentials, hosts/domains, TCP/UDP services, HTTP directories, etc. The extracted information is then internally stored in different PostgreSql database tables, which enables the continuous, structured enhancement and re-use of the collected intelligence by subsequently executed Kali Linux tools.

Additional features are:

  • pre-defined dependencies between Kali Linux tools ensure that relevant information like SNMP default community strings or default credentials is known to KIS before trying to access the respective services
  • remembering the execution status of each Kali Linux tool and API query ensures that already executed OS commands are not automatically executed again
  • data imports of scan results of external scanners like Masscan, Nessus, or Nmap
  • supporting the intelligence collection based on virtual hosts (vhost)
  • using a modular approach that allows the fast integration of new Kali Linux tools
  • parallel Kali Linux command execution by using a specifiable number of threads
  • enables users to kill Kali commands via the KIS user interface in case they take too long
  • access public APIs to enhance data with OSINT

Setup and Installation

The latest version of KIS is available at Docker.com. Follow the installation instructions there.

Information about manual installations can be obtained at INSTALL.

KIS’ Data and Collection Model

The following figure illustrates KIS’ data and collection model. Thereby, each node represents a table in the rational database and each solid line between the nodes documents the corresponding relationship. The dashed directed graphs document based on which already collected intelligence (source node) KIS is able to collect further information (destination node). The labels of the directed graphs document the techniques used by KIS to perform the collection.

Scoping the Engagement

Scoping is an essential feature of KIS, which specifies on which IP networks, IP addresses, host names, etc.,
KIS is allowed to collect data (e.g., via OSINT or active scans) from. Before diving into scoping, it is important to understand the following collection types, which are supported by KIS:

  • Passive: Passive collections do not directly interact with the targets but obtain the information from third-party sources like whois. Per default, KIS automatically executes these collections and, thereby, no scoping is required.
  • Active: Active collections directly interact with the targets by for example actively scanning them. Thus, in contrast to passive collections, these type of collection requires permission from the target’s owner and, therefore, KIS does not automatically perform active collections unless the targets are explicitly marked as in scope.
  • Active*: Active* collections are actually passive collections. Nevertheless, as accessing some third-party sources is somehow limited (e.g., querying certain sources like Shodan.io cost credits), they are treated like active collectors, and, as a result, targets must be marked as in scope in order to perform active* collections on them.

Scopes can be set on the following items by using the script kismanage:

  • IP networks and IP addresses: For IP networks the following scope types can be set:
    • all: Sets the given IP network (e.g., 192.168.1.0/24) together with all IP addresses (e.g., 192.168.1.1) that are within this network range in scope. As a result, KIS automatically executes any active and active* collectors on such IP networks and IP address. This scope type is useful during penetration tests where the scope is limited to certain IP networks and all their IP address. The following listing provides an example on how this scope type is set during the initial intel collection setup:

#create a new workspace example
docker exec -it kaliintelsuite kismanage workspace -a example
#add the network 192.168.1.0/24 to workspace example and set the scope to all (default)
docker exec -it kaliintelsuite kismanage network -w example -a 192.168.1.0/24
#add new IP address 192.168.1.1 to workspace example. IP address is automatically in scope due to the network’s scope all
docker exec -it kaliintelsuite kismanage host -w example -a 192.168.1.1
#verify the initial setup
docker exec -it kaliintelsuite bash
kis_shell> kisreport host -w example –csv | csvcut -c “Network (NW)”,”Scope (NW)”,”IP Address (IP)”,”In Scope (IP)” | csvlook
Network (NW) Scope (NW) IP Address (IP) “In Scope (IP)”
192.168.1.0/24 all 192.168.1.1 True
kis_shell> exi
t

  • exclude: Sets the given IP network (e.g., 192.168.1.0/24) together with all IP addresses (e.g., 192.168.1.1) that are within this network range out of scope. As a result, KIS does not execute any active and active* collectors on this IP network and its IP address.This scope type is the default type for all IP networks and IP addresses that are automatically identified by KIS (e.g., via whois, DNS resolution, etc.). Nevertheless, this scope type can be used to manually exclude networks from scope at a later time.

create a new workspace example
docker exec -it kaliintelsuite kismanage workspace -a example
add the second-level domain megacorpone.com to workspace example and set the scope to all (default)
docker exec -it kaliintelsuite kismanage domain -w example -a megacorpone.com
add new host names to workspace example. The host names are automatically in scope due to the second-level
domain’s scope all
docker exec -it kaliintelsuite kismanage hostname -w example -a www.megacorpone.com ftp.megacorpone.com
verify the initial setup
docker exec -it kaliintelsuite bash
kis_shell> kisreport domain -w example –csv | csvcut -c “Second-Level Domain (SLD)”,”Scope (SLD)”,”Host Name (HN)”,”In Scope (HN)” | csvlook
Second-Level Domain (SLD) | Scope (SLD) | Host Name (HN) | In Scope (HN) |
| megacorpone.com | all | megacorpone.com | True |
| megacorpone.com | all | www.megacorpone.com | True |
| megacorpone.com | all | ftp.megacorpone.com | True |
kis_shell> exit

strict: Sets the given second-level domains (e.g., megacorpone.com) in scope. In contrast to type all, any sub-level domains (e.g., www.megacorpone.com) are not automatically in scope, unless they are explicitly added. As a result, KIS automatically executes any active or active* collectors on such in-scope second-level domains and additionally on those sub-level domains that are explicitly added to the scope.

This type is useful during penetration tests where the scope is limited to certain sub-level domains.

The following listing provides an example on how this scope type is set during the initial intel collection setup:

create a new workspace example
docker exec -it kaliintelsuite kismanage workspace -a example
add the second-level domain megacorpone.com to workspace example and set the scope to strict
docker exec -it kaliintelsuite kismanage domain -w example -a megacorpone.com -s strict
add new host names to workspace example. They are automatically in scope due to kismanage’s default value.
docker exec -it kaliintelsuite kismanage hostname -w example -a www.megacorpone.com ftp.megacorpone.com
verify the initial setup
docker exec -it kaliintelsuite bash
kis_shell> kisreport domain -w example –csv | csvcut -c “Second-Level Domain (SLD)”,”Scope (SLD)”,”Host Name (HN)”,”In Scope (HN)” | csvlook
Second-Level Domain (SLD) | Scope (SLD) | Host Name (HN) | In Scope (HN) |
| megacorpone.com | strict | megacorpone.com | False |
| megacorpone.com | strict | www.megacorpone.com | True |
| megacorpone.com | strict | ftp.megacorpone.com | True |
kis_shell> exit
Note that KIS treats the second-level domain also as a host name. As it has not been explicitly put in scope, it
is still out of scope.

  • exclude: Sets the given second-level domains (e.g., megacorpone.com) together with all sub-level domains out of scope. As a result, KIS does not execute any active and active* collectors on these second-level domains. This scope type is the default type for all second-level domains and their sub-level domains that are automatically identified by KIS (e.g., via extraction from certificates, etc.). Thus, it is not necessary to explicitly set this scope type. Nevertheless, this scope type can be used to manually exclude second-level domains at a later time.

create a new workspace example
docker exec -it kaliintelsuite kismanage workspace -a example
add the second-level domain google.com to workspace example and set the scope to all (default)
docker exec -it kaliintelsuite kismanage domain -w example -a google.com

In this case, KIS is able to, among other things, enumerate any sub-level domains as well as resolve their corresponding IP addresses. In addition, to ensure that KIS scans any host with an IP address within the IP network range 172.217.0.0/16, this network range must be added to KIS with scope type all as well:

add the network 172.217.0.0/16 to workspace example and set the scope to all (default)

docker exec -it kaliintelsuite kismanage network -w example -a 172.217.0.0/16

Example 2: Let’s assume the second-level domain google.com together with all sub-level domains that resolve to any network range are in scope. In this case, the top-level domain google.com is added to the KIS database with scope type all as documented below:

create a new workspace example
docker exec -it kaliintelsuite kismanage workspace -a example
add the second-level domain google.com to workspace example and set the scope to all (default)
docker exec -it kaliintelsuite kismanage domain -w example -a google.com

In this case, KIS is able to, among other things, enumerate any sub-level domains as well as resolve their corresponding IP addresses. In addition, to ensure that KIS scans any host, the network range 0.0.0.0/0 must be added to KIS with scope type all as well:

add network 0.0.0.0/0 to workspace example and set the scope to all (default)
docker exec -it kaliintelsuite kismanage network -w example -a 0.0.0.0/0

List of KIS Collectors

The following table shows the list of existing collectors that are supported by KIS. These collectors are executed by the script kiscollect to create and execute actual OS commands.

The Priority column provides information about the order of execution; the lower the number, the earlier the respective OS commands are created and executed and subsequent collectors can profit from the already collected information. Collectors with a priority of - are not automatically executed as they either require user interaction or additional information (e.g., domain credentials) for execution.

The Name column contains the name of the collector. These names can be added as command line arguments to kiscollect (e.g. --httpnikto). The name also indicates, which underlying OS command is executed.

Column Level specifies whether the collector is operating on:

  • services: Scans services by using IPv4/IPv6 addresses and UDP/TCP port numbers
  • vhosts: Scans web services by using host names (instead of IP addresses) and TCP port numbers
  • hosts: Obtains information based on IPv4/IPv6 addresses
  • domains: Obtains information based on second-level domains and optionally sub-level domains
  • networks: Obtains information based on IPv4/IPv6 network ranges
  • emails: Obtains information based on emails
  • companies: Obtains information based on companies

Column Type specifies whether the collector actively approaches the target (active) or obtains the information from third-party sources (passive and active*).

The IP Support column specifies the IP versions, which are supported by the underlying Kali tool (e.g., gobuster). Kali uses this information to decide which operating systems commands can be created and successfully executed. This column is only relevant for host, network, service, and vhost collectors (see column Level).

Column Timeout specifies the number of seconds after which the collector is automatically terminated.

The column User specifies the user with which the respective operating system commands are executed.

PriorityNameLevelTypeIP SupportTimeoutUser
ftpdotdotpwnserviceActiveIPv4, IPv6nobody
httpdotdotpwnserviceActiveIPv4, IPv6nobody
tftpdotdotpwnserviceActiveIPv4, IPv6nobody
httphydraserviceActiveIPv4, IPv6nobody
rdphydraserviceActiveIPv4, IPv6nobody
smbhydraserviceActiveIPv4, IPv6nobody
smbmedusaserviceActiveIPv4nobody
smbmsfloginserviceActiveIPv4, IPv6root
sshhydraserviceActiveIPv4, IPv6nobody
125builtwithdomainActive*nobody
127hostiodomainActive*nobody
130censysdomaindomainActive*kali
131securitytrailsdomainActive*nobody
132dnsdumpsterdomainActive*nobody
133certspotterdomainActive*nobody
134crtshdomaindomainActive*nobody
135virustotaldomainActive*nobody
140dnssublist3rdomainActivenobody
141dnsamasspassivedomainActive*nobody
142dnsamassactivedomainActivenobody
143dnscrobatdomaindomainActive*nobody
144dnscrobattlddomainActive*nobody
150theharvesterdomainPassivekali
155awsslurpdomainActivenobody
160dnsenumdomainActivenobody
170dnsgobusterdomainActivenobody
180dnsrecondomainActivenobody
210whoisdomaindomainActive30nobody
215dnsspfdomainActivenobody
220dnsdmarcdomainActivenobody
235dnsdkimdomainActivenobody
240dnstakeoverdomainActivenobody
310dnshostdomainActivenobody
312dnshostpublicdomainPassivenobody
320dnsreverselookuphostActiveIPv4, IPv6nobody
360dnscrobatreversehosthostActive*IPv4, IPv6nobody
410hunterdomainActive*nobody
420haveibeenbreachemailActive*nobody
430haveibeenpasteemailActive*nobody
510whoishosthostPassiveIPv4, IPv630nobody
512whoisnetworknetworkPassiveIPv4, IPv630nobody
515reversewhoiscompanyActive*nobody
520shodanhosthostActive*IPv4, IPv6nobody
521shodannetworknetworkActive*IPv4, IPv6nobody
530censyshosthostActive*IPv4kali
540crtshcompanycompanyActive*nobody
550dnscrobatreversenetworknetworkActive*IPv4, IPv6nobody
1100tcpnmapnetworknetworkActiveIPv4, IPv6root
1150tcpnmapdomaindomainActiveIPv4, IPv6root
1200udpnmapnetworknetworkActiveIPv4, IPv6root
1250udpnmapdomaindomainActiveIPv4, IPv6root
1270icmpnmapnetworknetworkActiveIPv4, IPv6root
1300tcpmasscannetworknetworkActiveIPv4root
1305dnsaxfrdomaindomainActivenobody
1306dnsaxfrserviceserviceActive*Ipv4, Ipv6nobody
1320vhostgobusterserviceActiveIPv4, IPv6nobody
1350anyservicenmapserviceActiveIPv4, IPv6root
1820tcptraceroutehostActiveIPv4, IPv6nobody
1900httpmsfrobotstxtservice, vhostActiveIPv4, IPv6root
2000dnsnmapserviceActiveIPv4, IPv6root
2020telnetnmapserviceActiveIPv4, IPv6root
2040vncnmapserviceActiveIPv4, IPv6root
2100mssqlnmapserviceActiveIPv4, IPv6root
2150mysqlnmapserviceActiveIPv4, IPv6root
2200smbnmapserviceActiveIPv4, IPv6root
2250ftpnmapserviceActiveIPv4, IPv6root
2300smtpnmapserviceActiveIPv4, IPv6root
2400rpcnmapserviceActiveIPv4, IPv6root
2500rdpnmapserviceActiveIPv4, IPv6root
2700pop3nmapserviceActiveIPv4, IPv6root
2750msrpcenumserviceActiveIPv4, IPv6root
2800imapnmapserviceActiveIPv4, IPv6root
2850x11nmapserviceActiveIPv4, IPv6root
2900tftpnmapserviceActiveIPv4, IPv6root
2950nfsnmapserviceActiveIPv4, IPv6root
3100fingerserviceActiveIPv4nobody
3200ntpdateserviceActiveIPv4, IPv6nobody
3300ntpqserviceActiveIPv4, IPv6nobody
4000h323versionserviceActiveIPv4, IPv6root
4100sipnmapserviceActiveIPv4, IPv6root
4120sipmsfserviceActiveIPv4, IPv6root
4200stunnmapserviceActiveIPv4, IPv6root
11000vncmsfnoneauthserviceActiveIPv4, IPv6root
11010vncmsfloginserviceActiveIPv4, IPv6root
11100ftphydraserviceActiveIPv4, IPv6nobody
11200mssqlhydraserviceActiveIPv4, IPv6nobody
11400pgsqlhydraserviceActiveIPv4, IPv6nobody
11500snmphydraserviceActiveIPv4, IPv6nobody
11600sshchangemeserviceActiveIPv4, IPv6kali
11610httpchangemeservice, vhostActiveIPv4, IPv6kali
11700ipmiserviceActiveIPv4, IPv6root
11750rmiregistrymsfgatherserviceActiveIPv4, IPv6root
11760rmiregistrynmapserviceActiveIPv4, IPv6root
12100ftpfilelistserviceActiveIPv4, IPv6nobody
13000showmountserviceActiveIPv4, IPv6300nobody
13090smbcmeserviceActiveIpv4, Ipv6kali
13100smbclientserviceActiveIPv4, IPv6nobody
13200smbfilelistserviceActiveIPv4, IPv6nobody
13210smbmapserviceActiveIPv4nobody
21500nbtscanserviceActiveIPv4nobody
21600ldapsearchserviceActiveIPv4, IPv6nobody
21610ldapnmapserviceActiveIPv4, IPv6root
31100snmpcheckserviceActiveIPv4300nobody
31110snmpnmapserviceActiveIPv4, IPv6root
31200onesixtyoneserviceActiveIPv460nobody
31300snmpwalkserviceActiveIPv4nobody
31400oraclesidguessserviceActiveIPv4nobody
41200sshnmapserviceActiveIPv4, IPv6root
41300certnmapservice, vhostActiveIPv4, IPv6root
41310tlsnmapservice, vhostActiveIPv4, IPv6root
41315sslyzeservice, vhostActiveIPv4nobody
41320sslscanservice, vhostActiveIPv4, IPv6nobody
41330certopensslservice, vhostActiveIPv4, IPv6120nobody
51100httpgobusterservice, vhostActiveIPv4, IPv6nobody
51110httpgobustersmartservice, vhostActiveIPv4, IPv6nobody
51150httpkiterunnerservice, vhostActiveIpv4kali
51200httpnmapservice, vhostActiveIPv4, IPv6root
51205httpntlmnmapservice, vhostActiveIPv4, IPv6root
61400rpcclientserviceActiveIPv4, IPv6nobody
61500rpcinfoserviceActiveIPv4, IPv6nobody
71100ikescanserviceActiveIPv4root
91050httpwpscanserviceActiveIPv4, IPv6kali
91100enum4linuxserviceActiveIPv4nobody
91200httpniktoservice, vhostActiveIPv4nobody
91225httpburpsuiteprodomain, hostActiveIPv4, IPv6nobody
91250httpdavtestservice, vhostActiveIPv4, IPv6nobody
91260httpwhatwebserviceActiveIPv4, IPv6nobody
91300httpsqlmapservice, vhostActiveIPv4, IPv6nobody
91400smtpuserenumserviceActiveIPv4nobody
91600mysqlhydraserviceActiveIPv4, IPv6nobody
92200httpwapitiservice, vhostActiveIPv4, IPv6nobody
100000vncviewerserviceActiveIPv4nobody
100100httpeyewitnessservice, vhostActiveIPv4, IPv63600kali

Usage

After the setup, the following KIS commands are available.

kismanage

This script allows:

  • setting up and testing KIS
  • managing the database (re-creation, creating backups, restoring backups, etc.)
  • creating workspaces, networks, host names, emails, companies, etc.
  • importing Nmap, Nessus, and Masscan scan results
  • defining the scope

Run the following command to obtain more information and examples:

docker exec -it kaliintelsuite kismanage -h

kiscollect

This script implements a commandline interface to collect the intelligence.

Run the following command to obtain more information and examples:

docker exec -it kaliintelsuite kiscollect -h

kisreport

This script allows the analysis of the collected data via various filtering options. Supported report formats are:

  • Character-separated values (CSV): Export of the collected intelligence in the structured CSV format. This allows further processing via tools like grep, csvcut, or Aquatone
  • Microsoft Excel: Export of all collected intelligence into a Microsoft Excel file.
  • Text: Export of the collected raw text intelligence (e.g., text output of tool Nikto).
  • Raw: Export of additionally collected files like JSON objects from APIs like Shodan.io, or certificate files.

Run the following command to obtain more information and examples:

docker exec -it kaliintelsuite kisreport -h

LEAVE A REPLY

Please enter your comment!
Please enter your name here