TwiTi : Tool for extracting IOCs from tweet

0
31
Linux Smart Enumeration : Tool For Pentesting & CTFs With Verbosity Levels

TwiTi, a tool for extracting IOCs from tweets, can collect a large number of fresh, accurate IOCs.
TwiTi does

  • classifying whether a tweet contains IOCs or not.
  • extracting IOCs from a tweet and also from links mentioned in a tweet.

For more details please refer to our paper,
“#Twiti: Social Listening for Threat Intelligence” (TheWebConf 2021)
Also, you can find supplementary materials of the paper in data directory.

Requirements

Python

pip install -r requirements.txt

Python 3.7.0

Python 3.7.0 was the initial feature release of Python 3.7.

Note

There are now newer bugfix releases of Python 3.7 that supersede 3.7.0 and Python 3.8 is now the latest feature release of Python 3. Get the latest releases of 3.7.x and 3.8.x here. We plan to continue to provide bugfix releases for 3.7.x until mid 2020 and security fixes until mid 2023.

Among the major new features in Python 3.7 are:

  • PEP 539, new C API for thread-local storage
  • PEP 545, Python documentation translations
  • New documentation translations: Japanese, French, and Korean.
  • PEP 552, Deterministic pyc files
  • PEP 553, Built-in breakpoint()
  • PEP 557, Data Classes
  • PEP 560, Core support for typing module and generic types
  • PEP 562, Customization of access to module attributes
  • PEP 563, Postponed evaluation of annotations
  • PEP 564, Time functions with nanosecond resolution
  • PEP 565, Improved DeprecationWarning handling
  • PEP 567, Context Variables
  • Avoiding the use of ASCII as a default text encoding (PEP 538, legacy C locale coercion and PEP 540, forced UTF-8 runtime mode)
  • The insertion-order preservation nature of dict objects is now an official part of the Python language spec.
  • Notable performance improvements in many areas.

Please see What’s New In Python 3.7 for more information.

More resources

  • Online Documentation
  • PEP 537, 3.7 Release Schedule
  • Report bugs at https://bugs.python.org.
  • Help fund Python and its community.

Windows users

  • The binaries for AMD64 will also work on processors that implement the Intel 64 architecture. (Also known as the “x64” architecture, and formerly known as both “EM64T” and “x86-64”.)
  • There are now “web-based” installers for Windows platforms; the installer will download the needed software components at installation time.
  • There are redistributable zip files containing the Windows builds, making it easy to redistribute Python as part of another software package. Please see the documentation regarding Embedded Distribution for more information.

macOS users

  • For 3.7.0, we provide two binary installer options for download. The default variant is 64-bit-only and works on macOS 10.9 (Mavericks) and later systems. We also continue to provide a 64-bit/32-bit variant that works on all versions of macOS from 10.6 (Snow Leopard) on. Both variants now come with batteries-included versions oF Tcl/Tk 8.6 for users of IDLE and other tkinter-based GUI applications; third-party and system versions of Tcl/Tk are no longer used. Consider using the new 10.9 64-bit-only installer variant, unless you are building Python applications that also need to work on older macOS systems.
  • Both python.org installer variants include private copies of OpenSSL 1.1.0. Please carefully read the Important Information displayed during installation for information about SSL/TLS certificate validation and the Install Certificates.command.

Full Changelog

Files

VersionOperating SystemDescriptionMD5 SumFile SizeGPG
Gzipped source tarballSource release41b6595deb4147a1ed517a7d9a58027122745726SIG
XZ compressed source tarballSource releaseeb8c2a6b1447d50813c02714af4681f316922100SIG
macOS 64-bit/32-bit installermacOSfor Mac OS X 10.6 and laterca3eb84092d0ff6d02e42f63a734338e34274481SIG
macOS 64-bit installermacOSfor OS X 10.9 and laterae0717a02efea3b0eb34aadc680dc49827651276SIG
Windows help fileWindows46562af86c2049dd0cc7680348180dca8547689SIG
Windows x86-64 embeddable zip fileWindowsfor AMD64/EM64T/x64cb8b4f0d979a36258f73ed541def10a56946082SIG
Windows x86-64 executable installerWindowsfor AMD64/EM64T/x64531c3fc821ce0a4107b6d2c6a129be3e26262280SIG
Windows x86-64 web-based installerWindowsfor AMD64/EM64T/x643cfdaf4c8d3b0475aaec12ba402d04d21327160SIG
Windows x86 embeddable zip fileWindowsed9a1c028c1e99f5323b9c20723d7d6f6395982SIG
Windows x86 executable installerWindowsebb6444c284c1447e902e87381afeff025506832SIG
Windows x86 web-based installerWindows779c4085464eb3ee5b1a4fffd0eabca41298280SIG

NER

TwiTi utilizes NER model for text processing. NER model should be built before run.
Please refer to ner/README.md for more information.

Run

Run commands below in TwiTi directory

IOC extraction

python -m ioc_extractor –help

Tweet classification

python -m classifier –help