TwiTi : Tool for extracting IOCs from tweet

TwiTi, a tool for extracting IOCs from tweets, can collect a large number of fresh, accurate IOCs.
TwiTi does

• classifying whether a tweet contains IOCs or not.
• extracting IOCs from a tweet and also from links mentioned in a tweet.

For more details please refer to our paper,
“#Twiti: Social Listening for Threat Intelligence” (TheWebConf 2021)
Also, you can find supplementary materials of the paper in data directory.

Requirements

Python

pip install -r requirements.txt

Python 3.7.0

Python 3.7.0 was the initial feature release of Python 3.7.

Note

There are now newer bugfix releases of Python 3.7 that supersede 3.7.0 and Python 3.8 is now the latest feature release of Python 3. Get the latest releases of 3.7.x and 3.8.x here. We plan to continue to provide bugfix releases for 3.7.x until mid 2020 and security fixes until mid 2023.

Among the major new features in Python 3.7 are:

• PEP 539, new C API for thread-local storage
• PEP 545, Python documentation translations
• New documentation translations: Japanese, French, and Korean.
• PEP 552, Deterministic pyc files
• PEP 553, Built-in breakpoint()
• PEP 557, Data Classes
• PEP 560, Core support for typing module and generic types
• PEP 562, Customization of access to module attributes
• PEP 563, Postponed evaluation of annotations
• PEP 564, Time functions with nanosecond resolution
• PEP 565, Improved DeprecationWarning handling
• PEP 567, Context Variables
• Avoiding the use of ASCII as a default text encoding (PEP 538, legacy C locale coercion and PEP 540, forced UTF-8 runtime mode)
• The insertion-order preservation nature of dict objects is now an official part of the Python language spec.
• Notable performance improvements in many areas.

Please see What’s New In Python 3.7 for more information.

More resources

• Online Documentation
• PEP 537, 3.7 Release Schedule
• Report bugs at https://bugs.python.org.
• Help fund Python and its community.

Windows users

• The binaries for AMD64 will also work on processors that implement the Intel 64 architecture. (Also known as the “x64” architecture, and formerly known as both “EM64T” and “x86-64”.)
• There are now “web-based” installers for Windows platforms; the installer will download the needed software components at installation time.
• There are redistributable zip files containing the Windows builds, making it easy to redistribute Python as part of another software package. Please see the documentation regarding Embedded Distribution for more information.

macOS users

• For 3.7.0, we provide two binary installer options for download. The default variant is 64-bit-only and works on macOS 10.9 (Mavericks) and later systems. We also continue to provide a 64-bit/32-bit variant that works on all versions of macOS from 10.6 (Snow Leopard) on. Both variants now come with batteries-included versions oF Tcl/Tk 8.6 for users of IDLE and other tkinter-based GUI applications; third-party and system versions of Tcl/Tk are no longer used. Consider using the new 10.9 64-bit-only installer variant, unless you are building Python applications that also need to work on older macOS systems.
• Both python.org installer variants include private copies of OpenSSL 1.1.0. Please carefully read the Important Information displayed during installation for information about SSL/TLS certificate validation and the Install Certificates.command.

Full Changelog

Files

Version	Operating System	Description	MD5 Sum	File Size	GPG
Gzipped source tarball	Source release		41b6595deb4147a1ed517a7d9a580271	22745726	SIG
XZ compressed source tarball	Source release		eb8c2a6b1447d50813c02714af4681f3	16922100	SIG
macOS 64-bit/32-bit installer	macOS	for Mac OS X 10.6 and later	ca3eb84092d0ff6d02e42f63a734338e	34274481	SIG
macOS 64-bit installer	macOS	for OS X 10.9 and later	ae0717a02efea3b0eb34aadc680dc498	27651276	SIG
Windows help file	Windows		46562af86c2049dd0cc7680348180dca	8547689	SIG
Windows x86-64 embeddable zip file	Windows	for AMD64/EM64T/x64	cb8b4f0d979a36258f73ed541def10a5	6946082	SIG
Windows x86-64 executable installer	Windows	for AMD64/EM64T/x64	531c3fc821ce0a4107b6d2c6a129be3e	26262280	SIG
Windows x86-64 web-based installer	Windows	for AMD64/EM64T/x64	3cfdaf4c8d3b0475aaec12ba402d04d2	1327160	SIG
Windows x86 embeddable zip file	Windows		ed9a1c028c1e99f5323b9c20723d7d6f	6395982	SIG
Windows x86 executable installer	Windows		ebb6444c284c1447e902e87381afeff0	25506832	SIG
Windows x86 web-based installer	Windows		779c4085464eb3ee5b1a4fffd0eabca4	1298280	SIG

NER

TwiTi utilizes NER model for text processing. NER model should be built before run.
Please refer to ner/README.md for more information.

Run

Run commands below in TwiTi directory

IOC extraction

python -m ioc_extractor –help

Tweet classification

python -m classifier –help