TwiTi, a tool for extracting IOCs from tweets, can collect a large number of fresh, accurate IOCs.
TwiTi does
- classifying whether a tweet contains IOCs or not.
- extracting IOCs from a tweet and also from links mentioned in a tweet.
For more details please refer to our paper,
“#Twiti: Social Listening for Threat Intelligence” (TheWebConf 2021)
Also, you can find supplementary materials of the paper in data directory.
Requirements
Python
pip install -r requirements.txt
Python 3.7.0
Python 3.7.0 was the initial feature release of Python 3.7.
Note
There are now newer bugfix releases of Python 3.7 that supersede 3.7.0 and Python 3.8 is now the latest feature release of Python 3. Get the latest releases of 3.7.x and 3.8.x here. We plan to continue to provide bugfix releases for 3.7.x until mid 2020 and security fixes until mid 2023.
Among the major new features in Python 3.7 are:
- PEP 539, new C API for thread-local storage
- PEP 545, Python documentation translations
- New documentation translations: Japanese, French, and Korean.
- PEP 552, Deterministic pyc files
- PEP 553, Built-in breakpoint()
- PEP 557, Data Classes
- PEP 560, Core support for typing module and generic types
- PEP 562, Customization of access to module attributes
- PEP 563, Postponed evaluation of annotations
- PEP 564, Time functions with nanosecond resolution
- PEP 565, Improved DeprecationWarning handling
- PEP 567, Context Variables
- Avoiding the use of ASCII as a default text encoding (PEP 538, legacy C locale coercion and PEP 540, forced UTF-8 runtime mode)
- The insertion-order preservation nature of dict objects is now an official part of the Python language spec.
- Notable performance improvements in many areas.
Please see What’s New In Python 3.7 for more information.
More resources
- Online Documentation
- PEP 537, 3.7 Release Schedule
- Report bugs at https://bugs.python.org.
- Help fund Python and its community.
Windows users
- The binaries for AMD64 will also work on processors that implement the Intel 64 architecture. (Also known as the “x64” architecture, and formerly known as both “EM64T” and “x86-64”.)
- There are now “web-based” installers for Windows platforms; the installer will download the needed software components at installation time.
- There are redistributable zip files containing the Windows builds, making it easy to redistribute Python as part of another software package. Please see the documentation regarding Embedded Distribution for more information.
macOS users
- For 3.7.0, we provide two binary installer options for download. The default variant is 64-bit-only and works on macOS 10.9 (Mavericks) and later systems. We also continue to provide a 64-bit/32-bit variant that works on all versions of macOS from 10.6 (Snow Leopard) on. Both variants now come with batteries-included versions oF Tcl/Tk 8.6 for users of IDLE and other tkinter-based GUI applications; third-party and system versions of Tcl/Tk are no longer used. Consider using the new 10.9 64-bit-only installer variant, unless you are building Python applications that also need to work on older macOS systems.
- Both python.org installer variants include private copies of OpenSSL 1.1.0. Please carefully read the Important Information displayed during installation for information about SSL/TLS certificate validation and the Install Certificates.command.
Full Changelog
Files
| Version | Operating System | Description | MD5 Sum | File Size | GPG |
|---|---|---|---|---|---|
| Gzipped source tarball | Source release | 41b6595deb4147a1ed517a7d9a580271 | 22745726 | SIG | |
| XZ compressed source tarball | Source release | eb8c2a6b1447d50813c02714af4681f3 | 16922100 | SIG | |
| macOS 64-bit/32-bit installer | macOS | for Mac OS X 10.6 and later | ca3eb84092d0ff6d02e42f63a734338e | 34274481 | SIG |
| macOS 64-bit installer | macOS | for OS X 10.9 and later | ae0717a02efea3b0eb34aadc680dc498 | 27651276 | SIG |
| Windows help file | Windows | 46562af86c2049dd0cc7680348180dca | 8547689 | SIG | |
| Windows x86-64 embeddable zip file | Windows | for AMD64/EM64T/x64 | cb8b4f0d979a36258f73ed541def10a5 | 6946082 | SIG |
| Windows x86-64 executable installer | Windows | for AMD64/EM64T/x64 | 531c3fc821ce0a4107b6d2c6a129be3e | 26262280 | SIG |
| Windows x86-64 web-based installer | Windows | for AMD64/EM64T/x64 | 3cfdaf4c8d3b0475aaec12ba402d04d2 | 1327160 | SIG |
| Windows x86 embeddable zip file | Windows | ed9a1c028c1e99f5323b9c20723d7d6f | 6395982 | SIG | |
| Windows x86 executable installer | Windows | ebb6444c284c1447e902e87381afeff0 | 25506832 | SIG | |
| Windows x86 web-based installer | Windows | 779c4085464eb3ee5b1a4fffd0eabca4 | 1298280 | SIG |
NER
TwiTi utilizes NER model for text processing. NER model should be built before run.
Please refer to ner/README.md for more information.
Run commands below in TwiTi directory
python -m ioc_extractor –help
Tweet classification
python -m classifier –help
