TECH

kani – Harnessing Flexibility And Power In Language Model Integration

kani is a lightweight and highly hackable framework for chat-based language models with tool usage/function calling.

Compared to other LM frameworks, kani is less opinionated and offers more fine-grained customizability over the parts of the control flow that matter, making it the perfect choice for NLP researchers, hobbyists, and developers alike.

kani comes with support for the following models out of the box, with a model-agnostic framework to add support for many more:

  • OpenAI Models (GPT-3.5-turbo, GPT-4, GPT-4-turbo)
  • Anthropic Models (Claude, Claude Instant)
  • LLaMA v2 (via Hugging Face or ctransformers) & fine-tunes
  • Vicuna v1.3 (via Hugging Face) & fine-tunes

Interested in contributing? Check out our guide.

Read the docs on ReadTheDocs!

Read our paper on arXiv!

Features

  • Lightweight and high-level – kani implements common boilerplate to interface with language models without forcing you to use opinionated prompt frameworks or complex library-specific tooling.
  • Model agnostic – kani provides a simple interface to implement: token counting and completion generation. Implement these two, and kani can run with any language model.
  • Automatic chat memory management – Allow chat sessions to flow without worrying about managing the number of tokens in the history – kani takes care of it.
  • Function calling with model feedback and retry – Give models access to functions in just one line of code. kani elegantly provides feedback about hallucinated parameters and errors and allows the model to retry calls.
  • You control the prompts – There are no hidden prompt hacks. We will never decide for you how to format your own data, unlike other popular language model libraries.
  • Fast to iterate and intuitive to learn – With kani, you only write Python – we handle the rest.
  • Asynchronous design from the start – kani can scale to run multiple chat sessions in parallel easily, without having to manage multiple processes or programs.

Quickstart

kani requires Python 3.10 or above.

First, install the library. In this quickstart, we’ll use the OpenAI engine, though kani is model-agnostic.

$ pip install "kani[openai]"

Then, let’s use kani to create a simple chatbot using ChatGPT as a backend.

# import the library
from kani import Kani, chat_in_terminal
from kani.engines.openai import OpenAIEngine

# Replace this with your OpenAI API key: https://platform.openai.com/account/api-keys
api_key = "sk-..."

# kani uses an Engine to interact with the language model. You can specify other model 
# parameters here, like temperature=0.7.
engine = OpenAIEngine(api_key, model="gpt-3.5-turbo")

# The kani manages the chat state, prompting, and function calling. Here, we only give 
# it the engine to call ChatGPT, but you can specify other parameters like 
# system_prompt="You are..." here.
ai = Kani(engine)

# kani comes with a utility to interact with a kani through your terminal! Check out 
# the docs for how to use kani programmatically.
chat_in_terminal(ai)

kani makes the time to set up a working chat model short, while offering the programmer deep customizability over every prompt, function call, and even the underlying language model.

For more information click here.

Varshini

Varshini is a Cyber Security expert in Threat Analysis, Vulnerability Assessment, and Research. Passionate about staying ahead of emerging Threats and Technologies.

Recent Posts

Cybersecurity – Tools And Their Function

Cybersecurity tools play a critical role in safeguarding digital assets, systems, and networks from malicious…

15 hours ago

MODeflattener – Miasm’s OLLVM Deflattener

MODeflattener is a specialized tool designed to reverse OLLVM's control flow flattening obfuscation through static…

15 hours ago

My Awesome List : Tools And Their Functions

"My Awesome List" is a curated collection of tools, libraries, and resources spanning various domains…

15 hours ago

Chrome Browser Exploitation, Part 3 : Analyzing And Exploiting CVE-2018-17463

CVE-2018-17463, a type confusion vulnerability in Chrome’s V8 JavaScript engine, allowed attackers to execute arbitrary…

15 hours ago

Chrome Browser Exploitation, Part 1 : Introduction To V8 And JavaScript Internals

The blog post "Chrome Browser Exploitation, Part 1: Introduction to V8 and JavaScript Internals" provides…

16 hours ago

Chrome Browser Exploitation, Part 3: Analyzing and Exploiting CVE-2018-17463

The exploitation of CVE-2018-17463, a type confusion vulnerability in Chrome’s V8 JavaScript engine, relies on…

19 hours ago