How to Develop a Large Language Model Chatbot in 5 Minutes

0. Why This Article Exists

Since the release of ChatGPT, LLMs and LLM-based applications have proliferated over the past two years. Development tools like LangChain and LlamaIndex have emerged, and even some "low-code" drag-and-drop tools are now available. These tools are all powerful, each with its own specific roles and areas of expertise. Additionally, there are UIs like Next Web that are even more user-friendly than ChatGPT itself, making it seem unnecessary to simply develop a chatbot. However, this is the first step in experimenting with LLM interactions. By developing this application, we can gradually become familiar with how to build other powerful LLM-based applications. Today, we will explore how to develop a large language model chatbot in 5 minutes.

The source code for this article is available at: https://github.com/Kit086/LLMChatDemo

1. Preparation

1. First, I don't have a high-performance GPU, so I plan to use a free API. Groq (https://groq.com/) offers a free API (as well as a paid version). They host open-source large language models such as llama3, and their API generates tokens at an extremely fast speed. However, some users have reported that the quality is not great, though I have not compared it with other providers. You can register for a Groq account to generate a key and use their API for free. Groq also has its own ChatGPT-like application, which can be used as an alternative when ChatGPT is down or behaving unexpectedly.
2. This article requires Python version 3.8 or higher. If you have not installed Python, please refer to https://www.python.org/downloads/ for installation instructions.
3. This article uses poetry to manage Python dependencies, so poetry needs to be installed. If you have not installed poetry, please refer to https://python-poetry.org/docs/ for installation instructions. Learning how to use poetry is outside the scope of this article, so please consult the documentation on your own. You may also choose to use any other tool you prefer.

2. Create Project

Run the following command:

# 创建项目
poetry new LLMChatDemo

# 切换到项目目录
cd LLMChatDemo

# 添加依赖
poetry add gradio
poetry add llama-index
poetry add llama-index-llms-groq

3. Development

You will find a directory structure similar to the following in the current path:

LLMChatDemo
├── pyproject.toml
├── README.md
├── LLMChatDemo
│   └── __init__.py
└── tests
    └── __init__.py

First, create a config.json file in the current path and enter the key obtained from groq to avoid hardcoding it directly into the code:

{
    "groq_apikey": "<your-groq-api-key>"
}

I recommend that you immediately create a .gitignore file in the current directory and add config.json to it to prevent accidentally exposing your key when committing code to a git repository in the future.

We can use a tool such as VSCode to open the current directory, create a chatbot_demo.py file in the LLMChatDemo directory that contains __init__.py, and then import the required packages:

from llama_index.llms.groq import Groq
from llama_index.core.llms import ChatMessage
import gradio as gr

import json

llama_index is used to call the Groq API, build conversation messages, and so on. gradio is used to build a simple web interface, and json is used to read configuration files.

Then, we read the configuration file:

with open('config.json', 'r') as file:
    config = json.load(file)

As you can see, the address used in my configuration file is a relative path, because I intend to run this program from the project directory, and my config.json file is also located in the project directory. If you cd into a subdirectory and encounter issues when running the program, please carefully check the error messages.

Next, we create a Groq object named llm:

llm = Groq(model="llama3-70b-8192", api_key=config["groq_apikey"])

As you can see, I am using the llama3-70b-8192 model. This is a 70-billion parameter model that was recently open-sourced by Meta (formerly Facebook), and it ranks among the top open-source LLMs currently available. You can choose from different models provided by groq based on your specific needs.

Next, we create a chatbot function to handle user input messages:

def predict_llm(message, history):
    history_llama_index_messages = [ChatMessage(role="system", content="Please give your answer and translate it into Chinese like a Chinese native speaker.")]

    for human, ai in history:
        history_llama_index_messages.append(ChatMessage(role="user", content=human))
        history_llama_index_messages.append(ChatMessage(role="assistant", content=ai))

    history_llama_index_messages.append(ChatMessage(role="user", content=message))

    resp = llm.stream_chat(history_llama_index_messages)

    partial_message = ""
    for chunk in resp:
        partial_message = partial_message + chunk.delta
        yield partial_message

This function accepts two parameters:

• message: The message entered by the user, i.e., the new message the user just typed.
• history: The conversation history, containing the records of the dialogue between the user and the bot. Since I want our Chatbot to have the ability to remember the context of a specific conversation, I pass the history of messages exchanged in this conversation into this function.

Now let's look inside the function:

• history_llama_index_messages is a list used to store the conversation history, which includes the records of the dialogue between the user and the bot. First, we add the system message, i.e., the ChatMessage with role="system", to the list. The system message is like a prompt from God. Here, you are playing the role of God for llama3, instructing it to do anything. In this case, I instruct it to translate its response into Chinese after generating it, to make it easier for us to read;
• Then, every time the user sends a new message, this function is called once. We reconstruct history_llama_index_messages each time, adding the system message, the conversation history, and the user's new message to the list in order. This for loop is responsible for this task; it adds the conversation history records to the list in the llama_index format;
• history is a collection of tuples, where each tuple contains a human and an ai element, representing the message sent by the human to the AI and the AI's response, respectively. For messages sent by the human, we construct a ChatMessage with role="user"; for the AI's responses, we construct a ChatMessage with role="assistant";
• Then, we add the new message sent by the user to the list as a ChatMessage with role="user";
• Next, we call the llm.stream_chat method, passing in the conversation history to retrieve llama3's response. Here we use stream chat, so the AI outputs the text character by character, just like typing, similar to ChatGPT. However, since groq is extremely fast, you might not notice the difference. We obtain the AI's response response;
• Finally, we return the AI's response response to the user character by character based on the content of the delta field.

Finally, we use Gradio to build a simple web interface:

gr.ChatInterface(
    predict_llm,
    title="Kit's Chatbot",
    description="This is a demo.",
    examples=["你好！", "为什么周树人打了鲁迅，但是鲁迅没有选择用微信报警，而是在 twitter 上发了个帖子来抗议呢？", "在中国，高考满分才 750，怎么才能考 985？"]
    ).launch()

Here, we use Gradio's ChatInterface, passing in the predict_llm function we just defined, along with some parameters. The title and description here serve as the interface's title and description, while examples provides a set of examples to show users how to interact with the Chatbot.

I have prepared several strong questions to test the capabilities of llama3. You can also prepare your own questions to test llama3's capabilities.

Now we can run this program. However, if you want to start the program and allow your friends to access it as well, you can slightly modify the code as follows:

.launch(share=True)

This way, Gradio will automatically generate a URL for you, which you can share with your friends so they can also interact with your Chatbot.

4. Run

Still in the project root directory, which is where config.json is located, run the following command:

# 激活 poetry 环境
poetry shell

# 运行程序
python .\LLMChatDemo\chatbot_demo.py

5. Usage

Above, I asked it 为什么周树人打了鲁迅，但是鲁迅没有选择用微信报警，而是在 twitter 上发了个帖子来抗议呢？. This is a powerful question, and I wanted to see if llama3 could answer it. Here is the Chinese version of its answer:

It successfully cracked my question. This is a very powerful answer. I think llama3-70b will be a very good LLM, and I will continue to use it.

Then I tried a second question: 在中国，高考满分才 750，怎么才能考 985？. Here is its response:

It cracked my question again. It's amazing!

Now let's test if it still remembers the questions we asked:

Although it forgot to translate the answer into Chinese, it still remembers the questions we asked. This might be because my system message was not written well, causing it to misunderstand, or there could be other reasons.