In this blog post, we'll leverage the power of the llama 3 open-source model to build a fast inference chatbot capable of handling multiple PDF documents. We will use Groq, an LPU (Language Processing Unit) inference engine that enables the model to run incredibly fast. We'll use Nomic Embed Text for embeddings, which provides high-performance embeddings with a large token context window. We'll store these embeddings in Chroma DB, a vector store, to achieve exceptional performance. Additionally, we'll explore how to replicate this setup locally.
Components and Tools
- llama 3: Open-source language model
- Groq: LPU inference engine
- Nomic Embed Text: High-performance embedding model
- Chroma DB: Vector store
- Python Libraries: PyPDF2, langchain, Chroma, Groq, and others
Steps to Build the Chatbot
1. Setup Environment
- Install necessary libraries: PyPDF2, langchain, Chroma, Groq, etc.
- Create and activate a virtual environment.
2. Initialize Variables and Load Models
- Load the environment variables and initialize the Groq API key.
- Set up the conversational chain with memory and initialize the embedding model and vector store.
3. Upload and Process PDFs
- Create a function to handle PDF uploads.
- Split the text into chunks for embedding.
- Store the chunks and their metadata in Chroma DB.
4. Chat Functionality
- Implement the chat function to handle user queries.
- Retrieve relevant document sections from the vector store based on the user's question.
- Pass the query and retrieved sections to the llama 3 model via Groq.
- Return the model's answer along with references to the relevant document sections.
5. Local Setup (Optional)
- Adapt the code to run the setup locally using langchain and llama 3 without Groq.
Installationchainlit
langchain
langchain_community
PyPDF2
chromadb
groq
langchain-groq
ollama
python-dotenv
Code Implementation
import PyPDF2
from langchain_community.embeddings import OllamaEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ChatMessageHistory, ConversationBufferMemory
import chainlit as cl
from langchain_groq import ChatGroq
from dotenv import load_dotenv
import os
# Loading environment variables from .env file
load_dotenv()
# Function to initialize conversation chain with GROQ language model
groq_api_key = os.environ['GROQ_API_KEY']
# Initializing GROQ chat with provided API key, model name, and settings
llm_groq = ChatGroq(
groq_api_key=groq_api_key, model_name="llama3-70b-8192",
temperature=0.2)
@cl.on_chat_start
async def on_chat_start():
files = None #Initialize variable to store uploaded files
# Wait for the user to upload files
while files is None:
files = await cl.AskFileMessage(
content="Please upload one or more pdf files to begin!",
accept=["application/pdf"],
max_size_mb=100,# Optionally limit the file size,
max_files=10,
timeout=180, # Set a timeout for user response,
).send()
# Process each uploaded file
texts = []
metadatas = []
for file in files:
print(file) # Print the file object for debugging
# Read the PDF file
pdf = PyPDF2.PdfReader(file.path)
pdf_text = ""
for page in pdf.pages:
pdf_text += page.extract_text()
# Split the text into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=50)
file_texts = text_splitter.split_text(pdf_text)
texts.extend(file_texts)
# Create a metadata for each chunk
file_metadatas = [{"source": f"{i}-{file.name}"} for i in range(len(file_texts))]
metadatas.extend(file_metadatas)
# Create a Chroma vector store
embeddings = OllamaEmbeddings(model="nomic-embed-text")
docsearch = await cl.make_async(Chroma.from_texts)(
texts, embeddings, metadatas=metadatas
)
# Initialize message history for conversation
message_history = ChatMessageHistory()
# Memory for conversational context
memory = ConversationBufferMemory(
memory_key="chat_history",
output_key="answer",
chat_memory=message_history,
return_messages=True,
)
# Create a chain that uses the Chroma vector store
chain = ConversationalRetrievalChain.from_llm(
llm=llm_groq,
chain_type="stuff",
retriever=docsearch.as_retriever(),
memory=memory,
return_source_documents=True,
)
# Sending an image with the number of files
elements = [
cl.Image(name="image", display="inline", path="pic.jpg")
]
# Inform the user that processing has ended.You can now chat.
msg = cl.Message(content=f"Processing {len(files)} files done. You can now ask questions!",elements=elements)
await msg.send()
#store the chain in user session
cl.user_session.set("chain", chain)
@cl.on_message
async def main(message: cl.Message):
# Retrieve the chain from user session
chain = cl.user_session.get("chain")
#call backs happens asynchronously/parallel
cb = cl.AsyncLangchainCallbackHandler()
# call the chain with user's message content
res = await chain.ainvoke(message.content, callbacks=[cb])
answer = res["answer"]
source_documents = res["source_documents"]
text_elements = [] # Initialize list to store text elements
# Process source documents if available
if source_documents:
for source_idx, source_doc in enumerate(source_documents):
source_name = f"source_{source_idx}"
# Create the text element referenced in the message
text_elements.append(
cl.Text(content=source_doc.page_content, name=source_name)
)
source_names = [text_el.name for text_el in text_elements]
# Add source references to the answer
if source_names:
answer += f"\nSources: {', '.join(source_names)}"
else:
answer += "\nNo sources found"
#return results
await cl.Message(content=answer, elements=text_elements).send()
Conclusion
This setup demonstrates how to build a fast inference chatbot using llama 3 and Groq. By leveraging high-performance embedding models and a vector store, the chatbot can efficiently process and respond to queries based on multiple PDF documents. For those interested in running this locally, adapting the code to use langchain and llama 3 without Groq is straightforward.Happy coding!