Installing Software#
We’ll use LangChain, an open-source library for making applications with LLMs. We’ll use models from HuggingFace, a website that has tools and models for machine learning.
Exercise: Create new notebook
Create a new Jupyter Notebook called installing by clicking the File-menu in JupyterLab, and then New and Notebook.
If you are asked to select a kernel, choose “Python 3”.
Give the new notebook a name by clicking the File-menu in JupyterLab and then clicking Rename Notebook.
Use the name installing.
Virtual Environments
If you usually work with virtual environments on Fox, you should setup and activate a virtual environment before you continue, see Bonus: Virtual Environments. If you haven’t heard of virtual environments, you can continue without using virtual environments.
Python Packages#
We will use the python package manager pip for installing software.
pip installs software from the Python Package Index.
First, we update pip to the newest version:
pip install --upgrade pip
General LLM Software#
We will install LangChain and HuggingFace software first.
We use huggingface-hub to automatically download models when required.
pip install --upgrade huggingface-hub httpx
We need several packages for working with LangChain and HuggingFace.
pip install --upgrade langchain langchain-community langchain-huggingface
Transformers is the basic technology used in large language models,
so we install the library sentence-transformers.
pip install --upgrade sentence-transformers
Some models use the sentencepiece library, so we will install that as well.
pip install --upgrade sentencepiece
Software for Reading Text Documents#
We will use unstructured for reading documents. Unstructured supports different document formats, like PDFs, Word files and plain text documents.
pip install --upgrade unstructured[all-docs] langchain-unstructured
Search Index#
For the RAG chapter we will use FAISS to search for documents.
pip install --upgrade faiss-cpu
The Language Model#
We’ll use models from HuggingFace, a website that has tools and models for machine learning. Many models are suitable to use with a single GPU. We can use the open-weights models mistralai/Ministral-8B-Instruct-2410 or meta-llama/Llama-3.2-3B-Instruct for our tasks.
Ministral-8B-Instruct-2410 has 8 billion parameters. For comparison, one of the largest LLMs at the time of writing is Llama 3.1, with 405 billion parameters. Still, Ministral-8B-Instruct-2410 is around 16 GB, which makes it a quite large model. To run it, we must have a GPU with at least 20 GB memory. However, we must also have some GPU memory free for processing data. Therefore, practical use of Ministral-8B-Instruct-2410 might require a GPU with 40 GB memory.
The models can also be run without a GPU, but that will be much slower.
Model Location#
We should tell the HuggingFace library where to store its data. If you’re running on Educloud/Fox project ec443 the model is stored at the path below. If you’re running on your own computer, you probably don’t need to specify the model location.
import os
os.environ['HF_HOME'] = '/fp/projects01/ec443/huggingface/cache/'
Optional
If you’re running one of the models that is already downloaded Educloud/Fox project ec443 the model, you can skip this step. If you’re not running on Educloud/Fox project ec443 or want to use a model that isn’t already downloaded, you’ll need to download the model.
You will need a User Access Token from HuggingFace. If you don’t already have a user account on HuggingFace, you must first sign up for one. Click the button “Sign Up” in the upper right corner on HuggingFace.
When you have logged in to HuggingFace with your user account, you can create a User Access Token giving read access by following this guide.
from huggingface_hub import login
login()
Bonus: Virtual Environments#
By default, the pip command installs Python modules or libraries in your user profile, in your default Python environment.
If you use Python for different projects with different libraries, it might happen that your projects require different versions of the same library.
Python supports switching between different versions of the same library with virtual environments.
You can create a virtual environment for each of your projects.
Then, you install all the libraries related to a particular project in the virtual environment for that project.
The virtual environment is often stored in the project folder in a folder called .venv.
Creating a virtual environment#
Let’s create a virtual environment for running large language models.
There are several ways of doing this, but we recommend using python’s built-in venv command.
!python -m venv .venv
Activating the environment#
To activate the virtual environment on the command prompt you can use an activation script:
source .venv/bin/activate
JupyterLab kernel for the environment#
To use the virtual environment in JupyterLab, we must define a kernel for that environment.
! .venv/bin/python -m ipykernel install --user --name LLM --display-name "Python (LLM)"