Python Virtual Environments¶
Python has built-in virtual environments, which contains a particular version of python with additional packages/modules that can be activated/deactivated at will. This guide will cover the basics on how to create an environment and install packages on it.
Create a New Virtual Environment¶
This should be done from an interactive session on a compute node.
To get an interactive session on a compute node, run the following:
srun --time=01:00:00 --nodes=1 --cpus-per-task=8 --partition=interactive --pty /usr/bin/bash
Set Up¶
To setup a python virtual environment, you will need to load your specific version of python. To find a list of available versions, you can run:
module spider python
Next, I would have to see if there are any prerequesite modules for python.
module spider <version-of-python-here>
For example:
module spider python/3.8.8-ucekvff
The output of this says that the python/3.8.8-ucekvff module requires gcc/10.2.0-zuvaafu . So we must load it first before we can load the python module.
For example:
module load gcc/10.2.0-zuvaafu
module load python/3.8.8-ucekvff
Before proceeding, ensure that you are in your /work directory and not your /home directory.
After running your module, create your virtual environment by typing in:
python -m venv <name-of-environment>
This will create a directory that contains the environment.
Activate the Environment¶
Activate the environment by running:
$ source <name-of-environment>/bin/activate
Your terminal will now look like this:
(<name-of-environment>) [user@bigram2 jones-lab]
Install packages¶
To search for certain packages, you can type in:
pip search <package-name>
If you wish to install a certain package, you can use:
pip install <package-name>
If you want a certain version of said package, you can give the package name followed by "==" with the version number:
pip install <package-name>==1.0.0
Deactivate the Environment¶
To get out of the environment simply type:
deactivate
(This will not disconnect you from pronto/the node you are currently on)
Use a Virtual Environment in a Batch Job¶
# Load the python module. This should be the same version that was used to create the environment
module load gcc/10.2.0-zuvaafu
module load python/3.8.8-ucekvff
# Activate the environment as before. Replace /work/LAS/jones-lab/<name-of-environment> with the actual path to the environment.
source /work/LAS/jones-lab/<name-of-environment>/bin/activate
# Run your script. It will have access to all of the modules you installed in the environment.
python my_script.py
More Information¶
If you plan on creating more environments, please ensure that you are creating them in /work instead of /home.
For more commands and further documentation, please visit: https://docs.python.org/3/tutorial/venv.html