One of the best things I learnt from reading the amazing Good research code handbook, was the power of creating custom python packages for your projects. I had always known how to create packages in python, but never found a nice way to integrate it with my mostly jupyter based workflow. If you’ve been searching for this as well, buckle up!
Step 1 - Create a project env
In conda, you do this as follows:
# feel free to change the env_name and python version $ conda create --name "demo_env" python=3.8 $ conda activate demo_env
Step 2 - Create a custom package
My usual project code structure is as shown. The folder that we’re interested in is the
src folder. This is where our custom package resides. We can name our package whatever we want; in this case I’ve called it
demo_package . The package must contain an
__init__.py file along with the modules of the package.
. |-- data |-- notebooks |-- README.md |-- results |-- scripts |-- src |-- setup.py |-- demo_package |-- __init__.py |-- module1.py |-- module2.py |-- module3.py
Next, let’s take a look at the
setup.py file. My version usually looks something like this:
from setuptools import find_packages, setup setup( name='demo_package', packages=find_packages(), )
Step 3 - Installing the package
The next step is to install the package. We can do this as follows:
(demo_env) $ **~/demo_project/src** python -m pip install -e .
Remember to activate the
demo_env before installing it. Ensure that you run this command from inside the
src folder, where
setup.py is located.
Notes for the curious
python -m pip ensures that you’re using the right pip that is referenced in your virtual environment.
-e flag is used in order to not have to update the package each time you update it - What an amazing feature!
Step 4 - Integrating with Jupyter
Now for the fun part, making this work with jupyter notebooks. Whenever you normally use a custom package and you update it, you have to restart the jupyter kernel in order for the package to be updated in the kernel. It looks like the jupyter dev team were was also frustrated by this and created a magic command just to solve this: the autoreload command. Here’s how you use it:
%load_ext autoreload %autoreload 2
Just make sure to put this at the start of each jupyter notebook that you use. That’s it! Start using your custom package as you would any other! It’s that simple! :D