Creating custom python packages for your projects

One of the best things I learnt from reading the amazing Good research code handbook, was the power of creating custom python packages for your projects. I had always known how to create packages in python, but never found a nice way to integrate it with my mostly jupyter based workflow. If you’ve been searching for this as well, buckle up!

Step 1 - Create a project env

In conda, you do this as follows:

# feel free to change the env_name and python version
$ conda create --name "demo_env" python=3.8 
$ conda activate demo_env

Step 2 - Create a custom package

My usual project code structure is as shown. The folder that we’re interested in is the src folder. This is where our custom package resides. We can name our package whatever we want; in this case I’ve called it demo_package . The package must contain an __init__.py file along with the modules of the package.

.
|-- data
|-- notebooks
|-- README.md
|-- results
|-- scripts
|-- src
    |-- setup.py
    |-- demo_package
        |-- __init__.py
        |-- module1.py
        |-- module2.py
        |-- module3.py

Next, let’s take a look at the setup.py file. My version usually looks something like this:

from setuptools import find_packages, setup

setup(
    name='demo_package',
    packages=find_packages(),
)

Step 3 - Installing the package

The next step is to install the package. We can do this as follows:

(demo_env) $ **~/demo_project/src** python -m pip install -e .

Remember to activate the demo_env before installing it. Ensure that you run this command from inside the src folder, where setup.py is located.

Step 4 - Integrating with Jupyter

Now for the fun part, making this work with jupyter notebooks. Whenever you normally use a custom package and you update it, you have to restart the jupyter kernel in order for the package to be updated in the kernel. It looks like the jupyter dev team were was also frustrated by this and created a magic command just to solve this: the autoreload command. Here’s how you use it:

%load_ext autoreload
%autoreload 2

Just make sure to put this at the start of each jupyter notebook that you use. That’s it! Start using your custom package as you would any other! It’s that simple! :D