The blog is about...

Setting up a CI workflow pipeline for a Python project using GitHub actions. Automating CI workflows is a good practice to be followed for any project. For any Python project, the automated CI workflow includes managing dependencies for Python, code linting, code formatting, and handling some test cases. The workflow that is discussed in this blog is setup for a project that can be found in the following ML_NY_Taxi GitHub repo.

1.1) The main components in the CI workflow

The following are the main components, that are usually used, in a CI workflow for the Python Project.

• Setting up Python package dependencies (handled using pip and requirements.txt)

• Code linting (PyLint package has been utilized for this)

• Code formatting (Black package has been utilized for this)

• Handling some test cases (PyTest package has been utilized for this)

In the GitHub Actions workflow used for this project, only one thing is omitted and that is handling some test cases because of the need for the model files in the code repo. Since the model files were not added to the GitHub repo, it is omitted. Otherwise it is a best practice to include handling some test cases.

1.2) Using a Makefile for performing different components of the workflow

A Makefile has been used for adding different components of the workflow. The Makefile can be found in the repo . This includes steps for all the mentioned components.

Testing the various components of the Makefile locally can be performed with the following commands For managing package dependencies, run
make install
For code linting, run
make lint
For code formatting, run
make format
The test cases can be run manually, locally, because the model files were present inside the required directories. For handling test cases, run
make test
For running everything, run
make all

1.3) Setting up a GitHub actions workflow on a repo

In a GitHub repo, a workflow for various different tasks can be setup in the Actions tab. A number of predefined templates are available for automating various workflows. However, none of the available templates were in this project and a workflow was setup by me. Click on Actions -> Setup a workflow yourself -> Copy paste the workflow to the main.yml file -> Commit the changes to the repo.

A main.yml will be created inside the .github/workflows directory in the repo. The workflow file of this project can be found here . One can easily understand the structure of the main.yml file. There is a name which describes the workflow. There is field, push, on which the CI workflow needs to be performed. There are named steps that needs to be performed as part of the workflow. Most of the usual workflow steps are present. It can be noted that handling the test cases are skipped due to the nature of the tests in this particular project. But, it is a good practice to include this for other projects where the nature of test cases are different.

1.4) Displaying a badge in the readme for the GitHub actions workflow execution status

It can also be noted that a badge has been displayed in the project repo's README.md . If any of the steps in the workflow fails, the badge will display a Failing. For the badge to show Passing, all of the steps need to be successful.

Some challenges

Since, this is a relatively simple workflow for CI, I did not face much difficulties regarding the workflow except for handling the test cases using PyTest. I tried using the fixtures feature of PyTest package. The tests worked fine locally since the model directory is present but I could not get it working on the GitHub Actions hence this step has been omitted for this particular project.

Main takeaway

These are some of the new things that I learned in this project. The first is the usage of Makefile for handling different components of the CI automation workflow. The main learning is the use of GitHub Actions to automate some CI workflows.

Next Steps

• Work on some more workflows using the testing component.

• Work on different but useful workflows like deploying applications to cloud environments.