E2E Python solution in DevOps – Part 4: Create Azure DevOps pipeline – staging
In this serie I show you how you can create a PyPi ready solution which is stored on GitHub and has fully automatized build-test-deploy CI/CD pipeline in Azure DevOps. Furthermore you will be able to use it from command line like azure-cli or aws-cli.
Table of content
- Part 1: Basic Python Code
- Part 2: Basic Release diagram
- Part 3: Azure DevOps preparation
- Part 4: Create Azure DevOps pipeline – staging (current)
- Part 5: Create Azure DevOps pipeline – master
- Part 6: Add test cases to project
- Part 7: Expand Python project functionality
Last time we prepared our Azure DevOps project and created the external connections for Github and PyPi.
Today we do something exciting… After several preparation steps today we will create our staging pipeline in Azure DevOps.
Accordinglycreate a pipeline which observes if something is changing in your code in staging branch on Git. Then it compiles the code then uploads to Test-PyPi for testing purpose.
Staging is important from quality perspective. Because this ensures to check your code before production (live).
The pipeline creation for staging branch has the next steps:
- Create basic pipeline (check changes on Git)
- Compile and upload package
- Check result on Azure DevOps
- Check result on Test PyPi
- Test pipeline
1. Create basic pipeline (check changes on Git)
1. Let’s open our project landing page.
And click on “Pipelines” menu item on left side of screen.
Because we don’t have any pipeline you can see the following screen.
2. Here you merely click on New pipeline button
3. On next screen you choose the “Use the visual designer” option under the three other code.
4. Choose GitHub option at Select source screen.
5. Click on three point “…” at Repository part then select the right repository (pypi-project) and right branch (staging) from GitHub. Then click on Continue button.
6. On next screen you should define the pipeline template. Remember we would like to create a pipeline which will build and deploy Python package to Pypi. Therefore you should choose the Python Package template from the list. Then click on Apply button.
Wow. As you can see on next screen we have a pipeline which seem pretty good. 🙂
7. Here we need to add a Name for our pipeline. (Because this is the staging related pipeline, my suggestion is: staging – pypi-project)
Then choose Hosted VS2017 from list under Agent pool.
8. Now we configure – again – the code source. Click on Get sources on middle of screen.
9. In this step you need to repeat the steps you did in point 5. Additional activity to tick Report build status which is important from debug and tracing point of view.
10. Now we have to delete some steps and stages from the default pipeline. So please click on then choose Remove selected the following items:
- Flake
- pytest
And keep only the Build and Test stage.
11. Rename Build and Test stage to Build and Publish. Then choose the None under Parallelism section. And under Run this job choose the Only when all previous jobs have succeeded option.
12. Click on Triggers tab and under Continuous integration choose your git repository. Then on the right side tick Enable continuous integration. And be sure the staging branch is the selected under Branch filters.
2. Compile and upload package
In this section we configure steps inside pipeline to compile our code the upload to Test-PyPi.
13. Click on Variables tab and modify the python.version value to 2.7
14. Go back to Tasks tab then click on Install dependencies task and replace the Script content to this:
python -m pip install --upgrade pip && pip install -r pypi/requirements.txt
15. Now click on + to add a new Command Line task to Build and Publish stage. Then click on Add button.
16. Select newly added task, rename it to Build sdist & bdist_wheel. Then insert into script section the following:
python setup.py sdist bdist_wheel
This task is required to build python package.
17. Add a new Python Twine Upload Authenticate task with + on Tasks tab. Then click to Add button.
18. Select newly added task, rename it to Twine Authenticate to test PyPi – STAGING. Then under Feeds from external organizations choose the test Pypi related connection. (pypi-project on PyPi test)
This will create the connection with Test PyPi to upload your freshly built package.
19. Finally add a new Command Line task with + on Tasks tab. Then click to Add button.
20. Select newly added task, rename it to Use a custom twine task to publish. Then insert into script section the following:
twine upload dist/* -r the1bitdemo --config-file $(PYPIRC_PATH)
Please be sure the value after -r is same you defined at EndpointName in test PyPi connection.
This uploads out package to PyPi where we configured the connection in previous step.
21. Last but not least save out pipeline so click on Save & queue button on screen.
22. In the screen we should add some information to starting such as comment and commit. Then click on Save & queue button. It is in queue now.
3. Check result on Azure DevOps
It is in queue now.
23. To click on Builds on left side of screen then choose staging – pypi-project you can see the status of your build.
It’s great…Let’s check it on Test PyPi
4. Check result on Test PyPi
24. Open in your browser the https://test.pypi.org/
25. Type pypi-project into search field then hit enter.
26. Choose the from the list
YEEEEESSSSS!…This is ours. And anybody can install it.
5. Test pipeline
The final activity for today is testing our pipeline in work.
27. Open your code in a Visual Studio Code. It must be the staging branch.
28. Modify the version number to 0.0.0.3 in the right places.
# Modify: modules/version.py
__version__ = "0.0.0.3"
# Add this to docs/readme.md
* version 0.0.0.3
* Test Azure DevOps pipeline
29. Push the changes to Git
30. Check your pipeline
31. Check the latest package on Test PyPi
We are done. 🙂
Next time we will create the pipeline for master branch. Have a great time until that. 🙂
