fbpx
Menu

Jenkins

technicalthursdaylogo

Technical Thursday – Jenkins pipeline and AMArETTo

, , , , , , 1

Last week I showed you How you can integrate Git and Jenkins. Inside that post I did not provide script part for Azure related operation. Today I would like to show it.

In Step 4.4.5 we configured a file which is located on our Git. (pipeline/Jenkinsfile). This file is the “link” which can call an upload-to-azure method script. I know you ask: How?

At first I have a good news AMArETTo supports these operations from v0.0.2.9. AMArETTo is available on Git and on PyPi. 🙂

This is the best position for you to create a cool automation solution at your company.

And now let’s see how can we implement the Azure functionality to our Jenkins pipeline.

Step 1: Install AMArETTo to our Jenkins server.

  1. This step is quite easy because we merely should follow the installation steps for AMArETTo. 
    # Install from bash
sudo pip install amaretto

     

Step 2: Create Python script which calls AMArETTo

In this step we will create a small python script which execute the upload function from AMArETTo.

  1. Create uploadtoazure.py file into pipeline directory under your GitLab project’s root. 
    pipeline/uploadtoazure.py
  2. Write a short code which get some external parameters 
    #!/usr/bin/python

# import amaterro
import amaretto
from amaretto import amarettostorage

# import some important packages
import sys
import json

# Get arguments
fileVersion = str(sys.argv[1])
storageaccountName = str(sys.argv[2])
sasToken = str(sys.argv[3])
filePath = str(sys.argv[4])
modificationLimitMin = str(sys.argv[5])

print "--- Upload ---"
uploadFiles = amaretto.amarettostorage.uploadAllFiles(fileVersion = fileVersion, storageaccountName = storageaccountName, sasToken = sasToken, filePath = filePath, modificationLimitMin = modificationLimitMin)

try:
	result = json.loads(uploadFiles)
	print "--- Upload files' result: '{0}' with following message: {1}".format(result["status"], result["result"])
except:
	print "--- Something went wrong during uploading files."

print "-----------------------------"

     

  3.  Create the Jenkinsfile into pipeline directory under your GitLab project’s root. 
    pipeline/Jenkinsfile
  4. Write a valid and lightweight Jenkinsfile code for Python which call our uploadtoazure.py with the right parameters. 
    pipeline {
    agent any
    environment {
        FILE_VERSION = "1.0.0.0"
        AZURE_SA_NAME = "thisismystorage"
        AZURE_SA_SAS = "?sv=..."
        FILE_PATH = "./upload/" 
        MODIFICATION_LIMIT_IN_MINUTES = "30"
    }
    stages {
        stage('Build') {
            steps {
                withCredentials([azureServicePrincipal('c66gbz87-aabb-4096-8192-55d554565fff')]) {
                    sh '''
                        # Login to Azure with ServicePrincipal
                        az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET --tenant $AZURE_TENANT_ID

                        # Set default subscription
                        az account set --subscription $AZURE_SUBSCRIPTION_ID

                        # Execute upload to Azure
                        python pipeline/uploadtoazure.py "$FILE_VERSION" "$AZURE_SA_NAME" "$AZURE_SA_SAS" "$FILE_PATH" "$MODIFICATION_LIMIT_IN_MINUTES"

                        # Logout from Azure
                        az logout --verbose
                    '''
                }
            }
        }
    }
}

     

 

Let me explain the Jenkinsfile. As you can see there is a unfamiliar part above bash code withCredentials(). This comes from Jenkins and this contains the Azure Service Principal related data for our Storage Account. (this was configured in the Step 2 in the post from last week) When you use this credential you have well configured variables which contain the related values such as AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID and AZURE_SUBSCRIPTION_ID. These are fully enough to login Azure.

Step 3: Push files to Git

  1. Finaly we have to push these files to our Git
  2. Then push to Build now button in Jenkins
  3. And check the result 🙂

 

I hope together with previous post you can improve your own Pipeline and provide a cool solution to your management. 😉

 

technicalthursdaylogo

Technical Thursday – CI/CD with Git, Azure and Jenkins

, , , , , , , 3

“How to prepare our CI/CD process?” – This could be the subtitle of this article. Why? Because I will show you how you can start to build a fully automated CI/CD process.

 

What is CI/CD? You can read it on Wikipedia. Nevertheless this is a very important and useful thing nowadays when we work in a DevOps model.

Scenario

In my scenario I would like to copy files from Git to Azure with Jenkins when a commit/push happens to my GitLab. As you can see this is quite complex therefore it’s a good practice example.

Important to know, purpose of this post to show you how can you integrate within some minutes your GitLab and your Jenkins. (So we will use our personal git account for configure connection and we will create the connection between Jenkins and Git over https – and not SSH) This means due to testing purpose we won’t create a very secure integration. 😉

 

Integration

Assumptions and prerequisites

  • You have a Jenkins  environment for automation which is a general used tool.  Installation steps are here.
  • You have a configured GitLab environment
  • You configure your environment only for testing purpose. In other case you have to use different parameters or ssh keys during configuration from security point of view.
  • You store your application in Git.
  • You configured the pipeline solution in our Git.
  • You have an Azure subscription with owner privileges.

 

Step 1: Configure Jenkins

Here you have to install some plugins to Jenkins.

  1. Login to your Jenkins server
  2. Navigate to “Manage Jenkins > Manage Plugins”
  3. From the “Available” tab, find and select the following plugins:
    1. GitLab Plugin
    2. Azure CLI Plugin
    3. Azure Credentials
  4. Click the “Download now and install after restart” button to download it.
  5. Once the plugin has been downloaded, click the “Restart Jenkins…” checkbox and wait for Jenkins to restart.
  6. When Jenkins restarted navigate to “Manage Jenkins > Configure System”
  7. Find the “Git plugin” section and configure Git basic values
  8. Save configuration

 

Step 2: Service Principal in Azure

To be able to upload our files to Azure we have to create a Service Principal which has enough privileges to make it.

  1. Login to a computer where the Azure-Cli 2.0 is installed
  2. Login to the subscription where you would like to create the Service Principal 
    # check relevant cloud infra where you want to login (i.e. AzureGermanCloud, AzureCloud, AzureChinaCloud, ...)
az cloud set --name <name of Cloud>

# Please login to your azure account
az login -u <useraccount>

# Select your subscription
az account set --subscription <subscription ID>
  3. Create a Service Principal with the following command 
    az ad sp create-for-rbac --name <Service Principal name in Azure. eg. JenkinsGitAzure-the1bithu> --query '{"client_id": appId, "secret": password, "tenant": tenant}'

  4. Copy these values to a safety place because we will use it in Jenkins!

Step 3: Credentials in Jenkins

Now set some credentials such as Git, Azure.

  1. Navigate to “Credentials > System”
  2. Choose the “Global credentials” domain
  3. Click on “Add credentials” button
  4. Create GitLab user credential
    1. Kind: Username with password
    2. Username: <your git username>
    3. Password: <your git password>
    4. ID: <An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    5. Then click OK button
  5. Create Azure Service Principal credential (We need the data from Step 2)
    1. Kind: Microsoft Azure Service Principal
    2. Subscription ID: <Azure Storage account subscription ID>
    3. Client ID: <Azure Service Principal Client ID>
    4. Client Secret: <Azure Service Principal Client Secret>
    5. Tenant ID <Azure Service Principal Tenant ID>
    6. Azure Environment: <choose one according to your subscription location>
    7. ID: <An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    8. Then click OK button
  6. Create Azure Storage Account SAS token credential (because our pipeline solution requires the SAS token we have to store somewhere on secure way)
    1. Kind: Secret text
    2. Secret: <paste here the SAS token for storage account. It begins with ‘?sv=’>
    3. ID: <eg. sasTokenAzure | An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    4. Then click OK button
  7. Create Azure Storage Account Account Key credential (because our pipeline solution requires the Account Key we have to store somewhere on secure way)
    1. Kind: Secret text
    2. Secret: <paste here the Account Key for storage account.>
    3. ID: <eg. storageKeyAzure | An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    4. Then click OK button

 

Step 4: Create Jenkins project

  1. Click on ‘New Item’
  2. Type a name to “Enter an item name” field and choose “Pipeline project” then click on OK button
  3. Build Triggers
    1. Tick “Build when a change is pushed to GitLab. GitLab webhook URL:”
    2. Click on “Advanced” button
    3. Choose the “Filter branches by name” and write in the include filed your branch name
      If you receive an error please ignore because it will be fixed when you will integrate your project with Git. (in Step 5)
    4. Then click on “Generate” button to genetate token for this project.
  4. Pipeline
    1. Select “Piepeline script from SCM” at definition
    2. SCM: Git
    3. Repositories
      1. Paste the clonable url into “Repository URL”. (eg. https://gitlab.com/*****/*****.git)
      2. Credentials. Choose the credential which was created in Step 3.4.
    4. Branches to build. Put here your baranch instead of master. (eg. */master)
    5. Script Path (where the Jenkins file is stored in Git): pipeline/Jenkinsfile
  5. Click Save
  6. Click “Build Now” to test it from Jenkins
  7. Check Console output

Step 5: Integrate GitLab with Jenkins project

  1. Login to GitLab
  2. Step into your project
  3. Navigate to “Settings > Integrations”
  4. Paste the Jenkins project URL (Step 4.3.1)  and Token (Step 4.3.4) to the first two fields
  5. Choose the required Triggers
  6. Uncheck the “Enable SSL verification” (if you use self-signed certificate on Jenkins)
  7. Click “Add webhook” button
  8. Scroll down and find your newly created webhook at middle of screen
  9. Click the “Push events” under the “Test” button dropdown menu.
  10. If you receive a HTTP 200 message with blue background the integration was success

Step 6: Test integration

  1. Modify your project and commit that to this branch
  2. Open Jenkins and check the project
    1. Status after start by Git
    2. Changes where you can see the commit message
    3. Check Console

Awesome…As you can see it works. 🙂

Please kindly notes this is a very basic implementation. If you would like to use it in production you have to configure impersonated accounts for git connections and you have to configure the pipeline solution according to your storage account related data. Additionally the SSH based integration could be better later.