fbpx
Menu

Linux

technicalthursdaylogo

Technical Thursday – Scheduled VM start in Azure

, , , 0

In this week I would like to show you my latest automation solution for Azure. This is able to start a VM in Azure.

Everybody knows the automatic VM shutdown feature (Microsoft.DevTestLab/schedules) in Azure. It had been debutated in 2016. I love to use it for Azure developer servers because it can save cost and time for me. Nevertheless there is a small gap here. Why can’t start Azure my developer machine when I arrive to the office?

Now I make a ‘FIX” for this gap.

My Azure VM Manager solution helps you to start your VM in Azure at that time when you schedule. So when you arrive to your workplace your VM is up and running every time. 🙂

 

Prerequisites

At the moment the v18.6.0 supports only Linux machines. Especially I have only tested on CentOS 7.

Step-by-step installation

  1. Pull the solution from my git
  2. Create a Service Principal for your account. More information is here
  3. Edit configuration file in config directory 
    {
	"vmName": "<name of your vm>",
	"vmResourceGroup": "<vm resource group>",
	"azure": {
		"cloudName": "AzureCloud",
		"clientID": "<Service Principal ID>",
		"clientSecret": "<Service Principal Secret>",
		"tenant": "<Tenant ID>",
		"subscriptionID": "<Subscription ID>"
	}
}
  4. Save configuration file
  5. Configure crontab according to your update requirement 
    # Edit crontab settings
vim /etc/crontab

### Configure to start your vm at 9AM every weekdays
0 9 * * mon,tue,wed,thu,fri root cd /root/scripts/azvmmanager;bash azvmmanager.sh;

     

  6. Wait for the required time then check the logs according to execution in /var/log/azvmmanager directory 
    less /var/log/azvmmanager/azvmmanager20180614090001.log

    part of log:

    Thu Jun 14 09:00:29 CEST 2018 : # Login success 
Thu Jun 14 09:00:29 CEST 2018 : # Set default subscription 
Thu Jun 14 09:00:38 CEST 2018 : # Default subscription has been set 
Thu Jun 14 09:00:38 CEST 2018 : # Start VM: xxxxxxxxx

     

Let’s check your VM status in Azure. You can see it is up and running…

 

Please do not hesitate to contact me if you have any questions or feedback about this solution or Azure. 🙂

technicalthursdaylogo

Technical Thursday – Dynamic DNS client for Linux (in Azure)

, , , , 0

In the world of clouds there are some home servers and on-premise servers which work hard to do their daily tasks for their owners. The people who own these machines often try to reach them through internet. Due to internet suppliers – who provide dynamic IP for their customers – this is a real challenge sometimes. Luckily there are several good and free dynamic dns sites where we can register our home server to reach it with a dns name through the Internet. Here is a quite fresh list about the most popular: 17 Popular Sites Like No-ip

I used No-ip but I did not like the 30-days confirmation of my host there. I know, this is not a big deal. Additionally I am interested in Azure so the solution – I would like to show you – is a simple step on this way.

I had decided to make a solution on Azure basis which can replace No-ip client on my home server. And now that is ready and enough stable for “PROD” usage.

 

And now…I would like to introduce an alternative for dynamic DNS which works with Azure DNS zone. Sounds good? Let’s see..

 

This solution helps you to update your home server public IP dynamically. This is not 100% free. The monthly cost in case of a “Pay-AS-YOU-GO” subscription is about 1 EUR/month. Additionally you have to register a domain which you can use in Azure (you can do it in Azure).

Dynamic DNS in Azure (by the1bit)

Prerequisites

At the moment the v18.6.0 supports only Linux machines. Especially I have only tested on CentOS 7.

 

Step-by-step installation

  1. Pull the solution from my git
  2. Create a Service Principal for your account. More information is here
  3. Edit configuration file in config directory 
    {
	"zoneName": "<domain name>",
	"aRecordName": "<subdomain>",
	"dnsResourceGroup": "<DNS Zone resource group>",
	"azure": {
		"cloudName": "AzureCloud",
		"clientID": "<Service Principal ID>",
		"clientSecret": "<Service Principal Secret>",
		"tenant": "<Tenant ID>",
		"subscriptionID": "<Subscription ID>"
	}
}
  4. Save configuration file
  5. Configure crontab according to your update requirement 
    # Edit crontab settings
vim /etc/crontab

### Configure to execute at 7AM and 7PM every day
0 7 * * * root cd /root/scripts/azdns;bash azdns.sh;
0 19 * * * root cd /root/scripts/azdns;bash azdns.sh;

     

  6. Wait for the required time then check the logs according to execution in /var/log/azdns directory 
    less /var/log/azdns/azdns20180614070001.log

    part of log file:

    ...
Thu Jun 14 07:00:29 CEST 2018 : # Login success
Thu Jun 14 07:00:29 CEST 2018 : # Set default subscription
Thu Jun 14 07:00:38 CEST 2018 : # Default subscription has been set
Thu Jun 14 07:00:38 CEST 2018 : # Get current Puplic IP from Internet
...

This means you have your own dynamic DNS solution with Azure DNS Zone. I think this is quite cool…

 

Please do not hesitate to contact me if you have any questions or feedback about this solution or Azure. 🙂

 

technicalthursdaylogo

Technical Thursday – CI/CD with Git, Azure and Jenkins

, , , , , , , 3

“How to prepare our CI/CD process?” – This could be the subtitle of this article. Why? Because I will show you how you can start to build a fully automated CI/CD process.

 

What is CI/CD? You can read it on Wikipedia. Nevertheless this is a very important and useful thing nowadays when we work in a DevOps model.

Scenario

In my scenario I would like to copy files from Git to Azure with Jenkins when a commit/push happens to my GitLab. As you can see this is quite complex therefore it’s a good practice example.

Important to know, purpose of this post to show you how can you integrate within some minutes your GitLab and your Jenkins. (So we will use our personal git account for configure connection and we will create the connection between Jenkins and Git over https – and not SSH) This means due to testing purpose we won’t create a very secure integration. 😉

 

Integration

Assumptions and prerequisites

  • You have a Jenkins  environment for automation which is a general used tool.  Installation steps are here.
  • You have a configured GitLab environment
  • You configure your environment only for testing purpose. In other case you have to use different parameters or ssh keys during configuration from security point of view.
  • You store your application in Git.
  • You configured the pipeline solution in our Git.
  • You have an Azure subscription with owner privileges.

 

Step 1: Configure Jenkins

Here you have to install some plugins to Jenkins.

  1. Login to your Jenkins server
  2. Navigate to “Manage Jenkins > Manage Plugins”
  3. From the “Available” tab, find and select the following plugins:
    1. GitLab Plugin
    2. Azure CLI Plugin
    3. Azure Credentials
  4. Click the “Download now and install after restart” button to download it.
  5. Once the plugin has been downloaded, click the “Restart Jenkins…” checkbox and wait for Jenkins to restart.
  6. When Jenkins restarted navigate to “Manage Jenkins > Configure System”
  7. Find the “Git plugin” section and configure Git basic values
  8. Save configuration

 

Step 2: Service Principal in Azure

To be able to upload our files to Azure we have to create a Service Principal which has enough privileges to make it.

  1. Login to a computer where the Azure-Cli 2.0 is installed
  2. Login to the subscription where you would like to create the Service Principal 
    # check relevant cloud infra where you want to login (i.e. AzureGermanCloud, AzureCloud, AzureChinaCloud, ...)
az cloud set --name <name of Cloud>

# Please login to your azure account
az login -u <useraccount>

# Select your subscription
az account set --subscription <subscription ID>
  3. Create a Service Principal with the following command 
    az ad sp create-for-rbac --name <Service Principal name in Azure. eg. JenkinsGitAzure-the1bithu> --query '{"client_id": appId, "secret": password, "tenant": tenant}'

  4. Copy these values to a safety place because we will use it in Jenkins!

Step 3: Credentials in Jenkins

Now set some credentials such as Git, Azure.

  1. Navigate to “Credentials > System”
  2. Choose the “Global credentials” domain
  3. Click on “Add credentials” button
  4. Create GitLab user credential
    1. Kind: Username with password
    2. Username: <your git username>
    3. Password: <your git password>
    4. ID: <An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    5. Then click OK button
  5. Create Azure Service Principal credential (We need the data from Step 2)
    1. Kind: Microsoft Azure Service Principal
    2. Subscription ID: <Azure Storage account subscription ID>
    3. Client ID: <Azure Service Principal Client ID>
    4. Client Secret: <Azure Service Principal Client Secret>
    5. Tenant ID <Azure Service Principal Tenant ID>
    6. Azure Environment: <choose one according to your subscription location>
    7. ID: <An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    8. Then click OK button
  6. Create Azure Storage Account SAS token credential (because our pipeline solution requires the SAS token we have to store somewhere on secure way)
    1. Kind: Secret text
    2. Secret: <paste here the SAS token for storage account. It begins with ‘?sv=’>
    3. ID: <eg. sasTokenAzure | An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    4. Then click OK button
  7. Create Azure Storage Account Account Key credential (because our pipeline solution requires the Account Key we have to store somewhere on secure way)
    1. Kind: Secret text
    2. Secret: <paste here the Account Key for storage account.>
    3. ID: <eg. storageKeyAzure | An internal unique ID by which these credentials are identified from jobs and other configuration. Normally left blank, in which case an ID will be generated, which is fine for jobs created using visual forms. Useful to specify explicitly when using credentials from scripted configuration. >
    4. Then click OK button

 

Step 4: Create Jenkins project

  1. Click on ‘New Item’
  2. Type a name to “Enter an item name” field and choose “Pipeline project” then click on OK button
  3. Build Triggers
    1. Tick “Build when a change is pushed to GitLab. GitLab webhook URL:”
    2. Click on “Advanced” button
    3. Choose the “Filter branches by name” and write in the include filed your branch name
      If you receive an error please ignore because it will be fixed when you will integrate your project with Git. (in Step 5)
    4. Then click on “Generate” button to genetate token for this project.
  4. Pipeline
    1. Select “Piepeline script from SCM” at definition
    2. SCM: Git
    3. Repositories
      1. Paste the clonable url into “Repository URL”. (eg. https://gitlab.com/*****/*****.git)
      2. Credentials. Choose the credential which was created in Step 3.4.
    4. Branches to build. Put here your baranch instead of master. (eg. */master)
    5. Script Path (where the Jenkins file is stored in Git): pipeline/Jenkinsfile
  5. Click Save
  6. Click “Build Now” to test it from Jenkins
  7. Check Console output

Step 5: Integrate GitLab with Jenkins project

  1. Login to GitLab
  2. Step into your project
  3. Navigate to “Settings > Integrations”
  4. Paste the Jenkins project URL (Step 4.3.1)  and Token (Step 4.3.4) to the first two fields
  5. Choose the required Triggers
  6. Uncheck the “Enable SSL verification” (if you use self-signed certificate on Jenkins)
  7. Click “Add webhook” button
  8. Scroll down and find your newly created webhook at middle of screen
  9. Click the “Push events” under the “Test” button dropdown menu.
  10. If you receive a HTTP 200 message with blue background the integration was success

Step 6: Test integration

  1. Modify your project and commit that to this branch
  2. Open Jenkins and check the project
    1. Status after start by Git
    2. Changes where you can see the commit message
    3. Check Console

Awesome…As you can see it works. 🙂

Please kindly notes this is a very basic implementation. If you would like to use it in production you have to configure impersonated accounts for git connections and you have to configure the pipeline solution according to your storage account related data. Additionally the SSH based integration could be better later.

technicalthursdaylogo

Technical Thursday – Lightweight WebUI with Bottle for Python scripts

, , , , , , 0

When you have many python scripts and several very useful feature inside them maybe you are thinking about it: whether can I make a WebUI over them? Then you realize you are not a web developer and you have never made WebUI. So you feel you have to spend hundreds of hours to learn web development or you should skip this option.

I was in this situation then I had found Bottle and I created a website over my python scripts. I know there are some other way to do it such as Django or Flask. For the very first usage the Bottle is a perfect choice.

In this article I would like to provide you a “getting started” guide for Bottle installation and configuration. (more or less this is a link collection)

 

Basics

When I started this I had three things in my mind:

  • I use Visual Studio for development
  • I would like to use it from Azure Web App
  • I would like to use it from a stand alone CentOS Linux

At first we are a very lucky situation because there are several article on Internet regarding this topic. Please kindly notes I am focus on Visual Studio related usage. 🙂

 

Scenarios

1. Visual Studio and Azure

Here is a page for Visual Studio and python relationship: Python feature of Visual Studio

This is the easiest situation because we merely follow the instructions of this article: Publishing to Azure App Service

Then a page for Python web application project templates (this article contains that very useful information to install Python on App Service MS recommends using the site extensions. These extensions are copies of the official releases of Python, optimized and repackaged for Azure App Service.)

 

Note:
– Azure generally uses Windows machines under App services so your web application will be executed by an IIS
– You have to install the python extension under the App service where you would like to hosts your UI
– Bottle uses MVC modell

2. Visual Studio and CentOs

In this scenario we have to configure and prepare our server to bottle. Great thing…their is a very good article for this: How To Deploy Python Web Applications with the Bottle Micro Framework on CentOS 7

Here we need to be careful regarding the app.py related configuration because the original code is wrong for us.

Original code in app.py (service is not reachable from internet – only from localhost):

HOST = os.environ.get('SERVER_HOST', 'localhost')

Correct code:

HOST = os.environ.get('SERVER_HOST', '0.0.0.0')

Note:
– Bottle uses MVC modell

 

Tricks and Tips

Use git for code management

In the scenario 2 that is the most useful if you use a git for manage your code between Visual Studio and CentOs server

 

Modify ‘os.environ.get’ in ‘app.py’

Check the app.py related configuration because the original code is wrong for us.

Original code in app.py (service is not reachable from internet – only from localhost):

HOST = os.environ.get('SERVER_HOST', 'localhost')

Correct code:

HOST = os.environ.get('SERVER_HOST', '0.0.0.0')

 

‘reloader=True’ in ‘app.py’

Use ‘reloader’ in  app.py because in this way after you pull the latest code to your CentOs, Bottle will reload automatically and use the latest code.

bottle.run(server='wsgiref', host=HOST, port=PORT, reloader=True)

 

Use different names for GET and POST

I suggest to use different names for GET and POST method related functions. I mean when you have an upload function you should use def upload():for GET and use def do_upload(): for POST method in your code. Why? In this way you can call directly your upload (GET) function from another function.

 

Learn python code implementation

Finally you should improve your knowledge about existing python code implementation with Bottle: Building a Rest API with the Bottle Framework

 

Favicon 404

When you check the python console in debug mode you can see a 404 error for favicon:

"GET /favicon.ico HTTP/1.1" 404 743

You can manage this according to this article: Serve favicon with bottle.py

 

 

I think this is a very good fundamentals to make your codes available via browser… 🙂

azuredisks

Some words about Azure VM disks

, , 0

I think when you create a Linux VM in Azure there are some important things you need to know. Although most of information are written by Microsoft I feel the repetion and collection is a very good opportunity for everybody to find the related information.

More
AzureOnLinux

Azure command-line interface on Linux

, , , , 0

A day you will meet that situation when you should manage your VMs from a Linux based environment but your Linux skills are not on the top yet. 🙂

Of course Microsoft provides a very good description for Azure-Cli installation, some important Linux side configuration and prerequisites steps are missing from that. Nevertheless you are not in trouble because merely node.js which is required to install.

More