fbpx
Menu

Python

jupiter

Jupyter in Visual Studio Code

, , , , ,

When you work with python codes, day-by-day you want to reach the status where your code is well-managed and you can collaborate with contributors on the most efficient way.

You use git for code management and it most of the cases efficient from collaboration point of view. Nevertheless there are some situation or project where you need more or a little bit different.

For this there is a great “tool”. This is the Jupyter. Since 2014, they do their best to provide cool features for you. šŸ™‚

Today I wouldn’t like to make a Jupyter Notebook and -Hub introduction, merely I would like to share with you a great article about how you can configure your Visual Studio Code for Jupyter.

Getting Started to Work With Jupyter Notebooks in Visual Studio Code

In this article you can find an easy-to-follow description about installation and configuration.

I hope this helps you to start to use Jupyter if it makes sense. šŸ™‚

Don’t forget: you also can use it for another programming languages.

technicalthursdaylogo

Technical Thursday – JSON outputs in Python

, , , , , , 0

Outputs, output formatting…this is a quite interesting topic during your development activities. Every time when you create a function you want to provide reusable code with excellent outputs…but how can you achieve this?

Maybe you feel this is a simple question and the answer is also simple. Nevertheless you realize some weeks later this is a little bit complex area. I mean when you choose a result type or solution you have some direction you can choose.

  • True/False
  • Simple string
  • Nothing šŸ™
  • Result code (0, 1, …)
  • JSON

 

My personal choice is JSON because easy to manage and you can provide several important information for re-usage via that . For example in Azure-Cli every time you receive JSON object which contains the most important information for current activity. Moreover a JSON object is easily managed by Python.

 

And now I show some useful example for JSON output management by Python.

0. Use JSON in python

JSON management is very simple in Python. You merely need to import JSON module.

# Import JSON module
import json

# Define JSON format string
mystring = '{"name": "Python", "version": "2.7"}'

# Convert string to JSON
myjson = json.loads(mystring)

# Use JSON data
print "{0}".format(myjson["name"])

 

1. Return from string

When you have a string in JSON format in your function and you would like to use it as a JSON object you need to convert the string to JSON.

# Define JSON format string
mystring = '{"name": "Python", "version": "2.7"}'    

# my Function
def myFunction(inputString):
    # Import JSON module
    import json

    # Try to convert to JSON
    try:
        # Convert string to JSON
        myjson = json.loads(inputString)
        
        # Return with result
        return myjson["name"]
    except:
        # Error handling
        return False


# Call function
myFunction(mystring)

To convert string to JSON object you can use the “json.loads()” method.

 

2. Return from JSON

Sometimes you have a JSON object and you would like to give back it via your function in string format.

# Define JSON format string
mystring = '{"name": "Python", "version": "2.7"}'    

# my Function
def myFunction(inputString):
    # Import JSON module
    import json

    # Try to convert to JSON
    try:
        # Convert string to JSON
        myjson = json.loads(inputString)
        
        # Return with result
        return json.dumps(myjson)
    except:
        # Error handling
        return False


# Call function
myFunction(mystring)

To convert JSON object to string you can use the “json.dumps()” method.

 

3. Return custom string

Finally you can create string output which is built by you.

# my Function
def myFunction(inputString):
     # Return with result
     return '{"status": "success", "message": "%s"}' % (inputString) 

# Call function
myFunction("This is a string")

Here you should not import json module to create JSON format output

 

Of course can combine them and use other formats…Let’s try them…

technicalthursdaylogo

Technical Thursday – Lightweight WebUI with Bottle for Python scripts

, , , , , , 0

When you have many python scripts and several very useful feature inside them maybe you are thinking about it: whether can I make a WebUI over them? Then you realize you are not a web developer and you have never made WebUI. So you feel you have to spend hundreds of hours to learn web development or you should skip this option.

I was in this situation then I had found Bottle and I created a website over my python scripts. I know there are some other way to do it such as Django or Flask. For the very first usage the Bottle is a perfect choice.

In this article I would like to provide you a “getting started” guide for Bottle installation and configuration. (more or less this is a link collection)

 

Basics

When I started this I had three things in my mind:

  • I use Visual Studio for development
  • I would like to use it from Azure Web App
  • I would like to use it from a stand alone CentOS Linux

At first we are a very lucky situation because there are several article on Internet regarding this topic. Please kindly notes I am focus on Visual Studio related usage. šŸ™‚

 

Scenarios

1. Visual Studio and Azure

Here is a page for Visual Studio and python relationship: Python feature of Visual Studio

This is the easiest situation because we merely follow the instructions of this article: Publishing to Azure App Service

Then a page for Python web application project templates (this article contains that very useful information to install Python on App Service MS recommends using the site extensions. These extensions are copies of the official releases of Python, optimized and repackaged for Azure App Service.)

 

Note:
– Azure generally uses Windows machines under App services so your web application will be executed by an IIS
– You have to install the python extension under the App service where you would like to hosts your UI
– Bottle uses MVC modell

2. Visual Studio and CentOs

In this scenario we have to configure and prepare our server to bottle. Great thing…their is a very good article for this: How To Deploy Python Web Applications with the Bottle Micro Framework on CentOS 7

Here we need to be careful regarding the app.py related configuration because the original code is wrong for us.

Original code in app.py (service is not reachable from internet – only from localhost):

HOST = os.environ.get('SERVER_HOST', 'localhost')

Correct code:

HOST = os.environ.get('SERVER_HOST', '0.0.0.0')

Note:
– Bottle uses MVC modell

 

Tricks and Tips

Use git for code management

In the scenario 2 that is the most useful if you use a git for manage your code between Visual Studio and CentOs server

 

Modify ‘os.environ.get’ in ‘app.py’

Check the app.py related configuration because the original code is wrong for us.

Original code in app.py (service is not reachable from internet – only from localhost):

HOST = os.environ.get('SERVER_HOST', 'localhost')

Correct code:

HOST = os.environ.get('SERVER_HOST', '0.0.0.0')

 

‘reloader=True’ in ‘app.py’

Use ‘reloader’ inĀ  app.py because in this way after you pull the latest code to your CentOs, Bottle will reload automatically and use the latest code.

bottle.run(server='wsgiref', host=HOST, port=PORT, reloader=True)

 

Use different names for GET and POST

I suggest to use different names for GET and POST method related functions. I mean when you have an upload function you should use def upload():for GET and useĀ def do_upload(): for POST method in your code. Why? In this way you can call directly your upload (GET) function from another function.

 

Learn python code implementation

Finally you should improve your knowledge about existing python code implementation with Bottle: Building a Rest API with the Bottle Framework

 

Favicon 404

When you check the python console in debug mode you can see a 404 error for favicon:

"GET /favicon.ico HTTP/1.1" 404 743

You can manage this according to this article: Serve favicon with bottle.py

 

 

I think this is a very good fundamentals to make your codes available via browser… šŸ™‚

technicalthursdaylogo

Technical Thursday ā€“ Restore VM from RSV ā€“ unmanaged disk (with amaretto)

, , , , , , 0

I know…this is another restore related post for unmanaged disk. I promise this is the last in this period… šŸ™‚

So the concept is here for restore and you can find here the previous unmanaged disk related restore with several scripts. (the managed disk related steps are here) Now I would like to show you the amaretto related restore steps forl unmanaged disk based VM.

 

Some useful information before you start the restore:

Required naming convention

OS disk and Data disks related vhds must be in the following format:

  • OS disk:
    • [vmname]-osdisk.vhd
  • Data disk:
    • [vmname]-datadisk-[diskid].vhdĀ (where the diskid represents the value of lun)
    • (example for 1st data disk: myvm-datadisk-0.vhd)

Prerequisites

  • Linux OS
  • Azure-Cli 2.x
  • Python 2.7
  • amaretto (Azure management tools by the1bit) package for python. You can download it from pypi and git as well.

 

Amaretto

In amaretto you can also find a restoreUnmanagedDiskFromVhd function in amarettorestore module which ā€œdo your jobā€ regarding restore procedure.

What does this function do?

  1. Get restore file from restored container
  2. Download deploy file (generally this is a config.json with UTF-16 encoding(!))
  3. Check file encoding ā€“ if necessary it converts from UTF-16 to UTF-8
  4. Deallocate VM
  5. Delete VM object (ONLY)
  6. Get os disk information (restored vhd’s url)
  7. Delete old unmanaged disk
  8. Copy os disk to its original location
  9. Get data disk information (restored vhds’ url)
  10. Delete old unmanaged disk one-by-one
  11. Copy data disk to its original location
  12. Check restore result (whether all disks are restored or not)

 

And now let’s see the steps one-by-one:

1. Restore VM’s VHDs from backup vault

Choose the right restore point from Recovery Services vaults which belongs to target VM and Restore OSDisk and DataDisks to your storage account.

https://docs.microsoft.com/en-us/azure/backup/backup-azure-arm-restore-vms

2. Configure and execute “restoreUnmanagedDiskFromVhd” function from amaretto

You have to execute the following commands with your VM related parameters from python:

# Your VM name
vmNameĀ =Ā "thisismyserver-1"
# resource group name where the VM is located
resourceGroup = "thisismyrg"
# location where the resources are located. (westeurope, germanycentral, ...)
location = "westeurope"
# storage account name where the VM's restored vhds are stored
sourceStorageAccount = "thisismystorage"
# 1st or 2nd access key for sourceStorageAccount
sourceSecretKey = "d22j/rr+a7br7LW6KDKV8KZkO2wCIe3m0MTKVr3Tt9B9NMZZsYxny8bvWvPwUGgZpDkE8gyAePjWCVu2IZ4LYw=="
# name of container where the restored vhds are stored
sourceContainer = "vhd6bdda0e88c88408299246c468784656546a"
# Execute restore function
amaretto.amarettorestore.restoreUnmanagedDiskFromVhd(vmName,Ā resourceGroup,Ā location,Ā storageAccount,Ā secretKey,Ā sourceContainer)

3. Re-create target VM with your ARM Template

In this step you merely redeploy your vm from that ARM template you had created for original vm creation.

 

Next week I will provide a new topic to you… šŸ™‚

 

If you need some help regarding ARM Template for restore or other scenarios do not hesitate to contact me.

technicalthursdaylogo

Technical Thursday – Restore VM from RSV – managed disk

, , , , , , , 0

Last week I provided the unmanaged disk related restore description which based on the concept for restore and now I show the managed disk related restore steps on technical level.

 

Some useful information before you start the restore:

Required naming convention

OS disk and Data disks related vhds must be in the following format:

  • OS disk:
    • [vmname]-osdisk
  • Data disk:
    • [vmname]-datadisk-[diskid]Ā (where the diskid represents the value of lun)
    • (example for 1st data disk: myvm-datadisk-0)

Prerequisites

  • Linux OS
  • Azure-Cli 2.x
  • Python 2.7
  • amaretto (Azure management tools by the1bit) package for python. You can download it from pypi and git as well.

 

For this type of restore we can use the previously introduced amaretto python package. This package contains the restore steps related functions one-by-one. Additionally you can also find a restoreManagedDiskFromVhd function in amarettorestore module which “do your job” regarding restore procedure.

What does this function do?

  1. Get restore file from restored container
  2. Download deploy file (generally this is a config.json with UTF-16 encoding(!))
  3. Check file encoding – if necessary it converts from UTF-16 to UTF-8
  4. Deallocate VM
  5. Delete VM object (ONLY)
  6. Get os disk information (size, tags – it is important from billing and categorization point of view)
  7. Delete old managed disk
  8. Convert vhd to managed disk (with tags and sizes from step 6)
  9. Get data disks information (size, tags – it is important from billing and categorization point of view)
  10. Delete old managed disks one-by-one
  11. Convert vhds to managed disks (with tags and sizes from step 9)
  12. Check restore result (whether all disks are restored or not)

 

And now let’s see the restore steps one-by-one:

1. Restore VM’s VHDs from backup vault

Choose the right restore point from Recovery Services vaults which belongs to target VM and Restore OSDisk and DataDisks to your storage account.

https://docs.microsoft.com/en-us/azure/backup/backup-azure-arm-restore-vms

Note: In case of manage disk restore: Template based restore for managed disks is planned in June 2018. Thus far that we should identify the restored VHDs. This means after the restore you have vhd files instead of managed disks. šŸ™

2. Configure and execute “restoreManagedDiskFromVhd” function from amaretto

You have to execute the following commands with your VM related parameters from python:

# Your VM name
vmName = "thisismyserver-2"
# resource group name where the VM is located
resourceGroup = "thisismyrg"
# location where the resources are located. (westeurope, germanycentral, ...)
location = "westeurope"
# storage account name where the VM's restored vhds are stored
sourceStorageAccount = "thisismystorage"
# 1st or 2nd access key for sourceStorageAccount
sourceSecretKey = "d22j/rr+a7br7LW6KDKV8KZkO2wCIe3m0MTKVr3Tt9B9NMZZsYxny8bvWvPwUGgZpDkE8gyAePjWCVu2IZ4LYw=="
# name of container where the restored vhds are stored
sourceContainer = "vhd6bdda0e88c88408299246c468784656546a"
# managedDiskAccountType (optional): sku of disk. Possible values: Standard_LRS or Premium_LRS. Default value: Standard_LRS
# managedDiskAccountType = "Standard_LRS"
# Execute restore function
amaretto.amarettorestore.restoreManagedDiskFromVhd(vmName, resourceGroup, location, sourceStorageAccount, sourceSecretKey, sourceContainer)

Result:

>>> amaretto.amarettorestore.restoreManagedDiskFromVhd(vmName,Ā resourceGroup,Ā location,Ā sourceStorageAccount,Ā sourceSecretKey,Ā sourceContainer)
2018-02-22 13:18:40 - FUNCTION Restore Managed disk based VM's vhds
2018-02-22 13:18:40 - Get restore file from restored container
2018-02-22 13:18:43 - Download deploy file
Finished[#############################################################]  100.0000%
2018-02-22 13:18:45 - Check config.json file encoding
2018-02-22 13:18:45 - Deallocate VM: thisismyserver-2
2018-02-22 13:18:48 - Delete VM object: thisismyserver-2
2018-02-22 13:18:50 - - OS DISK
2018-02-22 13:18:50 - Get os disk information
2018-02-22 13:18:50 - Delete old managed disk: thisismyserver-2-osdisk
2018-02-22 13:18:54 - Convert os vhd to its original location
2018-02-22 13:19:00 - - DATA DISKS
2018-02-22 13:19:00 - Get data disks information
2018-02-22 13:19:00 - Delete old data disk
2018-02-22 13:19:05 - Convert data disk to its original location: thisismyserver-2-datadisk-0
2018-02-22 13:19:11 - OS and Data disks are restored
True

3. Re-create target VM with your ARM Template

In this step you merely redeploy your vm from that ARM template you had created for original vm creation.

 

I hope it helps to solve your restore VM problem. šŸ™‚

 

If you need some help regarding ARM Template for restore or other scenarios do not hesitate to contact me.

snake

Installation of Azure Monitoring libraries for python

, , 0

Sometimes you need a “strong” tool for Azure management and Azure Monitoring. In this case you have the Azure Monitoring libraries for python which could be installed with pip install azure-mgmt-monitor. Then when you are lucky you can use it immediately (MS documentation here). What happens if you are facing an unexpected issue when you would like to use this beautiful opportunity? You start to find the solution…

Situation

Here I would like to show you a strange situation…and its solution. šŸ™‚

I wanted to manage the Azure subscription related metrics and alerts from Python. I have started the discovering of capabilities of this area and I am facing an serious issue. (I was following this article:Ā Azure Monitoring libraries for python)

After this line I receive an error message.

for metric in client.metric_definitions.list(resource_id): 
    # azure.monitor.models.MetricDefinition
    print("{}: id={}, unit={}".format( metric.name.localized_value, metric.name.value, metric.unit ))

The error message is this:

AttributeError: 'MonitorManagementClient' object has no attribute 'metric_definitions'

 

Investigation

Then I checked the client object and I realized there is not “metric_definitions” in “MonitorManagementClient”

>>> dir(client)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_client', '_deserialize', '_serialize', 'activity_log_alerts', 'alert_rule_incidents', 'alert_rules', 'autoscale_settings', 'config', 'log_profiles', 'service_diagnostic_settings']

And I checked the installed package version of “azure-mgmt-monitor” with pip freeze (result: azure-mgmt-monitor==0.4.0) and print(azure.mgmt.monitor.__version__)(result: 0.2.1)

 

Solution

After some hours I had decided I will fully reinstall the package…and at that point I have found the solution. Both packages were installed to my computer. This situation caused the unexpected error.

To fix this issue you just need to follow these steps:

  1. Check the version of packages:Ā pip freezeĀ (result:Ā azure-mgmt-monitor==0.4.0)
  2. Uninstall the whole “azure-mgmt-monitor” package:Ā pip uninstall azure-mgmt-monitor
  3. Check the version of packagesĀ again:Ā pip freezeĀ (result:Ā azure-mgmt-monitor==0.2.1)
  4. Uninstall the whole “azure-mgmt-monitor” packageĀ again:Ā pip uninstall azure-mgmt-monitor
  5. The “azure-mgmt-monitor” was gone now.
  6. Reinstall “azure-mgmt-monitor” package:Ā pip install azure-mgmt-monitor
  7. Check the version of packagesĀ again:Ā pip freezeĀ (result:Ā azure-mgmt-monitor==0.4.0)

Checking

Finally I check the “metric_definitions” attribute at “MonitorManagementClient” object.

>>> dir(client)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_client', '_deserialize', '_serialize', 'action_groups', 'activity_log_alerts', 'activity_logs', 'alert_rule_incidents', 'alert_rules', 'autoscale_settings', 'config', 'diagnostic_settings', 'diagnostic_settings_category', 'event_categories', 'log_profiles', 'metric_definitions', 'metrics', 'operations', 'tenant_activity_logs']

This means we win and the required version of “azure-mgmt-monitor” is available from now.

 

Now you can list, create and modify all of your resources related metrics and alerts with python…

You can find the details of this investigation on Github