Category Archives: DevTools

Remote debugging Python with VSCode

I truly think that no matter what your platform is, you must have access to a comfortable development environment and a working debugger is one of the most important part of it. Remote debugging can be very helpful: it is possible to execute code on a remote machine and benefit from a nice debugging experience locally in your favorite code editor.

In this blog post I propose to review the setup of Python remote debugging with the portable and popular code editor VSCode. Actually VSCode documentation provides some very short instructions. In this blog post we will provide more explanations.

Use remote debugging capabilities of VSCode with Python

Use remote debugging capabilities of VSCode with Python

Prerequisite

We will assume that we do not have any security constraints. Precisely, we do not care about MITM interceptions between our client and remote server. We will discuss in appendix how we could solve this using SSH portforwarding.

We assume that the reader is familiar with the usage of a debugger in VSCode. In addition, we assume that the reader knows how to logon on a remote machine using SSH.

Our example

In this blog post we used an Ubuntu Azure Virtual Machine. Its configuration, RAM, GPU etc. are independent so you can basically choose anything.

We assume now that the reader has an Azure Ubuntu server running and is able to logon through SSH. Note that in VSCode documentation SSH portforwarding is mentioned but we will ignore it for now.

Let us present precisely what remote debugging is.
In this post, the name remote stands for our Ubuntu VM on Azure while the client is our local, e.g. MACOS, computer. With remote debugging only a single Python process is executed on the remote VM then, on client computer, VSCode “attach itself” to this remote process so you can match the remote code execution with your local files. Therefore, it is important to keep exactly the same .py files on client and in host so that the debugging process is able to match line by line the two versions.

The magic lies in a library called ptvsd that makes the bridge for attaching local VSCode to remotely executed process. The remotely executed Python waits until the client debugging agent is attached.

Obviously network communication is involved here and that is actually the major pitfall when configuring remote debugging. The VSCode documentation is fuzzy about whether to use IP or localhost which port to set etc. We will try to simplify things so the debugging experience becomes crystal clear.

Networking

To make things simpler we decided to show an example where the Python process is executed on a remote machine whose IP address is 234.56.45.89 (I chose this address randomly). We use the good old port 80 for the communication (the usual port for http).

Before doing anything else we need to make sure that our remote VM network configuration is ok. We will make sure that machine 234.56.45.89 can be contacted from the outside world on port 80.

Firstly, using an SSH session on remote machine we will start a webserver using the following Python3 command. You may need elevated privilege for listening on port 80 (for real production usage give this privilege to the current user, do not sudo the process).

sudo python3 -m http.server 80

Secondly on a client terminal you should be able request your machine using wget (spider mode to avoid file download). In this command the target machine is accessed with IP:PORT

wget --spider 234.56.45.89:80

You should get response from the server. If you see some errors, you mat need to open the 80 port in firewall configuration, see instructions here for Azure.

Make sure you can contact your machine on port 80 by running a one line Python server

Make sure you can contact your machine on port 80 by running a one line Python server

At this stage your network configuration is ok. You can stop the Python command that runs the webserver.

Configuring VSCode

Make sure that you have the VSCode Python extension installed. Follow the instructions here to add a new Debug configuration in your launch.json containing the following JSON configuration.

{
    "name": "Attach (Remote Debug)",
    "type": "python",
    "request": "attach",
    "localRoot": "${workspaceRoot}",
    "remoteRoot": "/home/benoitpatra",
    "port": 80,
    "secret": "my_secret",
    "host":"234.56.45.89"
}

It is important to understand that this configuration is only for VSCode. The host corresponds to the machine where the remote Python process is ran. The port corresponds to the port that will be used by the remote process to communicate with the client debugging agent, in our case it is 80.

You must specify the root folders, on both local environment and on the remote machine.

That’s it for VSCode configuration.

The code under debugging

Let us debug the following Python script

import os
import ptvsd
import socket
ptvsd.enable_attach("my_secret", address = ('0.0.0.0', 80))

# Enable the line of source code below only if you want the application to wait until the debugger has attached to it
#ptvsd.wait_for_attach()

ptvsd.break_into_debugger()

cwd = os.getcwd()

print("Hello world you are here %s" % cwd )
print("On machine %s" % socket.gethostname())

As explained in the introduction, the Python file must be the same on client and on remote machine. There is one exception yet, the line ptvsd.wait_for_attach() must be executed by remote Python process only. Indeed, it tells the Python process to pause and wait that the client is attached to continue.

Of course in order to execute it you may need to install dependencies (for example using Pip) so it executes on the remote machine.

REMARK: looks like at the time of the writing version of ptvsd>3.0.0 suffers some problems. I suggest that you force the install of version 3.0.0, see this issue.

It is important to understand that enable_attach, enable_attach, break_into_debugger are instructions for the remote Python process. The first line ptvsd.enable_attach("my_secret", address = ('0.0.0.0', 80)) basically instructs the remote Python process to listen on all network interfaces, on port 80 for any client debugger that would like to attach. This client agent must provide the right secret (here my_secret).

The line ptvsd.break_into_debugger() is important, it is the line that allows to break and navigate in code with client VSCode.

Putting things together

Now you are almost ready. Make sure your Python file is duplicated on both local and remote at root location. Make sure the ptvsd.wait_for_attach is uncommented and executes on remote environment.

Now using an SSH session on remote machine. Start the Python process using elevated privileges
sudo python3 your_file_here.py

This should not return anything right now and should be hanging, waiting for your VSCode to attach the process.

Set a VSCode break point just after ptvsd.break_into_debugger(), make sure that in VSCode the selected debugging configuration is Attach (Remote Debugger). Hit F5, you should be attached and breaking in code !

What a relief, efficient working ahead !

Breaking in VSCode

Breaking in VSCode

Going further

The debugging procedure described aboved is simplified and suffer some flaws.

Security constraints

Here anybody can intercept your traffic, it is plain unencrypted http traffic. A recommended and yet simple option to secure the communication is to use SSH port forwarding tunnelling. It basically creates an encrypted network communication between your localhost client and the remote machine. When an SSH tunnel is setup, you can talk to your local machine on a given port and the remote receives call on another port (magic, isn’t it?). Therefore the launch.json configuration should be modified and host value is localhost. Note also that the port in Python code and in launch.json may not be the same, you have two different ports now.

Copying files

We pointed out that the files must be the same between local env and remote. We advise to group in a shell script: the files mirroring logic (using scp) and the execution of the Python process on remote machine.

Handling differences between local and remote files

We said that the files must the same between local env and remote but we need some differences at least to allow the execution of ptvsd.wait_for_attach on remote.
This is definitely something that can be handled in an elegant manner using environment variables.

if os.environ.has_key("REMOTE"):
    ptvsd.break_into_debugger()
end

Of course you need to pass now the environment variable to you remote process with SSH, see this stackexchange post to know how to do that.

Debugging locally REST API webhooks with Visual Studio

Modern REST APIs such as Outlook REST Api, Microsoft Graph or Facebook Graph expose very powerful capabilities called webhooks. They allow push notifications. After subscription, when something change these API send notifications to your service by calling the URL you provided. For example, in Outlook REST API the push notification services will send a request when something has been modify in the user mailbox such as a mail received or an email marked as read.

I am not going to explain how you register subscriptions to a particular webhook. In this blog post, we provide a solution in order to be able to “break” with your Visual Studio debugger in a callback webhook you subscribed to. The approach is not windows/.NET specific, actually the mechanism exposed here is generic, but these are the tools I am using at the moment so they will serve as example in this post.

Problem: when you subscribe to a webhook you specify what would be your notification URL (see Outlook REST API example). This url must be https and visible from the ‘outside’ internet. Therefore, you cannot set an url such as https://localhost:44301/api/MyNotificationCallBack where https://localhost:44301 is the url of your local development website.  However, it would be convenient in order to ‘break’ directly in your server side code responsible for handling the request. In addition, if you are using Visual Studio and IIS express for development you cannot simply expose a website with custom domain and SSL to the outside internet.

Solution: take a (sub)domain name you own (e.g. superdebug.keluro.com) then create an A record to point to your public IP. If you are in a home network this IP is the one of your ISP box. Configure this box to redirect incoming traffic for superdebug.keluro.com on port 443 to your personal developer machine (still on port 443). In your machine, configure an IIS web server with a binding for https://superdebug.keluro.com on port 443 that will act as reverse proxy and will redirect incoming traffic to your IIS Express local development server  (e.g. https://localhost:44301). Finally, set a valid SSL certificate on the reverse proxy IIS server for superdebug.keluro.com. Now, you can now use https://superdebug.keluro.com/api/MyNotificationCallBack as notification Url and the routing logic will redirect incoming push notification requests to https://localhost:44301/api/MyNotificationCallBack where you can debug locally.

Debug IIS Express website visible from the outside internet

Debug locally IIS Express website visible from the outside internet

Pitfalls:

  • Unfortunately, in case of home network, I cannot give precise instructions on how to configure your ISP box to reroute incoming traffic. Also make sure that the box IP does not change and is static.
  • Take care of your own Firewall rules, make sure that 443 port is open for both Inbound and Outbound rules.
  • In IIS Application Request Routing (ARR), the module that may be used for creating the reverse proxy, an option is set by default that modifies ‘location’ request response Headers. It may break your application that probably uses OAUTH flow. See this stackoverflow response.
  • If you never setup IIS to work as a reverse proxy. That is quite simple now with ARR or Rewrite Request modules. In this previous blog post we explained how to setup a reverse proxy with IIS.

Tools for javascript TDD with Visual Studio

In this post I describe the tools that I have selected for efficient development of javascript tests within Visual Studio. Indeed, if you are developing sites with ASP.NET or Apps for Office then you are more or less committed to use Visual Studio. Therefore, you probably do not want to use another editor for the javascript development.

What were my requirements when I chose the tools that I will describe to you in this post? Actually, I needed tools which can provide me the same comfort that I have while I am developing in TDD with .NET. Precisely, the test should be run individually (or at least individually for a given set). They should be run on the continuous integration build, which is teamcity in my case. The test runner may use any browser for executing the tests even if flexibility for using all kind of browsers would be appreciated. In addition, the test should be easily debugged and this is a very important aspect for me.

Let me detail a little this latter requirement. Indeed, I have read recently this blog post and its written that

Debugging the tests
Wiseman said: “If you need to debug unit test then something is wrong with it’s unitness.”
PS: If it’s really needed you can debug it in browser as you normally debug JavaScript code.

Even if the rest of the blog is brilliant, I do not agree at all with this sentence.  If you practice regularly TDD then you’ll see that you have to debug, that’s part of the game. If you never have to debug maybe it’s because you are asserting too much trivialities and not enough your own complex logic. In addition, if you have to create the webpage on your own, for debugging, that is not acceptable to me either. You should be able to put your breakpoint anywhere in your code and hit it in less than five seconds. We will see that the tool chosen permits such quick debugging and that’s the main reason why I am using it.

At the time of the writing, the popular Visual Studio Addin Resharper with its version 8 supports javascript tests with the frameworks Jasmine or QUnit. Unfortunately, the debugger cannot be properly attached while debugging the scripts. Sadly, I had to reject Resharper that I appreciate so much for .NET development.

Finally, the best tool that I found is Chutzpah (which means Audacity in Yeddish). Firstly, there is the Chutzpah runner which is a javascript test runner that uses the headless browser phantomJs, we will invoke it from the command line in the continuous build. Sedondly, there is the Chutzpah Visual Studio extension which enables quick runs and debugging sessions from Visual Studio. Both runner and Visual Studio extension is fully compatible with JS test framework Jasmine 2.0. that I have chosen. Note that I also use the mocking library sinonJs but I won’t discuss it there.

You may retrieve the code samples below on my github repository.

To illustrate these tools we will use a very simple mockup project and we start its description with the following VisualStudio solution.

The Visual Studio solution describing the example of this post.

The Visual Studio solution describing the example of this post.

As usual there are two projects: WebApplication1, the core project, containing the app folder with our javascript logic, especially the one that we would like to test: sut.js (for SystemUnderTest). There is also the test project, WebApplication1.Tests. The file testing the logic contained in sut.js is simply sutTests.js. The testing framework Jasmine is added to the solution by referencing the folder lib/jasmine-2.0.0..

We will take a very simple example, where the object app.sut has a function for computing the factorial of an integer. The code of sut.js can be found in the following snippet.

(Note that in app we have implemented a basic namespacing function such as this one)

Let us now focus on the test suite in the sutTests.js code.

This piece of code is very basic Jasmine syntax for a test suite. We assert the values of the factorial function for the two corner cases where the input is 0 and 1 and we also test the situation with n=5. Remark also that we have a test which asserts that an exception is thrown when a non positive value is passed to the function. We have not covered this situation with the snippet above, consequently, we expect this test to fail… Remark that all the needed references to js files are handled with ///<reference of Visual Studio. By the way, we benefit from Visual Studio Intellisense here.

The Javascript intellisense with Visual Studio

The javascript Intellisense with Visual Studio

In the next screen shot, we run the tests with the Visual Studio Test Explorer that is compatible with Jasmine thanks to Chutzpah.

Run all our test with the Visual Studio Test Explorer. As expected we encounter a failure.

Run all our test with the Visual Studio test explorer. As expected we encounter a failure.

If we were in the situation where we do not know what is going wrong (which is often the case) we would need to debug the test. Unfortunately, like Resharper, you cannot debug the test directly in VisualStudio, the debugger does not attach well. Even Chutzpah creator does not know how we would do that, so I believe we will have to wait to debug javascript test as easily as we debug .NET within visual studio. Then, we will have to debug with a web browser. However, Chutzpah creates the webpage for bootstrapping your failing test. So just by clicking in Open in browser you will have the web page loaded and a link to run the failing test within the browser (see screenshots below).

Chutzpah Visual Studio extension creates web page for running/debugging tests in the web browser.

Chutzpah Visual Studio extension creates web page for running/debugging tests in the web browser.

Then the debugging sessions happens in your browser. Reclicking the links reexecutes the tests.

Then the debugging sessions happens in your browser (here Chrome). Clicking again the links reexecute the tests.

Now, we do have all the material for editing efficiently tests in Visual Studio. Let us have a few words for running the test using the command line on the server. Personally, I embed the Chutzaph runner in a /build directory within my sources and executes the following Powershell script.

Here is what it looks when ran it an Powershell console with teamcity options.

The execution of all javascript tests of the solution with Chutzpah runner using Powershell

The execution of all javascript tests of the solution with Chutzpah runner using Powershell

To conclude, I would say that Chutzpah is a great project and if you need a simple and ready to use test runner I would recommend it. The only limitation for now  would be that the runner only supports phantomJS. However, you may use another another test runner for executing the tests with different browser and you can keep the Chutzpah Visual Studio extension for the development. One last important thing to note is the fact that Chutzpah 3.0 supports RequireJs.

A git practical class

The link to the practical (pdf)

Since 2010, I teach C++  about 25 hours per year at ENSAE engineering school. During a short period we have to teach the students, with few computer science background, the complex concept of C++ (pointers, dynamic memory management, virtuality, templates etc.). The objective is to give them the basic material to be autonomous while they will be building their own C++ project during the second semester. This year we decided to sacrifice a little part of the core C++ teaching to have one lesson and a practical class dedicated to source control. Indeed, we have seen too many students struggling with sharing code on USB drives or via emails. Moreover, we do believe that this is something that a student should know before getting into any professional life involving code. To my point of view, source control is entirely a part of the software development process and could not be avoided anymore.

The source control that we decided to retain is git. Even if we do believe that SVN would suffice for the quite small project of our students. We think that git is the future in term of source control and git is much more powerful than SVN so, why have less when you can have more?

I publish this practical because it may be useful to others. Note that this practical is not a tutorial and it does not aim at being a git manual either. I think that their is plenty of very good documentation on the net starting by the famous ProGit by Scott Chacon. This practical is more about how do you work with git in a real life? It should be followed by a pair of student but you can do it alone if you manage split your brain in two.

It starts by forking an existing dummy C++ project on my personal github. After that, the two students are supposed to make some modifications (adding dummy classes, commenting code etc.) and share the work on their github using the most common git commands (add, commit, merge, fetch, push, pull). We examine at the end some more advanced features: blame, interactive rebasing etc.

gitextensions

Contrary to many practical that you may find on the web, this practical encourage the students to use GitExtensions. However, some tasks should be executed through the command line. I decided to promote GitExtensions because our students, for their great majority, are not computer geeks. They are not really friendly to the command line especially on windows (ENSAE computers have Win7 OS), so giving them  a “command line only” tool would probably not convince the majority. Secondly,  GitExtensions is really the best GUI over git that I have found, so why not use it? I use it at work and I switch from command line to GitExtensions depending on the task. Finally,  GitExtension is “verbose” by default, this means that it tells what command line is being executing underneath, so it helps you learning git internals.

To conclude, that some will find in this document a complementary to git documentation such as ProGit.

Powershell + ConEmu + PowerTab: a good combo

Edit: this post was written before the release of PSReadLine which seems to be a more reasonable choice than PowerTab for powershell.exe enhancement (completion, intellisense etc.). Indeed, this latter project looks like to be dead now and does not seem to be as rich as the former one. Regarding the rest of the post below, I encourage you to use Conemu console which is still far better than the usual CommandPrompt (this may change with the release of Win10). In addition, you may skip the “Aliasing part” because command in PATH are well recognized by PSReadLine which was not the case for PowerTab. You will discover also a lot of interesting features proposed by PsReadLine such as syntax coloring, custom key bindings….

Microsoft has released in 2006 Powershell (PoSh), a brilliant shell based on the .NET framework. However, the console (or terminal) stays really close to the usual windows command prompt. It is commonly admitted that this console suffers a lot of drawbacks which makes it painful for a everyday usage, see for example these blogs (artlogic , hanselman, etc.). Just to make things clear, let us recall the difference between a shell and a console. The former is the command interpreter, it is performing the hard work while the latter is just the window, i.e. the program in charge of prompting and displaying information.

conemu

ConEmu with PowerTab tabbing

In this post I will show you how to configure ConEmu an opensource console emulator to work with Powershell and to benefit from a well designed tab completion with PowerTab.

We will only refer to the latest version of Powershell, the 3.0 released in 2012. Let us mention that Powershell 3.0 came with a new version of Powershel ISE, the PoSh scripting environment. In this third version, ISE has Intellisense which is a great feature, especially when you are discovering PoSh. Unfortunately, ISE console does not support tabbing and suffers the same drawbacks has the original windows command prompt. In addition, ISE is perfectly suited when you are writing some script but is not really adapted to perform every day tasks. In one word ISE more an IDE than a console. In order to have a substitute to Intellisense in ConEmu we will use the PowerTab project.

You may start right now by checking that you have Powershell 3.0 installed (how to check version). Right after, you may start downloading ConEmu.

If you are an intensive Powershell user, you would want ConEmu to start directly with a new PoSh session. Nothing is more simple, just add the following .txt file somewhere in your disk (e.g. next to ConEmu .exe file).

shortcutProp

Target in properties of the desktop/taskbar shortcut

Next, create a shortcut in task bar or desktop, right click on it on go to the properties of  the link. You may enter the following command in the target property: <pathToConemu>conemu.exe /cmd @<pathtoyourfile>/startfile.txt

So now you would have a nice console which starts automatically with Powershell. Let us go a little bit further by importing PowerTab in ConEmu. Visit PowerTab website and download the sources rather than the packaged version. Indeed, many fixes are present in the source code but not packaged in the .zip. At the time of the writing I use version of commit 035310b4c93e, Follow the instruction for installing Powertab. Quickly it is: unzipping source under <YourHomePath>/WindowsPowerShell/Modules/PowerTab and execute command Import-Module PowerTab.

While importing PowerTab module, it will ask you if you want to import PowerTab on start. This config will be saved by updating your $PROFILE. If your not familiar with PoSh profiles now, I suggest you to read this. To sum up the $PROFILE is a special PoSh script executed each time your entering a new Powershell session (then each time you are starting ConEmu if you have done what is above). This is where you would put some custom scripts and any in-house configuration. NB: under commit 035310b4c93e, the function of your $PROFILE are recognized by PowerTab which does not seemed to be the case on the packaged version PowerTab 0.99.6.

Now we do have a console with efficient tabbing. The style is much more old school than the ISE intellisense but it does the job well and is quite pleasant to use.

I am not done yet, actually, our version of PowerTab does not see the .exe in you PATH environment variable for tabbing. This is quite frustrating not to have auto-complete for vim, notepad++, tracecrt.exe etc. This may be included in incoming version (I will see what I can do…) but here is workaround. As long as PowerTab recognizes Powershell command aliases, we are going to build an alias dictionary that includes the aforementioned .exe located in the PATH environment variable.

The following snippet shows a part of your $PROFILE where the aliases are imported just before the module PowerTab. They are imported from a file alias.txt which is located next to the $PROFILE .ps1 file. You create this alias.txt file by invoking the function Export-AliasWithEXEInPATH. You have to call this function at least once and to keep your aliases up-to-date you should run it when you modify the PATH variable. Remark the lightweight PoSh syntax for the .NET hashtable: @{}

Even if it will probably be outdated soon, this is my configuration at home and at work for now and I am quite pleased of it.