My new course : Continuous Delivery with Visual Studio Team Services published

Over the last 3 months I was busy working on my course on Continuous Delivery with Visual Studio Team Services. 

The course is published by Packt - a leading UK provider of Technology eBooks, Videos and Blogs, and their editors and marketing department has done a fantastic job of helping me with with creative graphics and animations to bring life to this course.

This course is intended for software development professionals who want to learn about Continuous Delivery in a technical value stream as supported by Visual Studio Team Services in order to continually deliver working software at scale. You'll learn how to create, build, test and release a containerized ASP.NET web api to kubernetes cluster on Azure.

The course consists out of seven sections:

• Introduction to continuous delivery - We'll learn about Continuous Delivery, the business goals and principles of Continuous Delivery
• Introduction to visual studio team services - We'll learn about the different elements of Visual Studio Team Services and how they function and how using these connected tools you can move your organization to a full-fledged continuous delivery implementation.
• Setting up the development environment for .net core applications using Docker:
• Source code management in VSTS : We'll have a look into the Git version control in VSTS and discuss branching strategies like trunk based development and how these concepts relate to continuous delivery.
• Build management in VSTS : We'll see how to setup a Continuous integration build in VSTS that helps the code stay robust enough that team members can fearlessly make changes to the code base without worrying about breaking existing functionality.
• Continuous testing : This section we’ll learn how to ensure that an application's components function correctly when assembled together. We'll see how to automate the tests and include them in your Continuous Integration scheme to get the most value from the tests.
• Release management in VSTS : This section shows you how to do automatic deployments by using Release Management. You'll learn Release Management in VSTS, how it works and how to use release management to deploy our sample application to a Kubernetes cluster on Azure.

The course follows a phased approach to continuous delivery to automate build, deploy and management of .net core applications to Azure Container Services using VSTS ALM. By the end of this course, you'll have the necessary knowledge to create a fully automated build and release pipeline using Visual Studio Team Services for your .NET applications. And don't forget to check the course at https://www.packtpub.com/virtualization-and-cloud/continuous-delivery-visual-studio-team-services-video

Don't ignore the 'Core' in Continuous Delivery

You see, I'm a person who loves to write code, travel around the world, take nice pictures and eat!!! But nothing less than that, I also enjoy running (especially in the morning!!!, ). It's amazing to have that feeling of the fresh air, smell of the grass and sense the muscles working while observing the surroundings. My running helps me to do my job better because it helps clear my mind, relieves stress and builds a lot of confidence. Its provides me with those moments with great ideas and solutions to most of the problems you have at work or life. By the time you finish running, you have already set up the goal for the day with a certain amount of clarity about the activities that you have to do.

The more I started running, the more I loved challenging myself with those extra miles and pace. But with my overgrowing excitement, I was making the obvious mistake.. I was running too much, too soon, too fast. While doing so, I forgot about the "little things" that help stay healthy and flexible. I started getting injuries, my hips and hamstrings were always tight. I was breaking down with back pain and injuries. I knew I was doing something wrong and it was time to get an expert advice. Having heard the back pain story many times, the physio explained me about the importance of 'Core' muscles and the exercises to do that would strengthen my muscles and improve my core stability. He suggested doing squats, swimming, pull ups, crunches etc. So I decided to give my running a break and focus on core building. It felt a bit off in the gym doing all those non equipment exercises while everyone else was lifting those 20/30 kg dumbbells, but a month after training, I started noticing the difference. I felt more stronger with increased muscle mass, my posture was better, lost some body fat and I was more agile!!! I started running again and this time I noticed the difference immediately. My core helped me to maintain an efficient running form  helping with better stability, balance, posture and overall control.

In short, lesson learnt from my running experience. "Ignoring core strength in your training program may look like working for a short term, but is a recipe for long-term disaster. Lack of core-strength and flexibility causes your body to find a path of resistance towards running or cause the body to adapt in a negative fashion that will most likely lead to complex injuries!!!"

Well how does this relate to continuous delivery and DevOps? The answer is , before immediately adopting a CD tool and start working on the pipeline, you should prepare the teams and the organization with the core of continuous delivery. The core practices like version control, build automation, test automation and continuous integration increases software process agility and helps create an organizational culture and structure to fully embrace CD.

In this post, I’ll describe some of the core practices of continuous delivery  that helps delivering software feature more frequently and reliably. 

Version control everything

Putting everything in version control (source code, build scripts, test scripts and data setup, monitoring scripts etc.) This helps in keeping track and control everything. Keeping track of the changes gives the flexibility of undo or replay any changes easily in all environments including production. You can also understand who made a change and why. Version controlling also creates the first level of built-in safety net, preventing any out of process, locally made changes or manual overrides from entering the production environment. Every change a developer makes must in the source control, any failure to do so will be overridden in the next deployment if not present in the source control.

Practice TDD

TDD is a software development approach where the development of code is driven by first writing an automated test case, followed by writing the code to fulfill the test and then refactoring. With continuous delivery, it’s very important to have the tests pushed into source control along with the implementation code. This helps in ensuring that every new code pushed will be tested in the same build process. This eliminates the risk of delivering to production the code that does not meet quality requirements.

These automated tests helps identify errors in a really short period of time. These tests give developers very fast feedback when something breaks. These tests are also called change detectors. When you make a change, the tests will detect very soon if there are any problems or mistakes. With continuous delivery organizations need the have the ability to make frequent releases of their software without the risk of breaking existing features or adding new bugs. The success of the business depends on how fast it can react to changing environments and address the new improvement suggestions with assurance. Test-driven development is the methodology that makes such flexibility, maintainability, and extensibility possible.

Continuous integration

"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly." - Martin Fowler

Continuous integration process runs ensures that the tests are executed every time a change is pushed into source control and ensures that the new changes didn’t break any other parts of the software. Every push automatically triggers multiple tests. Then if one fails it’s much easier to identify where the error is and start working to fix it.

Automate everything

Automation is the heart of every successful continuous delivery transformation process and the single biggest enabler for CD. The success of continuous delivery lies in stable environments, consistent build and test process and happy releases. Automation can be used to make the release process more deterministic and to bridge the gap between development and production. Development teams can learn from operations the value of automation and how to create automation scripts to improve the overall process. Together the team is responsible for defining the desired state of the infrastructure and convert this to code. The degree to which your organization can achieve successful one click deployment process that can be initiated by anyone whenever needed depends on the level of automation at every stage. At the processes like code analysis, testing, environment provisioning, defect detection and prevention at all stages should be automated. Any manual tasks should be treated as technical debt and later picked up as part of the sprint and addressed.

Test automation

The key to building quality into our software is making sure we can get fast feedback on the impact of changes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Companies adopting continuous delivery should ensure that when issues are identified they can be managed and resolved quickly and definitively. The easiest and effective way to achieve this is by having an automated test suite incorporated in the release pipeline and CI process. This kind of a process where automated regression, performance and security tests are constantly executed as part of the deployment strategy helps identify issues early and deal with them before they reach the later stages of the deployment cycle/pipeline.

Monitor and measure everything

Proper and effective monitoring of the entire process and environments is important to provide crucial information that ensure service uptime and optimal performance. It's important to measure the progress of the approach to know whether teams are improving or making progress. With the support of proper data and metrics, it’s easy for teams to inspect the current way of working and come up with ideas or processes to improve the overall CD/ DevOps movement. Measuring the current capabilities and process helps teams determine the problem areas and defined focus points for making a change.

An important DevOps practice is to work with application monitoring and insights to enable full end-to-end traceability in a product to provide both operational insights and  usage understanding

Keep focus on these core practices and move forward with your continuous delivery journey!!!

“….to satisfy the customer through early and continuous delivery of valuable software” - #1 of the twelve principles behind the agile manifesto.

PSScriptAnalyzer - Automate code checking for your PowerShell projects as part of TFS build

With WMF 5.0 we can make use of the PowerShell static code checker utility PSScriptAnalyzer to perform code analysis on PowerShell script files and modules. PSScriptAnalyzer checks the quality of Windows PowerShell code by running a set of rules. The rules are based on PowerShell best practices identified by PowerShell Team and the community. PSScriptAnalyzer generates DiagnosticResults (errors and warnings) to inform users about potential code defects and suggests possible solutions for improvements.

You can download the entire module from PowerShell gallery using the Install-Module cmdlet or create a pull request on the project at Github and build it yourself. Once you have the module available on the workstation you can perform code checking on PowerShell files by using the Invoke-ScriptAnalyzer cmdlet.

By making use of this cmdlets and the post-build script options from TFS build, you can now perform code checking on the PowerShell projects from the build server and choose to fail the build based on the results. Let’s see how this works.

First we need a function to do code checking using the Invoke-ScriptAnalyzer cmdlet and parse the results. The below content describes the process

function Get-DiagnosticIssue

{

       param

       (

              [Parameter(Mandatory=$true)]

              [ValidateNotNullOrEmpty()]

              [ValidateScript({Test-Path $_})]

              [string] $Path,

              [Parameter(Mandatory=$true)]

              [ValidateNotNullOrEmpty()]

              [string] $Severity,

              [bool] $FailOnIssues = $false

       )

       $logDirectory = Join-Path $Env:TF_BUILD_DROPLOCATION "Logs"

       if(-not(Test-Path $logDirectory -ErrorAction SilentlyContinue))

       {

              New-Item -ItemType Directory -Force -Path $logDirectory

       }

       $logFile = Join-Path $logDirectory "PSScriptAnalyzerLogs.log"

       Invoke-ScriptAnalyzer -Path $Path -Recurse |? {$_.Severity -ge $Severity} |fl | Out-File $logFile

       if($FailOnIssues)

       {

              if((Get-Content $logFile) -ne $null)

              {

                     $Host.UI.WriteErrorLine("PSScriptAnalyzer check failed. Please refer to the file $($logFile) for more details.")

              }

       }

}

Export-ModuleMember *

The Get-DiagnosticIssue function accepts a Severity parameter to filter the issues logged and a Path variable to perform analysis on all the scripts files and modules under that location. The FailOnIssues parameter can be used to decide whether the build should be failed or not. This module is called from a script file that is used as the post-build script path in the TFS build definition.

param

(

       [string] $Severity = "Information",

       [string] $Filter,

       [string] $FailOnIssues = "False"

)

$exitCode = 0

trap

{

    $Host.UI.WriteErrorLine($error[0].Exception.Message)

    $Host.UI.WriteErrorLine($error[0].Exception.StackTrace)

    if ($exitCode -eq 0)

    {

        $exitCode = 1

    }

}

$scriptName = $MyInvocation.MyCommand.Name

$scriptPath = Split-Path -Parent (Get-Variable MyInvocation -Scope Script).Value.MyCommand.Path

Push-Location $scriptPath

Import-Module .\PSScriptAnalyzerExtensions.psm1

$source = $env:TF_BUILD_BUILDDIRECTORY

if(-not ([string]::IsNullOrWhiteSpace($Filter)))

{

       $source = Get-ChildItem -Directory $env:TF_BUILD_BUILDDIRECTORY -Filter $Filter -Recurse |? {$_.FullName.ToUpper().Contains("Solution\$Filter".ToUpper())} | select -expand FullName

}

$failBuild = $FailOnIssues -eq "True"

Get-DiagnosticIssue -Path $source -Severity $Severity -FailOnIssues:$failBuild

Pop-Location

exit $exitCode

This script file is used in the process template as post-build script path with arguments as given below.

Next time the build is queued, it will trigger a code check using PSScriptAnalyzer and you can see the results in the log file generated.

Build and publish to NuGet as part of TFS build process

NuGet is a free, open source package management system for .NET platform, which helps in easy dependency management. Using NuGet very easy for project owners to build and define package dependencies which will be included in the package during the packaging process and later publish this to the repository for usage by external users.

Users can later install the package by searching in the NuGet source repository which will automatically download and install the dependencies as well. You can use the NuGet command line utility to create packages from a specification file and later publish them to the repository whenever needed. If you want to automate the process of package creation and publishing from a TFS build process, you can make use of custom PowerShell scripts that can be invoked as part of the build process. The Team Foundation Build template (TfvcTemplate.12.xaml) provides the capability to extend the build process by using PowerShell scripts at stages like Pre build and Post build.

You can use the default template to run PowerShell and batch (.bat) scripts before and after you compile your code, and before and after your run your tests. During the script execution you can make use of the Team foundation build environment variables to get key bits of data that you need for your build process logic.

I've created a PowerShell module that contains the functions that I need to build and publish a nuget package.

$ErrorActionPreference = "Stop"

Function Publish-NugetPackage

{

    param

    (

        [Parameter(Mandatory = $true, Position = 0)]

        [ValidateNotNullOrEmpty()]

        [string] $Project

    )

    # Read the product version from the assembly

    $version = Get-ChildItem -Filter "$($Project).dll" `

                             -Recurse `

                             -Path (Join-Path $env:TF_BUILD_BUILDDIRECTORY bin) `

                             -File | select -First 1 | select -expand VersionInfo | select -expand ProductVersion

    # Read the contents of the nuspec file

    $nuspecFile = Get-ChildItem -Filter "$($Project).nuspec" `

                                -Recurse `

                                -Path $env:TF_BUILD_SOURCESDIRECTORY -File | select -First 1

    $nuspecContents = Get-Content $nuspecFile.FullName

    # Replace the version of the nuspec file with the assembly version

    Set-Content -Path $nuspecFile.FullName `

                -Value ($nuspecContents -replace "[0-9]+(\.([0-9]+|\*)){1,3}", "$($version)

") `

                -Force

    # Create the nuget package

    New-NugetPackage $nuspecFile.FullName

    # Fetch the newly created nuget package

    $nugetPackage = Get-ChildItem -Filter "*.nupkg" `

                                  -Recurse `

                                  -Path $env:TF_BUILD_SOURCESDIRECTORY -File | select -First 1

    # Publish the nuget package to nuget repository

    Push-NugetPackage $nugetPackage.FullName

}

Function New-NugetPackage

{

    param

    (

        [Parameter(Mandatory = $true, Position = 0)]

        [ValidateNotNull()]

        [ValidateScript({Test-Path $_})]

        [String] $NuspecFile,

        [Parameter(Position= 1)]

        [ValidateScript({Test-Path $_ -PathType Leaf -Include "nuget.exe"})]

        [String] $NugetLocation = (Join-Path $PSScriptRoot "nuget.exe")

    )

    $command = $NugetLocation + " pack $($NuspecFile)"

    Invoke-Expression $command

}

Function Push-NugetPackage

{

     param

    (

        [Parameter(Position= 3)]

        [string] $Source = "http://prajeeshnuget.azurewebsites.net/nuget",

        [Parameter(Mandatory = $true, Position = 0)]

        [ValidateNotNull()]

        [ValidateScript({Test-Path $_})]

        [string] $PackagePath,

       

        [Parameter(Position= 2)]

        [string] $APIKey = "MY_NUGET_APIKEY",

        [Parameter(Position= 1)]

        [ValidateScript({Test-Path $_ -PathType Leaf -Include "nuget.exe"})]

        [String] $NugetLocation = (Join-Path $PSScriptRoot "nuget.exe")

    )

    $command = $NugetLocation + " push $($PackagePath) $($APIKey)"

    if(-not ([String]::IsNullOrEmpty($Source)))

    {

        $command += " -s "  + $Source

    }

    Invoke-Expression $command | Out-Null

}

Export-ModuleMember -Function Publish*

The module contains logic to update the NuSpec file with the latest product version from the assembly that is build. Once updated this file will be used to create a nuget package and then published to the repository. As you can see, I’ve used the TF_BUILD_SOURCESDIRECTORY and TF_BUILD_BUILDDIRECTORY to get the source location and build location respectively in the module. To read more about TF_BUILD variables, refer to the link https://msdn.microsoft.com/en-us/library/hh850448.aspx.

I’ve created a script that I use to invoke the functions in this module

param

(

       [string]$Project

)

$exitCode = 0

trap

{

    $Host.UI.WriteErrorLine($error[0].Exception.Message)

    $Host.UI.WriteErrorLine($error[0].Exception.StackTrace)

    if ($exitCode -eq 0)

    {

        $exitCode = 1

    }

}

Import-Module .\TFSNugetExtensions.psm1

Publish-NugetPackage $Project

exit $exitCode

The next step is to publish these scripts along with the Nuget.exe file to the source control, so that I can refer these files as part of my build definition. I’ve added these files into the BuildExtensions folder in my team project

Now we can go ahead and create the build definition for our project. While creating the build definition it’s important to create the source settings mapping for the scripts folder so that they are also available as part of the build process.

In the process tab, you can now call the script which we have created as part of the post build execution phase.

That’s it, next time you queue a build, you can see that the build creates and uploads a nuget package to the repository automatically.