Tuesday, July 26, 2016

Maintain a consistent development environment for your team using DSC and Chocolatey

PowerShell DSC configuration modes

PowerShell DSC offers two modes of applying a configuration to the nodes. PUSH and PULL.
In PUSH mode, you have to trigger the execution of the configuration on the node by using the Start-DSCConfiguration cmdlet. While in PULL mode the initiative to apply the configuration is given to the nodes itself. It now becomes the responsibility of the node to check for new configurations available in the sever and download and apply the configuration whenever there is an update available.

Compared to the PUSH mode, the PULL configuration is a bit more complex to setup. You need to setup a web service that uses an OData interface to make the configuration files available to the target nodes.

For more information on PowerShell DSC and configuration modes, please refer to the article here (https://msdn.microsoft.com/en-us/powershell/dsc/overview) .

In this post, I’ll explain the step by step process to configure a DSC pull server and setup your team’s development machines as clients that can download the latest configurations and modules from the pull server and apply it to install the required packages from chocolatey. This way you can manage the development configurations for the team in a central location and ensure that all the machines are running with the same configuration. We’ll also see how to check the compliance server to ensure that the machines are indeed in the desired state.

Step 1: Create and configure a pull server for publishing configurations and custom DSC resources

The first step is to create a DSC pull server that will be used to publish our development environment configurations and the custom resources and modules that we need to apply these configurations.
To setup a pull server, we need a server machine running WMF 5.0 or above, with IIS server role and DSC service added. I have created a VM in Azure (Windows 2012 R2 datacenter) and used as the pull server. The pull server creation is automated by DSC. The configuration script looks like.

param
(
    [Parameter(Mandatory=$false)]
    [String] $NodeName = 'localhost',

    [Parameter(Mandatory)]
    [String] $Key
)

Configuration PullServerConfiguration
{
    Import-DSCResource -ModuleName xPSDesiredStateConfiguration

    Node $NodeName
    {
        LocalConfigurationManager
        {
            ConfigurationMode = 'ApplyAndAutoCorrect'
            RefreshMode = 'Push'
            RebootNodeifNeeded = $node.RebootNodeifNeeded
        }

        WindowsFeature DSCServiceFeature
        {
            Ensure = 'Present';
            Name   = 'DSC-Service'          
        }

        xDscWebService PullServer
        {
            Ensure                  = 'Present';
            EndpointName            = 'PullServer';
            Port                    = $Node.Port;
            PhysicalPath            = "$env:SystemDrive\inetpub\PullServer";
            CertificateThumbPrint   = 'AllowUnencryptedTraffic';
            ModulePath              = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Modules";
            ConfigurationPath       = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Configuration";
            State                   = 'Started'
            DependsOn               = '[WindowsFeature]DSCServiceFeature'                        
        }

        File RegistrationKeyFile
        {
            Ensure          = 'Present'
            Type            = 'File'
            DestinationPath = "$env:ProgramFiles\WindowsPowerShell\DscService\RegistrationKeys.txt"
            Contents        = $Node.RegistrationKey
        }
    }
}

$ConfigParameters = @{
    AllNodes = @(
            @{
                NodeName = 'localhost'
                Port = 8080
                RegistrationKey = $Key
                RebootNodeifNeeded = $true
            }
        )
    }


PullServerConfiguration -ConfigurationData $ConfigParameters


I’ve saved this configuration to a file DSCPullServer.ps1. To apply the configuration, you need to create a Guid and pass it as a parameter to the configuration. This will be added as the registration key for the server to uniquely identify the server from the client nodes. This way clients can register to the server if they know the registration key and download configurations. The script to apply this configuration is


$ErrorActionPreference = 'Stop'
if((Get-ExecutionPolicy) -eq 'Restricted')
{
    throw 'Execution policy should be set atleast to RemoteSigned..'
}
if(-not(Test-WSMan -ErrorAction SilentlyContinue))
{
    Set-WSManQuickConfig -Force
}

if(-not(Get-Module -Name PackageManagement -ListAvailable))
{
    throw 'PackageManagement module should be installed to proceed'
}

if(-not(Get-Module -Name xPSDesiredStateConfiguration -ListAvailable))
{
    Install-Module -Name xPSDesiredStateConfiguration -Confirm:$false -Verbose
}

#Use [Guid]::NewGuid() | select -ExpandProperty Guid to generate a new key and paste the key here
$registrationKey =YOUR_GUID_HERE'

.\DSCPullServer.ps1 -NodeName 'localhost' -Key $registrationKey

Set-DscLocalConfigurationManager -Path .\PullServerConfiguration -Verbose -Force
Start-DscConfiguration .\PullServerConfiguration -Verbose –Force


Running the script will push the configuration on the server and create a site DSCPullServer. You can verify this by opening IIS and browsing the site. I’ve hosted the site on port 80 and removed the default website on port 80, but you can host this site on any other port like (8080).



Step 2: Create a Report/ Compliance server

If you need to know the status of each nodes and whether they are complaint to the latest configuration, you need to setup a report server an configure the LCM on the client nodes to send reports about its configuration status to the report server. Later you can retrieve this data by calling the reporting web service endpoint and check which nodes have succeeded applying the configuration and which ones have errors.

To configure the reporting server, we can use the same approach as the pull server. I have used a different server as my report server, but you can also use the same server which was used as  the pull server and run the report web service on a different port on the same server. The configuration script file is created as


param
(
    [Parameter(Mandatory=$false)]
    [String] $NodeName = 'localhost',

    [Parameter(Mandatory)]
    [String] $Key
)

Configuration ComplainceServerConfiguration
{
    Import-DSCResource -ModuleName xPSDesiredStateConfiguration

    Node $NodeName
    {
        LocalConfigurationManager
        {
            ConfigurationMode = 'ApplyAndAutoCorrect'
            RefreshMode = 'Push'
            RebootNodeifNeeded = $node.RebootNodeifNeeded
        }

        WindowsFeature DSCServiceFeature
        {
            Ensure = 'Present';
            Name   = 'DSC-Service'          
        }

        xDscWebService ComplainceServer 
        {
            Ensure                  = "Present"
            EndpointName            = "ComplainceServer"
            Port                    =  $Node.Port
            PhysicalPath            = "$env:SystemDrive\inetpub\wwwroot\ComplainceServer"
            CertificateThumbPrint   = "AllowUnencryptedTraffic"
            State                   = "Started"
        }

        File RegistrationKeyFile
        {
            Ensure          = 'Present'
            Type            = 'File'
            DestinationPath = "$env:ProgramFiles\WindowsPowerShell\DscService\RegistrationKeys.txt"
            Contents        = $Node.RegistrationKey
        }
    }
}

$ConfigParameters = @{
    AllNodes = @(
            @{
                NodeName = 'localhost'
                Port = 8080
                RegistrationKey = $Key
                RebootNodeifNeeded = $true
            }
        )
    }


ComplainceServerConfiguration -ConfigurationData $ConfigParameters


As the pull server, we need to create a Guid for the registration key and pass it as a parameter to this configuration when applied. I’ve saved the configuration to a file DSCComplainceServer.ps1. The script to apply the configuration looks like.


$ErrorActionPreference = 'Stop'
if((Get-ExecutionPolicy) -eq 'Restricted')
{
    throw 'Execution policy should be set atleast to RemoteSigned..'
}
if(-not(Test-WSMan -ErrorAction SilentlyContinue))
{
    Set-WSManQuickConfig -Force
}

if(-not(Get-Module -Name PackageManagement -ListAvailable))
{
    throw 'PackageManagement module should be installed to proceed'
}

if(-not(Get-Module -Name xPSDesiredStateConfiguration -ListAvailable))
{
    Install-Module -Name xPSDesiredStateConfiguration -Confirm:$false -Verbose
}

$registrationKey =YOUR_GUID_HERE'
.\DSCComplainceServer.ps1 -NodeName 'localhost' -Key $registrationKey

Set-DscLocalConfigurationManager -Path .\ComplainceServerConfiguration -Verbose -Force
Start-DscConfiguration .\ComplainceServerConfiguration -Verbose -Force


After the script is executed, you can verify the site in IIS the same way we tested the pull server.

Step 3: Create configurations for the development machines.

Before we configure the client machines, we need to create the configurations that need to be applied on the client nodes. I’ve decided to use the combination of PowerShell package management and Chocolatey to install the required dependencies on the nodes. To make the configurations easy and maintainable, we can make use of partial configurations on the nodes.

Partial configurations are basically named containers in the Local Configuration Manager (LCM) that can pull from different sources. Each configuration or container can have certain resources to be applied in the node. Later when the LCM is ready to validate the current configuration, it will put together all the partial configurations into one configuration to apply on the node. By making use of partial configurations we’ll split the resources that are needed for the base configuration and chocolatey packages in separate files.

In this demo, I’ll make use of two containers (BaseConfig and ChocoConfig), where the BaseConfig will contain resources to set some basic configuration on the machines and get it ready for installing the packages from chocolatey. To read more about partial configurations refer to the article here (https://msdn.microsoft.com/en-us/powershell/dsc/partialconfigs ). Let’s have a look into the code.


param
(
    [Parameter(Mandatory)]
    [string] $Path
)

configuration BaseConfig
{
    Node BaseConfig
    { 
             Registry DisableLUA
             {
                    Key = "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
                    ValueName = "EnableLUA"
                    ValueData = 1
                    ValueType = "DWORD"
                    Ensure = "Present"
             }
    }
}

BaseConfig -OutputPath $Path
param
(
    [Parameter(Mandatory)]
    [string] $Path
)

configuration ChocoConfig
{
    Import-DscResource -Module CChoco

    Node ChocoConfig
    { 
             cChocoInstaller InstallChocolatey
        {
            InstallDir = "C:\ProgramData\chocolatey"
        }

        cChocoPackageInstaller 7Zip
        {
            Name = '7zip.install'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller Pester
        {
            Name = 'Pester'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller NetFX40
        {
            Name = 'DotNet4.0'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller NetFX45
        {
            Name = 'DotNet4.5'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller AdobeReaderDC
        {
            Name = 'adobereader'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller ILSpy
        {
            Name = 'ilspy'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller NCrunch
        {
            Name = 'ncrunch-vs2015'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller PickPick
        {
            Name = 'picpick.portable'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller VisualStudioCode
        {
            Name = 'visualstudiocode'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }

        cChocoPackageInstaller VSCommunity
        {
            Name = 'visualstudio2015community'
            DependsOn = "[cChocoInstaller]InstallChocolatey"
        }
    }
}

ChocoConfig -OutputPath $Path


Save these configurations into BaseConfig.ps1 and ChocoConfig.ps1 respectively.

Step 4: Packaging the configurations and CChoco module in the pull server.

After creating the configurations, next step is to make this available on the pull server. Before we upload the configurations to the pull server, we need to ensure that the dependent module (cChoco) that contains the custom resources that are used in the configurations are also available on the pull server for the client nodes to download from.

If you have noticed the configuration applied on the pull server, we have mentioned the configuration path and module paths as


ModulePath = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Modules";
ConfigurationPath = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Configuration";


We need to upload the configuration and the CChoco module to this location. To upload the configurations and modules, we also need to provide a checksum associated with each configuration and module for the nodes to recognize any changes.

I’ve a script for doing all these J


$tempConfigPath = Join-Path $env:TEMP "PullConfigurations"
if(-not(Test-Path $tempConfigPath -ErrorAction SilentlyContinue)){
    New-Item -ItemType Directory -Force $tempConfigPath
}

$baseConfigFolder = Join-Path $tempConfigPath 'BaseConfig'
$chocoConfigFolder = Join-Path $tempConfigPath 'ChocoConfig'

.\BaseModuleConfig -Path $baseConfigFolder
.\ChocoPackageConfig -Path $chocoConfigFolder

"Creating checksum file at location $baseConfigFolder" | Write-Host
New-DscChecksum -ConfigurationPath $baseConfigFolder -OutPath $baseConfigFolder -Verbose -Force
"Creating checksum file at location $chocoConfigFolder" | Write-Host
New-DscChecksum -ConfigurationPath $chocoConfigFolder -OutPath $chocoConfigFolder -Verbose -Force


$configUploadFolder = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Configuration\"
"Copying the configuration and checksum files to $configUploadFolder" | Write-Host
Copy-Item -Path "$baseConfigFolder\*" -Destination $configUploadFolder
Copy-Item -Path "$chocoConfigFolder\*" -Destination $configUploadFolder


The above script will create a .mof file for each configuration, add a checksum associated to the .mof file and copy those files to the configuration folder on the pull server. Note that the script has to be executed on the pull server. Ideally you should use the invoke-command and remotely copy the files to the server. For this demo we’ll follow the simple scenario of creating these files directly on the pull server.

Next we need to pack the CChoco module as a zip and upload to the modules folder on the pull server. Again I have a script for doing that for us.


$parentFolder = "C:\Program Files\WindowsPowerShell\Modules"
$testModulePath = Join-Path $parentFolder 'CChoco'

"Compressing the module xDismFeature to $env:TEMP\CChoco_2.1.1.51.zip" | Write-Host
Add-Type -A System.IO.Compression.FileSystem
[IO.Compression.ZipFile]::CreateFromDirectory($testModulePath, "$env:TEMP\CChoco_2.1.1.51.zip")


$moduleUploadFolder = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Modules\"
"Moving the compressed folder to module share location $moduleUploadFolder" | Write-Host
Move-Item -Path $env:TEMP\CChoco_2.1.1.51.zip -Destination "$moduleUploadFolder"

"Creating checksum" | Write-Host
New-DSCCheckSum -path $moduleUploadFolder -force


This will copy the CChoco module available in the modules path to the pull server modules directory. Your configuration and modules folder on the pull server should now look like.



Step 5: Configure a dev machine to be the pull client

After having the server infrastructure ready, we are now all setup for adding our first pull client. To add a client node as a pull client and provide the required metadata to the LCM of the client node, we have to create a configuration and apply on the client. I’ve create the configuration for this and saved to a file ClientConfig.ps1.


param
(
    [Parameter(Mandatory)]
    [string] $ConfigurationServerUrl,

    [Parameter(Mandatory)]
    [string] $ConfigurationServerKey,

    [Parameter(Mandatory)]
    [string] $ModuleServerUrl,

    [Parameter(Mandatory)]
    [string] $ModuleServerKey,

    [Parameter(Mandatory)]
    [string] $ReportServerUrl,

    [Parameter(Mandatory)]
    [string] $ReportServerKey
)

[DSCLocalConfigurationManager()]
configuration PullClientConfiguration
{
    node localhost
    {
        Settings
        {
            AllowModuleOverwrite = $True;
            ConfigurationMode = 'ApplyAndAutoCorrect';
            ConfigurationModeFrequencyMins = 60;
            RefreshMode          = 'Pull';
            RefreshFrequencyMins = 30 ;
            RebootNodeIfNeeded   = $true;
        }

        #specifies an HTTP pull server for configurations
        ConfigurationRepositoryWeb DSCConfigurationServer
        {
            ServerURL          = $Node.ConfigServer;
            RegistrationKey    = $Node.ConfigServerKey;
            AllowUnsecureConnection = $true;
            ConfigurationNames = @("BaseConfig", "ChocoConfig")
        }

        PartialConfiguration BaseConfig
        {
            Description = "BaseConfig"
            ConfigurationSource = @("[ConfigurationRepositoryWeb]DSCConfigurationServer")
        }

        PartialConfiguration ChocoConfig
        {
            Description = "ChocoConfig"
            ConfigurationSource = @("[ConfigurationRepositoryWeb]DSCConfigurationServer")
            DependsOn = '[PartialConfiguration]BaseConfig'
        }

        #specifies an HTTP pull server for modules
        ResourceRepositoryWeb DSCModuleServer
        {
            ServerURL          = $Node.ModuleServer;
            RegistrationKey    = $Node.$ModuleServerKey;
            AllowUnsecureConnection = $true;
        }

        #specifies an HTTP pull server to which reports are sent
        ReportServerWeb DSCComplainceServer
        {
            ServerURL          = $Node.ComplainceServer;
            RegistrationKey    = $Node.ComplainceServerKey;
            AllowUnsecureConnection = $true;
        }
    }
}

$configParams = @{
    AllNodes = @(
        @{
            NodeName = 'localhost'
            ConfigServer = $ConfigurationServerUrl
            ConfigServerKey = $ConfigurationServerKey
            ModuleServer = $ModuleServerUrl
            ModuleServerKey = $ModuleServerKey
            ComplainceServer = $ReportServerUrl
            ComplainceServerKey = $ReportServerKey
        }
    )
}

PullClientConfiguration -ConfigurationData $configParams


To apply the configuration, you will need to provide the urls for the configuration server, report server and the registration keys for these servers. We can script that process as


$ErrorActionPreference = 'Stop'
if((Get-ExecutionPolicy) -eq 'Restricted')
{
    throw 'Execution policy should be set atleast to RemoteSigned..'
}
if(-not(Test-WSMan -ErrorAction SilentlyContinue))
{
    Set-WSManQuickConfig -Force
}

$configServerUrl = 'http://PULL-SERVER/psdscpullserver.svc/'
$reportServerUrl = 'http://REPORT-SERVER/PSDSCPullServer.svc/'
$configServerKey = 'PULL SERVER REG KEY'
$reportServerKey = 'REPORT SERVER REG KEY'

.\ClientConfig.ps1 -ConfigurationServerUrl $configServerUrl `
                    -ModuleServerUrl $configServerUrl `
                    -ReportServerUrl $reportServerUrl `
                    -ConfigurationServerKey $configServerKey `
                    -ModuleServerKey $configServerKey `
                    -ReportServerKey $reportServerKey


Set-DscLocalConfigurationManager -Path .\PullClientConfiguration -Verbose -Force


That’s all we need to setup automated dev machines using PowerShell DSC!!!. When the client machine wakes up next time to pull the configuration, it will get the partial configurations from the server and apply that on the machine. After executing the configuration, it will report the data back to the reporting server.

You can open the logs in the event viewer to check the status of execution or use the Get-DSCConfiguration cmdlet to check the applied configuration on the machine.