Category Archives: DevOps

CI/CD & Test Automation for Dynamics 365 in Azure DevOps/VSTS – Part 2- Build Definition

In the previous post, I showed you how you can export and extract the solution from Dynamics 365 CE instance and commit to VSTS.

In this blog post, I will tell you how to create the VSTS build definition.

Dynamics CRM CICD process

The below diagram illustrates the basic flow of the Dynamics 365 CE Workflow process.

In this part, we are going to see how to create a build definition in VSTS.

Pre-requisites

Install Dynamics 365 Build Tools by Wael Hamze in VSTS

To set up your tooling there is only one thing to do. You need to install an extension from the Visual Studio Marketplace within VSTS. Go to this link to add this extension to your VSTS account. You need to be an administrator to do this.

Setting up the variables for the connection to CRM

You can create variable groups in VSTS. This is useful for variables who are related. You will use this to create a group for credentials for your development environment. Later if you implement the automated deployment, you can store the credentials to other environments there too.

In VSTS go to “Build and Release” and select “Library”. Here you can create variable groups.

https://www.dynamichands.nl/wp-content/uploads/2018/07/LibraryLink.png

Next click on ‘+ Variable Group’. This will take you to a form where you can create a variable group.

https://www.dynamichands.nl/wp-content/uploads/2018/07/AddVariableGroup.png

Now you give your variable group a name. For our demo purpose, I am giving the example as a Development environment. Next, add the below variables

Connection string – AuthType=$(AuthType);Username=$(Username);Password=$(Password);Url=$(Url)

URL- Enter your instance URL.

Username – Enter the username of your instance

Password – Enter the password of your instance

Authtype – Office365

Below is the sample connection string details for your reference

AuthType=Office365;Username=jsmith@contoso.onmicrosoft.com; Password=passcode;Url=https://contoso.crm.dynamics.com

Reference Link https://docs.microsoft.com/en-us/previous-versions/dynamicscrm-2016/developers-guide/mt608573(v=crm.8)?redirectedfrom=MSDN

VSTS Build Definition

Once the developer checks-in the changes into BuildAutomation.Solutions project we have a Build Definition in VSTS which will trigger automatically and perform the following steps:

In Build Pipeline, we will be packing the CRM solution and store the solution file in Artifacts. You can create a build definition directly from Visual Studio Online(VSTS/Azure-DevOps) or from within Visual Studio. Firstly, I will show you how to create a build definition from within Visual Studio, navigate to the Builds tab in the Team Explorer:

https://www.insight.com/content/dam/insight-web/sitesections/knowledgebase/Cardinal/Images/BuildDefinitions-1.png
https://www.insight.com/content/dam/insight-web/sitesections/knowledgebase/Cardinal/Images/BuildDefinitions-2.png

Once there, you can click New Build Definition to be taken directly to Visual Studio Online. This is where you would start if you had decided to create the build definition directly from Visual Studio Online instead of starting in Visual Studio.

On the dialog box that pops up in the browser, we’ll select Visual Studio as our build template, but you can see there are other templates for use, such as Xamarin for Android or iOS and the Universal Windows Platform. The default settings for your build definition should be correct for the most part, but you’ll need to check the Continuous Integration checkbox. Here’s what they look like for this example:

https://www.insight.com/content/dam/insight-web/sitesections/knowledgebase/Cardinal/Images/BuildDefinitions-3.png

Because this is a simple example and we don’t need the additional flexibility the Default queue provides, we can leave the default Hosted option selected in the Default agent queue field. For more information on the restrictions on the Hosted pool.

 
You can see the checkbox for CI at the bottom of the dialog. This is enabled so that Visual Studio Online will execute the defined build definition for each check-in. The build definition will define whether or not this build code is published to an environment. Since we want to continually build and deploy to our web environment, we’ll check this box.

We can create the build definition from Azure Dev Ops too by following the below steps.

Navigate to Pipelines -> Builds -> Click New Pipeline

image003
image005

Click the Visual Designer, which will allow you to perform using GUI.

Select the Team Project, Repository, Branch and click on the continue button. For Demo purpose, I have given the name of the build definition as D365 CI

image007

In the next step, Select an empty job as shown below:

image010

After selecting an empty job, you can see the empty Agent Job and Select the Agent Pool as Hosted

image012

Once the above step is done, we need to associate the variable group which we have created in the previous step to the build definition variables. In the “Variables” tab, you can set up the variables which are connected to this definition. The first part is the process variables. These are variables specific and available only to this build definition. Just leave the default variables like they are.

The other part is that you can link variable groups. Earlier in the blog, you set up your credentials as a variable group. Here you click link variable group and select your variable group to link it to this build definition. After linking you can use these variables like they are process variables.

If you ever add your own variables, make sure that a variable in the process variable and a variable in the linked group(s) don’t have the same name. This can lead to unpredictable results.

https://www.dynamichands.nl/wp-content/uploads/2018/07/BuildLinkVariableGroup.png

System.Debug process variable

A good fact to know is the use of the ‘system.debug’ variable. If you set this to true, then verbose logging will be enabled on all build tasks. If you are stuck on some error in the process, this may help to get more information. It will create a massive amount of logging.

Next step is we need to configure the VSTS build definition as shown in the below diagram.

  1. UseNuGet4.4.1 & Nuget restore – The NuGet tasks are required to process the NuGet package references in the
  2. Build Solution – Then we use the Visual Studio Build task to build the complete solution
  3. Copy files – The Copy task copies all files in **\bin\$(BuildConfiguration)\** to the $(Build.ArtifactStagingDirectory).
  4. MSCRM Tool Installer – We are using the MSCRM tool installer. It will install the necessary files (Installs the Dynamics 365 tools required by all the tasks).
  5. MSCRM Set Solution Version – Then we are setting the solution version in the source environment using MSCRM set solution version component.

Note: We need to provide the connection string in the below format to establish the connection with Dynamics 365 CE instance which we have created initially.

  • MSCRM Pack Solution task creates an Unmanaged and managed solution from the files in Solution/.. within that project and we output the Zip file in the respective folder of the project.  The Copy task copies all files in **\bin\$(BuildConfiguration)\** to the $(Build.ArtifactStagingDirectory), we then Publish the Artifact.

In the MSCRM Set Version task, we must specify the Build Number Format under Options in the Build Definition. Here is an example of my build number $(Build.BuildNumber).

In my next blog, we will see how to create the VSTS Release definition(It will pick the CRM solution from artifacts and deploy it in the Dynamics 365 CE target instance).

If you are interested in this topic and would like to do some further self-study I encourage you to check out my blog on this.

CI/CD & Test Automation for Dynamics 365 in Azure DevOps/VSTS – Part 1

In this blog series, we will explore building out DevOps processes and practices for Dynamics 365 Customer Engagement (CE) by utilizing Wael Hamez MSCRM Build tools. In this first blog, we will cover the version control for Solutions.

What is DevOps?

DevOps is a new term emerging from the collision of two major related trends. The first was also called “agile infrastructure” or “agile operations”; it sprang from applying Agile and Lean approaches to operations work.  The second is a much-expanded understanding of the value of collaboration between development and operations staff throughout all stages of the development lifecycle when creating and operating a service, and how important operations has become in our increasingly service-oriented world (cf. Operations: The New Secret Sauce).

Problem Statement

I have started working on CI/CD when I was assigned as a Platform Engineer for a Dynamics 365 CE implementation project. At that time, I had a few key challenges with Dynamics 365 CE. I have listed those below:

  • Solution files were manually extracted and imported to target as a deployment process
  • No Unit testing or validation for deployed solution
  • Multiple deployment process is followed between release environments. For example, in Dev and Sit environment, the solution was migrated manually, and in UAT, Pre-Prod and Prod environment DB compare was applied to promote changes
  • Master data were mutually entered in each environment
  • Multiple developers working in the same organizations overwriting the changes.

Before we start solving the problem, let us take a moment to define some of the mandatory steps which we need to follow.

  • All the source code must be in source control. For example, Plugin Code, Custom workflows, actions, Warehouses (HTML, js, etc) Solution file, master Data, and user roles, etc.
  • Check-in regularly and every change should trigger the commit process
  • Commit process should be short and validates the committed component(Gated-Checkin)
  • Track changes and rollback as needed

Pre-Requisites

Here is a little bit of information regarding the environment.  We are using Dynamics 365 (Online) v9.0 and VSTS/Azure DevOps for version control, build and deployment.  I am using Visual Studio 2017 with the Microsoft Dynamics CRM SDK Templates extension installed.

I have also installed the Dynamics 365 Build Tools(By Wael Hamze) into our VSTS Organization so the tasks are available for all Build and Release Definitions.

Version control for solutions

Solutions in Dynamics 365 CE are in essence a package containing any customization we’ve done to our environment that we can export from one environment then import into various environments. When exported from an environment, solutions are in the form of a zip file. When that zip file is unzipped, the output directory contains folders for plugins, web resources, and any workflows we have made as well as XML files defining the schema of any customization we have done. In the zipped format, our schema definition is contained in one massive file. Consider this zip file as a binary, or in layman’s terms, a tidy package with a fancy bow, i.e. It may look nice but it’s not easy to see what’s inside and it’s a poor format for version control.

Solution packager is a tool that essentially takes our Dynamics 365 CE solution zip file and breaks it out into a logical folder structure by decomposing the contents. The resulting output shows a more granular view of our solution and is considerably more friendly for version control as you can see from the example screenshots below.

Dynamics 365 Instance Management

Below are the little regarding how we have managed the dynamics 365 instances for achieving this DevOps.

  1. Dev environment: It is the true source for all the customization and configuration changes.
  2. SIT: where you test all the functionality against different types of data by internal test users.
  3. UAT: where you test all the functionality against different types of data by business users.
  4. Production: Go Live Instance.

Dynamics CRM CICD process

The below diagram illustrates the basic flow of Dynamics 365 CE Workflow process.

In this part, we are going to see the highlighted one in detail.

The process followed by the developer is essentially these steps (pictured below), the two steps in the middle are handled by the SolutionExportDev.cmd script:

Items

It will export the solution from source instance and unpack the solutions into the folders in BuildAutomation.Solutions

  • Developers benefit using this process by making the solution deployment repeatable and predictable.
  • Developers can associate their work items with their check-ins
  • Very useful for tracking changes (traceability) and comparing solution files.
  • SolutionExport.ps1 – This will connect to the Dynamics 365 CE instance and export the Unmanaged and Managed solutions from that environment.
  • SolutionUnpack.cmd – This will extract the Unmanaged solution into the BuildAutomation.Solutions /BuildAutomation folder.  The Core Tools folder contains the Solution Packager tool from the Dynamics Core Tools NuGet package (pictured below)

Our solution consists of a CRM Deployment Packager project(BuildAutomation.SolutionPackager) and Solutions(BuildAutomation.Solutions) which is used as a container for the extracted files and folders of the solution. 

We added a Scripts(SolutionUnpack.cmd) folder to the project (BuildAutomation.SolutionPackager) which contains some below PowerShell scripts.

Powershell script for exporting the solution from the source environment.

$solutionName ="BuildAutomation"
$SolutionFilePath="C:\Users\DBALASU\Source\Workspaces\BuildAutomation\BuildAutomation.Solutions\BuildAutomation"
$CRMSourceUserName=""
$CRMSourcePassword=""
$CRMDestinationUserName=""
$CRMDestinationPassword=""
$CRMSourceUrl="https://instancename.crm.dynamics.com"
$CRMDestinationUrl="https://instancename.crm5.dynamics.com"

Set-StrictMode -Version latest
function InstallRequiredModule{
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force
$moduleName = “Microsoft.Xrm.Data.Powershell”
$moduleVersion = “2.7.2”
if (!(Get-Module -ListAvailable -Name $moduleName )) {
Write-host “Module Not found, installing now”
$moduleVersion
Install-Module -Name $moduleName -MinimumVersion $moduleVersion -Force
}
else
{
Write-host “Module Found”
}
}
function EstablishCRMConnection{
param(
[string]$CRMUserName,
[string]$CRMSecPasswd,
[string]$crmUrl)
Write-Host "UserId: $CRMUserName Password: $CRMSecPasswd CrmUrl: $crmUrl"
$CRMSecPasswdString = ConvertTo-SecureString -String $CRMSecPasswd -AsPlainText -Force
write-host "Creating credentials"
$Credentials = New-Object System.Management.Automation.PSCredential ($CRMUserName, $CRMSecPasswdString)
write-host "Credentials object created"
write-host "Establishing crm connection next"
$crm = Connect-CrmOnline -Credential $Credentials -ServerUrl $CrmUrl
write-host "Crm connection established"
return $crm
}

InstallRequiredModule

#Update Source CRM instance details below:
Write-Host "going to create source connection"
$CrmSourceConnectionString = EstablishCRMConnection -user "$CRMSourceUserName" -secpasswd "$CRMSourcePassword" -crmUrl "$CRMSourceUrl"
Write-Host "source connection created"
Set-CrmConnectionTimeout -conn $CrmSourceConnectionString -TimeoutInSeconds 1000

Write-Host "going to create destination connection"
$CrmSourceDestinationString = EstablishCRMConnection -user "$CRMDestinationUserName" -secpasswd "$CRMDestinationPassword" -crmUrl "$CRMDestinationUrl"
Write-Host "destination connection created"
Set-CrmConnectionTimeout -conn $CrmSourceDestinationString -TimeoutInSeconds 1000

Write-Host "Publishing Customizations in source environment"
Publish-CrmAllCustomization -conn $CrmSourceConnectionString
Write-Host "Publishing Completed in source environment."

Write-Host "Exporting Solution"
Export-CrmSolution -conn $CrmSourceConnectionString -SolutionName "$solutionName" -SolutionFilePath "$SolutionFilePath" -SolutionZipFileName "$solutionName.zip" 
Write-host "Solution Exported."
  1. $SolutionName – Name of the solution which you want to export the source instance.
  2. $SolutionFilePath – Desired folder path to export the solution
  3. $CRMSourceUserName – Username of the Dynamics 365 CE source instance
  4. $CRMDestinationUserName – Username of the Dynamics 365 CE destination instance
  5. $CRMDestinationPassword – Password of the Dynamics 365 CE destination instance
  6. $CRMDestinationPassword – Password of the Dynamics 365 CE source instance
  7. $CRMSourceUrl – URL of the Dynamics 365 CE source instance
  8. $CRMDestinationUrl – URL of the Dynamics 365 CE destination instance.

After updating the values in the above script, you can copy and paste in Windows PowerShell and run it. This will establish the connection to the Dynamics 365 CE source instance and export the solution to the $SolutionFilePath(Desired folder path to export the solution). Once this is done then run the below command file to extract the solution file.

Command Script:

rem @echo off
set solution=%1
set folder=%2

if "%solution%"=="" set solution=BuildAutomation
if "%folder%"=="" set folder=%solution%
if "%auth%"=="" set auth=ifd
set managed=false
set packagetype=unmanaged
set projectdir=BuildAutomation.Solutions

tools\solutionpackager.exe /action:Extract /packagetype:%packagetype% /zipfile:..\%projectdir%\%folder%\%solution%.zip /folder:..\%projectdir%\%folder%\ /allowDelete:No
exit /b
:EXIT
@pause

Developers need to do below the configuration in visual studio(Run the command prompt from inside Visual Studio) to execute the SolutionUnPack.cmd

To make the tool available, add it to the external tools list. Here are the steps:

  1. Open Visual Studio.
  2. Select the Tools menu, and then choose External Tools.
  • On the External Tools dialog box, choose the Add button. A new entry appears.
  • Enter a Title for your new menu item such as Command Prompt.
  • In the Command field, specify the file you want to launch, such as SolutionUnPack.cmd
  • In the Arguments field, please leave it as blank.
  • Choose a value for the Initial directory field, such as Project Directory.

Example:

Title – Name of the external tools to run

Command – Provide the physical path to the command file

Initial directory – Provide the directory path of the SolutionUnPack.cmd file.

Use Output Window – Enable

Prompt for Arguments ­– Disable

Close on exit–Enable

  • Choose the OK button.

The new menu item is added, and you can access the command prompt from the Tools menu as follows:

To run this, please click on ExportCrmSolution. It will export the solution from source instance and unpack the solutions into the folders in BuildAutomation. Solutions.

Once the above step is done, the developer now has some manual steps to perform depending on the files/changes that were extracted from the solution:

  1. Include any new file under the extracted folder Solutions\BuildAutomation (extracted) into the project- BuildAutomation. Solutions (so that they can be checked-in).
    NOTE: When doing the check-in, be sure to check the ‘Excluded Changes’ if additional files were detected and promote them to the ‘Included Changes’ as needed.
  2. For any .xaml file (under Solutions\BuildAutomation\Workflows or Solutions\BuildAutomation \Entities\xxxx\Formulas), make sure that Build Action is ‘None’ (not ‘Page’ or ‘XamlAppDef’). If this step is skipped, you may get one of this compilation error (s):
    – Project file must include the .NET Framework assembly ‘WindowsBase, PresentationCore, PresentationFramework’ in the reference list.
    – Assembly “System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ cannot be resolved”.
  • Ensure that DLL files under Solutions\..\PluginAssemblies are added to source control.
  • Undo all pending changes that do not specifically apply to your changes.
  • Associate the check-in with a work item(s), write a comment and check-in all required changes.

Note: Check-in to TFS will auto trigger the deployment process.

In my next blog, we will see how to pack the solution and deploy it in the Target instance using VSTS Build and Release definition.

If you are interested in this topic and would like to do some further self-study I encourage you to check out my blog on this.

Recent Entries »