SharePoint List Versioning & Change Control

Introduction

Whenever we have a solution that involves a SharePoint component such as SharePoint Online, or a SharePoint On-Premise (201*) environment there’s always a question around what to do with SharePoint Lists. There are times where we are using the Lists as data storage either for incoming data via an integration API or used by custom built Web-Parts as a data source.

Every time a developer or consultant needs to make changes to these SharePoint lists or needs to consolidate a list from their local or personal development instance with a Test or Production instance there isn’t an existing method for adequately manage these through source control mechanisms like the ones provided within Azure DevOps. Sure SharePoint can maintain a version history against a library or list, but in large scale solutions you might want a more central and robust versioning and branching strategy for all aspects of your solution, including critical SharePoint lists.

At the time of writing there is no pre-built method of determining the changes between two lists when scripting deployment or updates of SharePoint lists. With that in mind and a little discovery work we realised that the downloaded .stp files that we were storing within Azure DevOps in order to manage versioning and branching were simply .cab files and that mean that we could extract a fairly length .xml manifest file which contained all of the information that SharePoint uses to deploy the list. With this in mind it meant that I could create an application that could extract this file and parse the xml into something we could use to compare lists, examine list views, and aid in our change control requirements.

So with all of that in mind I set out to create a simple WPF (Windows Presentation Foundation) application written in C# to help me compare, version and branch our SharePoint lists as well as help with the creation of Release Notes for each push to production of our solution.

Extracting the .stp lists column headers & views

So because we want to be able to compare the column headers between two lists to see if any have been added, removed, or renamed we need to first extract this information from the manifest.xml document. Fortunately for us there is a node called ‘Fields’ within the metadata tag and this contains various attributes against each column, but all we really need is the display name. The application does also extract the column type, but at the moment this isn’t something we look at during the comparison analysis.

The XPath to fields within the Manifest.xml file is as follows:

/ListTemplate/UserLists/List/MetaData/Fields

This is an example of a standard Field node and it’s attributes within the manifest.xml file.

<Field DisplayName=”ColumnOne” Format=”Dropdown” MaxLength=”255″ Name=”ColumnOne” Title=”ColumnOne” Type=”Text”
 ID=”{7f1e1d78-4813-4902-b2c6-024033c51d24}” SourceID=”{5d8ded26-11b5-456d-a893-97ec21665918}” StaticName=”ColumnOne” ColName=”nvarchar4″ RowOrdinal=”0″/>  

The XPath to views within the Manifest.xml file is as follows:

/ListTemplate/UserLists/List/MetaData/Views.

This contains as many views as there are created for the particular SharePoint list being examined. The <View> xml node contains a few attributes that might be useful, but for us we just wanted to get the name of the view, so we simply extract the ‘DisplayName’ attribute and then we can proceed to extract the column headers within the view, these are referred to as ‘FieldRefs’ within the ‘ViewFields’ achild node of ‘View’

<ViewFields>
	<FieldRef Name=”LinkTitle”/>
	<FieldRef Name=”ColumnOne”/>
	<FieldRef Name=”ColumnTwo”/>
	<FieldRef Name=”ColumnThree”/>
	<FieldRef Name=”ColumnFour”/>
	<FieldRef Name=”ColumnFive”/>
</ViewFields>

Now this data can be extracted in three different ways. The first and easiest method is by simply by deserialising the manifest.xml into a class object built from the xml structure. That’s done using auto mapping provided by what’s called an XML Serialiser. This allows us to quickly and easily extract the data needed into our own internal models ready for comparisons.

There are instances however where this does not work, Microsoft SharePoint List Templates sometimes have undeclared namespaces within the manifest.xml often related to the data node that contains any rows that might be stored within the template as content. In this instance the manifest.xml fails to load into our application properly, so we have to go through one of two other methods of manually retrieving the information we need.

If the manifest.xml has loaded into an XML Document appropriately; no undeclared namespaces, but a different structure than we are expecting then we can use XPaths to navigate and select the appropriate nodes, this then allows us to extract the column headers and views fairly easily.

However if the manifest.xml is malformed with undeclared namespaces then we fall back to using something a little more messy; RegEx. RegEx stands for regular expressions and it’s used most commonly to find and match patterns in text. We use RegEx as our third method of manual data retrieval by searching through the manifest.xml for specific patterns that match the <Views> and <Fields> opening and closing xml tags.

By using all three potential methods of extracting the data we need, it means we can remain resilient to changes to the structure of the xml, or even malformations of the xml.

Working with the SharePoint Online/On-Premise lists

Thanks to the SharePoint CSOM library already available to us from Microsoft, working with the online lists and getting them in a format ready for comparison is quite straight forward. All we had to do for the application to work was allow the user to enter required SharePoint information and then specify whether it was an Online or On-Premise environment. The only difference being that the On-Premise SharePoint accepts ‘NetworkCredentials’ and SharePoint Online uses a specific ‘SharePointOnlineCredentials’ class to authenticate the user.

Once the user has entered this information they can simply scroll through a table of lists and select the ones they want to compare. We get this table of lists by performing a simple retrieval process in the code, as follows:

ClientContext clientContext = new ClientContext(_siteURL);
clientContext.Credentials = new SharePointOnlineCredentials(username, spSecurePassword);
Web web = clientContext.Web;
clientContext.Load(web.Lists, lists => lists.Include(list => list.Title, list => list.Id));
clientContext.ExecuteQuery();
ListCollection lists = web?.Lists;

This gives us a list  collection with all the list Titles and ID’s loaded ready for us to display to the user. We don’t actually pull all of the data for the lists for the user at this stage as any given site could have huge volumes of lists so we want to keep our calls to SharePoint fairly responsive and quick.

Once a user has selected lists they want to compare, we call SharePoint again and retrieve the information we are interested in, namely the column headers (Fields) and the Views for the list. This is done in a similar way to how we retrieved the SharePoint site lists. We create a method that takes a pointer to the SharePoint list the user selected and that we retrieved above.

if (list is List spList)
{
	clientContext.Load(spList.Views);
	clientContext.Load(spList.Fields);
	
	// We must call ExecuteQuery before enumerate list.Fields.
	clientContext.ExecuteQuery();
	
	// right now we need the view fields as well
	foreach (var view in spList.Views)
	{
		clientContext.Load(view.ViewFields);
	}
	
	// Call ExecuteQuery again
	clientContext.ExecuteQuery();
	return spList;
}

Development Considerations

Because it’s possible for a user to add in multiple lists with no upper limit it was necessary to consider an optimised approach to developing the utility application. This was done by implementing multi-threading and making use of the Parallel tasking library where possible.

Task.Run(() =>
{
	// Code here runs on it’s own thread assigned from the Thread Pool.
	Dispatcher.Invoke(() =>
	{
		//Runs on the UI thread.
	});
}

Tasks are used to contain long running code so that they don’t block the UI thread and can run in there own thread pool. Wherever necessary when we need to return information back to the UI we wrap it in the Dispatchers invoke method which tells the application to perform the requested action on the main UI thread. ‘Parallel ForEach’ statements were used where we needed to loop through lists of data but we didn’t care about maintaining the order, this meant that for a list that might contain 100 records, instead of looping through one record at a time, we could parallel this and look at multiple records in that list concurrently and do what we needed to do, meaning that the time to complete the process was reduced. All of this works to help improve performance when having to deal with a large volume of data processing.

Closing thoughts

I have to say it was rather fun to mess around with these manifest.xml files, but if i don’t end up having to use RegEx for awhile, I ‘m not going to complain. There’s definitely room to add additional functionality to the application, but for the moment it’s proving to be a useful tool in helping us version these lists as we amend them moving forward. This blog post is only a very brief overview of some of the concepts that had to be considered when working on this utility application, and it’s worth noting that the application is currently available on GitHub in it’s entirety and can be used and adjusted under the MIT licence, you can inspect the code and raise bugs and issues if you find them which will be reviewed and fixed if applicable. If you just want to download it, click on this link, otherwise click on the banner below to see the GitHub Repo.

About the author