Failed to acquire server box RAC and CLC for the tenant from ”.

2017-12-07T10:23:13+00:00 June 29th, 2015|Azure, Cloud|

While recently onsite with a customer deploying Azure Rights Management Services with Exchange 2010 Integration I had an interesting issue where OWA integration wasn’t playing nicely at all.

To summarise the setup, the customer had a hybrid setup with mailboxes in Office 365 and on premise and wished to deploy Azure RMS to provide protection capabilities within Office 365 and their on premise deployment.

I enabled Azure RMS and installed the RMS Sharing Application on an end user device, deployed a test template and verified everything worked as expected.

I then performed integration with Office 365 which worked perfectly.

In order to protect on premise applications, Azure RMS requires the RMS Connector be deployed. To provide high availability, I deployed two load balanced servers with the RMS Connector installed and added the Exchange servers containing the CAS and Hub Transport servers in to the RMS Connector. The RMS Connector was deployed in https mode with a certificate from an internal CA.

Having deployed the required infrastructure for the hybrid RMS deployment, I then performed the integration with Exchange 2010 and ran Test-IRMConfiguration Cmdlet which returned an overall PASS with no issues present. At risual, we perform end to end testing of our deployments before handovers to the customer – and it was while doing this that I uncovered the following issue. When attempting to send an e-mail within OWA on premise and selecting a template the following error message was displayed –

Failed to acquire server box RAC and CLC for the tenant from ”.

I verified once again that the Test-IRMConfiguration returned an overall PASS result. I checked once again that Exchange 2010 met the minimum requirements (SP3 UR3) and that there were no known issues with Server 2012 R2. I then investigated the event viewer logs on the RMS Connector servers where I discovered the following exception (sanitised):

Request
Url:
User: testuser
EX Address:
SMTP Address:
OWA version: 14.3.210.2
Mailbox server:

Exception
Exception type: Microsoft.Exchange.Data.Storage.RightsManagementPermanentException
Exception message: Failed to acquire server box RAC and CLC for the tenant from ”.

Call stack

Microsoft.Exchange.Data.Storage.MsgToRpMsgConverter.CallRM(RMCall call, LocalizedString error)
Microsoft.Exchange.Data.Storage.MsgToRpMsgConverter.InitTenantLicenses()
Microsoft.Exchange.Data.Storage.MsgToRpMsgConverter..ctor(MessageItem envelopeMessage, Participant conversationOwner, OrganizationId orgId, RmsTemplate restriction, OutboundConversionOptions options)
Microsoft.Exchange.Data.Storage.RightsManagedMessageItem.OnBeforeSave()
Microsoft.Exchange.Data.Storage.Item.SaveInternal(SaveMode saveMode, Boolean commit)
Microsoft.Exchange.Data.Storage.MessageItem.OnBeforeSend()
Microsoft.Exchange.Data.Storage.RightsManagedMessageItem.OnBeforeSend()
Microsoft.Exchange.Data.Storage.CoreItem.OnBeforeSend()
Microsoft.Exchange.Data.Storage.CoreItem.Submit(SubmitMessageFlags submitFlags)
Microsoft.Exchange.Data.Storage.MessageItem.InternalSend(StoreObjectId saveSentMessageFolder, SubmitMessageFlags submitFlags)
Microsoft.Exchange.Clients.Owa.Premium.EditMessageEventHandler.ProcessMessageRequest(MessageAction action)

Inner Exception
Exception type: Microsoft.Exchange.Security.RightsManagement.RightsManagementException
Exception message: Failed to acquire server box RAC from http://rmsconnector.CustomerDomain.local/_wmcs/certification/servercertification.asmx.

Call stack

Microsoft.Exchange.Data.Storage.RightsManagement.RmsClientManager.EndAcquireInternalOrganizationRACAndCLC(IAsyncResult asyncResult)
Microsoft.Exchange.Data.Storage.MsgToRpMsgConverter.b__10()
Microsoft.Exchange.Data.Storage.MsgToRpMsgConverter.CallRM(RMCall call, LocalizedString error)

Inner Exception
Exception type: System.Web.Services.Protocols.SoapException
Exception message: Exception of type ‘System.Web.Services.Protocols.SoapException’ was thrown. —> Microsoft.DigitalRightsManagement.Core.VerifyMachineCertificateChainFailedException: —> Exception of type ‘Microsoft.DigitalRightsManagement.Core.VerifyMachineCertificateChainFailedException’ was thrown.

Call stack

System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
Microsoft.Exchange.Net.WsAsyncProxyWrapper.EndInvoke(IAsyncResult result)
Microsoft.Exchange.Security.RightsManagement.SOAP.ServerCertification.ServerCertificationWS.EndCertify(IAsyncResult asyncResult)
Microsoft.Exchange.Security.RightsManagement.ServerCertificationWSManager.EndAcquireRac(IAsyncResult asyncResult)

Most of the items I found during my research in to the issue involved giving Read and Execute permissions to the ServerCertification.asmx file within the RMS Connector IIS site. I did this but the issue did not resolve.

I then did some further digging in to the C:ProgramDataMicrosoftDrmServer folder where I discovered that some of the customers Exchange Servers had multiple entries contained in this folder (note the Server folder is hidden, so you will need to manually enter the path). These will be in the form of folders that resemble SID entries (S-1-5-18, S-1-5-20). The issue here seems to be that there had been a previous RMS deployment of which the customer was unaware that never went in to production.

The fix here is to:

1) Turn off the IRM integration Set-IRMConfiguration -InternalLicensingEnabled $false

2) Back up and then delete the folder entries in C:ProgramDataMicrosoftDRMServer (I did this on all the CAS and Hub Transport boxes)

3) Reboot each Exchange box which has the CAS and Hub Transport role

4) Once all Exchange boxes have been rebooted turn IRM integration back on using Set-IRMConfiguration -InternalLicensingEnabled $true

5) Run Test-IRMConfiguration

You should notice that the Server folder re-populates with a single new folder and that OWA integration now works successfully.