Issue Spotlight #1 – Microsoft.Bing.Speech API – Bond.IO Issues

Recently we ran into an issue when we started working with the Bing Speech API as part of our Bot-framework demo initiative. When using the latest versions of the Microsoft.Bing.Speech SDK within a Web-API we saw it throw some interesting errors; so I thought we’d highlight our lessons learned and take a deeper look into the problem.

The Set-up:

Front-end: ASP.Net MVC Razer website.

• .Net Framework 4.6.1

Back-end: Bot-framework Web API (RESTful).

• .Net Framework 4.6

Back-Back-end: We use various Azure located cognitive services but in this case it’s just the Bing Speech API.

Relevant SDKs:

• Microsoft.Bing.Speech (Version: 2.0.2)
  • Bond.Core.CSharp (Version: 8.0.0) ~ dependency
  • Bond.CSharp (Version: 8.0.0) ~ dependency
  • Bond.Runtime.CSharp (Version: 8.0.0) ~ dependency

The above frameworks should work fine together, for most of the cognitive platform that Microsoft provides there’s no issue, and linking them all up to a ASP.net website as the front end is pretty straight forward, even if you don’t want to use Node.JS but rather standard JavaScript to complete your client side actions for interacting with the Bot-Framework.

We were using getUserMedia() on the website to record the users microphone upon request from some custom JavaScript code (We wanted to start by simply passing audio blocks, rather than streaming the audio real-time, at least to begin with). This creates a blob URL. I then pass the blob URL as the ContentUrl within an Attachment to an Activity. When this hits the Bot-framework I do some basic validation (nothing related to this problem), and then pass to a custom Dialog<T>.

Code Snippet:

public async Task Run(string audioFile, string locale, Uri serviceUrl)
{
    // create the preferences object
    var preferences = new Preferences(locale, serviceUrl, new CognitiveServicesAuthorizationProvider(subscriptionKey));

    using (var speechClient = new SpeechClient(preferences))
    {
        speechClient.SubscribeToPartialResult(this.OnPartialResult);
        speechClient.SubscribeToRecognitionResult(this.OnRecognitionResult);

        using (WebClient webClient = new WebClient())
        {
            using (Stream stream = webClient.OpenRead(audioFile))
            {
                var deviceMetadata = new DeviceMetadata(DeviceType.Near, DeviceFamily.Desktop, NetworkType.Ethernet, OsName.Windows, "1607", "Dell", "T3600");
                var applicationMetadata = new ApplicationMetadata("SampleApp", "1.0.0");
                var requestMetadata = new RequestMetadata(Guid.NewGuid(), deviceMetadata, applicationMetadata, "SampleAppService");

               var ms = new MemoryStream();
                stream.CopyTo(ms);
                ms.Position = 0;

               try
               {
                    await speechClient.RecognizeAsync(new SpeechInput(ms, requestMetadata), this.cts.Token).ConfigureAwait(false);
               }
               catch
               {
                    // Was just using this try/catch for debugging reasons
               }
           }
       }
   }
}

When you run the above code you’ll get an error about a Bond.IO dependency that the Microsoft Bing Speech SDK is trying to reference (Version 1.0.0.0) . Unfortunately the SDK only has a dependency requirement of Bond.IO version 8.0.0 minimum. In fact you can’t actually get an earlier version of Bond.IO, the earliest nuget package version is 7.0.1

If you download the sample code from Microsoft you’ll notice that they actually use a much older version of both the Bond.IO library and an older version of the Bing Speech SDK.

Error Snippet:

(Local paths are simply due to local debugging)

=== Pre-bind state information ===
LOG: DisplayName = Bond.IO, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///[project folder]
LOG: Initial PrivatePath = \bin
Calling assembly : Microsoft.Bing.Speech, Version=2.0.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file:\web.config
LOG: Using host configuration file: \aspnet.config
LOG: Using machine configuration file from \machine.config.
LOG: Post-policy reference: Bond.IO, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Attempting download of new URL file:///C:/Users/[USER]/AppData/Local/Temp/Temporary ASP.NET Files/vs/0f4bb63f/ca796715/Bond.IO.DLL.
LOG: Attempting download of new URL file:///C:/Users/[USER]/AppData/Local/Temp/Temporary ASP.NET Files/vs/0f4bb63f/ca796715/Bond.IO/Bond.IO.DLL.
LOG: Attempting download of new URL file:///C:/[USER]/[PROJECT PATH]/bin/Bond.IO.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

Unfortunately it looks as though right now support for the Microsoft.Bing.Speech API is fairly limited. Microsoft appear to be collecting their speech AI and ML tools and combining client access for them into a central Speech SDK.

The people working on Bond.IO project on GitHub introduced a breaking change between lower versions and the two latest versions that are currently listed on nuget (7.0.1 and 8.0.0).

This was an intentional breaking change between 5.x and 6.x to enable people outside of Microsoft to build and use strong-named signed Bond assemblies.

Breaking change Bond assemblies are now strong-name signed with the bond.snk key in > the repository instead of with a Microsoft key. This allows anyone to produce compatible > assemblies, not just Microsoft. Official distribution of Bond will continue to be > > Authenticode signed with a Microsoft certificate. Issue #414

The new public key for assemblies is now [Truncated a public key example]

Breaking change Bond assemblies now have assembly and file versions that correspond to their NuGet package version. Strong name identities will now change release-over-release in line with the NuGet package versions. Issue #325 ¹

This seemed to mean that upgrading the Microsoft.Bing.Speech api to it’s latest versions 2.0.1 and 2.0.2 (bear in mind these are the only two available on nuget) could only install Bond.IO 7.0.1 or above. However they still contained an internal requirement on version 1.0.0.0 of Bond.IO (or more explicitly any build before 7.0.1).

It’s also worth highlighting that if you manually install the packages from the microsoft sample project which target older versions of both the Microsoft.Bing.Speech assembly and the Bond.IO version 4.2.1 assembly the above code works without issue.²

There are also comments on one of the Microsoft Docs page by one of the contributors that the Microsoft.Bind.Speech assembly is on it’s way to being depreciated (would have been nice if they had marked it as such, am I right.)³

In short there are two options to solve the problem. Either grab the older Libraries from Microsoft’s sample project on GitHub. Or attempt to develop against the Speech SDK (although it is still in preview). It’s worth noting that you may also find some issues integrating the Speech SDK directly into your Bot-Framework API ⁴, so it would need to sit in it’s own container, perhaps an azure function.

Reference sources:

¹ Please see this issue against the Bond.IO Github

² Comment on a similar question supporting this.

³ Look under closed comments at the bottom of this page, the response by ‘Zhouwangzw’ suggests using the latest Speech SDK.

³ Found the GitHub issue that linked to the docs here

⁴ Current breaking error in a Web-API using the Speech SDK.