Showing posts with label OpenAI. Show all posts
Showing posts with label OpenAI. Show all posts

Thursday, March 12, 2026

Function Calling with Microsoft Agent Framework, C#, & Entity Framework

In this article, we will create a Microsoft Agentic Framework plugin that contains four functions that interact with live SQLite data. Entity Framework will be used to access the SQLite database. The end result is to use the powers of the OpenAI natural language models to ask questions and get answers about our custom data.

Source Code: https://github.com/medhatelmasry/EfFuncCallMAF

Pre-requisites

  • You will be using AI models hosted on GitHub. Therefore, you will need to obtain a personal access token from GitHub.
  • .NET Framework 10.0+

Getting Started

Let’s start by creating an ASP.NET Razor pages web application. Select a suitable working folder on your computer, then enter the following terminal window commands:

dotnet new razor --auth individual -o EfFuncCallMAF
cd EfFuncCallMAF

Te above creates a Razor Pages app with support for Entity Framework and SQLite.

Add these packages:

dotnet add package CsvHelper
dotnet add package Microsoft.Agents.AI.OpenAI --prerelease 
dotnet add package Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore
dotnet add package Microsoft.AspNetCore.Identity.EntityFrameworkCore
dotnet add package Microsoft.AspNetCore.Identity.UI
dotnet add package Microsoft.EntityFrameworkCore.Sqlite

The CsvHelper package will help us load a list of products from a CSV file named students.csv and hydrate a list of Student objects. The second package is needed to work with Microsoft Agent Framework. The rest of the packages support Identity, Entity Framework and SQLite.

Let’s Code

appsettings.json

Add these to appsettings.json:

"GitHub": {
  "Token": "PUT-GITHUB-PERSONAL-ACCESS-TOKEN-HERE",
  "ApiEndpoint": "https://models.github.ai/inference",
  "Model": "openai/gpt-4o-mini"
}

Of course, you need to adjust the Token setting with your GitHub personal access token.

Data

Create a folder named Models. Inside the Models folder, add the following Student class: 

public class Student {
   public int StudentId { get; set; }

   [Display(Name = "First Name")]
   [Required]
   public string? FirstName { get; set; }

   [Display(Name = "Last Name")]
   [Required]
   public string? LastName { get; set; }

   [Required]
   public string? School { get; set; }
 
   public override string ToString() {
      return $"Student ID: {StudentId}, First Name: {FirstName}, Last Name: {LastName}, School: {School}";
   }
}

Developers like having sample data when building data driven applications. Therefore, we will create sample data to ensure that our application behaves as expected. Copy CSV data from this link and save it to a text file wwwroot/students.csv.

Add the following code inside the Data/ApplicationDbContext class located inside the Data folder:

public DbSet<Student> Students => Set<Student>();    
 
protected override void OnModelCreating(ModelBuilder modelBuilder) {
    base.OnModelCreating(modelBuilder);
    modelBuilder.Entity<Student>().HasData(LoadStudents());
}  
 
// Load students from a csv file named students.csv in the wwwroot folder
public static List<Student> LoadStudents() {
    var students = new List<Student>();
    using (var reader = new StreamReader(Path.Combine("wwwroot", "students.csv"))) {
        using var csv = new CsvReader(reader, CultureInfo.InvariantCulture);
        students = csv.GetRecords<Student>().ToList();
    }
    return students;
}

Let us add a migration and subsequently update the database. Execute the following CLI commands in a terminal window.

dotnet ef migrations add M1 -o Data/Migrations
dotnet ef database update

At this point the database and tables are created in a SQLite database named app.db.

Helper Methods

We need a couple of static helper methods to assist us along the way. In the Models folder, add a class named Utils and add to it the following class definition:

public class Utils {
  public static string GetConfigValue(string config) {
    IConfigurationBuilder builder = new ConfigurationBuilder();
    if (System.IO.File.Exists("appsettings.json"))
      builder.AddJsonFile("appsettings.json", false, true);
    if (System.IO.File.Exists("appsettings.Development.json"))
      builder.AddJsonFile("appsettings.Development.json", false, true);
    IConfigurationRoot root = builder.Build();
    return root[config]!;
  }
 
  public static ApplicationDbContext GetDbContext() {
    var optionsBuilder = new DbContextOptionsBuilder<ApplicationDbContext>();
    var connStr = Utils.GetConfigValue("ConnectionStrings:DefaultConnection");
    optionsBuilder.UseSqlite(connStr);
    ApplicationDbContext db = new ApplicationDbContext(optionsBuilder.Options);
    return db;
  }
}

Method GetConfigValue() will read values in appsettings.json from any static method. The second GetDbContext() method gets an instance of the ApplicationDbContext class, also from any static method.

Plugins

Create a folder named Plugins and add to it the following class file named StudentPlugin.cs with this code:

public class StudentPlugin {
  [Description("Get student details by first name and last name")]
  public static string? GetStudentDetails(
    [Description("student first name, e.g. Kim")]
    string firstName,
    [Description("student last name, e.g. Ash")]
    string lastName
  ) {
      var db = Utils.GetDbContext();
      var studentDetails = db.Students
        .Where(s => s.FirstName == firstName && s.LastName == lastName).FirstOrDefault();
      if (studentDetails == null)
          return null;
      return studentDetails.ToString();
  }

  [Description("Get students in a school given the school name")]
  public static string? GetStudentsBySchool(
  [Description("The school name, e.g. Nursing")]
  string school
  ) {
      var studentsBySchool = Utils.GetDbContext().Students
        .Where(s => s.School == school).ToList();
      if (studentsBySchool.Count == 0)
          return null;
      return JsonSerializer.Serialize(studentsBySchool);
  }


  [Description("Get the school with most or least students. Takes boolean argument with true for most and false for least.")]
  static public string? GetSchoolWithMostOrLeastStudents(
  [Description("isMost is a boolean argument with true for most and false for least. Default is true.")]
  bool isMost = true
  ) {
      var students = Utils.GetDbContext().Students.ToList();
      IGrouping<string, Student>? schoolGroup = null;
      if (isMost)
          schoolGroup = students.GroupBy(s => s.School)
              .OrderByDescending(g => g.Count()).FirstOrDefault()!;
      else
          schoolGroup = students.GroupBy(s => s.School)
              .OrderBy(g => g.Count()).FirstOrDefault()!;
      if (schoolGroup != null)
          return $"{schoolGroup.Key} has {schoolGroup.Count()} students";
      else
          return null;
  }

  [Description("Get students grouped by school.")]
  static public string? GetStudentsInSchool() {
      var students = Utils.GetDbContext().Students.ToList().GroupBy(s => s.School)
        .OrderByDescending(g => g.Count());
      if (students == null)
          return null;
      else
          return JsonSerializer.Serialize(students);
  }
}

 In the above code, there are four methods with these purposes:

GetStudentDetails()Gets student details given first and last names
GetStudentsBySchool()Gets students in a school given the name of the school
GetSchoolWithMostOrLeastStudents()Takes a Boolean value isMost – true returns school with most students and false returns school with least students.
GetStudentsInSchool()Takes no arguments and returns a count of students by school.

Registering the Chat Client

In the Program.cs file, add the following code to register the MAF chat client so it is available for dependency injecton. The code goes before the "var app = builder.Build();" statement.

string? apiKey = builder.Configuration["GitHub:Token"];
string? model = builder.Configuration["GitHub:Model"] ?? "openai/gpt-4o-mini";
string? endpoint = builder.Configuration["GitHub:ApiEndpoint"] ?? "https://models.github.ai/inference";

builder.Services.AddSingleton<IChatClient>(_ =>
    new OpenAIClient(
        new ApiKeyCredential(apiKey!),
        new OpenAIClientOptions { Endpoint = new Uri(endpoint!) }
    ).GetChatClient(model!).AsIChatClient()
);

The User Interface

We will re-purpose the Index.cshtml and Index.cshtml.cs files so the user can enter a prompt in natural language and receive a response that comes from the OpenAI model working with our Microsoft Agent Framework plugin. 

Index.chtml.cs

Replace the IndexModel class definition in Pages/Index.cshtml.cs with:

public class IndexModel : PageModel {
  private readonly ILogger<IndexModel> _logger;
  private readonly IChatClient _chatClient;

  [BindProperty]
  public string? Reply { get; set; }

  public IndexModel(ILogger<IndexModel> logger, IChatClient chatClient) {
    _logger = logger;
    _chatClient = chatClient;
  }
  public void OnGet() { }
  // action method that receives prompt from the form
  public async Task<IActionResult> OnPostAsync(string prompt) {
    var response = await CallFunction(prompt);
    Reply = response;
    return Page();
  }

  private async Task<string> CallFunction(string question) {
    // Create tools from StudentPlugin methods
    var tools = new List<AITool> {
      AIFunctionFactory.Create(StudentPlugin.GetStudentDetails),
      AIFunctionFactory.Create(StudentPlugin.GetStudentsBySchool),
      AIFunctionFactory.Create(StudentPlugin.GetSchoolWithMostOrLeastStudents),
      AIFunctionFactory.Create(StudentPlugin.GetStudentsInSchool),
    };

    // Create the AI agent with tools
    var agent = _chatClient.AsAIAgent(
      instructions: "You are a helpful assistant that can look up student information.",
      name: "StudentAgent",
      tools: tools
    );

    // Run streaming and collect the response
    string fullMessage = "";
    await foreach (var update in agent.RunStreamingAsync(question)) {
      if (!string.IsNullOrEmpty(update.Text)) {
        fullMessage += update.Text;
      }
    }
    return fullMessage;
  }
}

In the above code, the prompt entered by the user is posted to the OnPostAsync() method. The prompt is then passed to the CallFunction() method, which returns the final response from Azure OpenAI.

The CallFunction() method sets up the AI agent with tools.

Note that the IChatClient object is available through dependency injection

All the tools (or plugins) are loaded into a list of AITool objects.

Index.chtml

Replace the content of Pages/Index.cshtml with:

@page
@model IndexModel
@{
    ViewData["Title"] = "Function Calling with Microsoft Agent Framework";
}
<div class="text-center">
    <h3 class="display-6">@ViewData["Title"]</h3>
    <form method="post">
        <input type="text" name="prompt" size="80" required />
        <input type="submit" value="Submit" />
    </form>
    <div style="text-align: left">
        <h5>Example prompts:</h5>
        <p>Which school does Mat Tan go to?</p>
        <p>Which school has the most students?</p>
        <p>Which school has the least students?</p>
        <p>Get the count of students in each school.</p>
        <p>How many students are there in the school of Mining?</p>
        <p>What is the ID of Jan Fry and which school does she go to?</p>
        <p>Which students belong to the school of Business? Respond only in JSON format.</p>
        <p>Which students in the school of Nursing have their first or last name start with the letter 'J'?</p>
    </div>
    @if (Model.Reply != null)
    {
        <p class="alert alert-success" id="reply">@Model.Reply</p>
    }
</div>

The above markup displays an HTML form that accepts a prompt from a user. The prompt is then submitted to the server and the response is displayed in a paragraph (<p> tag) with a green background (Bootstrap class alert-success).

Meantime, at the bottom of the page there are some suggested prompts to facilitate testing – namely:

Which school does Mat Tan go to?
Which school has the most students?
Which school has the least students?
Get the count of students in each school.
How many students are there in the school of Mining?
What is the ID of Jan Fry and which school does she go to?
Which students belong to the school of Business? Respond only in JSON format.
Which students in the school of Nursing have their first or last name start with the letter 'J'?

Trying the application

In a terminal window, at the root of the Razor Pages web application, enter the following command:

dotnet watch

The following page will display in your default browser:

You can enter any of the suggested prompts to ensure we are getting the proper results. I entered the last prompt and got these results:


Conclusion

We have seen how The Micrsoft Agenr Framework and Function Calling can be used with data coming from a database. In this example we are using SQLite. However, any other database can be used using the same technique.

Sunday, October 19, 2025

Small Language Models with AI Toolkit Extension in VS Code

In this article, we will see how we can work with small language models (SLM) from the AI Toolkit extension in VS Code. Though the toolkit can do other things, our focus is to consume an ONNX SLM hosted on Visual Studio Code from a C# application. We will first look at an example that is based on OpenAI packages. We will later use a similar example based on the Sematic Kernal approach.

Companion Video: https://youtu.be/V_eWAM2fxJg

Prerequisites

You will need:

  • The latest version of VS Code
  • .NET version 9.0 or higher

What are small language models (SLMs)?

Small Language Models (SLMs) are compact versions of large language models (LLMs), designed to deliver strong performance in natural language tasks while using significantly fewer computational resources.

What is the AI Toolkit Extension in VS Code?

The AI Toolkit Extension for Visual Studio Code is a powerful, all-in-one environment for building, testing, and deploying generative AI applications—especially useful for developers working with small language models (SLMs).

Getting Started

Install the following Visual Studio Code extension:


Click on the three dots (...) in the left navigation of VS Code, and choose "AI Toolkit".

Click on "Model Catalog".

Scroll down down the list until you find “Local Models” >> ONNX >> Minstral 7B – (CPU – Small, Standard) >> + Add Model.

Once the model is fully downloaded, it will appear under Models >> ONNX.

Right-click on the model and select “Copy Model Name”.

I copied the following name for the "Minstral 7B" model: 

mistral-7b-v02-int4-cpu

Using OpenAI packages

Create a C# console application named AIToolkitOpenAI and add to it required packages with the following terminal window commands:

dotnet new console -n AIToolkitOpenAI
cd AIToolkitOpenAI
dotnet add package OpenAI

Start VS Code with:

code .

Click on the "AI Toolkit" tab in VS Code and make sure that the "Minstral 7B" model is running.

Replace content of Program.cs with this code:

using OpenAI;
using OpenAI.Chat;
using System.ClientModel;
using System.Text;

var model = "mistral-7b-v02-int4-cpu";
var baseUrl = "http://localhost:5272/v1/"; // root URL for local OpenAI-like server
var apikey = "unused";

OpenAIClientOptions options = new OpenAIClientOptions();
options.Endpoint = new Uri(baseUrl);
ApiKeyCredential credential = new ApiKeyCredential(apikey);
ChatClient client = new OpenAIClient(credential, options).GetChatClient(model);

// Build the prompt
StringBuilder prompt = new StringBuilder();
prompt.AppendLine("You will analyze the sentiment of the following product reviews.");
prompt.AppendLine("Each line is its own review. Output the sentiment of each review in");
prompt.AppendLine("a bulleted list and then provide a general sentiment of all reviews.");
prompt.AppendLine();
prompt.AppendLine("I bought this product and it's amazing. I love it!");
prompt.AppendLine("This product is terrible. I hate it.");
prompt.AppendLine("I'm not sure about this product. It's okay.");
prompt.AppendLine("I found this product based on the other reviews. It worked");

// send the prompt to the model and wait for the text completion
var response = await client.CompleteChatAsync(prompt.ToString());
// display the response
Console.WriteLine(response.Value.Content[0].Text);

Run the application with:

dotnet run

The application does sentiment analysis on what customers think of the product.

This is a sample of the output:

* I bought this product and it's amazing. I love it!: Positive sentiment
* This product is terrible. I hate it.: Negative sentiment
* I'm not sure about this product. It's okay.: Neutral sentiment
* I found this product based on the other reviews. It worked for me.: Positive sentiment

General sentiment: The reviews contain both positive and negative sentiments. Some customers expressed their love for the product, while others expressed their dislike. Neutral sentiment was also expressed by one customer. Overall, the reviews suggest that the product has the potential to elicit strong feelings from customers, both positive and negative.

Sematic Kernel packages

Create a C# console application named AIToolkitSK and add to it required packages with the following terminal window commands:

dotnet new console -n AIToolkitSK
cd AIToolkitSK
dotnet add package Microsoft.SemanticKernel

Start VS Code with:

code .

Click on the "AI Toolkit" tab in VS Code and make sure that the "Minstral 7B" model is running.

Replace content of Program.cs with this code:

using System.Text;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;

var model = "mistral-7b-v02-int4-cpu";
var baseUrl = "http://localhost:5272/v1/";
var apikey = "unused";

// Create a chat completion service
var kernel = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion(modelId: model, apiKey: apikey, endpoint: new Uri(baseUrl))
    .Build();
var chat = kernel.GetRequiredService<IChatCompletionService>();
var history = new ChatHistory();
history.AddSystemMessage("You are a useful chatbot. Always reply in a funny way with short answers.");
var settings = new OpenAIPromptExecutionSettings
{
    MaxTokens = 500,
    Temperature = 1,
};

while (true)
{
    Console.Write("\nUser: ");
    var userInput = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(userInput)) break;

    history.AddUserMessage(userInput);

    var responseBuilder = new StringBuilder();
    Console.Write("\nAI: ");
    await foreach (var message in chat.GetStreamingChatMessageContentsAsync(userInput, settings, kernel))
    {
        responseBuilder.Append(message);
        Console.Write(message);
    }
}

This is a simple chat completion app.

Run the application with:

dotnet run

My prompt was:

Red or white wine with beef steak?

The response was:

AI:  Both red and white wines can pair well with beef steak, but a red wine is generally the more traditional choice. Red wines, such as Cabernet Sauvignon, Merlot, or Pinot Noir, have flavors that complement the rich and savory flavors of beef. However, if you prefer a lighter taste, a white wine such as Pinot Noir or Chardonnay can also work well with beef steak. Ultimately, it comes down to personal preference.

Conclusion

We have seen how to use SLMs hosted by VS Code through the AI Toolkit extension. We were able to communicate with the model from these two C# applications: (1) a app the uses OpenAI packages, and (2) an app that uses Sematic Kernel.

Monday, February 24, 2025

Using Azure OpenAI Whisper in an ASP.NET Razor Pages app

 Using Azure OpenAI, we will explore the audio-centric Whisper neural net from OpenAI. You can find more details about Whisper at https://github.com/openai/whisper.  The examples in this article assume that you have a developer account with Azure. These are the features we will explore:

  1. Transcribing audio into text
  2. Converting text into audio
  3. Translating audio from another spoken language into English text

Source Code: https://github.com/medhatelmasry/WhisperWebAzureOpenAI

Prerequisites:

  • You need a subscription with Azure.
  • The example uses Razor pages in ASP.NET 9.0
  • We will use the standard VS Code editor
  • You have installed the “C# Dev Kit” extension in VS Code

Getting Started

We will start by:

  1. creating an ASP.NET Razor Pages web app
  2. adding Azure packages to the project

Execute these commands in a terminal window: 

dotnet new razor -o WhisperWebAzureOpenAI
cd WhisperWebAzureOpenAI
dotnet add package Azure.AI.OpenAI -v 2.2.0-beta.2
dotnet add package Microsoft.Extensions.Azure

Start VS Code in the current project folder with:

code .

Add the following to appsettings.Development.json:

"AzOpenAI": {
  "Key": "YOUR-AZURE-OPENAI-KEY-HERE",
  "Url":  "YOUR-AZURE-OPENAI-ENDPOINT-HERE",
  "Audio2Text":
  {
    "Model": "whisper",
    "Folder": "audio2text"
  },
  "Text2Audio": {
    "Model": "tts",
    "Folder": "text2audio"
  },
  "Translation": {
    "Model": "whisper",
    "Folder": "translation"
  }
}

NOTE: Replace the value of the Key and Url settings above with your Azure OpenAI key and endpoint.

Model whisper is used for audio to text and audio translations. Model tts is used for converting text into audio.

Add this service to Program.cs:

builder.Services.AddAzureClients(clientBuilder =>
{
    // read key from configuration
    string? key = builder.Configuration["AzOpenAI:Key"];
    string? url = builder.Configuration["AzOpenAI:Url"];
    var credentials = new AzureKeyCredential(key!);
    
    // Register a custom client factory
    clientBuilder.AddClient<AzureOpenAIClient, AzureOpenAIClientOptions>(
        (options, _, _) => new AzureOpenAIClient(
            new Uri(url!), credentials, options)); 
});

Download a zip file from https://medhat.ca/images/audio.zip. Extract the file in the wwwroot folder.  This creates the following directory structure under wwwroot:

Note the presence of these audio files in the /wwwroot/audio/audio2text folder:

aboutSpeechSdk.wav
audio_houseplant_care.mp3
speechService.wav
TalkForAFewSeconds16.wav
wikipediaOcelot.wav

Also, note the presence of these audio files in the /wwwroot/audio/translation folder:

audio_arabic.mp3
audio_french.wav
audio_spanish.mp3

Add razor pages

In VS Code, view your project in the “Solution Explorer” tab:

Right-click on the Pages folder and add a razor page named Audio2Text:

Similarly, add these two razor pages:

  1. Text2Audio
  2. Translation

Make these code replacements into the respective files:

Audio2Text Razor Page

Audio2Text.cshtml.cs

using Azure.AI.OpenAI;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.AspNetCore.Mvc.Rendering;
namespace WhisperWebAzureOpenAI.Pages;
public class Audio2TextModel : PageModel {
    private readonly ILogger<Audio2TextModel> _logger;
    private readonly AzureOpenAIClient _azureOpenAIClient;
    private readonly IConfiguration _configuration;
    public List<SelectListItem>? AudioFiles { get; set; }
    public Audio2TextModel(
        ILogger<Audio2TextModel> logger, 
        AzureOpenAIClient client,
        IConfiguration configuration
    )
    {
        _logger = logger;
        _azureOpenAIClient = client;
        _configuration = configuration;
        // create wwroot/audio folder if it doesn't exist
        string? folder = _configuration["AzOpenAI:Audio2Text:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        if (!Directory.Exists(path)) {
            Directory.CreateDirectory(path);
        }
    }
    public void OnGet() {
        AudioFiles = GetAudioFiles();
    }
    public async Task<IActionResult> OnPostAsync(string? audioFile) {
        if (string.IsNullOrEmpty(audioFile)) {
            return Page();
        }
        string? deploymentName = _configuration["AzOpenAI:Audio2Text:Model"];
        var audioClient = _azureOpenAIClient.GetAudioClient(deploymentName);
        
        var result = await audioClient.TranscribeAudioAsync(audioFile);
        if (result is null) {
            return Page();
        }
        string? folder = _configuration["AzOpenAI:Audio2Text:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        ViewData["AudioFile"] = audioFile.StartsWith(path) ? audioFile.Substring(path.Length + 1) : audioFile;
        ViewData["Transcription"] = result.Value.Text;
        AudioFiles = GetAudioFiles();
        return Page();
    }
    public List<SelectListItem> GetAudioFiles() {
        List<SelectListItem> items = new List<SelectListItem>();
        string? folder = _configuration["AzOpenAI:Audio2Text:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        
        // Get files with .wav or .mp3 extensions
        string[] wavFiles = Directory.GetFiles(path, "*.wav");
        string[] mp3Files = Directory.GetFiles(path, "*.mp3");
        // Combine the arrays
        string[] list = wavFiles.Concat(mp3Files).ToArray();
        foreach (var item in list) {
            items.Add(new SelectListItem {
                Value = item.ToString(),
                Text = item.StartsWith(path) ? item.Substring(path.Length + 1) : item
            });
        }
        return items;
    }
}

Audio2Text.cshtml

@page
@model Audio2TextModel
@{ ViewData["Title"] = "Audio to Text Transcription"; }
<div class="text-center">
    <h1 class="display-4">@ViewData["Title"]</h1>
    <form method="post">
        <select asp-items="@Model.AudioFiles" name="audioFile"></select>
        <button type="submit">Submit</button>
    </form>
</div>
@if (ViewData["AudioFile"] != null) {
    <p></p>
    <h3 class="text-danger">@ViewData["AudioFile"]</h3>
}
@if (ViewData["Transcription"] != null) {
    <p class="alert alert-success">@ViewData["Transcription"]</p>
}

Text2Audio Razor Page

Text2Audio.cshtml.cs

using Azure.AI.OpenAI;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using OpenAI;
using OpenAI.Audio;
namespace WhisperWebAzureOpenAI.Pages;
public class Text2AudioModel : PageModel {
    private readonly ILogger<Text2AudioModel> _logger;
    private readonly AzureOpenAIClient _openAIClient;
    private readonly IConfiguration _configuration;
    const string DefaultText = @"Security officials confiscating bottles of water, tubes of 
shower gel and pots of face creams are a common sight at airport security.  
But officials enforcing the no-liquids rule at South Korea's Incheon International Airport 
have been busy seizing another outlawed item: kimchi, a concoction of salted and fermented 
vegetables that is a staple of every Korean dinner table.";
    public Text2AudioModel(ILogger<Text2AudioModel> logger,
        AzureOpenAIClient client,
        IConfiguration configuration
    )
    {
        _logger = logger;
        _openAIClient = client;
        _configuration = configuration;
        // create wwroot/audio folder if it doesn't exist
        string? folder = _configuration["AzOpenAI:Text2Audio:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        if (!Directory.Exists(path)) {
            Directory.CreateDirectory(path);
        }
    }
    public void OnGet() { 
        ViewData["sampleText"] = DefaultText;
    }
    public async Task<IActionResult> OnPostAsync(string inputText) {
        string? modelName = _configuration["AzOpenAI:Text2Audio:Model"];
        var audioClient = _openAIClient.GetAudioClient(modelName);
        BinaryData speech = await audioClient.GenerateSpeechAsync(inputText, GeneratedSpeechVoice.Alloy);
        // Generate a consistent file name based on the hash of the input text
        using var sha256 = System.Security.Cryptography.SHA256.Create();
        byte[] hashBytes = sha256.ComputeHash(System.Text.Encoding.UTF8.GetBytes(inputText));
        string hashString = BitConverter.ToString(hashBytes).Replace("-", "").ToLower();
        string fileName = $"{hashString}.mp3";
        string? folder = _configuration["AzOpenAI:Text2Audio:Folder"];
        string filePath = Path.Combine("wwwroot", "audio", folder!, fileName);
        // Check if the file already exists
        if (!System.IO.File.Exists(filePath)) {
            using FileStream stream = System.IO.File.OpenWrite(filePath);
            speech.ToStream().CopyTo(stream);
        }
        ViewData["sampleText"] = inputText;
        ViewData["AudioFilePath"] = $"/audio/{folder}/{fileName}";
        return Page();
    }
}

Text2Audio.cshtml

@page
@model Text2AudioModel
@{ ViewData["Title"] = "Text to Audio"; }
<h1>@ViewData["Title"]</h1>
<div class="text-center">
    <form method="post">
        <label for="prompt">Enter text to convert to audio:</label>
        <br />
        <textarea type="text" name="inputText" id="inputText" cols="80" rows="5" required>@if (ViewData["sampleText"]!=null){@ViewData["sampleText"]}</textarea>
        <br /><input type="submit" value="Submit" />
    </form>
    <p></p>
    @if (ViewData["AudioFilePath"] != null) {
        <audio controls>
            <source src="@ViewData["AudioFilePath"]" type="audio/mpeg">
            Your browser does not support the audio element.
        </audio>
    }
</div>

Translation Razor Page

Translation.cshtml.cs

using Azure.AI.OpenAI;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.AspNetCore.Mvc.Rendering;
using OpenAI;
namespace WhisperWebAzureOpenAI.Pages;
public class TranslationModel : PageModel {
    private readonly ILogger<TranslationModel> _logger;
    private readonly AzureOpenAIClient _openAIClient;
    private readonly IConfiguration _configuration;
    public List<SelectListItem>? AudioFiles { get; set; }
    public TranslationModel(ILogger<TranslationModel> logger,
        AzureOpenAIClient client,
        IConfiguration configuration
    )
    {
        _logger = logger;
        _openAIClient = client;
        _configuration = configuration;
        // create wwroot/audio folder if it doesn't exist
        string? folder = _configuration["AzOpenAI:Translation:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        if (!Directory.Exists(path)) {
            Directory.CreateDirectory(path);
        }
    }
    public void OnGet() {
        AudioFiles = GetAudioFiles();
    }
    public async Task<IActionResult> OnPostAsync(string? audioFile) {
        if (string.IsNullOrEmpty(audioFile)) {
            return Page();
        }
        string? modelName = _configuration["AzOpenAI:Translation:Model"];
        var audioClient = _openAIClient.GetAudioClient(modelName);
        var result = await audioClient.TranslateAudioAsync(audioFile);
        if (result is null) {
            return Page();
        }
        
        string? folder = _configuration["AzOpenAI:Translation:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        ViewData["AudioFile"] = audioFile.StartsWith(path) ? audioFile.Substring(path.Length + 1) : audioFile;
        ViewData["Transcription"] = result.Value.Text;
        AudioFiles = GetAudioFiles();
        return Page();
    }
    public List<SelectListItem> GetAudioFiles() {
        List<SelectListItem> items = new List<SelectListItem>();
        string? folder = _configuration["AzOpenAI:Translation:Folder"];
        string? path = $"wwwroot/audio/{folder}";
        // Get files with .wav or .mp3 extensions
        string[] wavFiles = Directory.GetFiles(path, "*.wav");
        string[] mp3Files = Directory.GetFiles(path, "*.mp3");
        // Combine the arrays
        string[] list = wavFiles.Concat(mp3Files).ToArray();
        foreach (var item in list) {
            items.Add(new SelectListItem {
                Value = item.ToString(),
                Text = item.StartsWith(path) ? item.Substring(path.Length + 1) : item
            });
        }
        return items;
    }
}

Translation.cshtml

@page
@model TranslationModel
@{ ViewData["Title"] = "Audio Translation"; }
<div class="text-center">
    <h1 class="display-4">@ViewData["Title"]</h1>
    <form method="post">
        <select asp-items="@Model.AudioFiles" name="audioFile"></select>
        <button type="submit">Submit</button>
    </form>
</div>
@if (ViewData["AudioFile"] != null) {
    <p></p>
    <h3 class="text-danger">@ViewData["AudioFile"]</h3>
}
@if (ViewData["Transcription"] != null) {
    <p class="alert alert-success">@ViewData["Transcription"]</p>
}

Adding pages to menu system

Let us see our new pages in action. But first, we need to add links to the three razor pages in the menu system. Open Pages/Shared/_Layout.cshtml in the editor and add these menu items inside the <ul> . . . </ul> block:

<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Audio2Text">Audio to Text</a>
</li>
<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Text2Audio">Text to Audio</a>
</li>
<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Translation">Translation</a>
</li>

Let’s try it out!

Start the application by executing the following command in the terminal window:

dotnet watch

Audio to Text Page


Text to Audio Page


Translation

Bonus - Streaming audio

Going back to the Text2Audio pages, bear in mind that the audio is being saved to the server's file system then linked to the <audio ..> element. We can instead stream the audio without the need of saving a file on the server. Let us see how that works. In the Text2Audio,cshtml.cs, add the following method:

public async Task<IActionResult> OnGetSpeakAsync(string text) {
  string? modelName = _configuration["AzOpenAI:Text2Audio:Model"];
  var audioClient = _openAIClient.GetAudioClient(modelName);
  BinaryData speech = await audioClient.GenerateSpeechAsync(text, GeneratedSpeechVoice.Alloy);
  MemoryStream memoryStream = new MemoryStream();
  speech.ToStream().CopyTo(memoryStream);
  memoryStream.Position = 0; // Reset the position to the beginning of the stream
  return File(memoryStream, "audio/wav");
}

Add this code to Text2Audio,cshtml just before the closing </div> tag:

<button id="speakBtn" class="btn btn-warning">Speak</button>
<audio id="audioPlayer" type="audio/wav" ></audio>
<script>
  document.getElementById('speakBtn').addEventListener('click', function () {
    var text = encodeURIComponent(document.getElementById('inputText').value);
    fetch('/Text2Audio?handler=Speak&text=' + text)
        .then(response => response.blob())
        .then(blob => {
            var url = URL.createObjectURL(blob);
            var audioPlayer = document.getElementById('audioPlayer');
            audioPlayer.src = url;
            audioPlayer.play();
        });
  });
</script>

Run the application and view the Text2Audio pages, you will notice a new "Speak" button:



Click on the speak button and you will be able to have the audio streamed back to you.

Conclusion

With the knowledge of how to use Azure OpenAI Whisper under your belt, I am sure you will build great apps. Happy Coding.

Thursday, February 20, 2025

Using OpenAI Whisper in an ASP.NET Razor Pages app

In this article, we will explore the audio-centric Whisper neural net from OpenAI. You can find more details about Whisper at https://github.com/openai/whisper.  The examples in this article assume that you have a developer account with OpenAI. These are the features we will explore:

  1. Transcribing audio into text
  2. Converting text into audio
  3. Translating audio from another spoken language into English text

Source Code: https://github.com/medhatelmasry/WhisperWebOpenAI

Prerequisites:

  • You need a developer subscription with OpenAI.
  • The example uses Razor pages in ASP.NET 9.0
  • The editor used is the standard VS Code
  • You have installed the “C# Dev Kit” extension in VS Code

Getting Started

We will start by:

  1. creating an ASP.NET Razor Pages web app
  2. adding the OpenAI package to the project

Execute these commands in a terminal window: 

dotnet new razor -o WhisperWebOpenAI
cd WhisperWebOpenAI
dotnet add package OpenAI

Start VS Code in the current project folder with:

code .

Add the following to appsettings.Development.json:

"OpenAI": {
  "Key": "YOUR-OpenAI-KEY",
  "Audio2Text": {
    "Model": "whisper-1",
    "Folder": "audio2text"
  },
  "Text2Audio": {
    "Model": "tts-1",
    "Folder": "text2audio"
  },
  "Translation": {
    "Model": "whisper-1",
    "Folder": "translation"
  }
}

NOTE: Replace the value of the Key setting above with your OpenAI key.

Model whisper-1 is used for audio to text and audio translations. Model tts-1 is used for converting text into audio.

Add this service to Program.cs:

// Add OpenAI service
builder.Services.AddSingleton<OpenAIClient>(sp =>
{
    string? apiKey = builder.Configuration["OpenAI:Key"];
    return new OpenAIClient(apiKey);
});

Download a zip file from https://medhat.ca/images/audio.zip. Extract the file in the wwwroot folder.  This creates the following directory structure under wwwroot:

Note the presence of these audio files in the /wwwroot/audio/audio2text folder:

aboutSpeechSdk.wav
audio_houseplant_care.mp3
speechService.wav
TalkForAFewSeconds16.wav
wikipediaOcelot.wav

Also, note the presence of these audio files in the /wwwroot/audio/translation folder:

audio_arabic.mp3
audio_french.wav
audio_spanish.mp3

Add razor pages

In VS Code, view your project in the “Solution Explorer” tab:

Right-click on the Pages folder and add a razor page named Audio2Text:


Similarly, add these two razor pages:

  1. Text2Audio
  2. Translation

Make these code replacements into the respective files:

Audio2Text Razor Page

Audio2Text.cshtml.cs

using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.AspNetCore.Mvc.Rendering;
using OpenAI;

namespace WhisperWebOpenAI.Pages;

public class Audio2TextModel : PageModel {
  private readonly ILogger<Audio2TextModel> _logger;
  private readonly OpenAIClient _openAIClient;
  private readonly IConfiguration _configuration;
  public List<SelectListItem>? AudioFiles { get; set; }
  public Audio2TextModel(ILogger<Audio2TextModel> logger,
    OpenAIClient client,
    IConfiguration configuration
  )
  {
    _logger = logger;
    _openAIClient = client;
    _configuration = configuration;
    // create wwroot/audio folder if it doesn't exist
    string? folder = _configuration["OpenAI:Audio2Text:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    if (!Directory.Exists(path)) {
      Directory.CreateDirectory(path);
    }
  }
  public void OnGet() {
    AudioFiles = GetWaveFiles();
  }
  public async Task<IActionResult> OnPostAsync(string? waveFile) {
    if (string.IsNullOrEmpty(waveFile)){
      return Page();
    }
    string? modelName = _configuration["OpenAI:Audio2Text:Model"];
    var audioClient = _openAIClient.GetAudioClient(modelName);
    var result = await audioClient.TranscribeAudioAsync(waveFile);
    if (result is null) {
      return Page();
    }
    string? folder = _configuration["OpenAI:Audio2Text:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    ViewData["AudioFile"] = waveFile.StartsWith(path) ? waveFile.Substring(path.Length + 1) : waveFile;
    ViewData["Transcription"] = result.Value.Text;
    AudioFiles = GetWaveFiles();
    return Page();
  }
  public List<SelectListItem> GetWaveFiles() {
    List<SelectListItem> items = new List<SelectListItem>();
    string? folder = _configuration["OpenAI:Audio2Text:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    
    // Get files with .wav or .mp3 extensions
    string[] wavFiles = Directory.GetFiles(path, "*.wav");
    string[] mp3Files = Directory.GetFiles(path, "*.mp3");
    // Combine the arrays
    string[] list = wavFiles.Concat(mp3Files).ToArray();
    foreach (var item in list) {
      items.Add(new SelectListItem
      {
          Value = item.ToString(),
          Text = item.StartsWith(path) ? item.Substring(path.Length + 1) : item
      });
    }
    return items;
  }
}

Audio2Text.cshtml

@page
@model Audio2TextModel

@{ ViewData["Title"] = "Audio to Text Transcription"; }

<div class="text-center">
  <h1 class="display-4">@ViewData["Title"]</h1>
  <form method="post">
    <select asp-items="@Model.AudioFiles" name="waveFile"></select>
    <button type="submit">Submit</button>
  </form>
</div>
@if (ViewData["AudioFile"] != null) {
  <p></p>
  <h3 class="text-danger">@ViewData["AudioFile"]</h3>
}
@if (ViewData["Transcription"] != null) {
  <p class="alert alert-success">@ViewData["Transcription"]</p>
}

Text2Audio Razor Page

Text2Audio.cshtml.cs

using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using OpenAI;
using OpenAI.Audio;

namespace WhisperWebOpenAI.Pages;

public class Text2AudioModel : PageModel {
  private readonly ILogger<Text2AudioModel> _logger;
  private readonly OpenAIClient _openAIClient;
  private readonly IConfiguration _configuration;
  const string DefaultText = @"Security officials confiscating bottles of water, tubes of 
shower gel and pots of face creams are a common sight at airport security.  
But officials enforcing the no-liquids rule at South Korea's Incheon International Airport 
have been busy seizing another outlawed item: kimchi, a concoction of salted and fermented 
vegetables that is a staple of every Korean dinner table.";
  public Text2AudioModel(ILogger<Text2AudioModel> logger,
      OpenAIClient client,
      IConfiguration configuration
  )
  {
    _logger = logger;
    _openAIClient = client;
    _configuration = configuration;
    // create wwroot/audio folder if it doesn't exist
    string? folder = _configuration["OpenAI:Text2Audio:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    if (!Directory.Exists(path)) {
        Directory.CreateDirectory(path);
    }
  }
  public void OnGet() { 
    ViewData["sampleText"] = DefaultText;
  }
  public async Task<IActionResult> OnPostAsync(string inputText) {
    string? modelName = _configuration["OpenAI:Text2Audio:Model"];
    var audioClient = _openAIClient.GetAudioClient(modelName);
    BinaryData speech = await audioClient.GenerateSpeechAsync(inputText, GeneratedSpeechVoice.Alloy);
    // Generate a consistent file name based on the hash of the input text
    using var sha256 = System.Security.Cryptography.SHA256.Create();
    byte[] hashBytes = sha256.ComputeHash(System.Text.Encoding.UTF8.GetBytes(inputText));
    string hashString = BitConverter.ToString(hashBytes).Replace("-", "").ToLower();
    string fileName = $"{hashString}.mp3";
    string? folder = _configuration["OpenAI:Text2Audio:Folder"];
    string filePath = Path.Combine("wwwroot", "audio", folder!, fileName);
    // Check if the file already exists
    if (!System.IO.File.Exists(filePath)) {
      using FileStream stream = System.IO.File.OpenWrite(filePath);
      speech.ToStream().CopyTo(stream);
    }
    ViewData["sampleText"] = inputText;
    ViewData["AudioFilePath"] = $"/audio/{folder}/{fileName}";
    return Page();
  }
}

Text2Audio.cshtml

@page
@model Text2AudioModel

@{ ViewData["Title"] = "Text to Audio"; }

<h1>@ViewData["Title"]</h1>
<div class="text-center">
    <form method="post">
        <label for="prompt">Enter text to convert to audio:</label>
        <br />
        <textarea type="text" name="inputText" id="inputText" cols="80" rows="5" required>@if (ViewData["sampleText"]!=null){@ViewData["sampleText"]}</textarea>
        <br /><input type="submit" value="Submit" />
    </form>
    <p></p>
    @if (ViewData["AudioFilePath"] != null) {
        <audio controls>
            <source src="@ViewData["AudioFilePath"]" type="audio/mpeg">
            Your browser does not support the audio element.
        </audio>
    }
</div>

Translation Razor Page

Translation.cshtml.cs

using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.AspNetCore.Mvc.Rendering;
using OpenAI;

namespace WhisperWebOpenAI.Pages;

public class TranslationModel : PageModel {
  private readonly ILogger<TranslationModel> _logger;
  private readonly OpenAIClient _openAIClient;
  private readonly IConfiguration _configuration;
  public List<SelectListItem>? AudioFiles { get; set; }
  public TranslationModel(ILogger<TranslationModel> logger,
      OpenAIClient client,
      IConfiguration configuration
  )
  {
    _logger = logger;
    _openAIClient = client;
    _configuration = configuration;
    // create wwroot/audio folder if it doesn't exist
    string? folder = _configuration["OpenAI:Translation:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    if (!Directory.Exists(path)) {
      Directory.CreateDirectory(path);
    }
  }
  public void OnGet() {
      AudioFiles = GetAudioFiles();
  }
  public async Task<IActionResult> OnPostAsync(string? audioFile) {
    if (string.IsNullOrEmpty(audioFile)) {
      return Page();
    }
    string? modelName = _configuration["OpenAI:Translation:Model"];
    var audioClient = _openAIClient.GetAudioClient(modelName);
    var result = await audioClient.TranslateAudioAsync(audioFile);
    if (result is null) {
      return Page();
    }
    string? folder = _configuration["OpenAI:Translation:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    ViewData["AudioFile"] = audioFile.StartsWith(path) ? audioFile.Substring(path.Length + 1) : audioFile;
    ViewData["Transcription"] = result.Value.Text;
    AudioFiles = GetAudioFiles();
    return Page();
  }
  public List<SelectListItem> GetAudioFiles() {
    List<SelectListItem> items = new List<SelectListItem>();
    string? folder = _configuration["OpenAI:Translation:Folder"];
    string? path = $"wwwroot/audio/{folder}";
    // Get files with .wav or .mp3 extensions
    string[] wavFiles = Directory.GetFiles(path, "*.wav");
    string[] mp3Files = Directory.GetFiles(path, "*.mp3");
    // Combine the arrays
    string[] list = wavFiles.Concat(mp3Files).ToArray();
    foreach (var item in list) {
      items.Add(new SelectListItem {
        Value = item.ToString(),
        Text = item.StartsWith(path) ? item.Substring(path.Length + 1) : item
      });
    }
    return items;
  }
}

Translation.cshtml

@page
@model TranslationModel

@{ ViewData["Title"] = "Audio Translation"; }

<div class="text-center">
  <h1 class="display-4">@ViewData["Title"]</h1>
  <form method="post">
    <select asp-items="@Model.AudioFiles" name="audioFile"></select>
    <button type="submit">Submit</button>
  </form>
</div>
@if (ViewData["AudioFile"] != null) {
  <p></p>
  <h3 class="text-danger">@ViewData["AudioFile"]</h3>
}
@if (ViewData["Transcription"] != null) {
  <p class="alert alert-success">@ViewData["Transcription"]</p>
}

Adding pages to menu system

Let us see our new pages in action. But first, we need to add links to the three razor pages in the menu system. Open Pages/Shared/_Layout.cshtml in the editor and add these menu items inside the <ul> . . . </ul> block:

<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Audio2Text">Audio to Text</a>
</li>
<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Text2Audio">Text to Audio</a>
</li>
<li class="nav-item">
    <a class="nav-link text-dark" asp-area="" asp-page="/Translation">Translation</a>
</li>

Let’s try it out!

Start the application by executing the following command in the terminal window:

dotnet watch

Audio to Text Page


Text to Audio Page


Translation

Bonus - Streaming audio

Going back to the Text2Audio pages, bear in mind that the audio is being saved to the server's file system then linked to the <audio ..> element. We can instead stream the audio without the need of saving a file on the server. Let us see how that works. In the Text2Audio,cshtml.cs, add the following method:

public async Task<IActionResult> OnGetSpeakAsync(string text) {
  string? modelName = _configuration["OpenAI:Text2Audio:Model"];
  var audioClient = _openAIClient.GetAudioClient(modelName);
  BinaryData speech = await audioClient.GenerateSpeechAsync(text, GeneratedSpeechVoice.Alloy);
  MemoryStream memoryStream = new MemoryStream();
  speech.ToStream().CopyTo(memoryStream);
  memoryStream.Position = 0; // Reset the position to the beginning of the stream
  return File(memoryStream, "audio/wav");
}

Add this code to Text2Audio,cshtml just before the closing </div> tag:

<button id="speakBtn" class="btn btn-warning">Speak</button>
<audio id="audioPlayer" type="audio/wav" ></audio>
<script>
  document.getElementById('speakBtn').addEventListener('click', function () {
    var text = encodeURIComponent(document.getElementById('inputText').value);
    fetch('/Text2Audio?handler=Speak&text=' + text)
        .then(response => response.blob())
        .then(blob => {
            var url = URL.createObjectURL(blob);
            var audioPlayer = document.getElementById('audioPlayer');
            audioPlayer.src = url;
            audioPlayer.play();
        });
  });
</script>

Run the application and view the Text2Audio pages, you will notice a new "Speak" button:



Click on the speak button and you will be able to have the audio streamed back to you.

Conclusion

With the knowledge of how to use OpenAI Whisper under your belt, I am sure you will build great apps. Happy Coding.