Using Sematic Kernel with AI models hosted on GitHub

Overview

In this article I will show you how you can experiment with AI models hosted on GitHub. GitHub AI Models are intended for learning, experimentation and proof-of-concept activities. The feature is subject to various limits (including requests per minute, requests per day, tokens per request, and concurrent requests) and is not designed for production use cases.

Getting Started

There are many AI models from a variety of vendors that you can choose from. The starting point is to visit https://github.com/marketplace/models. At the time of writing, these are a subset of the models available:

For this article, I will use the "Phi-3.5-mini instruct (128k)" model highlighted above. If you click on that model you will be taken to the model's landing page:

Click on the green "Get started" button.

The first thing we need to do is get a 'personal access token' by clicking on the indicated button above.

Choose 'Generate new token', which happens to be in beta at the time of writing.

Give your token a name, set the expiration, and optionally describe the purpose of the token. Thereafter, click on the green 'Generate token' button at the bottom of the page.

Copy the newly generated token and place it is a safe place because you cannot view this token again once you leave the above page. 

You should be able to return to the "Get Started" page by choosing the "Marketplace" tab in your browser.

On the "Get Started" page, note that you have examples using different programming languages.

If you choose C#, you can easily copy the basic sample code on your computer and try it out. We will, however, use alternative code with Semantic Kernel.

Scroll down the page to the "3. Run a basic code sample" section and note the values of endpoint & model variables. These values will later be used in our application.

Let's use Semantic Kernel

In a working directory, create a C# console app named GitHubAiModelSK inside a terminal window with the following command:

dotnet new console -n GitHubAiModelSK

Change into the newly created directory GitHubAiModelSK with:

cd GitHubAiModelSK

Next, let's add two packages to our console application with:

dotnet add package Microsoft.SemanticKernel -v 1.19.0

dotnet add package Microsoft.Extensions.Configuration.Json

Open the project in VS Code and add this directive to the .csproj file right below: <Nullable>enable</Nullable>:

<NoWarn>SKEXP0010</NoWarn>

Create a file named appsettings.json. Add this to appsettings.json:

{

    "AI": {

      "Endpoint": "https://models.inference.ai.azure.com",

      "Model": "Phi-3.5-mini-instruct",

      "PAT": "fake-token"

    }

}

Replace "fake-token" with the personal access token that you got from GitHub.

Next, open Program.cs in an editor and delete all contents of the file. Add this code to Program.cs:

using Microsoft.SemanticKernel;

using System.Text;

using Microsoft.SemanticKernel.ChatCompletion;

using OpenAI;

using System.ClientModel;

using Microsoft.Extensions.Configuration;

var config = new ConfigurationBuilder()

    .SetBasePath(Directory.GetCurrentDirectory())

    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)

    .Build();

var modelId = config["AI:Model"]!;

var uri = config["AI:Endpoint"]!;

var githubPAT = config["AI:PAT"]!;

var client = new OpenAIClient(new ApiKeyCredential(githubPAT), new OpenAIClientOptions { Endpoint = new Uri(uri) });

// Initialize the Semantic kernel

var builder = Kernel.CreateBuilder();

builder.AddOpenAIChatCompletion(modelId, client);

var kernel = builder.Build();

// get a chat completion service

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

// Create a new chat by specifying the assistant

ChatHistory chat = new(@"

    You are an AI assistant that helps people find information. 

    The response must be brief and should not exceed one paragraph.

    If you do not know the answer then simply say 'I do not know the answer'."

);

// Instantiate a StringBuilder

StringBuilder strBuilder = new();

// User question & answer loop

while (true)

{

    // Get the user's question

    Console.Write("Q: ");

    chat.AddUserMessage(Console.ReadLine()!);

    // Clear contents of the StringBuilder

    strBuilder.Clear();

    // Get the AI response streamed back to the console

    await foreach (var message in chatCompletionService.GetStreamingChatMessageContentsAsync(chat, kernel: kernel))

    {

        Console.Write(message);

        strBuilder.Append(message.Content);

    }

    Console.WriteLine();

    chat.AddAssistantMessage(strBuilder.ToString());

    Console.WriteLine();

}

Run the application:

I asked the question "How many pyramids are there in Egypt?" and the AI answered as shown above. 

Using a different model

How about we use a different AI model. For example, I will try the 'Meta-Llama-3.1-405B-Instruct' model. We need to get the model ID. Click on the model on the https://github.com/marketplace/models page.

Then, click on the "Get started" button. In the "3. Run a basic code sample" section, grab the modelName (meta-llama-3.1-405b-instruct) and paste it in our appsettings.json file instead of Phi-3.5-mini-instruct.

Run the application again. This is what I experienced with the AI model meta-llama-3.1-405b-instruct:

Conclusion

GitHub AI models are easy to access. I hope you come up with great AI driven applications that make a difference to our world.