Skip to content

About

This repository is intended for experimenting with small language models locally, with an initial focus on evaluating the Phi-4 model using C#.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

PhiLocalConsole

This project demonstrates how to run the Phi-4 Mini Instruct ONNX model locally using C# and the Microsoft.ML.OnnxRuntimeGenAI library. It is optimized for Intel Iris Xe GPUs but can be run on CPU as well.

Prerequisites

  • .NET 6.0 SDK or later
  • Visual Studio 2022 or later (or any compatible C# IDE)
  • Windows 10/11 (recommended for DirectML GPU acceleration)
  • Git LFS (For downloading large model files)
  • [Optional] Intel Iris Xe GPU for best performance

2. Download the Model

We use the INT4 ONNX version of Phi-4 for the best balance of speed and intelligence.

# Install Git LFS
git lfs install

# Clone the repository (approx 2.5GB)
git clone https://huggingface.co/microsoft/Phi-4-mini-instruct-onnx

# Add the DirectML package for GPU acceleration
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML --version 0.12.2

After cloning, extract or locate the model files. The folder must contain:

  • genai_config.json
  • .onnx model file(s)

Setup Instructions

  1. Clone this repository:

    git clone https://github.com/joshivilas/SLM-Phi-Local.git
    cd PhiSafeConsole
  2. Open the solution in Visual Studio.

  3. Restore NuGet packages:

    • Visual Studio will prompt you to restore packages on open, or right-click the solution and select Restore NuGet Packages.
  4. Set the model path:

    • In Program.cs, update the modelPath variable to the folder where you extracted the ONNX model files.
    • Example:
      string modelPath = @"C:\MyProjects\small-lang-proj\PhiLocal\Phi-4-mini-instruct-onnx\cpu_and_mobile\cpu-int4-rtn-block-32-acc-level-4";
  5. Build and run the project:

    • Press F5 in Visual Studio or run:
      dotnet run

Usage

  • When the console starts, type your prompt after You > and press Enter.
  • The model will generate a response in real-time.
  • Type exit or press Enter on an empty line to quit.

Troubleshooting

  • If you see [FATAL ERROR], ensure:
    • The modelPath is correct and points to the folder with genai_config.json and the .onnx file.
    • You have the correct INT4 ONNX model version.
    • Your hardware supports DirectML (for GPU acceleration).

About

This repository is intended for experimenting with small language models locally, with an initial focus on evaluating the Phi-4 model using C#.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages