Serving Scikit-LLM Pipelines as REST APIs with FastAPI

In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline, powered by a local Ollama model, as a REST API with FastAPI.

Topics we will cover include:

  • How to configure Scikit-LLM to use a local Ollama model instead of a paid cloud API.
  • How to build and serialize a zero-shot classifier using the familiar scikit-learn fit/predict syntax.
  • How to serve the classifier through an asynchronous FastAPI endpoint and test it from the interactive docs page.

Serving Scikit-LLM Pipelines as REST APIs with FastAPI

Introduction

It looks like Scikit-LLM is here to stay. This innovative Python framework pushes the boundaries of machine learning and LLMs by treating them as two sides of the same coin, enabling scikit-learn-based pipelines fueled by powerful pre-trained LLMs adapted to perform specific tasks — for instance, building a smart zero-shot classification engine. All of this sounds enthralling until you try to put it into production and instantly get chained to costly cloud APIs, latency spikes, and so on.

This article covers the first crucial steps toward deploying a deep semantic reasoning pipeline fueled by an LLM and combined with the reliability of standardized machine learning syntax — enter the .fit() and .predict() methods — as well as predictable outputs like those found in traditional machine learning pipelines.

We will put together an open-source stack composed of Scikit-LLM, FastAPI, and Ollama, teaching you how to wire up a local, self-hosted LLM that acts as a deterministic text labeler: all wrapped in an asynchronous, blazing-fast REST API, thereby bypassing cloud costs and commercial lock-in.

Setup and Initial Configurations

For this practical demonstration, I have used Visual Studio Code with a virtual environment using Python 3.11. I have also installed some basic libraries like pandas and scikit-learn in the virtual environment. In addition, you’ll need to install the following specialized libraries to make the code work:

You’ll also need to install Ollama and pull (locally download) llama3. This article explains the process to install Ollama in more detail, but chances are you won’t need it if you just visit the Ollama website: it’s pretty simple to install, I promise!

Building, Deploying, and Testing the Pipeline

Once everything is correctly installed and you have resolved any possible issues caused by version compatibility with your Python interpreter, let’s move on to the core of the process. We’ll create a new project folder in our IDE and create two Python files inside it, which will contain the code: name them train.py and app.py.

Let’s start with train.py. In this Python file, we will add the logic to train and serialize the Scikit-LLM pipeline. To keep things simple but scalable, we will just invoke the Ollama model and attach it to a one-stage pipeline consisting of a zero-shot classifier — in simple terms, a pre-trained LLM turned into a classifier by giving it the set of possible categories, or labels, from an example dataset of texts, without actual model training.

After that, we serialize the pipeline: once you run the code below, a file named scikit_llm_model.joblib containing the serialized pipeline will be created in your project folder. Here’s the code to do this:

The next step is to add the code for the app.py file. Run this code after generating the serialized pipeline. This file is needed to:

  1. Reset and re-apply the Scikit-LLM configuration needed for the serving environment.
  2. Create a FastAPI app instance and define the predict() endpoint with the necessary logic to enable inference by the model once deployed. Note that the service can take multiple input texts in a single call, thus returning multiple class predictions in one shot.
  3. Start the server using uvicorn, which the final lines of code take care of.

Once you run this code in your IDE, if everything goes smoothly, you should see the following messages in the output console:

Now your pipeline is accessible (well, locally accessible, I mean), and you can open the FastAPI interface by browsing to http://127.0.0.1:8000/docs. To test the app, click the “Try it out” button inside the expandable “/predict” block, and edit the “Request body” text box to paste the following input examples or any of your own:

Then click “Execute”, and your pipeline’s LLM will start doing its magic. In a few seconds, the predictions will pop up:

Final Remarks

Congratulations if you made it here! Just be aware of a few notes about what we have just done together:

  • The classifier we built and deployed relies on the label set y we passed to it in the fit() method early on. We didn’t update the LLM’s weights (this process would have taken considerably longer!). Instead, the classifier stored the specific closed-domain set of categories to choose from when performing inference on texts, just as if we had written a multiple-choice question in a prompt.
  • For this reason, when turning a pre-trained model into a zero-shot classifier, we are restricting its “view of the world”, forcing it to map any input (no matter what) to exactly one of the labels in the label set. In exchange for this closed-world view, we add a layer of reliability and predictability to an LLM that already had a profound level of knowledge.
  • We built an extremely simple, one-stage pipeline, but thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, we could have created a larger pipeline with additional preprocessing steps, similar to the example in this related article.

No comments yet.

Leave a Reply

Machine Learning Mastery is part of Guiding Tech Media, a leading digital media publisher focused on helping people figure out technology. Visit our corporate website to learn more about our mission and team.