Get a Taste of LLMs from GPT4All

Large language models have become popular recently. ChatGPT is fashionable. Trying out ChatGPT to understand what LLMs are about is easy, but sometimes, you may want an offline alternative that can run on your computer. In this post, you will learn about GPT4All as an LLM that you can install on your computer. In particular, you will learn

Large language models became popular recently. ChatGPT is fashionable. It is important to

  • What is GPT4All
  • How to install the desktop client for GPT4All
  • How to run GPT4All in Python

Let’s get started.

Get a Taste of LLMs from GPT4All
Picture generated by the author using Stable Diffusion. Some rights reserved.

Overview

This post is divided into three parts; they are:

  • What is GPT4All?
  • How to get GPT4All
  • How to use GPT4All in Python

What is GPT4All?

The term “GPT” is derived from the title of a 2018 paper, “Improving Language Understanding by Generative Pre-Training” by Radford et al. This paper describes how transformer models are demonstrated to be able to understand human language.

Since then, many people attempted to develop language models using transformer architecture, and it has been found that a model large enough can give excellent results. However, many of the models developed are proprietary. There are either provided as a service with paid subscription or under a license with certain restrictive terms. Some are even impossible to run on commodity hardware due to is size.

GPT4All project tried to make the LLMs available to the public on common hardware. It allows you to train and deploy your model. Pretrained models are also available, with a small size that can reasonably run on a CPU.

How to get GPT4All

Let’s focus only on using the pre-trained models.

At the time of writing, GPT4All is available from https://gpt4all.io/index.html, which you can run as a desktop application or using a Python library. You can download the installer for your OS to run a desktop client. The client is only a few hundred MB. You should see an installation screen as follows:

After you have the client installed, launching it the first time will prompt you to install a model, which can be as large as many GB. To start, you may pick “gpt4all-j-v1.3-groovy” (the GPT4All-J model). It is a relatively small but popular model.

Once the client and model are ready, you can type your message in the input box. The model may expect a specific form of input, e.g., a particular language or style. This model expects a conversation style (like ChatGPT) and generally handles English well. For example, below is how it responds to the input “Give me a list of 10 colors and their RGB code”:

How to use GPT4All in Python

The key component of GPT4All is the model. The desktop client is merely an interface to it. Besides the client, you can also invoke the model through a Python library.

The library is unsurprisingly named “gpt4all,” and you can install it with pip command:

Afterward, you can use it in Python in just a few lines of code:

Running the above code will download the model file if you haven’t yet. Afterward, the model is loaded, input is provided, and the response is returned as a Python dictionary like the following:

The example above use an input as a list of one dictionary. A more sophisticated input would be a list of many dictionaries, each containing the keys role and content. The role can be either "system", "assistant", or "user", while content is a string of text. If you’re using GPT4All-J model like the example, your role is "user" while the computer is "assistant". The input should be a sequence of conversations between these two parties. Below is how you can construct a conversation step by step:

Note that you invoked the model multiple times. Each time it responded, you took the output and appended it to the list of messages so you accumulated the context. Then you add a new dialog and invoke the model again. You need to collect the messages because each time you invoke the model, it starts afresh without knowing the previous conversation. So it is your job to keep the history to remind the model of its prior responses.

Below is an example of the output of the code above, in which the last input message to the model and its response is printed. You may get a better result from another model. You may also get a different result due to the randomness in the model:

Summary

GPT4All is a nice tool you can play with on your computer. It allows you to explore the interaction with a large language model and help you better understand the capability and limitation of a model. In this post, you learned that:

  • GPT4All has a desktop client that you can install on your computer
  • GPT4All has a Python interface that allows you to interact with a language model in code
  • There are multiple language model available

14 Responses to Get a Taste of LLMs from GPT4All

  1. Michael May 26, 2023 at 6:44 am #

    Using GPT4All is definitely one of the easiest ways to install an LLM model on a computer. The newest models you can download work quite well, not quite GPT-4 level but getting there, and over the next few months they will only get better. I like how by ticking the ‘enable web server’ check box you can set this up as an API service to allow for embedding into applications.

  2. K hwang May 27, 2023 at 3:14 pm #

    Hi

    It is good to try gpt4all

    However, I got the following strange characters from responses.

    How could I correct this error?

    thanks a lot

    ### Prompt:
    Name 3 colors
    ### Response:
    &64!!7G2;%&C8**”,[email protected])E$<A-E981)$;8(90BD;;4::=.,GABD2-61&4H$36!0);&&.7<=(E,%D:)
    {'model': 'ggml-gpt4all-j-v1.3-groovy',
    'usage': {'prompt_tokens': 239,
    'completion_tokens': 128,
    'total_tokens': 367},
    'choices': [{'message': {'role': 'assistant',
    'content': '&64!!7G2;%&C8**”,[email protected])E$<A-E981)$;8(90BD;;4::=.,GABD2-61&4H$36!0);&&.7<=(E,%D:)'}}]}

    • James Carmichael May 28, 2023 at 6:09 am #

      Hi K hwang…You may wish to try your model in Google Colab to rule out any issues with your local environment.

  3. K hwang May 30, 2023 at 12:52 pm #

    Thanks.
    When I tried in the COLAB, it was Ok.

    With my notebook, I have still got the same problem.
    I am using Korean font now.

    Is it a font problem?

    • Adrian Tam May 31, 2023 at 4:35 am #

      Font should not be a problem. But maybe you typed in “full-width” version of latin alphabets? e.g., ABC vs ABC

      • K hwang May 31, 2023 at 12:09 pm #

        Howdy Adrian

        I got the following responses when I attempted other Prompt (questions).

        I appreciate your kindness.

        Found model file.
        ### Instruction:
        The prompt below is a question to answer, a task to complete, or a conversation
        to respond to; decide which and write an appropriate response.

        ### Prompt:
        name capital for USA
        ### Response:
        %(9BH-G,>!.50>8%FA,)E=499C2″”3+,:,-5-165;[email protected]$,9<[email protected]<%+A,4"@H+3:-5"90F1$2:H!CH*F,+=$

        ### Instruction:
        The prompt below is a question to answer, a task to complete, or a conversation
        to respond to; decide which and write an appropriate response.

        ### Prompt:
        USA?
        ### Response:
        87957;;G,F%21D(C&,=94;&61<$=.9FDE4A81H&[email protected]@,$&*E%D,EHA1%&)G=H0GB$):2G2$139H*<<4:92A=C$<6A:,1-$.D8*7C

        ### Instruction:
        The prompt below is a question to answer, a task to complete, or a conversation
        to respond to; decide which and write an appropriate response.

        ### Prompt:
        who am I
        ### Response:
        ";D);)*)(8E2-3A54;8<,332$=22.8$+!$6(59)HH0=$>E7)=,A3>[email protected]+A8<DE6&,9C%04H*2%$A517A=86:(59&G.*:9:
        {'model': 'ggml-gpt4all-j-v1.3-groovy',
        'usage': {'prompt_tokens': 234,
        'completion_tokens': 128,
        'total_tokens': 362},
        'choices': [{'message': {'role': 'assistant',
        'content': '";D);)*)(8E2-3A54;8<,332$=22.8$+!$6(59)HH0=$>E7)=,A3>[email protected]+A8<DE6&,9C%04H*2%$A517A=86:(59&G.*:9:'}}]}

  4. John Warford May 31, 2023 at 1:43 am #

    I am having the same problem as K hwang. Iam using Windows 10 and the latest version of python and pycharm

    ### Prompt:
    Give me a list of 10 colors and their RGB code
    ### Response:
    71*F).D=&8;)6A&9B1″:&+1;H7:),E3+HGE4)$0H($8.0%GF.).H(5H06A37″=:2;;*0″2!H)>FE3+,@”@,,($5$93&5H+1&,!+<;53"[email protected])0()=!H%;9$;:
    {'model': 'ggml-gpt4all-j-v1.3-groovy', 'usage': {'prompt_tokens': 272, 'completion_tokens': 128, 'total_tokens': 400}, 'choices': [{'message': {'role': 'assistant', 'content': '71*F).D=&8;)6A&9B1":&+1;H7:),E3+HGE4)$0H($8.0%GF.).H(5H06A37″=:2;;*0″2!H)>FE3+,@”@,,($5$93&5H+1&,!+<;53"[email protected])0()=!H%;9$;:'}}]}

    Process finished with exit code 0

    • James Carmichael May 31, 2023 at 9:18 am #

      Hi John…Just curious…Have you tried your model in Google Colab?

      • John Warford May 31, 2023 at 4:25 pm #

        Hi James, my apologies I forgot to add that I was just using your first example from above. So I have now confirmed that it works in Colab, however I still get the same garbled output in pycharm, and at the command line uing a command prompt.

        —————
        import gpt4all

        gptj = gpt4all.GPT4All(“ggml-gpt4all-j-v1.3-groovy”)
        messages = [{“role”: “user”, “content”: “Give me a list of 10 colors and their RGB code”}]
        ret = gptj.chat_completion(messages)
        print(ret)

        100%|██████████| 3.79G/3.79G [01:29<00:00, 42.1MiB/s]
        Model downloaded at: /root/.cache/gpt4all/ggml-gpt4all-j-v1.3-groovy.bin
        ### Instruction:
        The prompt below is a question to answer, a task to complete, or a conversation
        to respond to; decide which and write an appropriate response.

        ### Prompt:
        Give me a list of 10 colors and their RGB code
        ### Response:
        Here is a list of 10 colors and their RGB code:Red (255, 0, 0) Blue (0, 255, 0) Green (0, 0, 255) Yellow (255, 255, 0) Orange (255, 127, 0) Purple (0, 128, 0) Pink (255, 192, 203) Gray (128, 128, 128) Black (0, 0, 0) White (255, 255, 255)
        {'model': 'ggml-gpt4all-j-v1.3-groovy', 'usage': {'prompt_tokens': 272, 'completion_tokens': 244, 'total_tokens': 516}, 'choices': [{'message': {'role': 'assistant', 'content': ' Here is a list of 10 colors and their RGB code:Red (255, 0, 0) Blue (0, 255, 0) Green (0, 0, 255) Yellow (255, 255, 0) Orange (255, 127, 0) Purple (0, 128, 0) Pink (255, 192, 203) Gray (128, 128, 128) Black (0, 0, 0) White (255, 255, 255)'}}]}
        ——-

  5. John Warford May 31, 2023 at 4:28 pm #

    Here is my full console output
    python main.py
    Hi, PyCharm
    Found model file.
    gptj_model_load: loading model from ‘C:\\\\Users\\\\jwarfo01\\\\.cache\\\\gpt4all\\ggml-gpt4all-j-v1.3-groovy.bin’ – please wait …
    gptj_model_load: n_vocab = 50400
    gptj_model_load: n_ctx = 2048
    gptj_model_load: n_embd = 4096
    gptj_model_load: n_head = 16
    gptj_model_load: n_layer = 28
    gptj_model_load: n_rot = 64
    gptj_model_load: f16 = 2
    gptj_model_load: ggml ctx size = 5401.45 MB
    gptj_model_load: kv self size = 896.00 MB
    gptj_model_load: done
    gptj_model_load: model size = 123.05 MB / num tensors = 1
    ### Instruction:
    The prompt below is a question to answer, a task to complete, or a conversation
    to respond to; decide which and write an appropriate response.

    ### Prompt:
    Give me a list of 10 colors and their RGB code
    ### Response:
    EAB8;-50B3B3)19&G*-$%3FCB+;%,B:-=E:F&B*)1(@+2!18(*2>;,H)*C)[email protected]+%9A19!)F
    {‘model’: ‘ggml-gpt4all-j-v1.3-groovy’, ‘usage’: {‘prompt_tokens’: 272, ‘completion_tokens’: 128, ‘total_tokens’: 400}, ‘choices’: [{‘message’: {‘role’: ‘assistant’, ‘content’: ‘EAB8;-50B3B3)19&G*-$%3FCB+;%,B:-=E:F&B*)1(@+2!18(*2>;,H)*C)[email protected]+%9A19!)F’}}]}

  6. Achim May 31, 2023 at 5:35 pm #

    Hello, does anyone have any idea how to process additional context information?

    • James Carmichael June 1, 2023 at 5:16 am #

      Hi Achim…Please clarify what is meant by “process additional context information”. That will enable us to better assist you.

      • Achim June 2, 2023 at 1:05 am #

        Hi, I want to make a a similarity search on dokuments based on the question I want to pass to GPT4All: this should be used as a context for the question.

        -> I want to feed the question and the context to GPT4All

        Thanks

  7. John Warford May 31, 2023 at 7:28 pm #

    FYI: the same script runs fine on Ubuntu. I set it up under the windows Linux subsystem.

Leave a Reply