Large language models have become popular recently. ChatGPT is fashionable. Trying out ChatGPT to understand what LLMs are about is easy, but sometimes, you may want an offline alternative that can run on your computer. In this post, you will learn about GPT4All as an LLM that you can install on your computer. In particular, you will learn
Large language models became popular recently. ChatGPT is fashionable. It is important to
- What is GPT4All
- How to install the desktop client for GPT4All
- How to run GPT4All in Python
Let’s get started.
Get a Taste of LLMs from GPT4All
Picture generated by the author using Stable Diffusion. Some rights reserved.
Overview
This post is divided into three parts; they are:
- What is GPT4All?
- How to get GPT4All
- How to use GPT4All in Python
What is GPT4All?
The term “GPT” is derived from the title of a 2018 paper, “Improving Language Understanding by Generative Pre-Training” by Radford et al. This paper describes how transformer models are demonstrated to be able to understand human language.
Since then, many people attempted to develop language models using transformer architecture, and it has been found that a model large enough can give excellent results. However, many of the models developed are proprietary. There are either provided as a service with paid subscription or under a license with certain restrictive terms. Some are even impossible to run on commodity hardware due to is size.
GPT4All project tried to make the LLMs available to the public on common hardware. It allows you to train and deploy your model. Pretrained models are also available, with a small size that can reasonably run on a CPU.
How to get GPT4All
Let’s focus only on using the pre-trained models.
At the time of writing, GPT4All is available from https://gpt4all.io/index.html, which you can run as a desktop application or using a Python library. You can download the installer for your OS to run a desktop client. The client is only a few hundred MB. You should see an installation screen as follows:
After you have the client installed, launching it the first time will prompt you to install a model, which can be as large as many GB. To start, you may pick “gpt4all-j-v1.3-groovy” (the GPT4All-J model). It is a relatively small but popular model.
Once the client and model are ready, you can type your message in the input box. The model may expect a specific form of input, e.g., a particular language or style. This model expects a conversation style (like ChatGPT) and generally handles English well. For example, below is how it responds to the input “Give me a list of 10 colors and their RGB code”:
How to use GPT4All in Python
The key component of GPT4All is the model. The desktop client is merely an interface to it. Besides the client, you can also invoke the model through a Python library.
The library is unsurprisingly named “gpt4all,” and you can install it with pip command:
|
1 |
pip install gpt4all |
Afterward, you can use it in Python in just a few lines of code:
|
1 2 3 4 5 6 |
import gpt4all gptj = gpt4all.GPT4All("ggml-gpt4all-j-v1.3-groovy") messages = [{"role": "user", "content": "Give me a list of 10 colors and their RGB code"}] ret = gptj.chat_completion(messages) print(ret) |
Running the above code will download the model file if you haven’t yet. Afterward, the model is loaded, input is provided, and the response is returned as a Python dictionary like the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
{'model': 'ggml-gpt4all-j-v1.3-groovy', 'usage': {'prompt_tokens': 272, 'completion_tokens': 301, 'total_tokens': 573}, 'choices': [ {'message': {'role': 'assistant', 'content': ' Here is a list of 10 colors and their RGB code:Red (255, 0, 0) Green (0, 255, 0) Blue (0, 0, 255) Yellow (255, 255, 0) Orange (255, 127, 0) Purple (0, 128, 255) Pink (255, 192, 203) Blue-Green (0, 0, 255) Green-Blue (0, 0, 255) Blue-Purple (0, 0, 255) Blue-Green (0, 0, 255) Blue-Purple (0, 0' } } ] } |
The example above use an input as a list of one dictionary. A more sophisticated input would be a list of many dictionaries, each containing the keys role and content. The role can be either "system", "assistant", or "user", while content is a string of text. If you’re using GPT4All-J model like the example, your role is "user" while the computer is "assistant". The input should be a sequence of conversations between these two parties. Below is how you can construct a conversation step by step:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
import json import gpt4all gptj = gpt4all.GPT4All("ggml-gpt4all-j-v1.3-groovy") messages = [{"role": "user", "content": "Can you explain what is a large language model?"}] ret = gptj.chat_completion(messages) messages.append(ret["choices"][0]["message"]) messages.append({"role": "user", "content": "Can you give some examples applications?"}) ret = gptj.chat_completion(messages) messages.append(ret["choices"][0]["message"]) messages.append({"role": "user", "content": "Are there any limitations?"}) ret = gptj.chat_completion(messages) messages.append(ret["choices"][0]["message"]) messages.append({"role": "user", "content": "Summarize the above in two sentences."}) ret = gptj.chat_completion(messages) print(json.dumps(messages, indent=4)) print(json.dumps(ret, indent=4)) |
Note that you invoked the model multiple times. Each time it responded, you took the output and appended it to the list of messages so you accumulated the context. Then you add a new dialog and invoke the model again. You need to collect the messages because each time you invoke the model, it starts afresh without knowing the previous conversation. So it is your job to keep the history to remind the model of its prior responses.
Below is an example of the output of the code above, in which the last input message to the model and its response is printed. You may get a better result from another model. You may also get a different result due to the randomness in the model:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
[ { "role": "user", "content": "Can you explain what is a large language model?" }, { "role": "assistant", "content": " A large language model is a type of artificial intelligence that can process and understand natural language, such as speech or text. It is trained on large amounts of data and can perform tasks such as language translation, sentiment analysis or even answering questions. Large language models are becoming increasingly popular in various industries, including finance and healthcare." }, { "role": "user", "content": "Can you give some examples applications?" }, { "role": "assistant", "content": " Large language models are being used in a variety of applications, such as:- Language translation and machine translation, where large language models are trained on vast amounts of data to improve accuracy and speed.- Sentiment analysis, where large language models are trained to analyze social media data and identify patterns in user opinions.- Question answering, where large language models are trained to answer questions based on a large corpus of data.- Healthcare, where large language models are being used to analyze patient data and identify patterns in health trends.- Finance, where large language models are being used to analyze financial data and identify patterns in" }, { "role": "user", "content": "Are there any limitations?" }, { "role": "assistant", "content": " Yes, there are some limitations to large language models. One of the main challenges is that they require a lot of data to train, which can be expensive and time-consuming. Additionally, large language models can be computationally expensive and may require specialized hardware to run. Another limitation is that large language models may not be able to understand the nuances of human language, which can make them less effective in certain situations. Finally, large language models may not be able to understand the context of a conversation or text, which can make them less effective in certain applications." }, { "role": "user", "content": "Summarize the above in two sentences." } ] { "model": "ggml-gpt4all-j-v1.3-groovy", "usage": { "prompt_tokens": 2113, "completion_tokens": 542, "total_tokens": 2655 }, "choices": [ { "message": { "role": "assistant", "content": " Large language models are a type of artificial intelligence that can process and understand natural language, such as speech or text. They are trained on large amounts of data and can perform tasks such as language translation, sentiment analysis or even answering questions. They are becoming increasingly popular in various industries, including finance and healthcare. However, there are some limitations such as expensive data and specialized hardware, computational expense, lack of understanding nuances in human language and context." } } ] } |
Summary
GPT4All is a nice tool you can play with on your computer. It allows you to explore the interaction with a large language model and help you better understand the capability and limitation of a model. In this post, you learned that:
- GPT4All has a desktop client that you can install on your computer
- GPT4All has a Python interface that allows you to interact with a language model in code
- There are multiple language model available
Using GPT4All is definitely one of the easiest ways to install an LLM model on a computer. The newest models you can download work quite well, not quite GPT-4 level but getting there, and over the next few months they will only get better. I like how by ticking the ‘enable web server’ check box you can set this up as an API service to allow for embedding into applications.
Hi
It is good to try gpt4all
However, I got the following strange characters from responses.
How could I correct this error?
thanks a lot
### Prompt:
Name 3 colors
### Response:
&64!!7G2;%&C8**”,[email protected])E$<A-E981)$;8(90BD;;4::=.,GABD2-61&4H$36!0);&&.7<=(E,%D:)
{'model': 'ggml-gpt4all-j-v1.3-groovy',
'usage': {'prompt_tokens': 239,
'completion_tokens': 128,
'total_tokens': 367},
'choices': [{'message': {'role': 'assistant',
'content': '&64!!7G2;%&C8**”,[email protected])E$<A-E981)$;8(90BD;;4::=.,GABD2-61&4H$36!0);&&.7<=(E,%D:)'}}]}
Hi K hwang…You may wish to try your model in Google Colab to rule out any issues with your local environment.
Thanks.
When I tried in the COLAB, it was Ok.
With my notebook, I have still got the same problem.
I am using Korean font now.
Is it a font problem?
Font should not be a problem. But maybe you typed in “full-width” version of latin alphabets? e.g., ＡＢＣ vs ABC
Howdy Adrian
I got the following responses when I attempted other Prompt (questions).
I appreciate your kindness.
Found model file.
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
name capital for USA
### Response:
%(9BH-G,>!.50>8%FA,)E=499C2″”3+,:,-5-165;[email protected]$,9<[email protected]<%+A,4"@H+3:-5"90F1$2:H!CH*F,+=$
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
USA?
### Response:
87957;;G,F%21D(C&,=94;&61<$=.9FDE4A81H&[email protected]@,$&*E%D,EHA1%&)G=H0GB$):2G2$139H*<<4:92A=C$<6A:,1-$.D8*7C
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
who am I
### Response:
";D);)*)(8E2-3A54;8<,332$=22.8$+!$6(59)HH0=$>E7)=,A3>[email protected]+A8<DE6&,9C%04H*2%$A517A=86:(59&G.*:9:
{'model': 'ggml-gpt4all-j-v1.3-groovy',
'usage': {'prompt_tokens': 234,
'completion_tokens': 128,
'total_tokens': 362},
'choices': [{'message': {'role': 'assistant',
'content': '";D);)*)(8E2-3A54;8<,332$=22.8$+!$6(59)HH0=$>E7)=,A3>[email protected]+A8<DE6&,9C%04H*2%$A517A=86:(59&G.*:9:'}}]}
I am having the same problem as K hwang. Iam using Windows 10 and the latest version of python and pycharm
### Prompt:
Give me a list of 10 colors and their RGB code
### Response:
71*F).D=&8;)6A&9B1″:&+1;H7:),E3+HGE4)$0H($8.0%GF.).H(5H06A37″=:2;;*0″2!H)>FE3+,@”@,,($5$93&5H+1&,!+<;53"[email protected])0()=!H%;9$;:
{'model': 'ggml-gpt4all-j-v1.3-groovy', 'usage': {'prompt_tokens': 272, 'completion_tokens': 128, 'total_tokens': 400}, 'choices': [{'message': {'role': 'assistant', 'content': '71*F).D=&8;)6A&9B1":&+1;H7:),E3+HGE4)$0H($8.0%GF.).H(5H06A37″=:2;;*0″2!H)>FE3+,@”@,,($5$93&5H+1&,!+<;53"[email protected])0()=!H%;9$;:'}}]}
Process finished with exit code 0
Hi John…Just curious…Have you tried your model in Google Colab?
Hi James, my apologies I forgot to add that I was just using your first example from above. So I have now confirmed that it works in Colab, however I still get the same garbled output in pycharm, and at the command line uing a command prompt.
—————
import gpt4all
gptj = gpt4all.GPT4All(“ggml-gpt4all-j-v1.3-groovy”)
messages = [{“role”: “user”, “content”: “Give me a list of 10 colors and their RGB code”}]
ret = gptj.chat_completion(messages)
print(ret)
100%|██████████| 3.79G/3.79G [01:29<00:00, 42.1MiB/s]
Model downloaded at: /root/.cache/gpt4all/ggml-gpt4all-j-v1.3-groovy.bin
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
Give me a list of 10 colors and their RGB code
### Response:
Here is a list of 10 colors and their RGB code:Red (255, 0, 0) Blue (0, 255, 0) Green (0, 0, 255) Yellow (255, 255, 0) Orange (255, 127, 0) Purple (0, 128, 0) Pink (255, 192, 203) Gray (128, 128, 128) Black (0, 0, 0) White (255, 255, 255)
{'model': 'ggml-gpt4all-j-v1.3-groovy', 'usage': {'prompt_tokens': 272, 'completion_tokens': 244, 'total_tokens': 516}, 'choices': [{'message': {'role': 'assistant', 'content': ' Here is a list of 10 colors and their RGB code:Red (255, 0, 0) Blue (0, 255, 0) Green (0, 0, 255) Yellow (255, 255, 0) Orange (255, 127, 0) Purple (0, 128, 0) Pink (255, 192, 203) Gray (128, 128, 128) Black (0, 0, 0) White (255, 255, 255)'}}]}
——-
Here is my full console output
python main.py
Hi, PyCharm
Found model file.
gptj_model_load: loading model from ‘C:\\\\Users\\\\jwarfo01\\\\.cache\\\\gpt4all\\ggml-gpt4all-j-v1.3-groovy.bin’ – please wait …
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 4096
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot = 64
gptj_model_load: f16 = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size = 896.00 MB
gptj_model_load: done
gptj_model_load: model size = 123.05 MB / num tensors = 1
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
Give me a list of 10 colors and their RGB code
### Response:
EAB8;-50B3B3)19&G*-$%3FCB+;%,B:-=E:F&B*)1(@+2!18(*2>;,H)*C)[email protected]+%9A19!)F
{‘model’: ‘ggml-gpt4all-j-v1.3-groovy’, ‘usage’: {‘prompt_tokens’: 272, ‘completion_tokens’: 128, ‘total_tokens’: 400}, ‘choices’: [{‘message’: {‘role’: ‘assistant’, ‘content’: ‘EAB8;-50B3B3)19&G*-$%3FCB+;%,B:-=E:F&B*)1(@+2!18(*2>;,H)*C)[email protected]+%9A19!)F’}}]}
Hello, does anyone have any idea how to process additional context information?
Hi Achim…Please clarify what is meant by “process additional context information”. That will enable us to better assist you.
Hi, I want to make a a similarity search on dokuments based on the question I want to pass to GPT4All: this should be used as a context for the question.
-> I want to feed the question and the context to GPT4All
Thanks
FYI: the same script runs fine on Ubuntu. I set it up under the windows Linux subsystem.