Intelligence
We are a start-up company developing open AI based on ChatGPT with algorithms that will make it smarter and more efficient
introduction
About Our Smart Product
MagnusAI generates varied and human-attractive text. But what makes a text “good”? This is subjective and depends on the context. For example, if you ask to write a story, you need to be creative. If you ask for information, you want it to be truthful. And if you ask to write code, you expect it to be executable.
More about data collection
AI is only as good as the data it is trained on. This is perfectly illustrated by the old famous saying garbage in, garbage out.
- 1Definition of toxicity in textsEmployees from different countries, for $2 an hour, marked up texts with examples of violence, incitement to hatred, etc. This data was used to train the toxicity detector and exclude harmful content.
- 2Programming trainingReal developers were hired to find bugs in the code and explain how the code works in human language. They were asked to describe in words how they approached the task. In case of a bug, they also did not just fix it, but described in words what was wrong and how it should be fixed.
- 3Markup of dialogues by peopleTo find errors or problems in the assistant’s response, as well as to determine creativity, etc. For different requests, the markers gave their ideal answer, and also evaluated the assistant’s different answers.
Reward Model
Reward Model - additional training based on predicting ratings given by people.
Building a reward model on human preferences is where the novelty in RLHF begins. This model should accept text and produce a numerical reward value - a reflection of the person's preference.
Explanations for the diagram:
In fact, it is at this step that the whole connection with “humanity” happens. No matter what criteria are important for a person - creativity, non-toxicity, an answer without errors - through the rating system we enable the machine to learn it and understand what is “bad” and what is “good” for a person.
Building a reward model on human preferences is where the novelty in RLHF begins. This model should accept text and produce a numerical reward value - a reflection of the person's preference.
Explanations for the diagram:
- Markers assigned Human Scoring scores to various responses from the Language Model. Thus, for each generated text, we know to what extent the model’s response corresponds to what a person would like to see.
- Using people's ratings, we train the Reward Model. Now we have a model that, based on any generated text, can predict how much this text meets human expectations.
In fact, it is at this step that the whole connection with “humanity” happens. No matter what criteria are important for a person - creativity, non-toxicity, an answer without errors - through the rating system we enable the machine to learn it and understand what is “bad” and what is “good” for a person.
Model evaluation
RLHF helps achieve good results in three main metrics:
- UsefulnessThe model can follow the user's instructions and draw correct conclusions.
- TruthfulnessThe model is not inclined to invent facts.
- SafetyThe model is not prone to responses that may be harmful or toxic.
Who waits, will wait!
