AI & Automation

AI with business data: API, RAG and privacy explained

AI with business data means that a language model answers on the basis of your own documents and data, which your software looks up for each question and sends along via an API. The model itself does not know your business: it was trained on general texts and knows nothing about your customers, prices or procedures. The technique of retrieving first and then letting the model answer is called RAG, retrieval-augmented generation.

At a glance

How does AI work with business data? The short answer

A useful answer about your business requires two kinds of knowledge. Model knowledge is what the language model learned during its training: language, general facts, reasoning. Business knowledge is what exists only in your systems. That second kind is not in the model, and does not need to be.

Instead of retraining the model, you give it the right information at the right moment. Your software looks up the passages relevant to the question, sends them along and asks the model to rely on those alone. Your documents stay where they are; only what is needed for that one question goes to the model.

As a result, three things become more important than the model itself: the quality of your documents, the access rights during retrieval and the agreements with the provider about what happens to the data you send.

AI with business data – from a question and your own documents to the language model and the answer

Comparison

Model knowledge and business knowledge: what is the difference?

A language model sounds as if it knows everything. Yet what it says comes from two clearly different sources.

Model knowledge and business knowledge compared
CharacteristicModel knowledgeBusiness knowledge
OriginThe texts the model was trained onYour own documents, databases and systems
ContentLanguage, general knowledge, reasoningProducts, prices, customers, procedures, arrangements
TimelinessGoes up to the point when training stoppedAs current as your own sources
UpdatingOnly with a new version of the modelYou update the document and the answer changes with it
Pointing to the sourceNot possibleThe answer can refer to the source document
AccessThe same for everyone who uses the modelDepends on who is asking

If a model gets a question about your business without having been given any business knowledge, it fills the gap with something that sounds plausible. Such a made-up answer is called a hallucination. The solution is not a smarter model, but the right information alongside the question.

Integration

The API: how your software talks to the model

Your software talks to a language model via an API: it sends a request with an instruction and the necessary text, and receives a response. Everything the model may use for that response is contained in that single request.

Two properties determine how you handle business data. The model itself remembers nothing between two calls: whatever you sent last time is gone. And there is a limit to the amount of text per call, so you cannot send your entire archive. Besides, more text does not automatically make the answer better.

This leads to the key question of this topic: which pieces of information do you send with which question? The answer is retrieval: looking up the right passages. How such an integration fits into existing software is explained in AI integrations: connecting AI to existing systems.

Technology

How does RAG (retrieval-augmented generation) work?

RAG stands for retrieval-augmented generation: first look things up (retrieval), then let the model answer (generation) with the retrieved text as a supplement (augmented). In six steps:

01

Preparing documents

Your sources are split into smaller chunks, such as a paragraph or a chapter, so that they can later be searched in a targeted way.

02

Index

An embedding is calculated for each chunk: a series of numbers that represents the meaning of the text. These series are stored in an index, often a vector database.

03

Retrieving

A question is converted in the same way and compared with the index. The chunks whose meaning is closest to the question come out on top, even when they use different words. This is often combined with keyword search.

04

Augmenting

The retrieved chunks are sent to the model along with the question and an instruction: answer only on the basis of this text, and say so when the answer is not in it.

05

Answering

The model formulates an answer from the passages supplied. It does not look anything up itself: it reads what the retrieval step supplied.

06

Referring to the source

The answer states which document it comes from, so a member of staff can check it.

Sources

Which documents and data are suitable?

An answer from RAG is never better than the source it comes from. Whatever is wrong, outdated or contradictory in your documents ends up in the answer in the same way.

Well suited

  • manuals, procedures and work instructions
  • product information and technical data sheets
  • frequently asked questions with a verified answer
  • terms and policy documents in their current version
  • documents with a clear title, date and owner

Sort out first

  • old versions that remain alongside the current one
  • documents that contradict each other
  • scans and images without readable text
  • knowledge that exists only in people’s heads or mailboxes
  • files that nobody knows are still accurate

Structured data, such as orders, stock or customer records, is better looked up directly in the database than via text passages. A good solution often combines both: passages from documents for explanations, a targeted query for figures and statuses. If the model is allowed to choose and run such queries itself, you are talking about an agent: see AI agents for businesses: 7 processes you can automate.

Agreements

Access rights, privacy and security

As soon as a model works with business data, it is no longer just about the quality of the answer, but also about who gets to see what and where the data ends up.

  • Permissions during retrieval The retrieval step only returns documents that the person asking is allowed to view. That check takes place in your software, before anything goes to the model, and not in the model.
  • Send as little as possible Only the passages the question requires are sent. Anything not needed, such as names or contact details, is left out or anonymised.
  • What leaves your systems Map out which data goes to the model provider, and whether it includes personal data or confidential information.
  • Where it is processed Check in which region the data is processed and how long the provider retains it.
  • Use for training Check the provider’s terms to see whether the data you send is used to train models, and whether you can opt out.
  • Keys and index The API key stays on the server. The index also contains your business information and deserves the same security as the documents themselves.

If the documents contain personal data, the GDPR is the framework within which you weigh all of this up. This is not legal advice: have your specific situation reviewed by someone with knowledge of privacy law.

Limitations

Why you should not simply put your business data into a model

Pasting into a public chat service

Anyone who pastes a contract or a customer list into a public chat service sends that data to an external party without your business having made any agreements about it. So set out internally which data may go into which tools, and provide an alternative that does fall under your own agreements.

Training a model on your own data

Further training a model, also called fine-tuning, mainly adjusts its style or behaviour. It is less suitable as a store of knowledge: you cannot remove or update a specific fact in it, the model cannot point to where an answer comes from, and it does not distinguish between who is asking. For knowledge that changes or is not meant for everyone, retrieving it at the moment of the question is the more logical route.

Human oversight remains necessary

Even with the right passages, a model can misread something or combine two sources incorrectly. Have a person check, approve or correct answers wherever a mistake has consequences, and always show the source. How I approach automation and AI integrations for businesses is explained on the page AI automation.

FAQ

Frequently asked questions about AI with business data

RAG is an open-book exam for a language model. Instead of answering from memory, the model receives the relevant passages from your own documents with each question and formulates an answer from them. The abbreviation stands for retrieval-augmented generation.

For knowledge, usually not. Retrieving at the moment of the question keeps your information current, respects access rights and lets you show the source. Further training is better suited to a consistent style or a very specific task.

At most, whatever was publicly available about it when the model was trained, and even that is incomplete and out of date. It does not know your internal documents, prices, customers or arrangements, unless you send them along with the question.

Yes. RAG reduces the chance that the model makes something up, because the answer is based on supplied text, but it does not rule out errors. The retrieval step can find the wrong passage, or the model can misread a passage. A source reference and a person who checks where it matters remain necessary.

That depends on the set-up. The deciding factors are the access rights during retrieval, where the index and the documents are stored, and the model provider’s terms. For highly sensitive data, a model on your own infrastructure may be an option.

Not always. For a limited set of documents, an ordinary search function or an extension to your existing database may be enough. A separate vector database becomes worthwhile with large volumes of text or when searching by meaning gives clearly better results.

The technique is not reserved for large companies. What determines the work is the state of your documents, the number of sources and the access requirements. A well-defined start, such as one set of manuals for one team, keeps it manageable. Bespoke work is priced on request.

More on this topic

Further reading

The related service

AI automation as a service

Would you like a language model to work with your own documents or systems? The service page explains how I approach automation and AI integrations, with attention to data, permissions and oversight.

Dries Nijs, Organic Web Specialist and author of this article

About the author

Dries Nijs, Organic Web Specialist

I am Dries Nijs, a developer from Maasmechelen. I have been building for the web since the 1990s: web applications in React, Next.js and Node.js, and websites in WordPress. In my own platforms, such as BM Jumpfun, Pelletkachelzorg and SEO Websites, I use AI via APIs. MadeByDries is the personal brand under which I work.

Want AI to work with your own data?

Tell me which questions the system needs to answer and where the information is stored today. Together we will then look at what is feasible and responsible.

Discuss your project

Tell me what you want to build or improve

You do not need a technical brief yet. Tell me where you are now and what you want to achieve, and I will help you find the right approach.

Choose at least one topic. You can select more than one.

I only use your details to respond to your enquiry.

Prefer to email me directly?

Location
Maasmechelen, Belgium