Most website chatbots don’t train a model on your website at all. They save a copy of your pages, and for each question they search that copy and give the AI the few passages that match, so it can answer from them. That means the quality of the answers depends on which pages the chatbot reads, what it leaves out, and what you add for the questions your website doesn’t answer.
This guide covers each of those steps: which pages to include, how sitemaps and robots.txt affect an import, what happens with JavaScript-built sites, how to fill the gaps, how to test before customers see it, and how to keep it current.
“Training” vs retrieval, in plain words
When a vendor says “train the chatbot on your website”, they nearly always mean retrieval. The technique has a name, retrieval-augmented generation (RAG), from a 2020 research paper. In practice it works like this:
- Import. The tool reads your pages and stores their text, cleaned of menus, footers and other repeated parts.
- Search. When a customer asks “Do you deliver to Austin?”, the tool searches the stored text for the passages most likely to answer it.
- Answer. Only those passages go to the AI model, with instructions to answer from them.
Real training, or fine-tuning, changes the model’s own weights. It’s slow, it costs money each time, and a fine-tuned model can still mix your facts up with what it learned elsewhere. Retrieval has three practical advantages for a support chat:
| Retrieval (what most chatbots do) | Fine-tuning | |
|---|---|---|
| Update after a price change | Import the page again; minutes | Train again; hours or days |
| Can show where an answer came from | Yes, the passage has a page address | No |
| Can tell when nothing matches | Yes, the search finds nothing | No, the model answers anyway |
| What decides the quality | Which pages it reads and how well it searches | The training data and the model |
The last two rows matter most. A chatbot that knows when its search found nothing can say “I’m not sure” instead of guessing. If you want to go deeper on that, read our guide to why chatbots give wrong answers.
So when you “train an AI chatbot on your website content”, the work is really choosing the content and checking the search finds it.
Which pages to include and leave out
A chatbot that reads every page on your site isn’t better informed. It’s noisier. A blog post from 2022 that mentions an old price will be found by the search just as readily as your current price page, and the AI can’t tell which one is right.
Start with the pages customers actually ask about:
| Include first | Usually leave out |
|---|---|
| Home page | Blog and news posts |
| Services or products, and their prices | Job and careers pages |
| Delivery, returns, refund and cancellation policies | Privacy and cookie pages |
| FAQ and help pages | Sign-in, cart and account pages |
| Contact page, opening hours, locations | Tag, category, archive and search-result pages |
| About page (who you are, how long you’ve run) | Error pages and old landing pages for finished offers |
Two rules decide the gray areas:
- Would a customer ask a question this page answers? A blog post titled “Five ways to style a linen shirt” rarely answers “How long does delivery take?”. A blog post titled “How our installation works” might, and is worth including by hand.
- Is the page current? If a page has old prices, dates or offers, either fix it or keep it out. The chatbot will repeat what the page says.
A good chatbot also strips the text every page repeats. If your footer lists your phone number and address on 80 pages, the search shouldn’t return that footer 80 times. It should keep it once, as site information.
Sitemaps, robots.txt and noindex
Most website chatbots find your pages the way a search engine does. Three files and tags decide what they see.
The sitemap. A sitemap is a list of your page addresses, usually at /sitemap.xml. Shopify, Wix, Squarespace and Webflow create one for you, and WordPress has had one built in since version 5.5 (at /wp-sitemap.xml, or /sitemap_index.xml with an SEO plugin). If your site has one, the import reads it and gets every page in one go. If not, it follows links from the home page, and pages that nothing links to are missed. Before importing, open your sitemap in a browser and check your important pages are listed.
robots.txt. robots.txt tells crawlers which parts of a site to stay out of. A well-behaved chatbot import respects it. If you’ve blocked /help/ from crawlers, your help pages won’t be imported either. That’s usually a leftover from a staging site, and worth fixing for Google’s sake too.
noindex. A page with <meta name="robots" content="noindex"> asks search engines not to list it (Google’s explanation). Imports often skip these pages as well. That’s right for thank-you pages and internal pages, but check you haven’t left noindex on a whole site after launch. It happens more often than you’d think, especially on sites copied from a staging copy.
JavaScript-built sites (React, Vue) and headless rendering
Some websites send the browser an almost empty page and build the content with JavaScript: single-page apps in React, Vue or Angular, and some site builders. A simple import downloads the HTML and finds nothing in it. Every address returns the same empty shell, so the chatbot ends up knowing one page of loading text.
You can check your own site in a minute:
- Open a page in Chrome.
- Right-click and choose View page source (not Inspect).
- Search the source for a sentence you can see on the page.
If the sentence is in the source, any import can read it. If it isn’t, the page is built in the browser, and you need a chatbot that opens pages in a real browser before reading them, called headless rendering. Google does the same when it indexes JavaScript sites, and its guide to JavaScript SEO explains the steps.
If your chatbot can’t render JavaScript, you have two options. Ask your developer to prerender or server-render the pages (frameworks such as Next.js and Nuxt do this), which also helps your search rankings. Or add the key facts by hand as Q&As and notes.
Filling the gaps: Q&As and notes
Your website was written to sell, not to answer support questions. Some answers customers want aren’t on any page:
- “Can I pick up instead of delivery?”
- “Do you have parking?”
- “Is the course fee payable in installments?”
- “Do you ship to Canada?”
Add these as short Q&As. Write the question the way a customer would type it, and the answer the way your best teammate would say it:
Q: Can I pay the course fee in installments?
A: Yes. The fee can be paid in two parts: half at admission and half before the second term starts. There’s no extra charge.
Our guide to turning your FAQ into a website chatbot has more rules for writing them, and a 25-question test.
For longer information that doesn’t fit a question, such as a delivery-zone list, a seasonal timetable or how a service works step by step, add a titled note. Keep each one on a single topic, so the search can find it.
Where do the gap questions come from? Three places:
- Your inbox and phone. The ten questions your team answers most often. If they aren’t on the website, they belong in Q&As.
- Chats your team took over. Every time a customer was handed to a person, ask: could the chatbot have answered this with one more Q&A?
- Your competitors’ FAQ pages. They show what customers in your field ask.
A tutoring center, for example, usually needs Q&As for class times, eligibility, the refund rule if a student drops out, and whether demo classes are free. Our page on chat for coaching institute websites goes through this in more detail.
One rule: don’t write a Q&A that contradicts a page. If the fee changed, change the page too. A chatbot that finds two different fees should say the information differs, not pick one, and that’s a sign you need to fix one of them.
Testing before customers see it
Never put a chatbot on your site without asking it questions first. A useful test takes about 20 minutes.
Write 20 questions in three groups:
| Group | Examples | What a good answer looks like |
|---|---|---|
| On your site (10) | “What are your opening hours?”, “How much is the basic plan?” | The right fact, with a link to the page it came from |
| Asked in other words (5) | “When do you shut?”, “What’s the cheapest option?” | The same facts, found despite different words |
| Not on your site (5) | “Do you price-match?”, “Can I bring my dog?” | “I’m not sure”, and an offer to reach a person, not a guess |
The third group is the one most guides skip, and the most important. A chatbot that answers everything confidently is a risk. You want one that says when it doesn’t know.
Then check two more things:
- Usage per item. After a few weeks live, look at which pages, Q&As and notes are used in answers. A page never used may be one customers don’t ask about, or one the search can’t find because it uses different words from your customers. Unused items are worth either rewriting or switching off.
- Answers after a change. Change a price on a page, import again, and ask the price. The chatbot should give the new one.
Keeping it current: re-import
Most chatbots hold a snapshot of your website from the moment you imported it. Change a price on your site tomorrow and the chatbot still knows the old one until you import again. Some tools re-crawl on a schedule; many, including SupportVia, re-import when you ask.
Make re-importing part of how you publish:
- After any change to prices, policies, hours or contact details, import the website again the same day.
- After adding a new service or product page, check it was picked up.
- After a redesign or a move to a new domain, import the site again from the new address. Old addresses in the knowledge mean broken links in answers.
- Once a month, look at the items that weren’t used and remove what’s out of date.
If you have several websites, for example an agency running client sites, put a re-import step in the checklist you already use when a client asks for a content change.
Doing it in SupportVia
SupportVia is an AI chat for your website that answers only from your own pages, Q&As and notes, shows the pages it read, and hands the chat to your team when it isn’t sure. It’s built around the steps above. Here is how the setup goes:
- Add your website address. SupportVia reads your sitemap, or follows links if there isn’t one. It respects
robots.txtand skips pages markednoindex. It drops the text your pages repeat, such as headers and footers. - Check the import preview. Before anything is imported, you see the pages it kept and what it left out, grouped: blog and news posts, job pages, privacy and cookie pages, sign-in and cart pages, tag and archive pages. The rest are ranked with your home, services, prices, help, policies and contact pages first. If it left out a page you want, add it by its link.
- JavaScript sites are handled. When a site builds its pages in the browser, SupportVia opens them in a sandboxed headless Chrome and reads them as a visitor sees them. If most pages still come back the same, it tells you, and suggests Q&As or prerendering.
- It fills in the chat for you. From the same import it reads your site’s name, logo and brand color, writes your company information, and drafts a greeting and up to five starter questions. You can edit all of them.
- Add Q&As and notes for what isn’t on the website. They can apply to one website or to all of them.
- Try it in the playground. Ask your 20 test questions. The playground uses your current knowledge, and nothing in it is stored or counted.
- Paste one line on your site. There are install guides for WordPress, Shopify, Wix, Squarespace, Webflow and any other website.
After that, each answer from the AI shows up to five “Read more” links to the pages it used. Your dashboard shows how often each page, Q&A and note was used in the last 30 days, and any item can be switched off. When the website changes, import it again.
A few honest limits. The search matches English word forms best; in other languages it matches exact words. SupportVia doesn’t take PDF or file uploads, and it doesn’t learn from your chats. You decide what it knows.
If you want a chat that does this for your site, see the AI chatbot trained on your website on our home page, or compare the plans and prices. You start with a free trial of 7 days or 50 AI conversations, with no card needed.
A short checklist
- Open your sitemap and check your key pages are listed.
- Check
robots.txtandnoindexaren’t hiding pages you want read. - View the source of one page; if the text isn’t there, you need headless rendering or prerendering.
- Import only useful pages; leave out blog, jobs, legal, account and archive pages.
- Fix pages with old prices or dates before importing.
- Add Q&As for the ten questions your team answers most.
- Test with 20 questions, including five that aren’t on your site.
- Import again every time prices, policies or hours change.