> For the complete documentation index, see [llms.txt](https://docs.voiceb.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.voiceb.ai/product-guides/automate-your-knowledge-base-with-web-scraping-beta.md).

# Automate Your Knowledge Base with Web Scraping (BETA)

A great agent starts with great knowledge — and now **Agent Studio builds that knowledge for you** directly from your website. Instead of writing prompts or uploading documents, you give Agent Studio a URL, and it scans, structures, and adds your public content to the agent's knowledge base automatically. You can even scrape your website to **create a brand-new agent from scratch** — Agent Studio uses the content to generate the goal, script, and objections for you.

<figure><img src="/files/ozfmRSCcjD4htQRX9dCj" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
This capability used to be a separate beta that generated a `.txt` file on request. It's now built directly into **Agent Studio** — self-serve, no email or manual file handling required.
{% endhint %}

#### 🔍 What it does

When you give Agent Studio a URL while building or editing an agent, it:

* Crawls the public pages linked from that address, including:
  * Product pages
  * Pricing
  * Help center / FAQs
  * Blog posts
  * Documentation and resources
* Extracts the useful content: headlines, paragraphs, lists, answers, calls-to-action
* Cleans and structures it into the agent's knowledge base, formatted for optimal AI comprehension

No prompts to write from scratch, no support topics to enter by hand.

***

#### 🧠 Why it matters

Your agent is only as good as the knowledge it has. Building the knowledge base from your website means your agent:

✅ **Covers all your public info** — from product specs to FAQs, it draws on your published content.\
✅ **Speaks your language** — it absorbs your tone, brand, and messaging from the way you present content online.\
✅ **Saves time** — no copying and pasting; the knowledge base builds itself from a URL.

***

#### 📂 What VoiceB extracts

Here's a simplified example of how VoiceB structures scraped content:

```
Section: Plans

The Mobile Unlimited plan costs $15/month and includes unlimited calls and 50GB at 5G speeds.

The Fiber 600Mb plan includes internet and mobile for $35/month.

Section: FAQs

Q: How long does installation take?
A: Installation usually takes 1–3 days depending on availability.

Q: Can I change my plan later?
A: Yes, all plans are flexible and can be upgraded at any time.
```

> This structure is optimized for AI processing, ensuring accurate, human-like responses from your agent. You can review and refine everything in Agent Studio before going live.

***

#### 🗣️ Getting started

You can use web scraping two ways:

* **Create a new agent from scratch** — give Agent Studio your website URL and it will generate the full agent (goal, script, objections, and knowledge base) from your content.
* **Enrich an existing agent** — add a URL to an agent you've already built to expand its knowledge base.

Steps:

1. Open **Agent Studio** and start building a new agent (or edit an existing one).
2. Give it your website URL (or a specific page — pricing, FAQ, product).
3. Agent Studio scans the content and adds it to the knowledge base.
4. Review, edit, or expand the result, then go live.

{% hint style="success" %}
**Recommendation:** Ask Agent Studio to consolidate the entire knowledge base into a **single script block**. Keeping it in one block makes the agent far easier to maintain and optimize — you update one place instead of hunting across multiple blocks.
{% endhint %}

This is especially valuable for SaaS products, eCommerce catalogs, and telcos, banks, and insurers with dynamic pricing and FAQs.

> New to the platform? Start with [Getting Started in 20 Minutes →](https://docs.voiceb.ai/product-guides/getting-started-with-voiceb.ai-in-20-minutes)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.voiceb.ai/product-guides/automate-your-knowledge-base-with-web-scraping-beta.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.