Automate Your Knowledge Base with Web Scraping (BETA)

A great agent starts with great knowledge — and now Agent Studio builds that knowledge for you directly from your website. Instead of writing prompts or uploading documents, you give Agent Studio a URL, and it scans, structures, and adds your public content to the agent's knowledge base automatically. You can even scrape your website to create a brand-new agent from scratch — Agent Studio uses the content to generate the goal, script, and objections for you.

This capability used to be a separate beta that generated a .txt file on request. It's now built directly into Agent Studio — self-serve, no email or manual file handling required.

🔍 What it does

When you give Agent Studio a URL while building or editing an agent, it:

• Crawls the public pages linked from that address, including:

  • Product pages

  • Pricing

  • Help center / FAQs

  • Blog posts

  • Documentation and resources

• Extracts the useful content: headlines, paragraphs, lists, answers, calls-to-action

• Cleans and structures it into the agent's knowledge base, formatted for optimal AI comprehension

No prompts to write from scratch, no support topics to enter by hand.

---

🧠 Why it matters

Your agent is only as good as the knowledge it has. Building the knowledge base from your website means your agent:

✅ Covers all your public info — from product specs to FAQs, it draws on your published content. ✅ Speaks your language — it absorbs your tone, brand, and messaging from the way you present content online. ✅ Saves time — no copying and pasting; the knowledge base builds itself from a URL.

---

📂 What VoiceB extracts

Here's a simplified example of how VoiceB structures scraped content:

Section: Plans

The Mobile Unlimited plan costs $15/month and includes unlimited calls and 50GB at 5G speeds.

The Fiber 600Mb plan includes internet and mobile for $35/month.

Section: FAQs

Q: How long does installation take?
A: Installation usually takes 1–3 days depending on availability.

Q: Can I change my plan later?
A: Yes, all plans are flexible and can be upgraded at any time.

This structure is optimized for AI processing, ensuring accurate, human-like responses from your agent. You can review and refine everything in Agent Studio before going live.

---

🗣️ Getting started

You can use web scraping two ways:

• Create a new agent from scratch — give Agent Studio your website URL and it will generate the full agent (goal, script, objections, and knowledge base) from your content.

• Enrich an existing agent — add a URL to an agent you've already built to expand its knowledge base.

Steps:

1. Open Agent Studio and start building a new agent (or edit an existing one).

2. Give it your website URL (or a specific page — pricing, FAQ, product).

3. Agent Studio scans the content and adds it to the knowledge base.

4. Review, edit, or expand the result, then go live.

Recommendation: Ask Agent Studio to consolidate the entire knowledge base into a single script block. Keeping it in one block makes the agent far easier to maintain and optimize — you update one place instead of hunting across multiple blocks.

This is especially valuable for SaaS products, eCommerce catalogs, and telcos, banks, and insurers with dynamic pricing and FAQs.

New to the platform? Start with Getting Started in 20 Minutes →