# Add pages from your website

> Teach the assistant your website's content by adding a single page by URL or many pages from your sitemap.

Source: https://agentmysite.com/docs/guides/content/add-a-website/

Add pages from your website so the assistant can answer from them. You can add one page by its URL, or pick many pages from your sitemap.

## Before you begin

- The pages must be public. AgentMySite fetches them over `http` or `https` without signing in.
- To add pages from a sitemap, your site needs a sitemap file, usually at `/sitemap.xml`.

## Add a single page

<Steps>

1. In the sidebar, select **Documents**.

2. Select **Add URL**.

3. In **Page URL**, enter the full address of the page, for example `https://www.acme.com/shipping`.

4. Select **Add page**.

</Steps>

![The Add a page by URL dialog with a page URL entered](../../../../assets/screenshots/add-url.png)

The page appears in the list with the status **Queued**, then **Processing**, then **Ready**.

If you see **This URL is already ingested — use Re-sync to refresh it.**, the page is already in your content. See [Keep content fresh](/docs/guides/content/keep-content-fresh/).

## Add pages from your sitemap

<Steps>

1. In the sidebar, select **Documents**.

2. Select **Add from sitemap**.

3. In **Website URL**, enter your site's address, for example `https://help.acme.com`, and select **Find pages**.

4. In the list of pages, select the ones you want. Use the filter box to narrow the list, or **Select all** and **Select none**.

5. Select **Add _N_ pages**.

</Steps>

![The Add pages from sitemap dialog listing three pages, all selected, with an Add 3 pages button](../../../../assets/screenshots/add-from-sitemap.png)

AgentMySite looks for your sitemap in this order:

1. The address you entered, if it ends in `.xml`.
2. `sitemap.xml` in the same folder as the address you entered. For example, `https://acme.com/support/` checks `https://acme.com/support/sitemap.xml`. This helps when your help center lives under a subpath.
3. `sitemap.xml` at the root of the domain, for example `https://acme.com/sitemap.xml`.

Sitemap index files are supported; AgentMySite follows them to the sitemaps they list. Very large sitemaps are cut off at 10,000 pages, and the dialog shows **This sitemap is large — only the first pages are shown.**

Pages you already added are skipped, so you can run **Add from sitemap** again later to pick up new pages.

## What you should see

The sitemap appears as one row with a summary such as **3/3 ready**, and each page appears as its own row. Each **Ready** page counts toward **Pages and documents** under **Plan limits** on your dashboard; the sitemap row itself doesn't.

![The Documents list with uploaded files and web pages in Ready, Processing, Queued and Failed states](../../../../assets/screenshots/documents.png)

## How pages are fetched

- AgentMySite respects your `robots.txt`. Pages it disallows fail with the reason **robots.txt disallows this URL**.
- It identifies itself with the user agent `AgentMySite-Bot/1.0`.
- Only HTML pages are added. A page must respond within 10 seconds and be no larger than 5 MB. Up to 3 redirects are followed.
- Addresses with a username and password in them, bare IP addresses, and private network addresses aren't allowed.

<Aside type="tip">
Pages that need a login, or that build all their text with JavaScript after loading, may have little or no text for the assistant to use. For that content, [upload a document](/docs/guides/content/upload-documents/) instead.
</Aside>

## Next steps

- [Keep content fresh](/docs/guides/content/keep-content-fresh/) when your pages change.
- [Upload documents](/docs/guides/content/upload-documents/) for content that isn't on your website.