How to Create an XML Sitemap and robots.txt for Your Website
· 3 min read · By PK Creative
The sitemap lists the pages you want search engines to find, and robots.txt tells crawlers which areas to avoid. Both are small text files, and mistakes in either can quietly keep a site out of search.
What an XML sitemap looks like
A sitemap is an XML file containing a url entry for each page, with its address and optionally the last modification date. Search engines mostly use the address and date, while priority and change frequency are largely ignored. A sitemap can hold up to 50,000 URLs; larger sites use a sitemap index that points to several files.
Generate it automatically
Writing the file by hand is error-prone. Most platforms generate it: WordPress has plugins, and Next.js can build it from a sitemap file in the app folder that returns your list of pages. Generating it from the same data that builds your pages ensures new pages appear automatically.
Rules for what to include
Keep the sitemap accurate.
- Include only canonical, indexable pages that return a normal response.
- Use the exact final address style, with the same host and https.
- Exclude redirecting pages, admin areas and pages marked noindex.
- Update the last modified date only when content really changes.
A sensible robots.txt
A typical file allows crawling of the whole site, blocks private areas such as the admin panel and points to the sitemap. For example: User-agent: *, Allow: /, Disallow: /admin/, followed by Sitemap: https://yourdomain.in/sitemap.xml. Be careful: a single line like Disallow: / blocks the entire site and is a classic launch mistake.
Test both files
Open yourdomain.in/robots.txt and /sitemap.xml in a browser, then use Search Console to submit the sitemap and the URL Inspection tool to confirm pages are not blocked. Remember that robots.txt prevents crawling but is not a security tool. For deeper detail see our technical SEO guide and how to submit a sitemap.
Frequently asked questions
- Does robots.txt hide pages from Google?
- It blocks crawling, but a blocked page can still appear if linked elsewhere. Use noindex to keep pages out of results.
- Do I need both files?
- They are not mandatory, but together they help control crawling and speed up discovery.
- Can a wrong robots.txt remove my site from Google?
- A rule that blocks all crawling can prevent indexing. Always check it after launches.
- create XML sitemap
- robots.txt example
- sitemap and robots.txt
- Next.js sitemap