RMDBbot

RMDBbot is the crawler for RMDB (trmdb.com), a site that tracks restaurant menus and how they change over time. If you run a restaurant website and see RMDBbot in your logs, this page explains what it does.

What it fetches

• Only the pages, PDFs and images that hold your menu, plus robots.txt.
• About once a week, and less often (every two weeks, then monthly) when your menu hasn't changed for a while.
• One request at a time, at least 10 seconds apart, or longer if your robots.txt sets a Crawl-delay.
• Conditional requests, so an unchanged page costs you a 304 Not Modified.

What it never does

• Fetch anything your robots.txt disallows for RMDBbot or for *.
• Get past a bot check, CAPTCHA or login.
• Use proxies, rotating addresses or a disguised browser identity.
• Republish your PDFs or photos. RMDB shows dish names, prices and dates, with a link back to your site.

Opting out

Add this to your robots.txt:

User-agent: RMDBbot
Disallow: /

Or email . We honour opt-outs within 48 hours.

Own the restaurant?

You can claim your restaurant's page on RMDB to send us menu updates directly and fix anything we got wrong.