User-agent: * Disallow: /wp-includes/ Disallow: /tmp/ Disallow: /images/2008/ Disallow: /images/2009/ Disallow: /images/2010/ Disallow: /images/2011/ Disallow: /images/2012/ Disallow: /images/2013/ Disallow: /images/2014/ Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /search/ Disallow: /author/ Disallow: /calendar/admin/ Disallow: /calendar/events/link/ Disallow: /wp-content/ Disallow: /ajax Disallow: /login/ Disallow: /archives/ Disallow: /login/ Disallow: /random-event-images/ Disallow: /*.css$ Disallow: /*.js$ Disallow: /wp-* Disallow: /trackback/ Disallow: /*.inc$ Disallow: /wp-comments-post.php Disallow: /admin-ajax.php Allow: /calendar/events/sitemap.php User-agent: msnbot Crawl-delay: 10 User-agent: bingbot Crawl-delay: 25 User-agent: Slurp Crawl-delay: 25 User-agent: Flamingo_SearchEngine Crawl-delay: 25 user-agent: AhrefsBot disallow: / user-agent: Baiduspider disallow: / User-agent: ia_archiver Disallow: / User-agent: duggmirror Disallow: / User-agent: Trapit Disallow: / User-agent: BLEXBot Disallow: / User-agent: ContextAd Disallow: / User-agent: ContextWeb Disallow: / User-agent: linkdexbot Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: MJ12bot Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: ScooperBot Disallow: / User-agent: psbot Disallow: / User-agent: Riddler Disallow: / User-agent: InfoPath Disallow: / User-agent: MaxPointCrawler disallow: / User-agent: MaxPointCrawler/Nutch-1.1 disallow: / User-agent: Ezooms Disallow: / User-agent: Ezooms.bot Disallow: / User-agent: Buck Disallow: / User-agent: ias_crawler Disallow: / User-agent: istellabot Disallow: / User-agent: Java 1.7 Apache HttpClient (Linux x86_64) / GnowitNewsbot / Contact information at http://www.gnowit.com Disallow: / User-agent: The Knowledge AI Disallow: / User-agent: weborama-fetcher (+http://www.weborama.com) Disallow: / User-agent: VelenPublicWebCrawler (velen.io) Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: sc-downloader/2.0 (http://www.searchconcepts.ch) Disallow: / User-agent: Owlin - https://www.owlin.com (Contains Googlebot for cookiewall) Disallow: / User-agent: omgili/0.5 +http://omgili.com Disallow: / User-agent: Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html) Disallow: / User-agent: googlebot Allow: / Allow: /images/2012/ User-agent: Googlebot-News Allow: / Disallow: /calendar/events/sitemap.php User-agent:adsbot Crawl-delay: 15 Sitemap: http://calcoastnews.com/calendar/events/sitemap.php Sitemap: http://calcoastnews.com/sitemap_index.xml