vlad/salvagedb_web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
salvagedb_web/templates/robots.txt
Vlad 12f0b91216 Init
2024-11-28 12:19:28 +03:00

836 lines
10 KiB
Plaintext
Raw Blame History

User-agent: *
Disallow: /detail/
Disallow: /search
Disallow: /info/
User-agent: Yandex
Disallow: /
User-agent: Amazonbot
Disallow: /
User-Agent: SemrushBot
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: AhrefsBot
Disallow: /
User-Agent: gigabot
Disallow: /
User-Agent: Gigabot/2.0
Disallow: /
User-Agent: msnbot
Disallow: /
User-Agent: msnbot/1.0
Disallow: /
User-Agent: ia_archiver
Disallow: /
User-Agent: libwww-perl
Disallow: /
User-Agent: NetStat.Ru Agent
Disallow: /
User-Agent: WebAlta Crawler/1.3.25
Disallow: /
User-Agent: Yahoo!-MMCrawler/3.x
Disallow: /
User-Agent: MMCrawler/3.x
Disallow: /
User-Agent: NG/2.0
Disallow: /
User-Agent: slurp
Disallow: /
User-Agent: aipbot
Disallow: /
User-Agent: Alexibot
Disallow: /
User-Agent: GameSpyHTTP/1.0
Disallow: /
User-Agent: Aqua_Products
Disallow: /
User-Agent: asterias
Disallow: /
User-Agent: b2w/0.1
Disallow: /
User-Agent: BackDoorBot/1.0
Disallow: /
User-Agent: becomebot
Disallow: /
User-Agent: BlowFish/1.0
Disallow: /
User-Agent: Bookmark search tool
Disallow: /
User-Agent: BotALot
Disallow: /
User-Agent: BotRightHere
Disallow: /
User-Agent: BuiltBotTough
Disallow: /
User-Agent: Bullseye/1.0
Disallow: /
User-Agent: BunnySlippers
Disallow: /
User-Agent: CheeseBot
Disallow: /
User-Agent: CherryPicker
Disallow: /
User-Agent: CherryPickerElite/1.0
Disallow: /
User-Agent: CherryPickerSE/1.0
Disallow: /
User-Agent: Copernic
Disallow: /
User-Agent: CopyRightCheck
Disallow: /
User-Agent: cosmos
Disallow: /
User-Agent: Crescent
Disallow: /
User-Agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-Agent: DittoSpyder
Disallow: /
User-Agent: EmailCollector
Disallow: /
User-Agent: EmailSiphon
Disallow: /
User-Agent: EmailWolf
Disallow: /
User-Agent: EroCrawler
Disallow: /
User-Agent: ExtractorPro
Disallow: /
User-Agent: FairAd Client
Disallow: /
User-Agent: Fasterfox
Disallow: /
User-Agent: Flaming AttackBot
Disallow: /
User-Agent: Foobot
Disallow: /
User-Agent: Gaisbot
Disallow: /
User-Agent: GetRight/4.2
Disallow: /
User-Agent: Harvest/1.5
Disallow: /
User-Agent: hloader
Disallow: /
User-Agent: httplib
Disallow: /
User-Agent: HTTrack 3.0
Disallow: /
User-Agent: humanlinks
Disallow: /
User-Agent: IconSurf
Disallow: /
User-Agent: InfoNaviRobot
Disallow: /
User-Agent: Iron33/1.0.2
Disallow: /
User-Agent: JennyBot
Disallow: /
User-Agent: Kenjin Spider
Disallow: /
User-Agent: Keyword Density/0.9
Disallow: /
User-Agent: larbin
Disallow: /
User-Agent: LexiBot
Disallow: /
User-Agent: libWeb/clsHTTP
Disallow: /
User-Agent: LinkextractorPro
Disallow: /
User-Agent: LinkScan/8.1a Unix
Disallow: /
User-Agent: LinkWalker
Disallow: /
User-Agent: LNSpiderguy
Disallow: /
User-Agent: lwp-trivial
Disallow: /
User-Agent: lwp-trivial/1.34
Disallow: /
User-Agent: Mata Hari
Disallow: /
User-Agent: Microsoft URL Control
Disallow: /
User-Agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-Agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-Agent: MIIxpc
Disallow: /
User-Agent: MIIxpc/4.2
Disallow: /
User-Agent: Mister PiX
Disallow: /
User-Agent: moget
Disallow: /
User-Agent: moget/2.1
Disallow: /
User-Agent: MSIECrawler
Disallow: /
User-Agent: NetAnts
Disallow: /
User-Agent: NICErsPRO
Disallow: /
User-Agent: Offline Explorer
Disallow: /
User-Agent: Openbot
Disallow: /
User-Agent: Openfind
Disallow: /
User-Agent: Openfind data gatherer
Disallow: /
User-Agent: Oracle Ultra Search
Disallow: /
User-Agent: PerMan
Disallow: /
User-Agent: ProPowerBot/2.14
Disallow: /
User-Agent: ProWebWalker
Disallow: /
User-Agent: psbot
Disallow: /
User-Agent: Python-urllib
Disallow: /
User-Agent: QueryN Metasearch
Disallow: /
User-Agent: Radiation Retriever 1.1
Disallow: /
User-Agent: RepoMonkey
Disallow: /
User-Agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-Agent: RMA
Disallow: /
User-Agent: searchpreview
Disallow: /
User-Agent: SiteSnagger
Disallow: /
User-Agent: SpankBot
Disallow: /
User-Agent: spanner
Disallow: /
User-Agent: SurveyBot
Disallow: /
User-Agent: suzuran
Disallow: /
User-Agent: Szukacz/1.4
Disallow: /
User-Agent: Teleport
Disallow: /
User-Agent: TeleportPro
Disallow: /
User-Agent: Telesoft
Disallow: /
User-Agent: The Intraformant
Disallow: /
User-Agent: TheNomad
Disallow: /
User-Agent: TightTwatBot
Disallow: /
User-Agent: toCrawl/UrlDispatcher
Disallow: /
User-Agent: True_Robot
Disallow: /
User-Agent: True_Robot/1.0
Disallow: /
User-Agent: turingos
Disallow: /
User-Agent: TurnitinBot
Disallow: /
User-Agent: TurnitinBot/1.5
Disallow: /
User-Agent: URL Control
Disallow: /
User-Agent: URL_Spider_Pro
Disallow: /
User-Agent: URLy Warning
Disallow: /
User-Agent: VCI
Disallow: /
User-Agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
User-Agent: Web Image Collector
Disallow: /
User-Agent: WebAuto
Disallow: /
User-Agent: WebBandit
Disallow: /
User-Agent: WebBandit/3.50
Disallow: /
User-Agent: WebCapture 2.0
Disallow: /
User-Agent: WebCopier
Disallow: /
User-Agent: WebCopier v.2.2
Disallow: /
User-Agent: WebCopier v3.2a
Disallow: /
User-Agent: WebEnhancer
Disallow: /
User-Agent: WebSauger
Disallow: /
User-Agent: Website Quester
Disallow: /
User-Agent: Webster Pro
Disallow: /
User-Agent: WebStripper
Disallow: /
User-Agent: WebZip
Disallow: /
User-Agent: WebZip
Disallow: /
User-Agent: WebZip/4.0
Disallow: /
User-Agent: WebZIP/4.21
Disallow: /
User-Agent: WebZIP/5.0
Disallow: /
User-Agent: Wget
Disallow: /
User-Agent: wget
Disallow: /
User-Agent: Wget/1.5.3
Disallow: /
User-Agent: Wget/1.6
Disallow: /
User-Agent: WWW-Collector-E
Disallow: /
User-Agent: Xenu's
Disallow: /
User-Agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-Agent: Zeus
Disallow: /
User-Agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-Agent: Zeus Link Scout
Disallow: /
User-Agent: EmailSiphon
Disallow: /
User-Agent: EmailCollector
Disallow: /
User-Agent: teoma
Disallow: /
User-Agent: NjuiceBot
Disallow: /
User-Agent: Scrapy
Disallow: /
User-Agent: Baiduspider
Disallow: /
User-Agent: SeznamBot
Disallow: /
User-Agent: crawler
Disallow: /
User-Agent: JS-Kit
Disallow: /
User-Agent: HybridBot
Disallow: /
User-Agent: Voyager
Disallow: /
User-Agent: PostRank
Disallow: /
User-Agent: DomainCrawler
Disallow: /
User-Agent: MegaIndex.ru
Disallow: /
User-Agent: ltx71
Disallow: /
User-Agent: Exabot
Disallow: /
User-Agent: CCBot
Disallow: /
User-Agent: DotBot
Disallow: /
User-Agent: GetIntent\ Crawler
Disallow: /
User-Agent: Butterfly
Disallow: /
User-Agent: libwww
Disallow: /
User-Agent: SWeb
Disallow: /
User-Agent: LinkExchanger
Disallow: /
User-Agent: Soup
Disallow: /
User-Agent: GrapeshotCrawler
Disallow: /
User-Agent: DnyzBot
Disallow: /
User-Agent: spbot
Disallow: /
User-Agent: DeuSu
Disallow: /
User-Agent: MLBot
Disallow: /
User-Agent: InternetSeer
Disallow: /
User-Agent: BUbiNG
Disallow: /
User-Agent: FairShare
Disallow: /
User-Agent: Yeti
Disallow: /
User-Agent: Birubot
Disallow: /
User-Agent: YottosBot
Disallow: /
User-Agent: gold\ crawler
Disallow: /
User-Agent: Linguee
Disallow: /
User-Agent: Ezooms
Disallow: /
User-Agent: Purebot
Disallow: /
User-Agent: kmSearchBot
Disallow: /
User-Agent: SiteBot
Disallow: /
User-Agent: CamontSpider
Disallow: /
User-Agent: ptd-crawler
Disallow: /
User-Agent: HTTrack
Disallow: /
User-Agent: suggybot
Disallow: /
User-Agent: ttCrawler
Disallow: /
User-Agent: Nutch
Disallow: /
User-Agent: msnbot-media
Disallow: /
User-Agent: Abonti
Disallow: /
User-Agent: aggregator
Disallow: /
User-Agent: BDCbot
Disallow: /
User-Agent: BLEXBot
Disallow: /
User-Agent: Bullseye
Disallow: /
User-Agent: ca\-crawler
Disallow: /
User-Agent: Cegbfeieh
Disallow: /
User-Agent: coccoc
Disallow: /
User-Agent: CyotekWebCopy/1\.7
Disallow: /
User-Agent: CyotekHTTP/2\.0
Disallow: /
User-Agent: discobot
Disallow: /
User-Agent: Download Ninja
Disallow: /
User-Agent: EasouSpider
Disallow: /
User-Agent: FeedBooster
Disallow: /
User-Agent: Genieo
Disallow: /
User-Agent: grub\-client
Disallow: /
User-Agent: Harvest
Disallow: /
User-Agent: ieautodiscovery
Disallow: /
User-Agent: Incutio
Disallow: /
User-Agent: IstellaBot
Disallow: /
User-Agent: JamesBOT
Disallow: /
User-Agent: k2spider
Disallow: /
User-Agent: Keyword Density/0\.9
Disallow: /
User-Agent: libWeb
Disallow: /
User-Agent: linko
Disallow: /
User-Agent: LinkScan/8\.1a Unix
Disallow: /
User-Agent: lmspider
Disallow: /
User-Agent: lwp\-trivial
Disallow: /
User-Agent: magpie
Disallow: /
User-Agent: MaxPointCrawler
Disallow: /
User-Agent: MegaIndex
Disallow: /
User-Agent: memoryBot
Disallow: /
User-Agent: Mippin
Disallow: /
User-Agent: Missigua Locator
Disallow: /
User-Agent: Niki\-Bot
Disallow: /
User-Agent: NPBot
Disallow: /
User-Agent: OLEcrawler
Disallow: /
User-Agent: panscient\.com
Disallow: /
User-Agent: ProPowerBot/2\.14
Disallow: /
User-Agent: Python\-urllib
Disallow: /
User-Agent: Riddler
Disallow: /
User-Agent: serf
Disallow: /
User-Agent: SISTRIX
Disallow: /
User-Agent: sitecheck\.Internetseer\.com
Disallow: /
User-Agent: Serpstat
Disallow: /
User-Agent: SnapPreviewBot
Disallow: /
User-Agent: Sogou
Disallow: /
User-Agent: Spinn3r
Disallow: /
User-Agent: SpyFu
Disallow: /
User-Agent: Szukacz/1\.4
Disallow: /
User-Agent: Titan
Disallow: /
User-Agent: UbiCrawler
Disallow: /
User-Agent: UnisterBot
Disallow: /
User-Agent: Unknown
Disallow: /
User-Agent: uptime files
Disallow: /
User-Agent: User-Agent
Disallow: /
User-Agent: Vedma
Disallow: /
User-Agent: WBSearchBot
Disallow: /
User-Agent: Web Downloader/6\.9
Disallow: /
User-Agent: WebmasterWorldForumBot
Disallow: /
User-Agent: WebReaper
Disallow: /
User-Agent: Wotbox
Disallow: /
User-Agent: wsr\-agent
Disallow: /
User-Agent: WWW\-Collector\-E
Disallow: /
User-Agent: Zao
Disallow: /
User-Agent: ZyBORG
Disallow: /
User-Agent: ahrefs
Disallow: /
User-Agent: qwantify
Disallow: /
User-Agent: qwant
Disallow: /
User-Agent: semrush
Disallow: /
User-Agent: Detectify
Disallow: /
User-Agent: LinkpadBot
Disallow: /
User-Agent: FlipboardProxy
Disallow: /
User-Agent: aiHitBot
Disallow: /
User-Agent: trovitBot
Disallow: /
Sitemap: https://www.salvagedb.com/sitemap.xml
Host: https://www.salvagedb.com
Reference in New Issue View Git Blame Copy Permalink