GAS(Google Apps Script)からのクロール拒否
ユーザーエージェントがMozilla/5.0 (compatible; Google-Apps-Script)からのアクセス数が急増。 原因を調べたら欠品中の在庫を定期的に調べているみたい。
.htaccess に以下を追加しアクセス禁止(403-Forbidden)に設定
RewriteEngine On # User Agent でアクセス拒否 SetEnvIfNoCase User-Agent "Google-Apps-Script" ua_key=on RewriteCond %{ENV:ua_key} on RewriteRule ^(.*)$ - [F,L]
GASってクローラー代わりに使えるようですね。 しかもアクセス元が複数ありました。
Google Apps ScriptのIP調査(拒否設定の為)
参考にさせて頂いたページ
GoogleAppsScript(GAS)のIPアドレス範囲を調べる
# nslookup -q=TXT _cloud-netblocks.googleusercontent.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: _cloud-netblocks.googleusercontent.com text = "v=spf1 include:_cloud-netblocks1.googleusercontent.com include:_cloud-netblocks2.googleusercontent.com include:_cloud-netblocks3.googleusercontent.com include:_cloud-netblocks4.googleusercontent.com include:_cloud-netblocks5.googleusercontent.com ?all" Authoritative answers can be found from:
include:cloud-netblocks1.googleusercontent.com include:cloud-netblocks2.googleusercontent.com include:cloud-netblocks3.googleusercontent.com include:cloud-netblocks4.googleusercontent.com include:_cloud-netblocks5.googleusercontent.com
# nslookup -q=TXT _cloud-netblocks1.googleusercontent.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: _cloud-netblocks1.googleusercontent.com text = "v=spf1 include:_cloud-netblocks6.googleusercontent.com include:_cloud-netblocks7.googleusercontent.com ip6:2600:1900::/35 ip4:8.34.208.0/20 ip4:8.35.192.0/21 ip4:8.35.200.0/23 ip4:23.236.48.0/20 ip4:23.251.128.0/19 ip4:34.64.0.0/11 ip4:34.96.0.0/14 ?all" Authoritative answers can be found from:
1~5(6~7)について繰り返し、アドレスの範囲がわかってのWAFの拒否設定に追加
34.100.0.0/16 34.102.0.0/15 34.104.0.0/14 34.124.0.0/18 34.124.64.0/20 34.124.80.0/23 34.124.84.0/22 34.124.88.0/23 34.124.92.0/22 34.125.0.0/16 35.184.0.0/14 35.188.0.0/15 35.190.0.0/17 34.100.0.0/16 34.102.0.0/15 34.104.0.0/14 34.124.0.0/18 34.124.64.0/20 34.124.80.0/23 34.124.84.0/22 34.124.88.0/23 34.124.92.0/22 34.125.0.0/16 35.184.0.0/14 35.188.0.0/15 35.190.0.0/17 35.190.128.0/18 35.190.192.0/19 35.190.224.0/20 35.190.240.0/22 35.192.0.0/14 35.196.0.0/15 35.198.0.0/16 35.199.0.0/17 35.199.128.0/18 35.200.0.0/13 35.208.0.0/13 35.216.0.0/15 35.219.192.0/24 35.220.0.0/14 35.224.0.0/13 35.232.0.0/15 35.234.0.0/16 35.235.0.0/17 35.235.192.0/20 35.235.216.0/21 35.235.224.0/20 35.236.0.0/14 35.240.0.0/13 104.154.0.0/15 104.196.0.0/14 107.167.160.0/19 107.178.192.0/18 108.59.80.0/20 108.170.192.0/20 108.170.208.0/21 108.170.216.0/22 108.170.220.0/23 108.170.222.0/24 130.211.4.0/22 130.211.8.0/21 130.211.16.0/20 130.211.32.0/19 130.211.64.0/18 130.211.128.0/17 146.148.2.0/23 146.148.4.0/22 146.148.8.0/21 146.148.16.0/20 146.148.32.0/19 146.148.64.0/18 162.216.148.0/22 162.222.176.0/21 173.255.112.0/20 192.158.28.0/22 199.192.112.0/22 199.223.232.0/22 199.223.236.0/23 208.68.108.0/23