農家SEの覚書き

忘れっぽいので、自分用の備忘録です。

GAS(Google Apps Script)からのクロール拒否

ユーザーエージェントがMozilla/5.0 (compatible; Google-Apps-Script)からのアクセス数が急増。 原因を調べたら欠品中の在庫を定期的に調べているみたい。

.htaccess に以下を追加しアクセス禁止(403-Forbidden)に設定

RewriteEngine On

# User Agent でアクセス拒否
SetEnvIfNoCase User-Agent "Google-Apps-Script" ua_key=on
RewriteCond %{ENV:ua_key} on
RewriteRule ^(.*)$ - [F,L]

GASってクローラー代わりに使えるようですね。 しかもアクセス元が複数ありました。

Google Apps ScriptのIP調査(拒否設定の為)

参考にさせて頂いたページ

GoogleAppsScript(GAS)のIPアドレス範囲を調べる

# nslookup -q=TXT _cloud-netblocks.googleusercontent.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
_cloud-netblocks.googleusercontent.com  text = "v=spf1 include:_cloud-netblocks1.googleusercontent.com include:_cloud-netblocks2.googleusercontent.com include:_cloud-netblocks3.googleusercontent.com include:_cloud-netblocks4.googleusercontent.com include:_cloud-netblocks5.googleusercontent.com ?all"

Authoritative answers can be found from:

include:cloud-netblocks1.googleusercontent.com include:cloud-netblocks2.googleusercontent.com include:cloud-netblocks3.googleusercontent.com include:cloud-netblocks4.googleusercontent.com include:_cloud-netblocks5.googleusercontent.com

# nslookup -q=TXT _cloud-netblocks1.googleusercontent.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
_cloud-netblocks1.googleusercontent.com text = "v=spf1 include:_cloud-netblocks6.googleusercontent.com include:_cloud-netblocks7.googleusercontent.com ip6:2600:1900::/35 ip4:8.34.208.0/20 ip4:8.35.192.0/21 ip4:8.35.200.0/23 ip4:23.236.48.0/20 ip4:23.251.128.0/19 ip4:34.64.0.0/11 ip4:34.96.0.0/14 ?all"

Authoritative answers can be found from:

1~5(6~7)について繰り返し、アドレスの範囲がわかってのWAFの拒否設定に追加

34.100.0.0/16
34.102.0.0/15
34.104.0.0/14
34.124.0.0/18
34.124.64.0/20
34.124.80.0/23
34.124.84.0/22
34.124.88.0/23
34.124.92.0/22
34.125.0.0/16
35.184.0.0/14
35.188.0.0/15
35.190.0.0/17
34.100.0.0/16
34.102.0.0/15
34.104.0.0/14
34.124.0.0/18
34.124.64.0/20
34.124.80.0/23
34.124.84.0/22
34.124.88.0/23
34.124.92.0/22
34.125.0.0/16
35.184.0.0/14
35.188.0.0/15
35.190.0.0/17
35.190.128.0/18
35.190.192.0/19
35.190.224.0/20
35.190.240.0/22
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/13
35.216.0.0/15
35.219.192.0/24
35.220.0.0/14
35.224.0.0/13
35.232.0.0/15
35.234.0.0/16
35.235.0.0/17
35.235.192.0/20
35.235.216.0/21
35.235.224.0/20
35.236.0.0/14
35.240.0.0/13
104.154.0.0/15
104.196.0.0/14
107.167.160.0/19
107.178.192.0/18
108.59.80.0/20
108.170.192.0/20
108.170.208.0/21
108.170.216.0/22
108.170.220.0/23
108.170.222.0/24
130.211.4.0/22
130.211.8.0/21
130.211.16.0/20
130.211.32.0/19
130.211.64.0/18
130.211.128.0/17
146.148.2.0/23
146.148.4.0/22
146.148.8.0/21
146.148.16.0/20
146.148.32.0/19
146.148.64.0/18
162.216.148.0/22
162.222.176.0/21
173.255.112.0/20
192.158.28.0/22
199.192.112.0/22
199.223.232.0/22
199.223.236.0/23
208.68.108.0/23