Disclaimer: most of the code in this post was written by AI
It’s been a constant irritation of mine that my website (https://nosearch.online) seems to get a fair amount of rogue traffic. What I mean by that is that I constantly get poked for addresses such as:
Bot Fight
It doesn’t really affect the site, and to some extend, it’s been mildly useful with some of the talks and research I’ve been doing. However, enough is enough, and so I had a look in Cloudflare to see if I could find something that might help - and sure enough, I did:
If I’m honest, it probably didn’t do everything I’d hoped - I still got (broadly) similar traffic; however, something interesting happened. My availability alerts started firing.
Availability Crisis
Looking at the graphs, it’s fairly clear that something was rotten in the state of Denmark:
Guess when I made the change.
It took me a while to realise this (since this was via Cloudflare, and my changes were manual, there was more than one change that was done to the site around that time - a very good indication that any infrastructure changes should be done via a build). Also, and this probably should have tipped me off sooner than it did - the site wasn’t down - only the availability tests - and not all of them.
Anyway, fast forward a few days, I came back to it and started investigating in earnest - finally coming the the disturbing conclusion that Cloudflare sees Azure as a bot (which… it is).
The solution seems fairly simple at this point: go into Cloudflare and allow the Azure endpoints.
Allowing Endpoints in Cloudflare
This is the interface in Cloudflare that lets you do that:
You select an IP or a range and type them in one at a time.
This is the link to Azure’s IP Ranges. If you look, the availability checks are fairly hefty.
Cloudflare has an API, so it should be fairly straightforward to leverage that…
Create a Custom CF token
I hate it when blog posts lay things out methodically - i.e. do dependency one, then dependency two, etc… I much prefer “here’s the solution, but now you need dependency one and two”. Anyway, I’m going to do exactly what I’m objecting to, because there’s only three steps to this, and they’re all fairly quick. Sorry!
The first step is that you need an API token - that’s found in your profile, and is fairly easy to create:
You will need to make sure that it has:
Zone / Zone / Read
Account / Account WAF / Edit
Zone / Firewall Services / Edit
When you’ve created it, you can test it (you don’t need to if you’re in a rush). Mine token was:
RQ39J0112MdV5tcauztAlzASb0UEO9OBKbaxB67c
A powershell script such as this will do the job:
$headers = @{
"Authorization" = "Bearer RQ39J0112MdV5tcauztAlzASb0UEO9OBKbaxB67c"
"Content-Type" = "application/json"
}
Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/user/tokens/verify" -Method Get -Headers $headers
This will result in something like the following:
PS C:\Users\pcmic\source\repos\cf-scripts> .\test-api.ps1
result success errors messages
------ ------- ------ --------
@{id=a52f8613e4cc1c2dda01b302f2327921; status=active} True {} {@{code=10000; message=This A...
All good! The next thing you’ll need is a zone - this is (or was at the time of writing) in the Overview tab:
Script
You now have all the dependencies, so you can run the following script (as I stated at the start this was written by AI):
# Cloudflare API details
$apiToken = "RQ39J0112MdV5tcauztAlzASb0UEO9OBKbaxB67c"
$zoneId = "32fda1749432d3f9e37fdb9f3ab1a361"
$apiUrl = "https://api.cloudflare.com/client/v4/zones/$zoneId/firewall/access_rules/rules"
# List of Azure Availability Test IP ranges (CIDR notation)
$ipRanges = @(
"13.86.97.224/28", "20.37.156.64/27", "20.37.192.80/29", "20.38.80.80/28",
"20.40.104.96/28", "20.40.124.176/28", "20.40.129.32/28", "20.40.129.96/28",
"20.42.4.64/27", "20.42.35.32/28", "20.42.129.32/27", "20.43.40.80/28",
"20.43.64.80/29", "20.43.128.96/29", "20.45.5.160/28", "20.189.106.64/29",
"23.100.224.16/28", "40.74.24.80/28", "40.80.186.128/26", "40.91.82.48/28",
"40.119.8.96/27", "51.104.24.80/29", "51.105.9.128/28", "51.137.160.80/29",
"51.144.56.96/28", "52.139.250.96/28", "52.140.232.160/28", "52.158.28.64/28",
"52.229.216.48/28", "191.233.26.176/28", "191.235.224.80/29"
)
# Headers for Cloudflare API request
$headers = @{
"Authorization" = "Bearer $apiToken"
"Content-Type" = "application/json"
}
# Function to convert CIDR range to individual IPs
function Convert-CIDRToIPs {
param (
[string]$cidr
)
$parts = $cidr -split '/'
$ip = $parts[0]
$prefix = [int]$parts[1]
# Convert IP to binary
$ipParts = $ip -split '\.'
$ipBinary = ([System.Net.IPAddress]::Parse($ip)).GetAddressBytes()
[Array]::Reverse($ipBinary)
$ipBinary = [BitConverter]::ToUInt32($ipBinary, 0)
# Calculate the number of IPs in range
$numIPs = [math]::Pow(2, (32 - $prefix))
# Generate list of individual IPs
$ipList = @()
for ($i = 0; $i -lt $numIPs; $i++) {
$newIP = $ipBinary + $i
$ipBytes = [BitConverter]::GetBytes($newIP)
[Array]::Reverse($ipBytes)
$ipList += [System.Net.IPAddress]::new($ipBytes)
}
return $ipList
}
# Function to add a single IP rule to Cloudflare
Function Add-IPRule {
param ($ip)
$body = @{
mode = "whitelist"
configuration = @{
target = "ip"
value = $ip.ToString()
}
notes = "Allow Azure Availability Test IP"
} | ConvertTo-Json -Depth 3
try {
$response = Invoke-RestMethod -Uri $apiUrl -Method Post -Headers $headers -Body $body
if ($response.success -eq $true) {
Write-Host "Successfully added $ip" -ForegroundColor Green
} else {
Write-Host "Failed to add $ip : $($response.errors | ConvertTo-Json -Depth 3)" -ForegroundColor Red
}
} catch {
Write-Host "Failed to add $ip : $_" -ForegroundColor Red
}
}
# Loop through CIDR ranges, expand them to individual IPs, and add to Cloudflare
foreach ($cidr in $ipRanges) {
Write-Host "Expanding CIDR: $cidr..."
$ipAddresses = Convert-CIDRToIPs -cidr $cidr
foreach ($ip in $ipAddresses) {
Add-IPRule -ip $ip
}
}
I’m aware that there are credentials in this script - they are not valid credentials, but I always prefer to see that so I know what the script will actually look like.
You’ll notice that it takes the CIDR range and splits it up, then adds the IPs individually.
And just like that, we’re back in the room: