I post to Bluesky from PHP scripts that cron runs for me, and I pull data from a handful of other APIs the same way. After enough of these I’ve settled on one small cURL function that I paste into everything. This post is that function, plus the three things that made it worth having: timeouts, retries and a cache. I tested all of it on PHP 8.2 before writing this up.
cURL or Guzzle?
I use plain cURL. Guzzle is a lovely library, but it needs Composer, and on shared hosting I always found Composer an enormous faff. cURL is built into PHP, so there’s nothing to install and nothing to update. If your host has PHP, you almost certainly have it already (check with php -m | grep curl).
A GET request that doesn’t hang or lie to you
Most examples you’ll find call curl_exec() and carry on, which goes wrong in two ways. With no timeout, a slow API can hold your script up for minutes. And an error page from the API still counts as a successful request as far as cURL is concerned, so you decode the error as if it were your data. This is the function I use:
function api_get(string $url, array $headers = [], int $tries = 3): array
{
for ($try = 1; $try <= $tries; $try++) {
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 10,
CURLOPT_CONNECTTIMEOUT => 5,
CURLOPT_HTTPHEADER => array_merge(['Accept: application/json'], $headers),
CURLOPT_USERAGENT => 'my-script/1.0',
]);
$body = curl_exec($ch);
$status = (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
$error = curl_error($ch);
curl_close($ch);
$retryable = ($body === false || $status === 429 || $status >= 500);
if ($retryable && $try < $tries) {
sleep($try * 2);
continue;
}
if ($body === false) {
return ['ok' => false, 'status' => 0, 'error' => $error];
}
$data = json_decode($body, true);
return [
'ok' => $status >= 200 && $status < 300 && $data !== null,
'status' => $status,
'data' => $data,
];
}
}It does four things:
- Gives up after 10 seconds of waiting (and 5 seconds just to connect), so cron never piles up
- Reads the HTTP status code rather than assuming all is well
- Retries up to 3 times if the API says 429 (too many requests), returns a 5xx error, or doesn’t answer at all
- Always returns an array with
ok,statusanddata, so the calling code never has to guess
Using it is one line:
$r = api_get('https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=bsky.app');
if ($r['ok']) {
echo $r['data']['handle']; // bsky.app
} else {
error_log('API failed, status ' . $r['status']);
}That URL is a real one and needs no key, so you can paste it in and try it straight away. I ran it just now and got a 200 and the handle back.
Does the retry actually work?
I didn’t want to take my own word for it, so I wrote a tiny test page that answers 429 twice and then succeeds, and ran it with PHP’s built-in server (php -S 127.0.0.1:8099). With the default 3 tries, api_get() came back with the data after 6.0 seconds. That’s the 2 second wait after the first failure plus the 4 second wait after the second. With only 2 tries it gave up and returned status 429, as it should.
It also copes with a server that isn’t there at all. Pointing it at a closed port returned ok as false, status 0 and cURL’s own error message (“Couldn’t connect to server”) instead of a crash.
One more thing I noticed while testing. When I asked Bluesky for a profile that doesn’t exist, it replied with a 400 and a JSON message saying “Profile not found”, not a 404. APIs aren’t always consistent about status codes, so if something odd happens, log $r['data'] as well as the status.
Cache it, so you don’t hit the limit in the first place
The best way to avoid rate limits is to not make the request. If the data only changes once an hour, there’s no point fetching it every minute. This function saves the result in a file and only calls the API again once the file is older than the number of seconds you give it:
function cached(string $key, int $ttl, callable $fetch)
{
$file = __DIR__ . '/cache/' . md5($key) . '.json';
if (is_file($file) && time() - filemtime($file) < $ttl) {
$data = json_decode(file_get_contents($file), true);
if ($data !== null) {
return $data;
}
}
$data = $fetch();
if ($data !== null) {
if (!is_dir(dirname($file))) {
mkdir(dirname($file), 0755, true);
}
file_put_contents($file, json_encode($data), LOCK_EX);
}
return $data;
}And you wrap the API call like this:
$profile = cached('bsky-profile', 3600, function () {
$r = api_get('https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=bsky.app');
return $r['ok'] ? $r['data'] : null;
});I called that three times in a row and the API received exactly 1 request. Notice that a failed request returns null and isn’t saved, so one bad response doesn’t get cached for an hour.
Where to keep your API keys
Don’t put keys in the script, and don’t put them anywhere that is inside your website folder. I keep mine in an .ini file outside the web root, and read them with PHP’s built-in parse_ini_file(). The file looks like this:
[weather]
api_key = abc123And this reads it:
function secret(string $section, string $key): string
{
static $ini = null;
$ini ??= parse_ini_file('/home/me/config/secrets.ini', true) ?: [];
return $ini[$section][$key] ?? '';
}Then secret('weather', 'api_key') gives you the key. Set the file so only your own user can read it (chmod 600 on a Linux server) and keep it out of Git.
Running it automatically
Once the script works from the command line, cron does the rest. This line runs it at the top of every hour and keeps a log, which you’ll be glad of when something breaks at 3am:
0 * * * * /usr/bin/php /home/me/scripts/fetch.php >> /home/me/logs/fetch.log 2>&1Your PHP path may be different, so run which php to find out. On my server it’s /usr/local/bin/php, so the line above would fail there!