Using cloudflare workers as a timer to defer actions

In this article I talk about my experience with setting up deploy hooks for front end websites on Cloudflare pages and rate-limiting build triggers.

Table of contents

Backstory

In the early stages of moving to a headless setup for Ghost CMS, we at IFF decided to host our static jam stack based website on Cloudflare pages. The challenge was to set up a meaningful build pipeline that can be triggered automatically from Ghost, without abusing or overwhelming the deployments’ quota at Cloudflare pages.

Initial Architecture

Cloudflare pages offers a git based integration along with direct uploads. Site builds are automatically triggered for every push.

This meant that all we had to do was to connect the frontend repo (Gatsby) to Cloudflare pages and changes are propagated after the build succeeds.

However, not all changes are done via git. Ghost has been a great way for us to manage all the content related to Posts and Blog related updates.

Ghost CMS has webhook suppoprt that helps us to use events like site.changed which is triggered whenever any content changes in your site data or settings. In order to trigger a new build, Cloudflare conveniently supports deploy hooks. Setting up the webhook endpoint can be done through the ghost dashboard itself. The endpoint has to be set to the deploy hook (URL) retrieved from the Cloudflare pages dashboard.

The Challenge

Initially, the setup worked perfectly. The build took on an average of about 5-6 minutes (this was before build cache was supported by Cloudflare Pages, it is much faster now). However, as we have multiple staff working on Ghost simultaneously, builds were being triggered very frequently. The free tier allows for 500 builds per month on Cloudflare Pages.

If a build is already in queue, the next trigger would have to wait in queue before being taken up for build. In other words, build are not concurrent (well, 1 build at a time for free tier, 5 for Pro). Does this mean that we should just get the Pro tier? Ideally Yes, but for our use case, we didn’t have a need for it. Let me explain why.

Truth be told, most of the time, the edits being made by the staff would be highly frequent before actually getting published. Sometimes minor edits were being made right before or after posts were published or some site settings were being changed manually. Concurrent builds is something we don’t really need. Thus, ideally multiple builds could be bundled and summited as a single build request, without any business impact. Right out of the box, there is no inbuilt solution to control the number of events being produced (Ghost) or consumed (CF Pages) in a given interval - This was the core problem that I tried solving.

The Solution

Since we were already in the Cloudflare ecosystem, I wanted to explore products that I can leverage to solve this problem. We could afford a delay of about 5 minutes (max) - from the time the webhook was triggered, and before the content is visible on the website. For instance, even if 100 requests were triggered in a span of 5 minutes, only one request should be sent and the rest should be ignored (gracefully). This is different from rate-limiting because from the client’s perspective there are no limits on the amount of events(webhooks) being triggered to the deploy hook.

The solution was to set a build some sort of timer and bundler that kept track of the build request. Cloudflare’s workers was a perfect platform for such a minor solution. Additionally, Cloudflare KV is a low-latency, key-value data storage that can be used from within workers. KV was used to store timestamp of the last build being triggered.

Here’s a snippet of part of the final solution that was deployed:

async function triggerBuild(env: Env, timestamp: string) {
  console.log("Queue triggered");
  await new Promise(r => setTimeout(r, COOLING_PERIOD));
  const res = await fetch(env.CF_HOOK, {
    method: "POST",
  });
  await env.ghost_build.put("timestamp", timestamp, {
    metadata: <metadata>{
      last_build_triggered_at: Date.now().toString(),
      hook_status: res.statusText,
    },
  });
  console.log("Queue processed");
}

It is important to be mindful while using setTimeout(), as the maximum duration of a worker is 30 seconds (at the time the solution was being built). To overcome this limitation, one could spin up additional workers in succession until the desired time has been reached. However, 30 seconds was enough for us to nullify undesired frequent builds.

Additionally, to ensure authenticity of the webhook, the webhook signature is validated before sending a request to trigger build. Here’s how to do it in workers:

async function checkSignature(
  secret: string,
  signature: string,
  req: Request
): Promise<Boolean> {
  const payload = await req.json();
  const [externalHmac, timestamp] = signature.split(",");
  const hmac = createHmac("sha256", secret)
    .update(JSON.stringify(payload))
    .digest("hex");
  console.log("Computed HMAC", hmac);
  console.log("External HMAC", externalHmac);
  return `sha256=${hmac}` === externalHmac;
}

Conclusion

In the end, the objective to bundle and defer build triggers was achieved using Cloudflare workers and KV. The solution is fully serverless and can be deployed using the free tier. The biggest takeaway for me was to not disrupt the happy path of end users (the staff in my case). People are generally happy and productive when they’re allowed to continue using the tooling of their choice without having to settle for obscure workarounds, alternatives and compromises.

The complete source code can be found here. Please don’t judge my code, I’m no front end developer, neither do I speak typescript. Lately, Cloudflare has announced Queues which allows Batching, Retries and Delays. I believe queues can be a better alternative, although I haven’t tried. If you have a better way to solve this problem, I’d love to hear about it. Anyway, I hope this gets baked as an inbuilt feature on Cloudflare Pages someday soon.