⚡ Performance - Decrease the production deploy time from 42 minutes to 20 minutes

Start with the listed workflow files, .dockerignore, scripts/crawl-sitemap.js, and the rule page data functions in app/[filename]/page.tsx; the issue groups work into three phases and links existing pull requests for several tasks. Read the phase plan and run logs first, then take one unstarted task and measure it as described: use /deploy when no build is in progress, or dispatch the workflow from a branch to staging for workflow changes. Done means the task's listed behavior is in place and its timing or page behavior is recorded.

Pain

A production deploy takes 42 minutes. The site is live after 30 minutes. When TinaCloud is slow, or when a second build runs at the same time, the deploy takes much longer. On 12 Nov the site was live after 50 minutes. A failed deploy costs the same time again, so one release can take hours.

On such a day the build also puts broken pages on the site. The build that went live on 12 Nov at 05:41 UTC had 20 rules that showed NOT FOUND and 11 rules with a false "Broken Related Rule Reference" banner. The next build went live at 06:42 UTC with 22 NOT FOUND pages (21 rules and 1 category) and 6 rules with a false banner.

Evidence

Production run 42969510432 on 11 Nov took 42.1 minutes. The runs on 9 and 10 Nov agree within one minute.

Part	Minutes
Deploy infrastructure (the build waits for it)	2.4
Build job: steps before Docker (1.3 is disk cleanup)	3.5
Docker: static generation of 4,209 pages	11.1
Docker: list of all rules for generateStaticParams	1.5
Docker: all other steps (Tina build, compile, install, push)	5.0
Build job: steps after Docker (0.7 is disk cleanup)	1.0
Deploy and slot swap	4.9
Sitemap crawl (the site is live before it starts)	12.3

Static generation time for the same number of pages (approximately 4,200):

Run	Builds at the same time	Static generation	Docker step	Throttling errors	Fallback pages
9 to 11 Nov, for example 42969510432	1	11.1 to 11.5 min	17.4 to 18.8 min	0 to 6	0 to 1
12 Nov 05:02 UTC, 43158664723	1	19.5 min	26.3 min	62	20
12 Nov 06:01 UTC, 43165212271	1	19.7 min	26.6 min	56	22
12 Nov 04:09 UTC, 43152870018 (main)	2	32.7 min	38.5 min	388	131
12 Nov 04:10 UTC, 43153068798 (release/v1.8.18)	2	32.4 min	38.8 min	400	134

The compile step was not slower in the slow runs. No commit on main between 11 Nov and 12 Nov added a request to the page render path. The difference is wait time for TinaCloud.

A throttling error is this text in the build log: Error retrieving data from DynamoDB: Throughput exceeds the current capacity of your table or index. A fallback page is a rule page that the build rendered as ClientFallbackPage after a Tina error. The build completes with no error.

On 12 Nov at 05:56 UTC, all 20 fallback pages of the live build showed NOT FOUND on production, with cache-control: s-maxage=31536000. In the same build, 11 other rules have a "Broken Related Rule Reference" banner, but those rules have no related rules. Their ruleData request was throttled, and the page code read that as a broken reference.

A fallback page has HTTP status 200 and cache-control: s-maxage=31536000. The sitemap crawl thus counts it as a good page, and the page stays until the next deploy. On 12 Nov at 21:45 UTC, the 22 fallback pages of run 43165212271 were all on production.

Causes

1. The build reads each page from TinaCloud.

• tinacms/tinacms#6636 added the local content build to stop these calls. Its usage is tinacms build --content=local -c "next build".
• #2572 tested it on this repo in April. That test had the -c command and set LOCAL_CONTENT_RELATIVE_PATH=../content in the Dockerfile. It was closed and not merged.
• #2631 then added only the flag. package.json runs tinacms build --content=local && next build, and the Dockerfile does not set the content path.
• The flag has no effect in this form. The Tina CLI keeps its local server and sets TINA_LOCAL_URL only for a -c command. With &&, the CLI stops first, and then next build reads from TinaCloud.
• Without the content path, Tina looks for the rules in /app/public/uploads/rules, which does not exist. The log line "Indexing local files" completes in 0.1 seconds.
• Each rule page sends 2 requests. getRuleData calls ruleDataBasic and then ruleData. The page and generateMetadata each call getRuleData, and the disk cache of the Tina client answers the second call.
• #2773 reported that TinaCloud throttles reads during each build. Builds that run at the same time make each other slower.
• The page code catches each Tina error. After a throttled request, getRuleData returns null and the page becomes a fallback page, or it returns the rule with no related rules and a warning banner.

2. The sitemap crawl runs at a quarter of its design rate. scripts/crawl-sitemap.js sends 5 requests and then waits 250 ms. That is approximately 5 requests each second, and its limit is 20. The log shows an estimate of 4 minutes, but the crawl of 3,707 pages takes 12. The crawl is a report only and cannot fail the deploy.

3. The disk cleanup is not necessary on the current runners. The runner had 80 GB free of 145 GB before the cleanup in 3 of 3 runs. The pre-build cleanup takes 1.3 to 2.6 minutes. The post-build cleanup takes 0.7 to 1.0 minutes, and GitHub discards the runner after the job.

4. The build waits for the infrastructure job. The build needs only the registry name from that job. The name is always acrsswrulesprod or acrsswrulesstaging. The infrastructure job takes 2.1 to 3.0 minutes.

5. Docker receives 3.3 GB that the build does not read. The build context includes the content checkout with its .git folder and all images.

6. Rule pages render again at run time, and each render reads TinaCloud.

• Since release v1.8.18 (live on 12 Nov), the build gives each rule page a revalidate period of 6 hours. The route table of run 43165212271 shows /[filename] 6h 1y. The 11 Nov build of v1.8.16 shows no period. The probable source is the 6 hour cache of the Giscus discussion numbers (#2776).
• After the first render at run time, the period of that page is 60 seconds (cache-control: s-maxage=60). The root layout query has revalidate: 60. Why the build does not count that value is not yet confirmed.
• Cloudflare does not cache the HTML (cf-cache-status: DYNAMIC). On 12 Nov at 21:28 UTC, a request for a rule page returned x-nextjs-cache: STALE, and the same request 12 seconds later returned HIT. This sequence repeated each 60 seconds. A HEAD request has the same effect as a GET request.
• So from 6 hours after a deploy, a request for a rule page can start a render that reads TinaCloud. With the current page code, a failed render can replace a good page with a NOT FOUND page (#2794).
• The crawl that a deploy starts requests pages that are minutes old, so it starts no renders. A crawl that a person starts by hand more than 6 hours after a deploy can start up to 3,700 renders.
• The load from visitors is possibly small. It was not measured.

Status (13 Nov)

Pull request	Task
#2802	Phase 2, tasks 1 to 3: retry throttled requests, fallback only for a missing record, one request for each rule page
#2803	Phase 2: rule list from a file, and removal of the category request that fails in each build
#2804	Phase 3, task 1: the same image URLs with local Tina data as with TinaCloud data
#2805	Phase 2: no revalidate period on rule pages (cause 6). Merge its second commit after #2802.
#2806	Phase 1: disk cleanup, parallel build, smaller Docker context, faster crawl after a deploy

• #2806 changes workflow files, so /deploy cannot test it. A staging deploy from its branch is the test.
• Not started: the build script change of phase 3, and the TinaCMS follow-ups.

Plan

Each task group is one pull request. Measure each pull request with /deploy when no other build is in progress. Record the times from the run log in the pull request.

A /deploy run uses the workflow files of main, because pr-deploy.yml starts on issue_comment. It thus cannot measure the workflow changes of phase 1. Measure phase 1 with one workflow_dispatch run of build-and-deploy.yml from the branch to staging.

The first task of phase 2 has the highest priority, because it stops broken pages. Phase 1 changes only workflow files, so it can be done at the same time.

Phase 1 - Workflow only (no application code)

• Crawl: for the crawl that a deploy starts, set SITEMAP_BATCH_SIZE=20 and SITEMAP_DELAY_MS=0. Keep the current rate for a crawl that a person starts by hand (see cause 6). Remove the pnpm install steps in crawl-sitemap.yml, because the script has no dependencies. Expected: 12.3 to approximately 3.5 minutes.
• Disk cleanup: remove the post-build cleanup. Run the pre-build cleanup only when free disk space is less than 30 GB. Expected: 2.0 to 3.6 minutes less.
• Infrastructure: remove needs: infrastructure from the build job in build-and-deploy.yml and pr-deploy.yml, and calculate acr_name from the environment. The deploy job continues to wait for the two jobs. Expected: 2.1 to 3.0 minutes less.
• Docker context: add content to .dockerignore. Expected: 0.5 minutes less.

Phase 2 - Fewer TinaCloud requests (page code)

The first three tasks are in pull request #2802.

• In getRuleData and getCategoryData, handle the two error types differently. For "Unable to find record", keep the current fallback. For all other errors (throttling, 5xx, network), try again after a delay and then throw. The build then fails and does not ship a broken page. This task is also the build part of the fix for #2794. Do it one time only.
  • In the CI build, the Tina client has a disk cache (tinacms build adds cacheDir, tinacms dev does not). A request with errorPolicy: "all" stores its result in that cache also when the result has errors. A second try of the same request then reads the stored result. getCategoryData uses errorPolicy: "all", so its second try must not use the stored result.
  • During the build, Next.js also stores each fetch response that has status 200, and gives it to later requests with the same body. TinaCloud returns a throttling error with status 200. A second try must thus not read that stored response. One way is cache: "no-store", which does not make a force-static page dynamic.
• In getRuleData, call ruleData first. Call ruleDataBasic only after an "Unable to find record" error.
• Put getRuleData and getCategoryData in React cache(), so that the page and generateMetadata use one result.
• Write the rule folder names to a JSON file before the Docker build. The content checkout is on disk at that time. Read the file in generateStaticParams. Keep the TinaCloud query as the fallback for local development.
• Stop the renders of rule pages that did not change (cause 6): the 6 hour period and the 60 second period. Keep the revalidation on demand for content changes. Expected: fewer TinaCloud requests from the live site, and a crawl that a person starts by hand is safe at each rate.

Expected: 1 request for each rule page, not 2. A local build in #2802 measured 7,541 and then 4,076 rule requests. Static generation in CI is then approximately 6 minutes (estimate). The rule list takes seconds, not 1.5 minutes. This phase also decreases the TinaCloud requests at run time, and it decreases the load on the local Tina server in phase 3.

Phase 3 - Build from local content (the goal of #2631)

• Make image paths absolute in all components that show a Tina image field. Examples are ImageEmbed, the markdown img component and the rule thumbnail. rewriteUploadsToAssets already does this for file links. Release this change first. It does not change the result with TinaCloud data.
• Apply the two changes from #2572: run next build as the -c command of tinacms build --content=local, and set LOCAL_CONTENT_RELATIVE_PATH=../content in the builder stage of the Dockerfile only.
• Send only the MDX and JSON files of the content checkout to Docker. They are approximately 24 MB.
• Compare a pull request slot with production: the number of prerendered pages, the image URLs on 10 rule pages, and the sitemap crawl result.

[!CAUTION]
Do not change the build script before the components accept relative image paths. The local Tina server returns image paths as /uploads/rules/... (useRelativeMedia: true). TinaCloud returns https://assets.tina.io/..., and the components expect that form. On production, https://www.ssw.com.au/rules/uploads/rules/3-steps-to-a-pbi/Testing-task.png gives a 404. If the order is wrong, the images on rule pages break.

Expected: in the April test (#2572), the build job in CI took 19.4 and 20.9 minutes. The usual build job took 24 to 27 minutes in the same days. With phase 2, the estimate for static generation is approximately 3 minutes. The build time does not depend on the TinaCloud speed or on other builds. The content freshness check also becomes exact, because the build uses the checkout that the workflow records.

Expected result

	Site live (minutes)	Run complete (minutes)
Now	30	42
After phase 1	25	28
After phase 2 (estimate)	19	22
After phase 3 (estimate)	16	19

The numbers for "Now", phase 1 and phase 2 are for a day when TinaCloud has its usual speed.

Acceptance Criteria

1. A production deploy makes the site live in 18 minutes or less (now 30), and the run completes in 22 minutes or less (now 42).
2. next build reads rule and category data from the local Tina server. Static generation takes 5 minutes or less.
3. A build is not slower when a second build runs at the same time.
4. The site does not change. The same pages are prerendered, the images on rule pages load, and the sitemap crawl reports status 200 for all pages.
5. A TinaCloud error during the build cannot put a broken page on the site. The build log has no [ClientFallbackPage] line for a rule that exists, and no rule gets a false broken reference banner.
6. The disk cleanup continues to run when free disk space is low.
7. Each pull request records the times before and after the change.

TinaCMS follow-ups

These are changes in tinacms/tinacms. This PBI does not need them, but they prevent the same problem in other projects.

• tinacms build --content=local without a -c command has no effect and gives no message. The CLI can stop with an error, or show a warning. SSW.Website has the same script form.
• In a --content=local build, the local server returns relative image paths. TinaCloud returns assets.tina.io URLs. A site with a separate content repository gets different output from the two sources. If the CLI returns the TinaCloud form in this mode, the first task of phase 3 is not necessary.

Not in this PBI

• The slot swap takes 3.3 to 4.9 minutes. No quick fix is known.
• The image is large (.next/standalone is 1.7 GB).
• The build prerenders all rules, which includes 479 archived rules.

Related

• tinacms/tinacms#6636 added the flag. tinacms/tinacms#6779 documented it. tinacms/tinacms#6780 plans its adoption.
• #2572 tested the local content build on this repo. #2631 added the flag to the build script.
• #2773 and #2774 removed hasReferences from the rule queries to decrease the TinaCloud load.
• #2794 is the bug for rule pages that show NOT FOUND. The first task of phase 2 is the build part of its fix.
• #1380 is the older issue about build speed for the Gatsby site.

How this was measured

The times are from the job and step data of the workflow runs and from the Docker build logs. The Tina CLI behavior is from the tinacms/tinacms source and the published @tinacms/cli 4.0.4 package. The numbers marked as estimates were not measured.

One more possible cause was checked and is not a cause. The Tina resolver parses the MDX body of each document that a query loads. For the 3,861 rules, that parse takes 0.9 ms for each rule on a laptop.

---

🤖 Issue created by Claude Fable 5.2 via Claude Code