The Charlotte Mercury was about twice as slow as our other publications, on every page, and we did not know why until July 24, when we found the reason in an hour of measuring and fixed it in an afternoon. The server's share of a homepage load went from about 542 milliseconds to 208. The sports section went from about a second to 188 milliseconds.
I am including this in a series about ownership because it is the clearest example I have of what owning the stack lets you do that renting it does not: look inside.
What was wrong
Every page on every publication is assembled from the same layout, the frame that holds the header, the navigation and the footer around whatever the page is. Two of our publications, the Charlotte paper and the racing paper, ask that frame for one extra thing: the navigation for the racing series they cover. That request went to the database three times in a row, uncached, on every page load. Not on the racing pages. On every page, because it lived in the frame.
The Farmington paper never made that request, which is why it was fast, and why the difference looked like "Charlotte is heavy" when it was actually "Charlotte's frame does three uncached queries before the page even starts."
What we got wrong first
The first attempt cached the homepage's own queries and moved nothing at all: 614 milliseconds before, 633 after. It was aimed at the page, and the cost was never on the page. That failed attempt is the lesson. The instinct is to optimize the thing you are looking at. The discipline is to measure first, and to measure in pieces: how much of the time is the network, how much is the server, and which part of the server. Eight to twelve samples, read the floor and the median, and always measure a second publication as a control so you cannot mistake a change in the weather for a win.
Once the numbers pointed at the frame, the fix was ordinary: cache the read with a tag that the database already clears when something publishes, and stop making the frame's five separate trips to the database one after another when only one of them depends on another.
The part that matters
The cache, the tag, the waterfall, none of that is interesting. What is interesting is that we could see it. On a rented content system the frame is the vendor's, the caching is the vendor's, and "your site is slow" is a support ticket. On ours, the whole path from a reader's request to the rendered page is code we can read and a database we can query, and a half-second that had been costing every reader on the flagship was found by looking.
There is a second lesson, and it is the one I keep learning. A change was verified by rendering the page and comparing it, element by element, against the live site, not by the build passing. A green build proves the code compiles. It proves nothing about what the reader sees. Since then, that is the standard for any change that touches a page: render it, read it, compare it.
Half a second is not a small thing on a news site. On a phone on a poor connection it is the difference between the page and the back button. We found it because it was ours to find.
This is No. 09 in We Built Our Own. Previous: The people, places and bodies behind the news. Next: The site checks itself every night.