Home
Articles
Products
Likes

What can we learn from the W3C CMS selection process

janus-boye Profile

By Janus Boye, Boye & Co

There’s a million different ways to select a new content management system and having been a part of the marketplace for the past two decades, I thought I had almost seen them all.

Still, the recent CMS selection as a part of the World Wide Web Consortium (W3C) redesign project was a different one. A project done out in the open, with a controversial tool selection and an emphasis on accessibility.

Given the public profile of the W3C, this naturally led to increased attention among agencies, analysts, vendors and other industry pundits alike.

Craft CMS came out the winner, but let’s take a step-by-step look at what happened and see what others might be able to use in their own selection processes.

Finding a modern and accessible CMS starts by getting some help

A good place to start to get an overview of this process is the W3C website redesign phase 1 RFP. This was issued back in November 2019 and as you see, W3C asked for help with more than selecting a CMS. To quote from the project summary

“W3C is looking to incrementally redesign its Website and revise the information architecture”

As the customer, W3C asked for cost estimates on services like design, SEO, development and consulting on things like migration strategy. Objectives included:

  • Solidify brand authority, consistent design, uniform appearance
  • Create an engaging and easy to navigate experience
  • Increase engagement
  • Create a long-lasting partnership with a web design agency to continue to work with us as our needs and organization evolve

The specific requirements included this bullet with three sub bullets:

  • Provide advice on a modern replacement for the custom CMS used for the current site
    • We need a CMS that is long-lived and easy to maintain, because we run our systems for decades.
    • Full change histories identifying who made each change.
    • The CMS may need to enable collaborative editing

Finally, the RFP also included this sentence on budget:

“The average cost estimate of $200,000 is based on figures given by people who are in the ecosystem. We would like vendors to propose what they recommend even if it exceeds $200,000.”

Several agencies responded during December, including UK-based Studio 24.

Studio 24 which was awarded the contract in early February. Although slightly delayed by a few weeks, given the holidays, that’s not a much longer response time than many other projects.

If you look at the project tweets from W3C, you can see a more detailed timeline for selection

Our W3C Website redesign RFP has now concluded.
We awarded the project to @studio24 who we are very much looking forward to working with.
Thanks to the 31 other bidders for the excellent quality of their proposals.https://t.co/RCtmGY6VBn

— W3C (@w3c) February 7, 2020

You might also want to see the winning proposal from Studio 24 (PDF)

My take:

Nothing out of the ordinary in the opening. A high-profile customer. A reasonable budget to relaunch a notable website. The project pretty much looks like any other website redesign including a new CMS to power the website. Also worth mentioning, that a customer showing an ability to make decisions at the beginning is a good starting point.

CMS strategy to inform the selection

The project kicked off in mid-March and following user group research and design discussions, then moved onto CMS strategy and requirements in May.

This led to the creation of a comprehensive CMS strategy and requirements document. This covers technical requirements, accessibility, content editing, forms, search and much more.
There’s also a section on licensing models. To quote:

“W3C has a strong preference for an open-source license for the CMS platform.”

And later in the same chapter:

“We will consider commercial software that fits the requirements for W3C.”

The document also outlines questions to be answered for each potential system as well as a section on headless CMS and static site generators.

My take:

While there’s no set formula for how a CMS strategy should look like and any rules on required chapters, I found the work done here comprehensive and understandable, also by non CMS experts.

From the perspective of an analyst or customer, I would also consider the importance of the ecosystem around the system and the quality of the documentation. Depending on who you ask, some might also consider a look at company/community vision and roadmap. Are they aligned with what you want to achieve?

Narrowing down the shortlist

As the project moved forward, the time also came to narrow down the list of potential systems.

Given the public nature of the project, you can read more in this document: W3C CMS platform selection - full status update. To quote from the chapter on shortlisting:

“We then looked at a series of CMSes from a high level. We looked at their marketing material to understand their philosophy and their main features; got a sense of the level of activity of their development team and community; scanned their user and developer documentation.

Based on this first review, we for example excluded all static site builders (e.g. Gatsby, Jekyll). Jekyll and Gatsby can have performance issues when building large sites, which we would rather avoid. Others such as Hugo are built on technology we are not familiar with in the studio.”

This led to a shortlist of three candidates, all based on the PHP technology stack familiar to the W3C:

Later in the document, there’s strengths and weaknesses with each of the three finalists. To me they looked balanced, but I in particular noticed this sentence on WordPress:

“Because we have a lot of experience with WordPress, we are also very much aware of its limitations in the context of a project of this size and longevity.”

My take:

This is a phase of the project, which can be either extremely difficult without outside help or left to random circumstances. Working with an agency with real implementation experiences like Studio 24 provides valuable insights to finding the right system.

It’s refreshing to read how marketing material provides a starting point for understanding the systems. In other processes, industry analysts like Forrester, Gartner or Real Story Group are also pulled in to provide insights to making the technology decision. Either via their reports or through their analyst services.

According to Simon Jones, the Gartner Magic Quadrant was reviewed as a part of the process, although this wasn’t noted in the project documentation.

As a part of our conversation for this article, Simon also mentioned that the PHP technology stack was a big decider on limiting CMS choice. Using the tech stack to narrow down the list is quite common.

Not selecting WordPress

Judging from the outside, it looks like the most controversial part of the process was removing WordPress from the final three.

In the CMS selection review process, Studio 24 wrote this about the process they used to test the finalists:

“This is how we tested Craft and Statamic (we didn’t submit WordPress to that test as we already have extensive experience working with it).”

And later in the document this quote:

“We have already rejected the use of Gutenberg in the context of this project due to accessibility issues.”

A few weeks later in the CMS selection project weeknotes from 11 Sept, there’s this sentence:

“We have shortlisted two CMS platforms at present.”

In other words: WordPress was out.

This created quite some debate in the community. Let’s look at the timeline:

Simon Jones from Studio 24 also responded with a separate update on not choosing WordPress. This post includes chapters on the challenges of Gutenberg. Specifically, the issue with Gutenberg for agencies that create bespoke websites is the development process. To quote Marie Manandise from Studio 24:

“It is very difficult to create custom blocks and make it profitable. And that is due to the necessity of using React for making any changes, including simple ("front-end") changes to the markup rendered by Gutenberg.”

In the post, Simon also addressed the topic of licensing and the different types of open source.

This tweet also sums up the issues around front-end complexity or “Reactification of frontend development.”

"The use of React complicates frontend build. We have very talented frontend developers, however, they are not React experts - nor should they need to be. I believe frontend should be built as standards-compliant HTML/CSS with JavaScript used to enrich functionality" @simonrjones

— Ben Furfie (@frontendben) September 27, 2020

My take:

It’s far from unusual for vendors or even champions of open source projects to respond negatively to not making the final cut. As a customer it’s good to be prepared for this and the tactics tend to come in different shapes and sizes.

In the case of a very public project done out in the open, it’s no surprise that the discussion made it to Twitter, Hacker News with 100s of comments, and other websites. Simon from Studio 24 spent quite a bit of time responding to different questions and comments.

To quote Simon Jones:

“it's not easy working in the open but it's a really good thing to do.”

Finally, it is worth noting that while the WordPress community came out loud, the Drupal community responded much more understandingly and with positive feedback, even though they were also not on the final shortlist.

The importance of accessibility

As is now clear, accessibility became a major theme in this selection process. Still, accessibility was not more of a decisive a factor as the development process for example.

According to Studio 24:

“The reason we had to spend so much time looking into accessibility is because information about the level of accessibility of software is very hard to come by. CMS developers themselves are often not in a good position to give that information. It requires a lot of time and niche expertise to make that assessment. And also because none of the CMSes out there fit the bill, we had to be creative in coming up with a solution to meet the accessibility requirement.”

There’s now also an ATAG Report Tool, which was released in June. The tool helps evaluators report on the accessibility of authoring tools and guides you through the Authoring Tool Accessibility Guidelines (ATAG) requirements.

The Web Accessibility Initiative (a working group in W3C) are trying to use this tool to help create a list of accessible authoring tools (e.g. CMSs). This list doesn't exist yet since they don't have enough data.

In a recent talk on advancing authoring tool accessibility by Hidde de Vries he looks at how better “authoring tools” can improve accessibility (for content editors and for web users), and share some of the new resources that the W3C’s Web Accessibility Initiative (WAI) is working on.

My take:

These days, it’s far from unusual for accessibility to be a topic when it comes to website design, also due to government regulations. Still, it’s quite rare that editor accessibility is such a decisive factor in selecting which platform to choose.

When we covered this case study in our recent CMS Expert group meeting, we talked about accessibility not being high enough on the list for most customers.

Kudos to W3C as a standards organization for raising the topic!

Final thoughts: Improving the CMS marketplace

In our conversation with the CMS Experts, several vendors said that their clients are asking more and more for an accessible CMS. This increased focus on accessibility is now visible on roadmaps and will hopefully lead to tangible improvements soon.

On the agency side, one of the Boye members had to switch the CMS at the last minute in order to use an accessible tool.

As a community, we are big fans of this type open of sharing and reflection. While vendors move quickly and clearly keep innovating, more openness about the process can help us all push the marketplace forward.

W3C
CMS