Building Symfony Docs

The documentation of the Symfony project is Open Source and managed using the
github.com/symfony/symfony-docs repository. Thousands of people from all
over the world have contributed to these docs. It’s one of the most popular
doc projects of the entire Open Source space.

Symfony Docs use reStructuredText (or RST for short) format to write its
contents. Then, we use Sphinx, a documentation building tool, to transform
those RST files into the HTML contents served on our website.

This setup has served us well for some years, but it had some serious problems
for us. First, Sphinx and most of the tools related to RST are written in Python
programming language, which we don’t know well. Second, updating and maintaining
Sphinx caused many troubles for us, because it’s too strict and unforgiving.

That’s why around 2019 we started to seriously look for an alternative based on PHP.

Sticking to RST format

An obvious solution would have been to switch Symfony Docs to Markdown, the most
popular doc format. This was quickly discarded because of the colossal amount of
contents of Symfony Docs (converting all of them to Markdown would take us forever).

Before continuing, let’s play a game: can you guess the size (in lines, words
and characters) of the entire Symfony Docs? I’ll show you the answer at the end
of this blog post.

Apart from the contents size, there was another problem. Markdown is great for
simple contents, but it’s not that great for complex technical docs. RST provides
all the features that you’ll need when writing complex docs, such as cross references
to internal doc sections and programmable „directives“ to create custom content blocks.
In Markdown it’s impossible to do that or you need to use hacks which aren’t part
of the standard.

Building a RST parser in PHP

Building a doc parser is not a simple feat, much less for a full-featured format
like RST. Luckily, Symfony was not the only PHP-related project that needed a
RST parser. Doctrine project uses RST for its docs too and also wanted to
get rid of Sphinx to build their docs.

That’s why some Doctrine people created (and open sourced) the doctrine/rst-parser
project based on a previous but incomplete RST parser. It’s not a fully-compliant
RST parser, but it’s super fast, easy to use and covers all our RST needs.

The RST parser problem was now solved. Next, we needed to create a full
„doc builder“ to replace Sphinx.

Creating a Doc Builder in PHP

Thanks to the flexibility of Doctrine’s RST parser, we could build an
application on top of it called symfony-tools/doc-builder. In short, this
tool finds the RST files, parses them into HTML (including code highlighting),
and saves them in JSON files with a certain structure expected by symfony.com.

This doc builder also defines and processes all the custom RST directives used
in Symfony Docs. For example, when you browse the Symfony routing article,
you can select the format of certain code blocks (annotations, attributes, YAML,
XML, etc.) This is not part of the RST standard but, contrary to Markdown, in
RST there’s a standard way of extending the format.

That’s why Symfony Docs use a proprietary .. configuration-block: directive
which is processed in the ConfigurationBlockDirective class of our doc builder.

Updating Doc Internals in symfony.com

All the necessary pieces to replace Sphinx/Python with our own PHP Doc Builder
were ready. The last step was to update symfony.com code to make use of it.
This took us a bit longer than expected because of the „technical debt“
accumulated during the past years.

Thankfully we had some PHPUnit tests in place to check that the docs worked as
expected (e.g. expected page titles and table-of-contents, doc navigation, etc.)
This gave us some confidence before starting this big refactoring.

We ended up deleting most of the existing code related to docs and replaced it
with some DTOs created from the JSON files generated by the doc builder. These
DTOs contain all the data needed to fully render a page (its table-of-contents
items, the locale of the contents, the other Symfony versions in which the page is
published, etc.) This helped us remove all the logic from controllers and templates.

Sphinx and Python were now gone. The RST parser and the Doc Builder had been
fully integrated in symfony.com. What was next? Redesigning the doc section!

Redesigning the Website

A few months ago we unveiled the new Symfony Bundles Doc section. This was
the first symfony.com section to use the new RST parser and Doc Builder. We
also created a unique design for it, completely different to the rest of the site.

From the very beginning, this bundles section was a test ground for the Symfony
Docs redesign. It allowed us to test the new doc building for real, to perform
some experiments and to get feedback from some members of the community. We used
all that to build the new Symfony Docs website.

The main ideas behind the new design were:

Overall design should be simpler, with less „noise“
There should be more „breathing space“ between contents (pages are wider now)
Readability should be the top priority (larger and better typography)

Here’s a comparison between the old (left) and the new (right) index page
(click to make it larger):

Here’s a screenshot of a typical book page under the new design (click to make it larger):

The new design is full of little details that help improve readability. It’s
also fully responsive and provides a dark mode alternative. The default mode is
the same as your operating system. If you want to set it to light or dark explicitly,
use the UI element located in the footer of all pages.

Thanks to All Who Made this Possible

The new Symfony Docs website is a good example of the advantages of Open Source
in practice. We leveraged (and contributed to) different projects created and
maintained by different organizations. The result is a PHP doc builder which is
faster and much easier to maintain than the previous non-PHP builder.

Thanks to all contributors who made this possible:

List of contributors of doctrine/rst-parser
List of contributors of symfony-tools/doc-builder

Did you guess the Symfony Docs size?

At the beginning of this article I asked you to try to guess the size of the
Symfony Docs contents. Here’s the answer: as of October 2021, and according to
wc command, the entire Symfony Docs contents across its 25 versions (from 2.0
to 6.0) and including all code examples, comprises 2,712,374 lines,
9,787,338 words and 97,698,319 characters.

To put things in perspective, the „Lord of the Rings“ trilogy contains
576,459 words, making Symfony Docs 17 times bigger.