Discovering and recovering from PostgreSQL corruption on Matrix.org

Richard van der Hoff, a member of the team that runs the Matrix.org homeserver,
has written
a detailed blog post about diagnosing and fixing a problem where Matrix rooms
would simply stop working:

We know that there are plenty of users out there who will have been
affected by the problem, and found themselves unable to communicate as
a result. We very much share your frustration, and we'd like to
apologise for the disruption to service.

With that said, we're glad that we were able to get to the bottom
of most of the problem, and get the lost data restored within a
relatively short time. If nothing else, hopefully this blog post will
be of use to future generations faced with Postgres index
corruption!