Building my own RSS Reader/Blogroll

RSS is having a bit of a renaissance in techie circles lately, with some prominent voices arguing that we should bring back RSS as an alternative to the corporate controlled social media platforms.

I’m in full agreement! For those who don’t know, RSS (which stands for Really Simple Syndication) is a specification for “web feeds” where a website publishes an ever-updating list of their content. Readers can then subscribe to a website’s feed by using a “Feed Reader”, which aggregates and displays the content from multiple different feeds.

If you want to get into RSS today, choose a reader (I recommend Inoreader or Artemis) and start looking for the RSS logo () which will let you subscribe to that website’s feed! Here’s mine for starters :P

Context

RSS is old tech – the first official version came out in 1999, and it was widely adopted during the naughties. News sites started publishing RSS feeds of their content, blogging platforms created RSS feeds automatically for their users, and web browsers had RSS utilities baked in as a first-class feature. Facebook and Twitter even published RSS feeds of an account’s content when they first launched.

Now, twentyish years later, the situation is much different. Most big web walled gardens no longer publish RSS feeds of their content (Facebook and Twitter killed theirs in 2021). The browser consortium (i.e., Google) is removing decades old browser features that made using RSS more pleasant, and RSS has thoroughly fallen out of the mainstream.

In the eyes of the big tech companies, RSS is now an enemy, because it lets us keep out of their ad-infested cesspools, and limits their ability to track what we do online.

But, despite their efforts, RSS hasn’t been killed completely, and, in fact, I think it’s going through a bit of a resurgence as we speak!

There’s been a big push in the “indieweb” scene to repopularize blogging itself, and RSS feeds with them. Most of the existing blog platforms (blogspot, wordpress, etc) still automatically create RSS feeds for their users’ blogs, with Blogger (ironically owned by Google) still allowing you to favorite feeds and show them on your own blog.

You can still find an RSS feed for most major news sites (see Aljazeera and the BBC), Youtube channels still publish RSS feeds (though you have to use a tool like https://feedfinder.davidbreder.com/ to find them), and Mastodon and Bluesky publish RSS feeds for their users.

The biggest modern proponent of RSS is, surprisingly, the podcasting industry, which uses it as their main distribution mechanism. Every podcast publishes their own RSS feed (here’s the feed for Cool People Who Did Cool Stuff), and then all the streaming platforms (Spotify, Apple Music, etc) read the RSS feed and simply display the content.

According to this github comment (from a podcast industry web developer protesting Google’s removal of RSS features), there’s currently some 4.5 million podcasts with their own RSS feeds.

The heydays of RSS may be past us, but I think the internet would be a much healthier place if we used it more. Below I’ll talk about my own journey with RSS Readers, and why I ultimately decided to build my own!

My RSS Journey

I only really got into RSS myself in the past year.

For the longest time, my approach to following people’s blogs was to manually check them, multiple times a day. This sorta worked, but it was tedious, and I would often forget to check at least one of the blogs I liked.

I was aware of RSS, but I’d never used a feed reader, and didn’t really know where to start. I tried a few of the popular ones (like Feedly and Newsblur) but the need to have my own account and application on my computer put me off, and I found the UI a bit like my email inbox – i.e. overwhelming.

Fortunately, one of the indie web people I follow James had just launched his new “calm web reader” Artemis, and I decided to give it a try.

Artemis’ main feature is that it only updates each feed at midnight (in your local time), to try to break people of the habit of continuously checking for updates. Additionally, the UI is very simple, presenting basically as just a list of links.

You can see an example of its presentation here: https://artemis.jamesg.blog/u/posts-about-the-indieweb.

Many RSS Reader programs are built with the assumption that you want to read the content inside the program itself, but I wanted to read each item on its the original website. Most of the feeds I follow are small blogger types, and I enjoy the experience of visiting their websites directly. Artemis was built with this in mind, so it seemed like it suited me perfectly!

I used it for a few months, but found that I was getting frustrated with the core feature, the 24hr fetch time. I felt like I was always a day behind the times.

This feeling is arguably unhealthy, a sign of my web addiction, but regardless, I decided to give some other readers a try.

The next one I ended up with is the unfortunately named vore.website. Like Artemis, it presents your feeds as a simple list of links, additionally with the ability to save a link for future reading.

Unlike Artemis, it refreshes the feeds more regularly, every 12 hours, which suited me better. It also makes your list of feeds public by default, which wouldn’t work for everyone, but was nice for me, letting me easily share my list with people.

I happily used it for several months, until it had an outage a few weeks ago. Vore was down for around two days, and during that time I had the dangerous thought – what if I made my own feed reader?

Programming your own reader seems to be the indie web challenge du jour, with Artemis, Vore, and David Bushell’s Croissant all being examples. Embracing RSS already feels like taking more control of your internet consumption, so building your own program to do it is the obvious next step 😅.

And thus, the Roly-Poly Blog-A-Roly was born!

Building my own Reader

I knew I wanted my feed reader to basically be the same as vore.website – a set of RSS feeds that I could update on the main page, with the aggregation being a list of links, sorted by date.

Additionally, I wanted to refresh the RSS feeds more frequently – currently I have it as every hour, unless the feed server itself requests otherwise.

With these criteria in mind, I started programming!

I chose to build the reader in Go, a language I’ve been getting more into recently, and which is particularly well-suited for http stuff. I use the gofeed library to handle the XML parsing, and the zombiezen sqlite package to talk to my database, but other than that there’s no dependencies (and I might see about writing my own XML parser one of these days…).

The code isn’t very exciting – all an RSS reader is at the end of the day is a program that makes HTTP GET calls on a regular schedule, and writes down the responses into an HTML file.

Still, there were some interesting bits, which I’d like to go into.

Being a good netizen

The difference between a DDOS attack and your viral blog post getting a lot of traffic is one of intent, not of form.

Webmasters have to pay for every request their server receives, and so RSS readers should be careful about how many requests they make. Besides, blogs often only update a few times a month – checking its feed every second is wasteful.

I wanted to make sure that my Reader wasn’t spamming other people’s servers, and that it followed as much of the web server social contract as possible (looking at you, evil LLM scrapers).

rachelbythebay’s Feed Reader Behavior Project was of great help to me with this. Rachel has curated this wonderful list of behaviors that a Reader should adhere to, and I refered back to it frequently while I was coding.

The most important behaviors are around so-called conditional requests, where you add the If-Modified-Since and ETag headers in your request headers, which allow the server to return cached content if the feed hasn’t been updated since you last checked.

Rachel also has a feed reader testing tool, which will check that my reader is doing things correctly in production – I plan to sign up soon!

Editing Data

Copying vore.website, I decided to make my feed list editable on the “homepage”, with no separate admin console or anything like that.

I stole his <textarea> interface, where my feeds (and tags) are just listed in an editable text-box, and hitting the submit button sends the updated list back to my server.

The server then does the work of figuring out which feeds or tags were added or removed, makes the necessary updates to the database, and regenerates the HTML page.

This editing functionality is only possible if you’re logged into the page, with (hopefully) only me knowing the password. But it’s a good template I think if I ever want to host the reader as a service so that people could have their own lists.

RSS Reader vs Blogroll

I started making my reader with the intention that it’d be private. My own personal RSS reader that would live on my web server and only be accessible by me.

I changed my mind pretty early on, deciding to publish my feed aggregation on my website under the /blogroll URL. The indie TTRPG scene has a robust culture of publishing each other’s blogs in blogrolls (e.g., see Elm Cat here), and I’ve found a load of good blogs by following the trail of links, so it felt only fair to publish my own.

I did have some concerns though: Some of the content I want to follow isn’t exactly corporate America appropriate – would I be closing employment doors if I admitted on my homepage that I like to read the wonderful sex-positive comic site Oh Joy Sex Toy? Maybe.

But in this time of facism and censorship, where LGBTQ (esp. trans) art and artists are under attack, I felt I would be a coward if I self-censored myself. Worse, I’d be helping the facist agenda, rather than fighting it.

So instead, I’ll do this (very) small act of resistance, and I encourage you all to do the same. Let’s keep the internet weird.

What’s next?

I’ve got a few features I want to add, with the top of the list being an RSS link finder (though maybe it’ll be a browser extension), and have some more tests to write, but other than that I think it’s mostly complete for now. I’ve only been using it for a couple weeks, so I’m not really sure what, if anything, needs changing yet!

I do want to open source the code so people can self-host it (I’ve got some ragman.net specific stuff I need to remove before I can do that), and I might look at hosting blogrolls for people, like vore.website and Artemis do, but that’ll be a ways off.

As always, thanks for reading, and if you’ve got any feedback about the /blogroll page, or want to talk RSS, I’d love to hear from you! :)