Feed Me!

Although I perhaps should have been using the time in other ways, I’ve spent considerable time and effort building a custom news aggregator.  I was able to leverage SimplePie and PHP to do a lot of the heavy lifting, but I also wanted to scour the Internet, social media, and all the trendy Web 2.0 networks for the best content related to the hobby of miniature painting.

I learned a few things along the way, like not all feeds are created equal, and not everyone uses hashtags or even content management systems to their fullest. Some preferences/prejudices were reinforced and some new tricks and tools were discovered, so since I’m still unemployed and you can use these tips and tools to monitor just about anything, I thought I’d list my Top 10 sources/techniques you can use to get RSS or Atom feeds.

1. Blogs It goes without saying there are a lot of blogs online. Blogger/blogspot.com and WordPress seem to be the two most popular blogging systems, both can produce feeds, but WordPress does a much better job of it. Even using FeedBurner to improve the default Blogspot.com Atom feed doesn’t give you the granularity and control that WordPress does.
2. Twitter is the world’s number one spot to break news or catch up the latest happenings and trends. Twitter has an API which you can use to search, but there are also third party tools that make searching and monitoring Twitter easier.
3. Topsy is a “real-time search engine for the social web“. It can search Twitter and Google+, but doesn’t work with every social network. For Twitter it does a great job, better than the default Twitter search. It can create RSS feeds and it has an API.
4. Tumblr gets a lot of press but I could list a number of reasons why I prefer WordPress and other publishing or cataloging tools. Tumblr’s RSS support isn’t as good as WordPress’s and it’s default search sucks. It is possible to use tools to monitor Tumblr for a particular topic or hashtag, such as this Yahoo Pipe, but it is definitely not as easy or as thorough as using Topsy to search Twitter.
5. Pinterest was at first dismissed as a Tumblr clone but it has really caught on, especially with women. Text is a second class citizen on Pinterest with image being everything. Pinterest had built in support for RSS feeds which I was able to use as part of my miniature painting news aggregator. It too stopped working and I haven’t had time to determine if I can fix it.
6. Instagram is another billion dollar image sharing network. It is a less useful for following topics and trends but folks do use it to share photos, you can subscribe to an RSS feed for a hashtag, the problem with this approach is hashtags are not consistently and universally applied.
7. Flickr has long had great support for RSS feeds and an API. Yet Pinterest and Instagram get all the press these days. Flickr is still popular with serious photographers. Yahoo actually sold Flickr to SmugMug. Pinterest is for sharing photos you found online, Flickr and Instagram for sharing photos you took yourself. Flickr’s RSS feeds use an enclosure which includes a thumbnail, this is something Pinterest should be doing too. And I actually discovered that Album feeds had died on Flickr, apparently they have risen unlike some of these other feeds I documented.
8. Google Alerts are one of the oldest and still one of the best ways to monitor the Internet for a specific topic or keyword. I’ve used them for years, but every now and then an old dog can learn a new trick. It is possible to create a Google Alert and have it output RSS. Combined with say ‘site:domain.name‘  plus keywords and you can create custom search agents specifically for Tumblr or other sources.
9. Feed of Feeds is a technique where you take lots of smaller independent information sources and merge them together into one easier to monitor feed. There are a number of blog networks in the miniature wargaming hobby, a few use tools like SimplePie to create one single feed that can be subscribed to, that includes the latest posts from a variety of sources. I built my own feed of feeds and it was how I discovered a probable bug in SimplePie.
10. YouTube isn’t something I’ve traditionally tried to monitor, but it has an API which can produce an RSS feed. You can use these features to create a keyword search and then subscribe to the results, so if you want to see all the videos uploaded on a particular topic or in a particular category, no problem. Just like in Pinterest it is possible to subscribe to someone else’s uploads or favourites too. Videos get added and taken down quickly on YouTube and I don’t think the search result feed updates that frequently, but it is definitely another tool in the toolbox.

I looked at a lot of other tools and techniques, but if you have a killer one I missed, let me know in a comment. If you want to know more about how I used or created various feeds I’ll try to explain it better or find someone who can.

Feeds Die and APIs can Restrict your Access

Topsy now requires an API key to create the RSS feeds I talk about above and use in my news aggregator. These feeds were formerly a free service, but now they want you to transition to a commercial account. I have a 30 day free trial API key but unless they make an exception for me, I’m going to have to revise all my news aggregators and my other PHP code.

Google Alerts stopped supporting RSS feeds and later they forced everyone to upgrade to version 3.0 of the YouTube API which killed off more easy to use RSS feeds. Google seems down on RSS even killing off Google Reader. When you base your code on free web services, they sometimes disappear, I’m just a guy, but companies have learned this lesson in a much more expensive manner. I will have to revisit this list as there are Google Alert competitors, some of which will no doubt tout their support for the RSS standard:

• Trackur
• Talkwalker

Instagram chooses API removes RSS feeds

Instagram of course changed their API and I believe is also abandoning RSS there are workarounds but I think the writing has long been on the wall. RSS is just too easy to use and difficult to monitor, companies want you to agree to use their API and then they can limit your usage and potentially charge you after you build something on top of it.

Time is Money

I’ve been busy with my now completed CFA® studies, but RSS feeds have continued to die. Sometimes if you ask nicely they will reappear other times you can use an API to create a hack or a workaround. I still don’t have time for hobbies, but I’m gradually fixing bugs and improving my web empire. I’m going to need to do something about my feed reader, as NewsFire no longer seems to work. But a bigger priority is actually finding a new job, I’m not sure fixing old bugs in HTML, JavaScript, CSS, and PHP will help, so I’m going to learn Python, just to add another keyword to my resume and LinkedIn profile.

In the time of the Coronavirus

During self-isolation or quarantine people are spending a lot of time online, they are using social media and with the exception of Twitter, which is truly open to anyone even Chinese government officials, social media is generally a walled garden. You can’t just get a feed anymore and avoid seeing ads. Some RSS and Atom feeds even have ads in them. I got a new newsreader, cleverly called Reader but I haven’t been using it lately. Nowadays people post to multiple social media platforms and cross share their content there. They may not even have a blog or a website choosing to make Instagram or Facebook their homepage. The other thing people seem to be doing in the time of the Coronavirus is spamming old blog posts of mine.

If you have thoughts on feeds and APIs or advice on fighting spammers you can leave a comment below.

This entry was originaly posted on April 25th, 2012, it was last edited on December 19th, 2021 and is filed under: Technology and tagged: News Aggregator, RSS, SimplePie.