The day Fintechbot died (sort of)

Halloween 2016 was a sad day. It is the day the Bit.ly owned Twitterfeed closed it’s doors. It is a brilliantly simple service I have used for years to power Fintechbot. Now it is gone and I need some new services to help replicate what Twitterfeed does/did.

What is Fintechbot and what it needs to keep it running.

Fintechbot is an automated fintech news bot for Twitter. It takes stories from a number of RSS sources and then publishes the post title and the URL in a tweet. It appends each tweet with the twitter account of the source of that story.

I currently feed through 120 RSS feeds. All my sources are here, this list probably needs a prune / check to see if these RSS feeds are all still working.  85 of those are unfiltered as they are pure fintech sources. Whenever a new story appears on those feeds it gets posted to Fintechbot. The other 35 feeds are filtered i.e. they only post stories that feature key words that I have assigned. It would be very nice to have a list of words that can be easily applied to multiple feeds. Editing this list (adding/removing key words) would impact all that are filtered.

I also control the outputs so that no single source can post more than 3 stories at any one time in a set period e.g. per hour.

And that is it really.

How to remake it

The obvious fix is to choose a solution such as Buffer, Social Pilot, Twibble or Dlvr.it. The only issue is the cost for such a great number of RSS feeds. Dlvr.it looks like the best option as I use it for my filtered posts today but that would be $30 per month.

I could merge the RSS feeds into a single mega feed but that means I would lose source and be unable to credit the posters. I do not want to do that. I have however been looking at Yahoo! Pipes alternatives….most of them have gone the way of that amazing service too unfortunately.

IFTTT and Zapier have been mentioned as solution but their filtering and control of RSS feeds is very basic indeed. My first few tests with IFTTT failed to retrieve any articles from an RSS feed at all.

I have been looking for pieces that allow me to host this myself. This would be my preferred option…the big issue is that I do not have the skills to build something. Feedparser looks like a very useful tool for bringing in the posts and potentially formatting them ready for publishing. I would then need a publishing mechanism. My python skills are none existent though and I am a bit unsure where to begin.

I was hoping there maybe a couple of wordpress plugins that might be able to cover my needs but I have not found anything that fits the bill just yet. http://www.wprssposter.com/ and http://www.bloggerposter.com/ have some of the features need but they are mostly aimed at taking stories from RSS and feeding them into a WordPress blog unsurprisingly.

My most promising avenue at the moment is using Google Docs. It looks like I could use the IMPORTFEED function in Google Sheets to pull in stories from the feeds. This little tutorial explains it well and how you can use it as an RSS reader.

From there I need a means of publishing each new RSS item into a new row and adding the twitter account of the source ready for publishing. This autotweeter Google Sheets template looks like just the ticket. If I can feed that spreadsheet with IMPORTRANGE I might be on to a winner.

I do however feel that may be a longshot. The tutorial that template is built on is by Zach Whelan and he states one thing missing is the ‘pull live data from a realtime API’. Zapier to Google Docs could create new row eveytime a new item appears. You can’t set up 120 Zapier zaps for each RSS for free though.

Nothing is easy for idiots that can’t code and expect something for free. I feel like I am missing some obvious pieces of functionality here I also feel I am overlooking some obvious issues too.

If anyone has any tips / ideas on alternative solutions then please do let me know. If anyone fancies helping me with Feedparser Python app then also please get in touch. I am sure you all want to save my sad little alter ego on Twitter as much as I do.

Update 05/11/16: My esteemed former colleague, Paul Dougan, reminded me about Feed2OMB a service for publishing RSS Feeds to Microblogs. He used it at our former employers to build something that would feed Fintechbot to an internal Statusnet. He also mentioned Microsoft’s new IFTTT clone, Flow. On first glance it looks like just the ticket. Far more RSS feed control than IFTTT and with better options for adding other conditions to flows so hopefully I can work out how to post limited numbers of links per hour. I am going to add some RSS feeds to it this weekend to test it further.

Update 11/01/17: Not got round to doing much with this yet. The services like IFTTT and Flow cannot give me the output I want as they cannot include @ symbols. Python looks like a decent option at the moment and the two programs mentioned above cover most of the functionality. A third is needed to work with the Twitter API, https://github.com/bear/python-twitter looks good to me. As does Tweepy. This article about making bots also of interest. Let’s see how far I get.

Update 18/01/17: I failed. Miserably. This is what I think I need to do. I think there are a few parts to this

1. Hold a list of RSS feeds to be checked (Phase 2 check feeds still working / have updated in the last x days)
2. Check periodically for new items (Phase 2 requirement allow checking by keywords i.e. if new post does not contain a list of key words don’t queue, only applied to some feeds i.e. ones that are not pure Fintech news)
3. When new items are posted hold them in a queue ready for publishing
4. Publish to Twitter in a controlled manner e.g. no more than x posts at a time, no more than x in an hour? (Phase 2 look at other publishing points e.g. Linked In, Tumblr)
5. The ability to add a prefix/suffix to the tweet e.g. by @nameofpersonwhowrotethestory or #Podcast
6. I think that is it…

I think the initial functionality is provided by Feed2OMB but that then needs an update to allow it to post to Twitter using a Python to Twitter tool like the ones mentioned in the comment above. I have no ide how to stictch those things together or how to install on my webserver.