That moment where you should have automated but didn't
TLDR; I migrated blogs over to Hugo and I didn’t automate because I was only doing it once, I should have automated because I actually migrated 450+ times (at least once per post. Find results at testerhq.com
Because I’m slightly risk aware I have the following concerns about my main blogs:
blogger.comis free so at some point Google will cancel it and I will lose everything
- wordpress uses php and a database and at somepoint someone will take advantage of a security hole and I’ll get hacked and lose everything
When I post to linkedin I have the concern that:
- there is no RSS feed
- linkedin might go down and I’ll lose everything
And if I ever start posting to medium or other platforms I will have the same issues.
Therefore I decided to ‘archive’ everything into a static site so that if everything goes down I have a backup.
I chose to use Hugo for this.
This has the advantage that:
- all posts in Hugo are written in markdown
- I already write all my posts in markdown (and covert into html for pasting into the other platforms)
- I get to use my testerhq.com domain
- I get to try out the multiple site hosting on my vidahost cloud plan
- much harder to hack a static site (and if you do, I just re-upload)
- I can eventually put all posts on github and have an offsite source back up as well
- I can repurpose all the posts later
I will use testerhq.com as my aggregator and I will probably extend it to cover:
- stuff I post to slideshare
- tweets I think are useful
- youtube videos etc.
I looked around for tools that could easily migrate from Wordpress to Hugo and blogger to Hugo but didn’t find a lot that fit my workflow.
I thought I should really write my own but since I saw so many failed attempts out there I thought it might be too hard and take too long.
- I really should have done
I thought that since I’ll just migrate once, that I really couldn’t justify the time to automate it.
- I might migrate each blog once (4 of them), but there are over 450 posts, that might be worth automating.
My final solution was:
- Use wordpress blogger import
- Use a jekyll export Wordpress plugin
- import from Jekyll to Hugo
I used a Wordpress bitnami VM and imported my blogger blogs to that, then exported to Jekyll and then imported to Hugo.
I exported my wordpress blogs directly to Jekyll and imported to Hugo.
And then the pain started.
I then had to check every post for mistakes during the various migration.
Sometimes it was:
- html for iframes of slideshare and youtube messed up
- images not convered to markdown properly
- mad html not converted properly due to a missing tag somewhere
- various special chars not converted properly
Pretty much every post needed to be hand checked and amended with find and replace in some form or other.
It took me ages. I suspect it would have been faster for me to automate this last part.
I dug out my old HTTP Testing Web Crawler and amended it to check for characters on the page this made it easy for me to check for:
- images that hadn’t converted properly e.g. and left
!(type chars on the html
I still have a lot of broken links to stuff I’ve linked to over the years, and I’ll slowly fix that up over time.
I was looking at an old Choose Your Own Adventure Game that I created a few years ago in Twine with my Son. And thought about converting it into more of a CYOA game book format e.g.
walk out of the door (goto 8)listed on the page rather than a link.
I started hand converting it from a twine archive into markdown because I looked around and could not find any tools available that did this, and I saw a few failed attempts and I was about to say ‘this is hard’.
But I knocked up a quick converter using Java and JSoup and it didn’t take long at all.
The benefit is, that because I have automated:
- I am more likely to do it again
- I will probably write more CYOA games
- I learned more about JSoup
- I have yet more code on github
- I re-used the work I put into pandocifier
- I’ve created an archive for my main blogs over at testerhq.com.
- This has an RSS feed that covers - SeleniumSimplified.com, EvilTester.com, JavaForTesters.com, and CompendiumDev.co.uk, and LinkedIn
- I will continue to add the posts to the other blogs as well, so if you’re only interested in one strand of my work you can keep using that
- Sometimes we should automate, even if we are only doing a task once.