Recently I’ve struggled to find the time and energy to write blog posts, for various reasons of which I cannot deny that Pokémon Go might’ve been one. However, there were also some valid reasons too, I realised that while I might do some automation, write some Python tools and generally dabble in programming, none of that is visible really from my LinkedIn profile or on my CV.

Sure, I can say that I have experience with Python 2.7.X and Python 3.4/3.5, but where’s the evidence? I can’t exactly put my code from my work repositories online, because that code belongs quite rightly to the company I work for.

Thus if I ever want to make it clear that I can write programs and that I am familiar with Python, I decided that I needed to put a Github repo up with some examples of the kinds of things I am capable of doing in Python. Great idea so far and I quickly spun up a repo and initialised it and… Then what?

I started asking around in the Tester’s Slack ( http://www.testers.io/ , if you’re a tester and you’re not on that Slack, you’re really missing out on a wealth of resources and advice) about what people would look for in a portfolio, did I want to build a gigantic project with lots of functionality to show what I could do, or did I want to build lots of smaller tools? Luckily the consensus on Slack was that the smaller tools route was the right way to go. I love writing little tools, so that worked out well for me.

Now I had a plan, a rough idea of what I needed  to do, so I sat down and I started to do it.

Or I tried to.

Turns out that I had no ideas about what it was that I could write, normally when I write Python it’s to solve an immediate problem. For example, I started my job 6 months ago and I’ve filled a small repo with ten or so small scripts that all solve an issue I’ve been facing in my work. One spins up a suite of VMs for regression checking, one resets variables and enabled dev options in builds, one is a simple https server that swaps out a served file at a command. Essentially very small, self contained scripts that make my day to day life easier.

So, once again I turned to the community to get some ideas and maybe someone to help me, after all I am not the best programmer, I would appreciate some input from someone who’s more experienced and nothing says “I can work in a team” like a collaborative project. For this I turned to Robert Page, a fellow tester from the Testers Slack (spotting a pattern yet? ) who’s kindly agreed to be the person I bounce my ideas off and hopefully throw a few pull requests my way.

Caution: Python technical part.
But what will we be actually making? Well, nothing groundbreaking but I liked the idea of making a dead link scanner. Not a new idea, but it seems like a fun thing to write . My idea was at first to just use scrapy and pull all the URLs in a domain, then I’d run those URLs through requests and see if I get a 404 on a requests.get(url).status_code . If I did then I would append the url to a list and display it at the end of the run.
Robert convinced me that that wasn’t enough, and he was right, what about pages that return 200, but display a set “We couldn’t find your page” message? Or when you hit a 403? Basically scanning for simple 404s isn’t enough.

But it’s definitely my minimum viable product, so I plan to start with the link scanner taking a url and a recursion depth argument. As I commit and push this, I’ll try and make my comments and commit messages explanatory enough to allow them to support this narrative. Github is at https://github.com/Dominic-Kua/Random-things , please have a look and enjoy watching me learn.