Data Inspired Insights

Author: Brett Romero (Page 3 of 6)

Data Science: A Kaggle Walkthrough – Introduction

This article on understanding the data is Part I in a series looking at data science and machine learning by walking through a Kaggle competition. The other parts in this series can be found here.

In a futile attempt to shed some light on the field of Data Science, I have put together a multi-part series looking at what data science involves and some of the techniques most commonly used. This series is not intended to make everyone experts on data science, rather it is intended to simply try and remove some of the fear and mystery surrounding the field. In order to be as practical as possible, this series will be structured as a walk through of the process of entering a Kaggle competition and the steps taken to arrive at the final submission.

Continue reading

The argument for taxing capital gains at the full rate

Politicians, both in Australia and the US, when asked how they will find the money to fund various policy proposals, often resort to the magic pudding of funding sources that is “closing the loop holes in the tax code”. After all, who can argue with stopping tax dodgers rorting the system? But as Megan McArdle recently pointed out, raising any significant revenue from closing loop holes requires denying deductions for things that a lot of middle and lower class people also benefit from. This includes, among other things, deductions for mortgage interest, employee sponsored health insurance, lower (or no) tax on money set aside for pensions and no tax on capital gains when the family house is sold.[1]

Broadly, I agree with McArdle’s point. The public, in general, are far too easily convinced by simplistic arguments about changes to taxation – as if after decades of tax policy changes there are still simple ways to increase revenues without anyone suffering. Any changes made at this point are going to cause winners and losers, and often, the people intended to be the losers (usually the rich) are less affected than some other group that also happened to be taking advantage of a particular deduction.

That said, there is one point, addressed breifly in McArdle’s article, that I thought deserved greater attention – the concessional taxation of capital gains. In the list provided in the article, it was the second most expensive tax deduction in the US at $85 billion a year[2]. You see, for a while now I have been somewhat of a closet skeptic of the need for lower tax rates on capital income (i.e. capital gains and dividends). The reason for my skepticism is two fold:

  1. Everyone seems to be in agreement that concessional rates for capital income are absolutely necessary, but no one seems to really understand why.
  2. Capital income makes up a much larger percentage of income for the wealthy than for the lower or middle class. When you hear that story about billionaire Warren Buffet paying a lower rate of tax than his secretary, it is because of the low rate of tax on capital income.

So, now that I am finally voicing my skepticism, this article is going to look at what arguments are made for lower tax rates on capital income (focusing on capital gains for individuals) and whether those arguments hold water.

Why are capital gains taxed at a lower rate?

Once you start digging, you quickly find there is a range of arguments (of variable quality) being made for why capital gains should be taxed at a lower rate. These arguments can largely be grouped into the following broad categories:

  1. Inflation
  2. Lock-In
  3. Double Taxation
  4. Capital is Mobile
  5. The Consumption – Savings tradeoff

Inflation

Taxing capital gains implies taxing the asset holder for any increases in the price of that asset. In an economy where inflation exists (i.e. every economy) this means you are taxing increases in the price of the asset due to inflation, as well as any increase in the value of the asset itself. Essentially, even if you had an asset which had only increased in value at the exact same rate as inflation (i.e. the asset was tradable for the same amount of goods as when you bought it), you would still have to pay capital gains tax.

The inflation argument although legitimate, is relatively easy to legislate around by allowing asset holders to adjust up the cost base of their assets by the inflation rate each year.

Lock In

‘Lock-in’ is the idea that investors, to avoid paying capital gains tax, will stop selling their assets. An investor holding onto assets to avoid tax implies they are being incentivized, through the tax system, to invest suboptimally – something economists really dislike. However, as far as ‘lock-in’ would occur, it cannot be considered anything other than an irrational reaction. Holding onto assets does not avoid tax, it only delays it, and given inflation is factored into the asset price (as discussed above), there is not even the benefit of time reducing the tax burden. The bottom line is this – to pay more capital gains tax, there must be larger capital gains. That is, even if the capital gains tax rate was 99%, an investor would still be better off making larger capital gains than smaller ones.

The other point to remember when it comes to ‘lock-in’ is that in both the US and Australia, the lower rate of capital gains tax only applies to assets held for more than a year. That means if ‘lock-in’ exists, it is already a major problem. Because asset holders can access a lower rate of tax by holding an asset for a year, they are already strongly incentivized to hold onto their underperforming assets longer than is optimal to access the concessional tax rate. In fact, increasing the long-term capital gains tax rate to the same level as the short-term rate should actually reduce lock-in by removing this incentive.

Double Taxation

The double taxation argument is a genuine concern for economists. The double tax situation arises because companies already pay tax on their profits. Taxing those profits in the hands of investors again, either as capital gains (on that company’s stock) or dividends, implies some high marginal tax rates on investment. This is one of the main reasons capital income is taxed at low rates in most countries.

Ideally, to avoid this situation, the tax code would be simplified by removing company tax altogether, as McArdle herself has argued in the past. However, we should probably both accept that, at best, the removal of corporate tax is a long way away. Nevertheless, this idea can form the basis for policies that achieve similar goals without the political issue of trying to sell the removal of corporate tax.

For dividends, for example, double taxation can be avoided by providing companies with a deduction for the value of dividends paid out to investors. Investors would then pay their full marginal tax rate on the dividends, more than replacing the lost company tax revenues.

Preventing double taxation of capital gains is a little more complicated, but the answer may lie in setting up a quarantined investment pool that companies can move profits into. Profits moved into this pool would not be subject to tax and, once in the pool, the money could only be used for certain legitimate investment activities. This would effectively remove taxation on profits going toward genuine reinvestment, as opposed to fattening bonus checks.

The overall point here is not that I have the perfect policy to avoid double taxation of company profits, but that there are other worthwhile avenues worth exploring that are not simply giving huge tax breaks to wealthy investors.

Capital is Mobile

This is one of the two arguments McArdle briefly mentions in her article. The ‘capital is mobile argument’ is the argument that if we tax wealthy investors too much, they will do a John Galt, take their money and go to another country that won’t be so “mean” to them.

When it comes to moving money offshore, obviously, not everyone is in a position to make the move. Pension funds and some investment vehicles cannot simply move country. Companies and some other investment vehicles do not receive a capital gains tax discount currently, meaning raising tax rates for capital gains for individuals would not impact them at all. Finally, even for investors that would be affected and do have the means, a hike in the capital gains rate does not automatically move all their investments below the required rate of return.

This argument also overlooks the vast array of complications in moving money offshore and the risks involved with that action. Moving assets offshore exposes investors to new risks such as exchange rate risk[3] and sovereign risk[4]. It also significantly complicates the administrative, compliance and legal burden the investor has to manage.

However, even if we concede that yes, some money would move offshore as a result of higher taxes on capital gains, let’s look at the long term picture. What is the logical end point for a world where each country employs a policy of attracting wealthy investors by lowering taxes on capital? A world where no country taxes capital!

Of course, there are alternatives. Countries (and developed countries in particular should take the lead on this) can stop chasing the money through tax policy and focus on other ways of competing for investment capital. Education, productivity, infrastructure, network effects, low administrative and compliance costs are all important factors in the assessment of how attractive a location is for investors. California, for example, is not the home of Silicon Valley because it has low taxes on capital. Pulling the ‘lower taxes to attract investment’ lever is essentially the lazy option.

Consumption vs. Savings

The second point raised by McArdle is the argument that if you reduce the returns from investing (by raising tax rates), people will substitute away from saving and investing (future consumption) and instead spend the money now (immediate consumption).

The way to think of this is not of someone cashing in all their assets and going on a spending spree because the capital gains tax rate increased. That is extremely unlikely to happen and would actually make no sense. The change will come on the margin – because the returns on investment have decreased slightly (for certain asset types), there will be slightly less incentive to save and invest. As a result, over time, less money ends up being invested and is instead consumed.

But let’s consider who would be affected. If we think about the vast majority of people, their only exposure to capital gains is through their pension fund and the property they live in, neither of which would be affected by increasing the individual capital gains tax rate. Day traders, high frequency traders and anyone holding stocks for less than a year on average would also be unaffected. Most investors in start-ups do so through investment vehicles that are, again, not subject to individual capital gains tax[5]. That leaves two main groups of investors impacted by an increase in the capital gains tax rate for individuals:

  1. Property investors
  2. High net worth individual investors

Given property investing is not what most people are thinking about when concerns about capital gains tax rates reducing investment are raised, let’s focus on high wealth investors.

The key issue when considering how these investors would be affected by an increase in the capital gains tax rate is identifying what drives them to invest in the first place. Many of them literally have more money than they could ever spend, which means their investment decisions cannot be driven by a desire for future consumption. Many of their kids will never want for anything either, so even ensuring the financial security of their kids is not an issue. The only real motivation that can be left is simply status, power and prestige. Or as the tech industry has helpfully rebadged it – ‘making the world a better place.’

If that is the motivation though, does a rise in the capital gains tax rate change that motivation?

To my mind, the answer to that question is ‘No’. These people are already consuming everything they want, or in economic parlance, their desire for goods and services has been satiated. They will gain no additional pleasure (‘utility’) from diverting savings to consumption, so there is no incentive to do so even when the gains from investing are reduced.

Of course, there are exceptions, and it is quite possible (even likely) that there are high net worth individuals who live somewhat frugally and as a result of this policy change would really start splashing out. The question is how significant is this amount of lost investment, and does the loss of that investment capital outweigh the cost to society more widely of a deduction that flows almost entirely to the wealthy.

The Research

Putting this piece together, I have studiously attempted to avoid confirmation bias.[6] Despite the fact that I would benefit personally from lower tax rates on capital gains (well, at least I would if my portfolio would increase in value for a change), I definitely want to believe that aligning capital gains tax rates with the tax rates on normal income would raise significant amounts of tax, mostly from wealthy individuals, with few negative consequences.

In my attempts to avoid confirmation bias, I have deliberately searched for articles and research papers that provide empirical evidence that lower capital gains tax rates were found to lead to higher rates of savings, investment and/or economic growth. I have not been able to find any. There were some papers that claimed to show that decreasing capital gains tax rates actually increased tax revenue, but reading the Australian section of this paper (about which I have some knowledge), it quickly became clear this conclusion had been reached using a combination of cherry picking dates[7] and leaving out important details.[8]

I did also find some papers that, through theoretical models, concluded higher taxes on capital income would cause a range of negative impacts. But the problem with papers that rely on theoretical models is that for every paper based on a theoretical model that concludes “… a capital income tax… reduces the number of entrepreneurs…” there is another paper based on a theoretical model that concludes “… higher capital income taxes lead to faster growth…

Leaving research aside, there were a number of articles supporting the lowering or removing of capital income taxes. The problem is they all recite the same old arguments (“it will cause lock-in!”) and tend to come from a very specific type of institution. Without going too much into what type of institution, let me just list where almost all the material I located was coming from (directly or indirectly):

Even when I found an article from a less partisan source (Forbes), it turned out to be written by a senior fellow at the Cato Institute, and was rebutted by another article in the same publication.

Of course we should not ignore what people say because they work for a certain type of institution – just because they have an agenda does not mean they are wrong. In fact, it stands to reason that organizations interested in reducing taxation and limiting government would research this particular topic. The problem is that if there are genuine arguments being made, they are being lost amongst the misleading and the nonsensical.

Take this argument for lower taxes on capital as an example. First there is a chart taken from this textbook:

Capital per Worker vs. Income per Worker

The article then uses this as evidence to suggest more capital equals more income for workers. As straightforward as this seems, what this conclusion misleadingly skips over is:

  • income per worker is not equivalent to income for workers, and
  • almost all the countries towards the top right hand corner of this chart (i.e. the rich ones) got to their highly capital intensive states despite having high taxes on capital.

A Change in Attitude?

The timing of this article seems to have conveniently coincided with the announcement by Hilary Clinton of a new policy proposal – a ‘Fair Share Surcharge’. In short, the surcharge would be a 4% tax on all income above $5 million, regardless of the source. Matt Yglesias has done a good job of outlining the details in this article if you are interested.

The interesting aspect of this policy is, given the lower rate of tax typically applied to dividends and capital gains, it is a larger percentage increase in taxes on capital income than wage income. Of course, unless something major changes, this policy is very unlikely to make it past Congress and so may simply be academic, but at least it shows one side of politics may be starting to question the idea that taxes on capital should always be lower.

The Data

Finally, I want to finish up with a few charts. The charts below show how various economic indicators changed as various changes were made to the rate of capital gains tax, historically and across countries. Please note, these charts should not be taken as conclusive evidence one way or the other. The curse of economics is the inability to know (except in rare circumstances) what would have happened if a tax rate had not been raised, or if an interest rate rise had been postponed. The same applies with changes to the capital gains tax rate. Without knowing what would have happened if the capital gains tax rate had not been changed, we cannot draw firm conclusions as to what the result of that change was.

However, what we can see is that the indicators shown below do not seem to be significantly affected by changes in the capital gains tax rate, one way or the other – the effects appear to be drowned out by larger changes in the economy. That could be considered a conclusion in itself.

Chart1 – Maximum Long Term CGT Rate vs. Personal Savings rate, US 1959 to 2014

Chart 2 – Maximum Long Term CGT Rate vs. Annual GDP Growth, US 1961 to 2014

Chart 3 – Maximum Long Term CGT Rate vs. Gross Savings, Multiple Countries, 2011-2015 Average

Gross savings are calculated as gross national income less total consumption, plus net transfers. This amount is then divided by GDP (the overall size of the economy to normalize the value across countries.

Chart 4 – Maximum Long Term CGT Rate vs. Gross Fixed Capital Formation, Multiple Countries, 2011-2015 Average

Gross fixed capital formation is money invested in assets such as land, machinery, buildings or infrastructure. For the full definition, please see here. This amount is then divided by GDP (the overall size of the economy to normalize the value across countries.

Chart 5 – Maximum Long Term CGT Rate vs. Gini Index, 2011-2015 Average

The Gini index is a measure of income inequality within a country. A Gini index of 100 represents a country in which one person receives all of the income (i.e. total inequality). An index of 0 represents total equality.

 

[1] Interestingly, two of these four deductions (mortgage interest and employee sponsored health insurance) will be completely foreign to Australians.

[2] A similar policy (50% tax discount for capital gains) in Australia costs around AUD$6-7 billion per year.

[3] The risk that the exchange rate changes and has an adverse impact on the value of your investments.

[4] The risk that the government of the country you are investing in will change the rules in such a way to hurt your investments.

[5] Capital Gains Tax Policy Toward Entrepreneurship, James M. Poterba, National Tax Journal, Vol. 42, No. 3, Revenue Enhancement and Other Word Games: When is it a Tax? (September, 1989), pp. 375-389

[6] Confirmation basis is the tendency of people, consciously or subconsciously, to disregard or discount evidence that disagrees with their preconceived notions while perceiving evidence that confirms those notions as more authoritative.

[7] “After Australian CGT rates for individuals were cut by 50% in 1999 revenue from individuals grew strongly and the CGT share of tax revenue nearly doubled over the subsequent nine years.” Note the carefully selected time period includes the huge run up in asset prices from 2000 to 2007 and avoids the 2008 financial crisis, which caused huge declines in CGT revenues.

[8] “Individuals enjoyed a larger discount under the 1999 reforms than superannuation funds (50% versus 33%), yet yielded a larger increase in CGT payable.” This neglects to mention that even after the discounts were applied, the rate for of capital gains tax for almost all individuals was still higher than for superannuation funds.

Web Analytics – Looking Under the Hood

On occasion I get the sense from bloggers that talking about your traffic statistics is a bit like talking about salary – not something to be done amongst polite company. However, unlike discussing pay, which can generate bad feelings, jealousy, poor morale and a range of other negative side effects, discussing website stats should provide a great learning opportunity for everyone taking part. With that said, in the name of transparency, let me offer a peak under the hood here at BrettRomero.com.

Overall Traffic

For those that have not looked at web traffic statistics, first a quick introduction. When it comes to web traffic, there are two primary measures of volume – sessions and page views. A session is a continuous period of time that one user spends on a website. One session can result in multiple page views – or just the one if the user leaves after reading one article as is often the case. Chart 1 below shows the traffic to BrettRomero.com, as measured in sessions per day.

Chart 1 – All Traffic – Daily


There are a couple of large peaks worth explaining in this chart. The first peak, on 3 November 2015, was the day I discovered just how much traffic Reddit.com can generate. Posting to the TrueReddit subreddit, I posted what, to that point, had been by far my most popular article – 4 Reasons Working Long Hours is Crazy. The article quickly gained over 100 upvotes and, over the course of the day, generated well over 500 sessions. To put that in perspective, the traffic generated from that one post on Reddit in one day is greater than all traffic from LinkedIn and Twitter combined… for the entire time the blog has been online.

The second big peak on 29 December 2015 was also a Reddit generated spike (in fact, all four spikes post 3 November were from Reddit). In this instance it was the posting of the Traffic Accidents Involving Cyclists visualization to two subreddits – the DataIsBeautiful subreddit and the Canberra subreddit.

Aside from these large peaks though, the data as represented in Chart 1 is a bit difficult to decipher – there is too much noise on a day-to-day basis to really see what is going on. Chart 2 shows the same data at a weekly level.

Chart 2 – All Traffic – Weekly


Looking at the weekly data the broader trend seems to show two different periods for the website. The first period, from March to around August has more consistent traffic, around 200 sessions a week, but with smaller spikes. The second period, from August onwards shows less consistent traffic, around 50 sessions a week, but with much larger spikes. But how accurate is this data? Let’s break some of the statistics down.

Breakdown by Channel

When looking at web traffic using Google Analytics, there are a couple of breakdowns worth looking at. The first is the breakdown by ‘channel’ – or how users got to your website for a given session. The four channels are:

  1. Direct – the user typed your website URL directly into the address bar
  2. Referral – the user navigated to your site from another (non-social media) website by clicking on a link
  3. Social – the user accessed your website from a social media website (Facebook, Twitter, Reddit, LinkedIn and so on)
  4. Organic Search – a user searched for something in a search engine (primarily Google) and clicked on a search result to access your site.

The breakdown of sessions by channel for BrettRomero.com is shown in Table 1 below:

Table 1 – Breakdown by Channel

Channel Grouping

Sessions

Direct

2,923

Referral

2,776

Social

2,190

Organic Search

567

Total

8,456

Referral Traffic

Looking at referral traffic specifically, Google Analytics allows you to view which specific sites you are getting referral traffic from. This is shown in Table 2.

Table 2 – Top Referrers

Rank Source

Sessions

1 floating-share-buttons.com

706

2 traffic2cash.xyz

177

3 adf.ly

160

4 free-share-buttons.com

152

5 snip.to

74

6 get-free-social-traffic.com

66

7 www.event-tracking.com

66

8 claim60963697.copyrightclaims.org

63

9 free-social-buttons.com

57

10 sexyali.com

50

Total All Referral Traffic

2,776

Looking at the top 10 referrers to BrettRomero.com, the first thing you may notice is that these site addresses look a bit… fake. You would be right. What you are seeing above is a prime example of what is known as ‘referrer spam’. In order to generate traffic to their sites, some unscrupulous people use a hack that tricks Google Analytics into recording visitors to your site coming from a URL they want you to visit. In short, they are counting on you looking at this data, getting curious and trying to work out where all this traffic is coming from. Over time these fake hits can build up to significant levels.

There are ways to customize your analytics to exclude traffic from certain domains, and initially I was doing this. However, I quickly realized that this spam comes from an almost unlimited number of domains and trying to block them all is basically a waste of time.

Looking at the full list of sites that have ‘referred’ traffic to my site, I can actually only find a handful of genuine referrals. These are shown in Table 3.

Table 3 – Genuine Referrers

Rank Source

Sessions

17 uberdriverdiaries.com

35

18 vladimiriii.github.io

33

72 australiancraftbeer.org.au

3

76 alexa.com

2

95 opendatakosovo.org

1

Total Genuine Referral Traffic

74

Total Referrer Spam

2,702

What does the total traffic look like if I exclude all the referrer spam? Chart 3 below shows the updated results.

Chart 3 – All Traffic Excluding Referrals


As can be seen, a lot of the traffic in the period March through August was actually coming from referrer spam. Although May still looks to have been a strong month, April, June and July now appear to be hovering around that baseline 50 sessions a month.

Search Traffic

Search traffic is generally the key channel for website owners in the long term. Unlike traffic from social media or from referrals, it is traffic that is generated on an ongoing basis without additional effort (posting, promotion and so on) on the part of the website. As you would expect though, to get to the first page of search results for any combination of key words that is searched regularly is very difficult. In fact it is so difficult, an entire industry has developed around trying to achieve this – Search Engine Optimization or SEO.

For BrettRomero.com, search traffic has been difficult to come by for the most part. Below is a chart showing all search traffic since the website started:

Chart 4 – Search Traffic – All


Keeping in mind the y-axis in this chart is on a smaller scale than the previous charts, there doesn’t seem to be much pattern to this data. August again seemed to be a strong month, as well as the weeks in late May and early June. Recent months have been flatter, but more consistent.

Going one step further, Table 4 shows the keywords that were searched by users to access BrettRomero.com.

Table 4 – Top Search Terms

Rank Keyword

Sessions

1 (not provided)

272

2 beat with a shovel the weak google spots addons.mozilla.org/en-us/firefox/addon/ilovevitaly/

47

3 erot.co

45

4 непереводимая.рф

40

5 “why you probably don’t need a financial advisor”

33

6 howtostopreferralspam.eu

32

7 sexyali.com

16

8 vitaly rules google ☆*:.。.゚゚・*ヽ(^ᴗ^)丿*・゚゚.。.:*☆ ¯\_(ツ)_/¯(•ิ_•ิ)(ಠ益ಠ)(ಥ‿ಥ)(ʘ‿ʘ)ლ(ಠ_ಠლ)( ͡° ͜ʖ ͡°)ヽ(゚д゚)ノʕ•̫͡•ʔᶘ ᵒᴥᵒᶅ(=^. .^=)oo

14

9 http://w3javascript.com

13

10 ghost spam is free from the politics, we dancing like a paralytics

11

Again, we see something unexpected – most of the keywords are actually URLs or nonsensical phrases (or both). As you might suspect, this is another form of spam. Other website promoters are utilizing another hack – this one tricks Google Analytics into recording a search session, with the keyword being a message or URL the promoter wants to display. Looking at the full list, the only genuine search traffic appears be the records for which keywords are not provided[1]. Chart 5 shows search traffic with the spam excluded.

Chart 5 – Search Traffic – Spam Removed


With the spam removed, we see something a little bit more positive. After essentially nothing from March through July, we see a spike in activity in August and September, before falling back to a new baseline of around 5-10 sessions per week. Although this is obviously still miniscule, it does suggest that the website is starting to show up regularly in people’s searches.

Referring back to the total sessions over time, Chart 6 shows how removing the spam search impacts our overall number of sessions chart.

Chart 6 – All Traffic Excluding Referrals and Spam Search

Social Traffic and the Reddit Effect

As was shown in Table 1, one of the two main sources of (real) traffic for the website is social media.

Social media provides a real bonus for people who are starting from zero. Most people now have large social networks they can utilize, allowing them to get their content in front of a lot of people from a very early stage. That said, there is a line and spamming your friends with content continuously is more likely to get you muted than generate additional traffic.

Publicizing content on social media can also be a frustrating experience. Competing against a never-ending flood of viral memes and mindless, auto-generated content designed specifically to generate clicks, can often feel like a lost cause. However, even though it seems like posts simply get lost amongst the tsunami of rubbish, social media is still generally a good indicator of how ‘catchy’ a given article is. Better content will almost always generate more likes/retweets/shares.

In terms of the effectiveness of each social media platform, Reddit and Facebook have proven to be the most effective for generating traffic by some margin. Table 5 shows sessions by social media source.

Table 5 – Sessions by Social Media Source

Rank Social Network

Sessions

1 Reddit

999

2 Facebook

868

3 Twitter

224

4 LinkedIn

69

5 Blogger

26

6 Google+

3

7 Pocket

1

When looking at the above data, also keep in mind, I only started posting to Reddit at the start of November, effectively giving Facebook a 7 month head start. This means Reddit is by far the most effective tool I have found to date to get traffic to the website. However, there is a catch to posting on Reddit – the audience can be brutal.

Generally on Facebook, Twitter and LinkedIn, people who do not agree with your article will just ignore it. On Reddit, if people do not agree with you – or worse still, if they do not like your writing – they will comment and tell you. They will not be delicate. They will down vote your post (meaning they are actively trying to discourage other people from viewing it). Finally, just to be vindictive, they will down vote any comments you make as well. If you are planning to post on Reddit, make sure you read the rules of the subreddit (many explicitly ban people from promoting their own content) and try to contribute in ways that are not just self‑promotional.

Pages Visited

Finally, let’s look at one final breakdown for BrettRomero.com. Table 5 shows the top 10 pages viewed on BrettRomero.com.

Table 6 – 10 Most Viewed Pages

Rank Page

Pageviews

1 /

4,345

2 /wordpress/

1,450

3 /wordpress/4-reasons-working-long-hours-is-crazy/

1,038

4 /cyclist-accidents-act/

773

5 /wordpress/climbing-mount-delusion-the-path-from-beginner-to-expert/

306

6 /wordpress/the-dark-side-of-meritocracy/

205

7 /wordpress/why-australians-love-fosters-and-other-beer-related-stories/

194

8 /blog.html

192

9 /?from=http://www.traffic2cash.xyz/

177

10 /wordpress/visualizations/

165

As mentioned earlier, 4 Reasons Working Long Hours is Crazy has been by some margin my popular article. Although Reddit gave this article a boost traffic wise, it was also by some margin the best performing article I have posted to Reddit with over 100 upvotes. The next best performing, the Traffic Accidents Involving Cyclists visualization, only managed 20 upvotes.

Overall

As I mentioned at the outset, web traffic statistics tend to be a subject that is not openly discussed all that often. As a result, I have little idea how good or bad these statistics are. Given I have made minimal effort to promote my blog, generate back links (incoming links from other websites) or get my name out there by guest blogging, I suspect that these numbers are pretty unimpressive in the wider scheme of things. Certainly I am not thinking about putting up a pay wall any time soon anyway.

As unimpressive as the numbers may be though, I hope they have provided an interesting glimpse into the world of web analytics and, for those other bloggers out there, some sort of useful comparison.

 

Spotted something interesting that I missed? Please leave a comment!

 

[1] For further information on why the keywords are often not provided, this article has a good explanation.

5 Things I Learned in 2015

2015 has been an interesting year in many respects. A new country[1], a new language, a new job, and plenty of new experiences – both at work and in life in general. To get into the year-end spirit, I thought I would list out 5 key things I learned this year.

1. I Love Pandas

Yes, those pandas as well, who doesn’t? But I knew that well before 2015. The pandas I learned to love this year is a data analysis library for the programming language Python. “Whoa, slow down egg head” I hear you say. For those that are not regular coders, what that means is that pandas provides a large range of ways for people writing Python code to interact with data that makes life very easy.

Reading from and writing to Excel, CSV files and JSON (see lesson number 2) is super easy and fast. Manipulating large datasets in table like structures (dataframes) – check. Slicing, dicing, aggregating – check, check and check. In fact, as a result of pandas, I have almost entirely stopped using R[2]. All the (mostly basic) data manipulation for which I used to use R, I now use Python. Of course R still has an important role to play, particularly when it comes to complex statistical analysis, but that does not tend to come up all that regularly.

2. JSON is Everywhere

JSON, JavaScript Object Notation for the uninitiated, is a data interchange format that has become the default way of transferring data online. Anytime you are seeing data displayed on a webpage, including all the visualizations on this website, JSON is the format the underlying data is in.

JSON has two big advantages that have led to its current state of dominance. The first is that, as the name suggests, it is native to JavaScript – the key programming language, alongside HTML, that is interpreted by the browser you are reading this on. The second is that JSON is an extremely flexible way of representing data.

However, as someone who comes from a statistics and data background, as opposed to a technology background, JSON can take a while to get used to. The way data is represented in JSON is very different to the traditional tables of data that most people are used to seeing. Gone are the columns and rows, replaced with key-value pairs and lots of curly brackets – “{“ and “}”. If you are interested in seeing what it looks like, there are numerous CSV to JSON convertors online. This one even has a sample dataset to play with.

If you do bother to take a look at some JSON, you will note that it is also much more verbose than your standard tabular format. A table containing 10 columns by 30 rows – something that could easily fit into one screen on a spreadsheet – runs to 300+ lines of JSON, depending on how it is structured. That does not make it easy to get an overview of the data for a human reader, but that overlooks what JSON is designed for – to be read by computers. The fact that a human can read it at all is seen as one of JSON’s strengths.

For those interested in working with data (or any web based technology), knowing how to read and manipulate JSON is becoming as important as knowing how to use a spreadsheet.

3. Free Tools are Great

There are some people working for software vendors who will read this and be happy I have a very small audience. Having worked in the public sector, for a large corporate and now for a small NGO, one thing I have been pleasantly surprised by in 2015 is the number and quality of free tools available online.

For general office administration there are office communicator applications (Slack), task management tools (Trello) and Google’s free replacements for Excel, Word and PowerPoint. For version control and code management there is GitHub. For data analysis, the aforementioned Python and R are both free and open source. For data storage, there is a huge range of free database technologies available, in both SQL (PostgreSQL, MySQL, SQLite3) and NoSQL (MongoDB, Redis, Cassandra) variations.

To be fair to my previous larger employers and my software-selling friends, most of these tools/applications do have significant catches. Many operate on a ‘freemium’ model. This means that for individuals and small organizations with relatively few users, the service is free (or next to free), but costs quickly rise when you need larger numbers of users and/or want access to additional features, typically the types of features larger organizations need. Many of the above also provide no tech support or guarantees, meaning that executives have no one to blame if the software blows up. If you are responsible for maintaining the personal data of millions of clients, that may not be a risk you are willing to take.

For small business owners and entrepreneurs however, these tools are great news. They bring down barriers to entry for small businesses and make their survival more dependent on the quality of the product rather than how much money they have. That is surely only a good thing.

4. Blogging is a Full Time Job

Speaking of starting a business, a common dream these days is semi-retiring somewhere warm and writing a blog. My realization this year from running a blog (if only part time) is just how difficult it is to get any traction. Aside from being able to write reasonably well, there are two main hurdles that anyone planning to become a full time blogger needs to overcome – note that I have not come close to accomplishing either of these:

  1. You have to generate large amounts of good quality content – at least 2-3 longer form pieces a week if you want to maintain a consistent audience. That may seem easy, but after you have quickly bashed out the 5-10 article ideas you have been mulling over, the grind begins. You will often be writing things that are not super interesting to you. You will often not be happy with what you have written. You will quickly realize that your favorite time is the time immediately after you have finished an article and your least favorite is when you need to start a new piece.
  2. You will spend more time marketing your blog than writing. Yep, if you want a big audience (big enough to generate cash to live on) you will need to spend an inordinate amount of time:
    • cold emailing other blogs and websites, asking them to link to your blog (‘generating back links’ in blogspeak)
    • ensuring everything on your blog is geared towards your blog showing up in peoples’ Google search results (Search Engine Optimization or SEO)
    • promoting yourself on Facebook
    • building a following on Twitter
    • contributing to discussions on Reddit and LinkedIn to show people you are someone worth listening to, and
    • writing guest blogs for other sites.

None of this is easy. Begging strangers for links, incorporating ‘focus words’ into your page titles and headings, posting links on Facebook to something you spend days writing, only to find you get one like (thanks Mum!). Meanwhile, some auto-generated, barely readable click-bait trash from ‘viralnova’ or ‘quandly’ (yes, I am deliberately not linking to those sites) is clocking up likes in the 5 figures. It can be downright depressing.

Of course, there are an almost infinite number of people out there offering their services to help with these things (I should know, they regularly comment on my articles telling me how one weird trick can improve my ‘on page SEO’). The problem is, the only real help they can give you is adding more things to the list above. On the other hand, if you are thinking about paid promotion (buying like’s or a similar strategy) I’d recommend watching this video:

Still want to be a blogger? You’re welcome.

5. Do not be Afraid to Try New Things

One of the things that struck me in 2015 is how attached people get to doing things a certain way. To a large degree this makes sense, the more often you use/do something, the better you get at it. I am very good at writing SQL and using Excel – I have spent most of the last 10 years using those two things. As a result, I will often try to use those tools to solve problems because I feel most comfortable using them.

Where this becomes a problem is when you start trying to shoehorn problems into tools not just because you are comfortable with the tool, but to avoid using something you are less comfortable with. As you have seen above, two of the best things I learned this year were two concepts that were completely foreign to a SQL/Excel guy like me. But that is part of what made learning them so rewarding. I gained a completely new perspective on how data can be structured and manipulated and, even though I am far from an expert in those new skills, I now know they are available and which sorts of problems they are useful for.

So, do not be afraid to try new things, even if the usefulness of that experience is not immediately apparent. You never know when that skill might come in handy.

 

Happy New Year to everyone, I hope you have a great 2016!

 

[1] Or ‘Autonomous Province’ depending on your political views

[2] R is another programming language designed specifically for statistical analysis, data manipulation and data mining.

Traffic Accidents Involving Cyclists in the ACT

I’ve had a few days off lately and I decided to try something a bit different. Instead of writing an(other) lengthy article, I thought I would go back to my roots and actually look at some data. To that end I recently discovered a website for open data in Australia, data.gov.au. This website has literally thousands of interesting datasets released from all levels of government, covering everything from the tax bills of Australia’s largest companies to the locations of trees in Ballarat.

One of the first datasets that caught my eye was one published by the Australian Capital Territory (ACT) Government on traffic accidents involving cyclists. For those that don’t know, Canberra (the main city in the ACT) is a very bike friendly city and is home to a large number of recreational and more serious cyclists, so seeing where the accidents were/are occurring was something I thought would be interesting.

Using a few new things I have not used before (primarily Mapbox and leaflet.js), I put (slapped?) together an interactive map that uses the data provided and also gives you a few different ways of viewing it. The full version of the map can be accessed by clicking the picture below:

cyclist-map

 

See a bug? Found it particularly useful? Hate it? Leave a comment below!

Should the Wealthy be able to pay for Better Healthcare?

Commenting on an article on reddit.com, I recently got into an argument[1] with someone about healthcare and more specifically the role of private healthcare. The article was this NY times piece that talks about how US hospitals provide a range of benefits for wealthier ‘clients’ (at significant additional cost of course). These benefits can be anything from nicer rooms to gourmet food and access to business centers.

My first reaction to the piece was, what I expect, the desired response – indignation. In a country like the US where there are countless healthcare horror stories (the story of a carpenter having to choose which fingers to reattach as covered in Sicko is particularly famous), this seems outrageous. How can some people not afford access to healthcare at all, and yet others are paying huge sums to stay in private rooms and eat soft cheeses?

I believe in my case, this sense of indignation was particularly strong because I come from one of the many non-US developed countries in the world with a basic but functioning universal healthcare system. No one avoids going to hospital for fear of being bankrupted by the cost. No one has to make horrible decisions about which appendages to reattach. The only major drawback in most universal healthcare systems is procedures that are non-life threatening can have significant wait times.

A good example of this is getting surgery to repair an ACL. You can get it done through the public health system (Medicare in Australia), free of charge – or close to free. However, because you are not going to die from a ruptured ACL, you are likely to have to wait for 1-2 years to get that surgery done through Medicare. If, on the other hand, you have something like $5,000-$10,000 you can have it done next week[2].

As you may have observed from this example though, this sounds very close to what I was getting all indignant about in the first place – wealthy people buying access to better healthcare. In fact, in most universal healthcare systems, including Australia’s, the wealthy do have the option to pay more to receive access to better care and/or skip to the front of the queue. In reality, the NY Times article could easily have been talking about Australian hospitals. What is more, the ability of richer patients to pay for better service is often viewed as necessary for the system in Australia – the extra money paid by wealthy patients helps to fund the system for others. So why does it feel different?

After several days of mentally dissecting this issue I think I have come to a conclusion as to why the NY Times story got such a reaction out of me and yet I had a generally positive impression of the private health system in Australia. The key difference (at least in my mind) is the extent of the privatization of the healthcare system. In the US, healthcare has been privatized to such an extent that some people have been priced out of the market completely. When this is contrasted with the opposite end of the spectrum – private rooms, nicer robes, lobster stuffed with tacos – it highlights that the problem with the system is not an overall lack of resources, but that those resources are being allocated in such a way that some people do not get access.

Contrast this to the existence of private health systems in countries with universal healthcare. Even though some patients are able to access better facilities (and potentially doctors), everyone has access to a (generally) good level of healthcare, regardless of wealth or insurance policy. Because of this, the fact that some people can pay extra for nicer rooms seems much less important. The system has enough resources for everyone – so it is not perceived as resources being taken from poorer patients.

However, it is worth asking the question of whether this is right or simply a convenient piece of logic.

To assess the morality of the wealthy having the ability to purchase better healthcare services, we have to recognize the two main constraints on a healthcare system. The first constraint is the supply of personnel, equipment and medical supplies. The second constraint is the supply of money. These constraints are not unrelated. An endless supply of money will not help if there is a shortage in equipment/personnel at a given point in time. But money can help to increase the supply of these things in the future.

If we accept the premise that wealthy patients benefit healthcare systems by adding additional money into the system, going back to the constraints above, we can see that essentially this is a short term sacrifice for a longer term gain. Assuming that the demand for healthcare will always exceed supply, wealthy patients skipping to the front of the queue will take resources away from poorer patients in the present. They occupy beds, take time away from doctors and require access to equipment just like any other patient. However, they also pay money into the system that allows future patients to access treatment they might not otherwise have had access to.

Here is where it gets a bit murkier. If we are saying that payments from wealthy patients are needed for the system to function in the future, are we not then implying that the system is underfunded? Why can that money not come from other sources such as higher tax rates or lower spending in other areas of the budget? The problem with that line of thinking is that in any realistic government budget, there will always be room for additional healthcare funding. No government is ever likely to fund a healthcare system to the point that everyone gets the best possible treatment instantaneously[3]. So even in a much better funded public health system than currently exists in most countries, additional funds provided by wealthy patients will still allow for better treatment of other patients in the future.

All this does not mean we have to like the US model of healthcare where money plays far too big a role for the comfort of many. Denying patient access to healthcare (or bankrupting them for emergency care) in a modern developed country is a deplorable situation. But my overall conclusion is that it is best to focus your indignation on the real issues with the system – the excessive insurance premiums, the tying of affordable insurance to employment, the huge markups charged by many hospitals and the unnecessary expensive treatments added to patients bills.

As outrageous as it seems to picture wealthy patients receiving lavish treatment in private rooms while others are avoiding necessary treatment for fear of the cost, it is not the real issue. In fact it is probably providing a net benefit in a deeply flawed system.

 

[1] Those that know me will find this very unsurprising

[2] This cost, it should be noted, is still a fraction of the $55,000+ my health insurance company paid for that procedure in the US.

[3] If that was the case, you would also have idle resources for much of the year

The Darker Side of Meritocracy

Meritocracy. An ideal world where everyone is rewarded based on his or her individual qualities. The intelligent and the hardworking become the rock stars, the lazy and ignorant are doomed to a life of mediocrity. But would a true meritocracy be as idyllic as it sounds?

As was covered in Part I, there are a number of problems with defining and identifying merit. In Part II, we are going to overlook these issues and imagine what a real meritocracy might look like, and why, despite what they might say, the vast majority of people actively undermine meritocracy on a regular basis.

What Would a True Meritocracy Look Like?

Assuming we have some agreed upon way to define and identify merit, what would a true meritocracy look like? For a pure meritocracy (i.e. one in which the success of a given person is solely determined by their own actions and intelligence) to exist, each individual’s merit needs to be determined solely by his or her own individual quality.

The problem with this is, in the real world, parents have a huge influence on a child’s chances of success. This influence comes in an infinite number of forms, but includes intangible things like advice, help with homework, introductions to influential people, and being a positive role model, as well as tangible resources such as money and access to the best schools.

If parents have such a large influence on the success (or failure) of their children, how can a true meritocracy exist? Realistically, to achieve a true meritocracy, the government (or some independent body) needs to equalize parents’ influence on their child.

This equalization can take two forms. The first form is providing resources and assistance to less well off parents to try bring them up to level of parents in the upper classes. This typically includes things like welfare payments, subsidized/free health care and housing assistance, but also includes scholarships and other programs offered to help disadvantaged kids.

The second form of equalization is typically more controversial and involves reducing the ability of upper class parents to provide advantages to their children. These types of measures are far more rare, but they do exist – policies such as inheritances taxes and the removal or restriction of private schools[1] are two examples.

The reason this second type of measure is so rare is because it starts to reveal the underlying tradeoff. The tradeoff being that ensuring everyone gets the same quality upbringing means that, for some children, the quality of their upbringing has to decrease.

But even if we were willing to accept more extreme policies, they can only realistically go so far. No government can legislate to ensure every child is read to at night, and nor can they implement a ban on reading to children to make sure no child gets an advantage. No government can legislate away deadbeat Dads or Moms that get drunk in front of the kids. Which means that if you are going to create a true meritocracy, there is only really one option – take the parents out of the picture completely. This is where things start to get a little scary.

To guarantee every child receives the exact same upbringing and education, the government (or some independent body) would need to remove parents from their children’s lives. This could take various forms. A Logan’s Run style scenario where everyone is ‘terminated’ at age 30 – essentially creating a nation of orphans is one potential option. Another would be taking children at birth and raising them in industrial scale nurseries and boarding schools out of reach of parents, somewhat akin to Aldous Huxley’s Brave New World (without the presence of castes or the extreme social conditioning).

There a numerous ways that one can envisage removing a parents’ influence from their children, but the difficulty is imagining one that does not sound like a good premise for a movie about a dystopian future. In fact, the options are so unappealing that even the most repressive and extreme regimes in history have shied away from this kind of intervention.

If this is what a society would need to do to implement a true meritocracy, are there at least some upsides?

A Fairer System?

One of the key arguments made for meritocracy is that it is a ‘fairer’ system. But is it fairer (whatever that means), or are we simply replacing one lottery with another?

The current system is one in which your future success is dictated by some combination of who your parents and/or role models are (‘nurture’) and your own individual abilities (‘nature’). A true meritocracy, as we have been describing it, is simply a system in which the ‘nurture’ component has been standardized.

Is that actually fairer though? There will still be winners and losers, but now the people born with a dud genetic hand are probably worse off then in our current less meritocratic world. Unlike the current world we live in, there is no chance that a superior work ethic instilled by charmingly humble parents will get someone ahead. There are no inspiring stories of underdogs beating their better-credentialed rivals through pure determination. Rocky Balboa never even gets to fight against Apollo Creed. In a true meritocracy, the favorite always wins – that is the point of system.

A Better System?

By ensuring that the best and brightest are the ones that rise to the most influential positions, are we at least guaranteeing the fastest possible rate of progress for humanity? The answer to that question depends on how you believe progress is made.

Someone who believes that progress is only really made by rare transformative geniuses, like Einstein and Hawking, should be in favor of a more meritorious society. The risk is that a genius will be born to bad parents or in the wrong country, and as a result, that genius is wasted and substantial progress is forgone. To minimize the risk of this happening, a rational person should be willing to sacrifice certain freedoms (through more government intervention) to make sure that fewer geniuses are ‘wasted’.

On the other hand, if a person believes that progress is made by the cumulative effort of many, many intelligent (but not unique people), they should not be so worried about a true meritocracy. In this case, the loss of some geniuses to bad upbringings and poverty is much less consequential as they will be replaced by other equally or slightly less intelligent people. Maximizing the overall level of child welfare should be the priority, which, to most people, would mean allowing parents to raise their own children as far as possible.

Saying versus Doing

Stepping away from the theoretical, there is a lot that can be learned about people’s preferences in regard to meritocracy by simply looking at their actions in our world today. There is an Italian proverb that I enjoy reciting from time to time to make myself sound intelligent:

“Between saying and doing, many a pair of shoe is worn out”

Aside from the aforementioned reason, I bring this up now because people’s actions often reveal their true preferences much more accurately than their words. This is particularly true when it comes to meritocracy. In my experience, there are few people that do not actively attempt to give themselves (or those they care about) some advantage over others, and even fewer that would not take advantage of an opportunity that was presented to them.

A common example is private schools. These schools, by definition, are unmeritorious. Their business model is that parents will pay money (often large amounts of it) to send their children to a certain school exactly because they believe it will provide their child with an advantage over other children that don’t go to that school. If they did not believe it provided their children with an advantage, no rational parent would pay to send their child there.

Inheritances, giving someone a job because you know them, private tutors, moving to a better (i.e. more expensive) school district, helping out the kids with homework or even reading to them at night are just some of the endless ways that everyone, myself included, undermines a true meritocracy.

Summary

Despite the platitudes and mainstream acceptance, a true meritocracy is not what we really want as a society. Any serious thought on the subject quickly reveals a true meritocracy it is all but impossible to implement, and if implemented, the reality would be a dystopian world worthy of a George Orwell novel.

However, once the realization is made that a true meritocracy is impossible and undesirable, the remaining conclusion is that no one is truly arguing for or against meritocracy, everyone is simply arguing for a different shade of grey. The introduction and removal of various policies simply makes that shade slightly darker or lighter.

This is an important conclusion because it changes the perspective of the argument. There is no right vs. left, haves vs. have nots, good vs. evil. There is just people arguing for incremental changes. Each country, with every election, is simply working out what shade of grey they prefer.

 

[1] Many countries in Europe do not have private school systems, including education pinup nation Finland.

The Dark Side of Meritocracy

In recent years, discussion about economic concepts like inequality and income mobility has been everywhere. Thrust into the spotlight by the global financial crisis, they have rarely left the front pages thanks to Thomas Piketty’s Capital in the Twenty-First Century and a series of rolling financial crises in Europe. These days you can’t even enjoy your artisanal quail egg omelet and fair trade coffee without some bearded, tweed wearing, artisanal whiskey distilling, overgrown trust fund baby complaining about how unfair it all is, in between cashing rent checks from his parents.

When it comes to discussions of inequality though, one of the underlying assumptions that few are willing to challenge is that the drivers of inequality largely boil down to nepotism and inherited wealth, while the answer to most inequality based problems comes down to one idea: meritocracy.

What is a Meritocracy?

Meritocracy is a system in which the people who hold power (through democratically elected means or otherwise) are those that are most deserving based on individual merit. In common use, it is usually taken to be slightly broader than that – a world in which money and success are allocated, perfectly, to those who are deemed to deserve it the most.

In an increasingly polarized political system in the US, meritocracy – or ‘the right to rise’ – is often the only thing that politicians on both sides of the aisle seem to agree on. The ideal of meritocracy is so ingrained in the US that Americans are famous for their belief that hard work will be rewarded with untold wealth and success. But this belief is far from unique to the US. In Australia meritocracy has long been considered part of the national identity with politicians of all stripes often talking in jingoistic terms about ‘the fair go’.

For all the talk of meritocracy though, how feasible is it in the real world? What would be some of the major hurdles to implementing a more meritocratic system?

Defining Merit

The first question that should arise whenever meritocracy is discussed is how is merit defined? There are 4 basic criteria that most commonly are thought of as contributing to merit:

  1. Qualifications
  2. Work ethic
  3. Intelligence
  4. Experience

For almost all competitions where there are winners and losers – jobs, university positions or other – some combination of these traits will generally be used to decide a winner. To keep things simpler, let’s focus on the job market for now.

The first thing to consider when defining merit for a given job is that to have an accurate measure of merit, the criteria need to be modified for every position. Jobs requiring manual labor place a higher value on work ethic but little value on qualifications. Jobs in tech often place higher value on intelligence, but less on formal qualifications and, depending on the role, large amounts of experience can be seen as detrimental. Most jobs will require applicants to possess experience in one or more specific areas.

For the most part, this customization of criteria for each job is already being done – a job advertisement is essentially a statement of the criteria that merit will be assessed by. But the question is, are those criteria actually the correct ones to identify the best possible person for a given job? I believe the answer to that question is a resounding “no”. Let me explain why.

Let’s look at a common example that anyone who has tried changing sector, industry or country in his or her career will be able to relate to.

Imagine you have been working for around 10-15 years and have spent all of that time in one industry[1]. During that time you have picked up a lot of useful workplace skills, spreadsheets, experience with various applications, writing skills, general how-not-to-piss-everyone-off skills and so on. Now you want a new challenge that will require a lot of the skills you have, but in a different industry. You approach a recruiter, bright eyed and excited by the possibilities, but despite your best efforts to sell your skills as relevant, the recruiter basically discards your experience as worthless and tries to push you towards a low level role or something in your old industry.

This experience is a simple example that reveals an underlying truth – if we were being truly meritorious, there could be no fixed criteria for merit for any job because there is no way to preemptively identify what combination of skills and experience will ultimately prove to be the most valuable.

The possibilities for what combination of skills and experience lead to the best performance in a role are endless. Many successful business owners do not have MBAs. Many of the best investors on Wall Street do not come from finance backgrounds. Some of the best NFL punters are ex-Australian Rules Football players. What people who excel tend to have in common is a combination of skills and experiences that allows them to bring a different perspective to a problem.

Yet, despite history proving time[2] and time again[3] that different perspectives are often vital to important insights, it is a rare employer or recruiter that will take a bet on a candidate with ‘unusual experience’ rather than a candidate who ticks all the boxes. The reason for that is simple – it is safer. Choosing the candidate that ticks the boxes provides cover (“I gave you what you asked for”) and it gives a better guarantee of an acceptable level of performance. The unusual candidate could be fantastic – but they could also be a complete flop who turns out to be way out of their depth.

Unfortunately, this is only the first hurdle for a true meritocracy – if merit is a difficult thing to define, it is an even more difficult thing to measure.

Measuring Merit

Once we get past the step of deciding what criteria will determine the most deserving applicant, the next step is deciding who best meets those criteria. A quick look at the application process for college admission or a technical job will give you an instant appreciation for the lengths that people will go to try and get an accurate assessment of an applicant’s true merit.

Tests, interviews and essays are probably the most common tools used to assess merit but all can be (and are) gamed by people who understand the system. Material for tests can be rote learned with little to no understanding necessary. Interviews are notorious for being poor predictors of talent, which makes sense when you consider that the most confident people are often delusional. Essays, aside from providing evidence of basic writing skills, are assessed subjectively.

Even if these tools for assessing merit were designed in such a way as to prevent gaming the system, these are still three very narrow tests of ability. As Megan McArdle explains, the experience in China shows that selecting for people who do well on exams gives you… a selection of people who do well on exams.

Assumptions and Prejudice

One interesting side effect of the difficulty in determining merit is it leads to people basing their assessment on completely superficial qualities (at least partially). A good dress sense, physical attractiveness, and being an eloquent speaker are just some examples of relatively superficial qualities people use to assess intelligence and merit. As frustrating as this can be for the unshapely, poorly dressed, mumblers out there, these are all things that can be improved and worked on (at least to some degree). Others are subject to prejudices that cannot be addressed – the impact of race on the ability to get interviews, for example, is well established.

Another concerning trend is the increasing use of someone’s current level of success/wealth as an indicator of merit. That is, if someone is wealthy and/or successful, they must be someone who is highly intelligent and works harder than everyone else. This line of thinking is dangerous for two reasons:

  1. Too much value is placed on the opinions of wealthy and successful people – particular on topics outside their domain. Anyone who has listened to Clive Palmer or Donald Trump speak should know that is a mistake.
  2. The implicit assumption made when you believe wealthy and successful people are fully deserving of their place in the world is that anyone who is poor and unsuccessful is also fully deserving of their situation.

Evidence of this thinking is present everywhere to some degree, but seems particularly prevalent in the US[4], where TV shows like Shark Tank are extremely popular and prominent CEOs are regularly asked for their opinions on public policy issues.

This belief system can largely be explained as the flip side of the optimistic view American’s have of their economic prospects. As this paper from the Brookings Institution highlights, American’s are far more likely to believe hard work and intelligence will be rewarded and yet are second only to the UK in terms of how closely correlated a son’s earnings are to his fathers (i.e. hard work has the least chance of improving your situation). If you truly believe that hard work and intelligence is all you need to be successful, you must also believe that people who are currently experiencing success have those attributes.

Best Person for the Job

Going back to our problems with creating a meritocracy, everything discussed so far has overlooked a key factor in this endless quest to find the most deserving – people are not cogs that can be simply transferred in and out of a machine seamlessly. The person who ‘deserves’ the job on merit (provided we can define it and measure it accurately) is often NOT the best person for the job. The best person for the job is often determined by qualities such as:

  • how that person fits in with the team culture,
  • their personality type, and
  • how they respond to authority (or the lack thereof).

These traits are all key factors in how well someone will perform in a given role and yet none would typically be thought of as meritorious qualities.

This realization is not new. Employers and hiring firms have been pushing the idea of the ‘beer test’ (asking yourself which of the candidates would you most like to go for an after-work beer with) for some time now. But it does beg the question – what would happen if a company simply hired the ‘best’ candidates for each position without considering whether these people will work well together? Would that team be more productive than a team that hired less ‘deserving’ candidates but aimed to build a harmonious work place? The entire body of management knowledge (and every buddy cop movie ever made) would tell us otherwise.

Summary

One thing that becomes obvious when you start thinking about how a true meritocracy would actually work is how difficult it would be to implement:

  • Many of the criteria we associate with someone being deserving of a role or position are subjective or exclude applicants who would in fact be far superior.
  • Our methods of assessment are often deeply flawed, subject to gaming and our own prejudices.
  • Selecting the most objectively deserving candidates is not guaranteed to provide the best results anyway.

Yet, despite the reasons above hopefully being enough to give pause the next time someone begins expressing frustration with the current lack of meritocracy, all of these issues are only really logistical problems.

There is a good argument to be made that we can and should try to improve on all of these things – that we should aim to get better at identifying the right people and refine our methods of assessing skills. We should pay more attention to team fit and personalities when selecting the best candidate. It is hard to imagine the world being a worse place if employers were more open minded about what skills might be valuable to their company and sociopaths were less likely to impress during an interview.

However, the next question is how far should we take this. What does a truly meritocratic society look like and is it something we really want? That is the subject of discussion in Part II in this series on meritocracy.

 

[1] You can easily replace ‘industry’ in this story with ‘company’, ‘field’ or ‘country’.

[2] Einstein famously worked at a patent office where his work often exposed him to the transmission of electric signals and electrical-mechanical synchronization of time. Exposure to these topics helped him to arrive at his conclusions about the nature of light and the connection between space and time.

[3] Steve Jobs often talked about the importance of a calligraphy class he took in shaping what fonts were best used in operating systems.

[4] Australia’s proclivity to tall poppy syndrome does have some positive side effects

Climbing Mount Delusion – The Path from Beginner to Expert

In our careers there are various skillsets that we will be required to develop over time. Whether that is carrying multiple plates at a time, while working in a restaurant, or something more technically challenging, such as learning a programming language or learning to write good. Regardless of the skillset, there is always a learning curve that must be conquered.

It is tempting to think of this learning curve as a steady slope where knowledge is accumulated over time, or perhaps a steep initial slope that flattens out. In my experience though, this is rarely the case. I believe there is a reoccurring pattern in the way most people move from beginner to expert in a given subject, with distinct phases. What’s more, I believe many others will identify with these phases.

If you do identify with these phases, you will also realize there are risks that emerge at different times, and that being aware of those risks can help you avoid them. These risks typically occur where a persons’ belief in their mastery of a given subject diverges from their actual abilities. Sometimes it will be a lack of confidence that causes more experienced people to not speak up when they should. Other times, the person will exhibit there far too much confidence relative to their knowledge. The latter case is so common it has become cliché: A little bit of knowledge is a dangerous thing.

To help illustrate the various phases of the journey from beginner to expert, I am going to tell the story through a fictional character, Fred, who is learning Economics.

1. Initial Optimism

md_pic_1

When Fred first begins to learn economics, he has a large burst of excitement. Everything is new and interesting, he is learning new ways to think about problems, and he can’t seem to get enough. He knows very little about the subject but is enthralled with how quickly he is absorbing all this new information, and how quickly the pieces seem to be fitting together.

For Fred though, the best part is that he can clearly see the point at which he believes he will be an expert.

2. The Summit of Mount Delusion

md_pic_2

Finally, after months/years of working hard, Fred reaches the peak of Mount Delusion. He finishes his degree and he can feel the knowledge coursing through him. Fred loves spending hours enlightening his friends and family about the intricacies of interest rate policy and why minimum wage increases are wrong headed. He feels great. He set out to master something and did it. Already his mind is turning to what is next on the list of topics to master.

The problem with standing on the summit of Mount Delusion is the fog often blocks the view.

Fred, like many who have stood on the summit of Mount Delusion, espouses advice without realizing the risks of that advice. He provides clear, unambiguous recommendations because he lacks the experience and/or knowledge to realize what caveats are needed. Ironically, this often makes Fred all the more convincing to his colleagues. While the true experts are hedging their responses, Fred is completely convinced option 1 is the best. People like decisiveness and, as a result, they like and trust Fred.

3. The Clearing of the Fog of Ignorance

md_pic_3

For Fred (and most people), a moment comes when the fog clears. Someone who is much further along this journey than Fred clears the fog completely unintentionally. With an innocuous comment and a simple question, this person – who does not even regard themselves as an expert – completely shakes Fred’s confidence to the core. For a horrible moment, Fred is left looking out over the vast expanse of knowledge and concepts he had not even known existed until 30 seconds ago. All the knowledge and experience accumulated to that point only seems to highlight how little he really knows. From here, it is a long way down …

It should be noted at this point that, for some people, the fog never clears. They simply lack the level of self-reflection required to ever critically review their performance and continue their development. They will go through life claiming they are an advanced user of X or an expert on Y without ever realizing just how misguided they are. To be frank, these people are often some of the most dangerous in the workplace.

4. The Valley of Self-Pity

md_pic_4

After that horrifying moment when the fog cleared, our former expert Fred was left in a depressed state. His mind continually racing through all the times he fearlessly dispensed his advice, advice he now realizes was off base or often just completely wrong. What’s worse, he now realizes that anyone with any real knowledge could have identified him as a fraud based purely on that misguided advice. In short, he feels amazingly stupid.

He revises his resume, removes all words like “advanced” and “expert” and prays his ill formed advice doesn’t come back to haunt him. People who used to rely on Fred for unambiguous advice are completely mystified as to what happened. Where did his confidence go? They will speculate about what happened but most will never really realize the truth.

At this point in the learning process, there are two main risks. The first is that Fred gives up on economics altogether. In his depressed state, he feels like he is back at square one. He views his own skills as trivial and meaningless, while over valuing the skills of others. Many people will never exit the valley of self-pity for this reason.

The second risk is that, in this state, Fred (and people like him) begins to significantly undersell his expertise. He defers decision making to those around him, even though in many instances he will be much better placed to make decisions.

5. Exiting the Valley

After what feels like the world’s longest meal of humble pie, some strange things start happening to Fred.

Firstly, he will start bumping into people who are still standing on the peak of Mount Delusion. He will identify them, because, despite their claims of being experts, he knows significantly more than them. He will realize that they do not even realize what they do not know yet, exactly like he did, not so long ago. This provides comfort because he realizes he is unlikely to be the first or last person to fall from the top of Mount Delusion. In fact, compared to some of the people he is now meeting, he was amazingly restrained.

Secondly, Fred will meet people who didn’t study economics and realize that skills and knowledge that, in the Valley of Self-Pity, he assumed everyone had are, in fact, exceedingly rare. Fred will realize that many of the basic skills he has are actually not so basic and are quite valuable.

With each of these encounters, Fred’s confidence begins to recover. He will remain painfully conscious of how much he still has to learn, but for the first time since this journey began, his actual knowledge level and his assumed knowledge level will come into alignment.

6. The Never-ending Slope of True Mastery

md_pic_5

Fred is finally on a sustainable path. He has acquired a large amount of knowledge and experience, but is fully aware of the limits of his knowledge. He has revised his resume again to include words like “advanced” and “expert”, but now seeks to play these down.

He continues to run into many people standing on the summit of Mount Delusion, but mostly just feels sorry for them – most have a large and embarrassing fall coming, and many will not recover from it. He attempts to coach these people where possible, to help lessen the pain from their fall. Some take his advice, some do not.

How I Can Relate to Fred

In my own life, I have taken the journey to the summit of Mount Delusion several times. With each subsequent visit I have learned to be more cautious, to pay more attention to people who have more experience than I do, but the scars of previous falls remain.

From SQL to Excel, writing blogs to learning Spanish, there has always been a specific depressing moment when the illusion of expertise disappeared and only a sense of inadequacy remained. I would always recover and continue to build knowledge (I have a reputation for being a little stubborn), but to this day, the words “expert” and “advanced user” continue to stick in the throat, the fear of being exposed as a fraud (again) always present.

So far I have been fortunate. Even my most reckless declarations and advice have only served to cause personal embarrassment rather than any significant damage to my career. It could have been so much worse.

To those that are beginning the journey, my only advice is to remain humble. To those that have already endured a fall or two, don’t give up. The world will be a better place for your continued contributions.

« Older posts Newer posts »

© 2021 Brett Romero

Theme by Anders NorenUp ↑