Data Inspired Insights

Tag: advice

5 Things I Learned in 2015

2015 has been an interesting year in many respects. A new country[1], a new language, a new job, and plenty of new experiences – both at work and in life in general. To get into the year-end spirit, I thought I would list out 5 key things I learned this year.

1. I Love Pandas

Yes, those pandas as well, who doesn’t? But I knew that well before 2015. The pandas I learned to love this year is a data analysis library for the programming language Python. “Whoa, slow down egg head” I hear you say. For those that are not regular coders, what that means is that pandas provides a large range of ways for people writing Python code to interact with data that makes life very easy.

Reading from and writing to Excel, CSV files and JSON (see lesson number 2) is super easy and fast. Manipulating large datasets in table like structures (dataframes) – check. Slicing, dicing, aggregating – check, check and check. In fact, as a result of pandas, I have almost entirely stopped using R[2]. All the (mostly basic) data manipulation for which I used to use R, I now use Python. Of course R still has an important role to play, particularly when it comes to complex statistical analysis, but that does not tend to come up all that regularly.

2. JSON is Everywhere

JSON, JavaScript Object Notation for the uninitiated, is a data interchange format that has become the default way of transferring data online. Anytime you are seeing data displayed on a webpage, including all the visualizations on this website, JSON is the format the underlying data is in.

JSON has two big advantages that have led to its current state of dominance. The first is that, as the name suggests, it is native to JavaScript – the key programming language, alongside HTML, that is interpreted by the browser you are reading this on. The second is that JSON is an extremely flexible way of representing data.

However, as someone who comes from a statistics and data background, as opposed to a technology background, JSON can take a while to get used to. The way data is represented in JSON is very different to the traditional tables of data that most people are used to seeing. Gone are the columns and rows, replaced with key-value pairs and lots of curly brackets – “{“ and “}”. If you are interested in seeing what it looks like, there are numerous CSV to JSON convertors online. This one even has a sample dataset to play with.

If you do bother to take a look at some JSON, you will note that it is also much more verbose than your standard tabular format. A table containing 10 columns by 30 rows – something that could easily fit into one screen on a spreadsheet – runs to 300+ lines of JSON, depending on how it is structured. That does not make it easy to get an overview of the data for a human reader, but that overlooks what JSON is designed for – to be read by computers. The fact that a human can read it at all is seen as one of JSON’s strengths.

For those interested in working with data (or any web based technology), knowing how to read and manipulate JSON is becoming as important as knowing how to use a spreadsheet.

3. Free Tools are Great

There are some people working for software vendors who will read this and be happy I have a very small audience. Having worked in the public sector, for a large corporate and now for a small NGO, one thing I have been pleasantly surprised by in 2015 is the number and quality of free tools available online.

For general office administration there are office communicator applications (Slack), task management tools (Trello) and Google’s free replacements for Excel, Word and PowerPoint. For version control and code management there is GitHub. For data analysis, the aforementioned Python and R are both free and open source. For data storage, there is a huge range of free database technologies available, in both SQL (PostgreSQL, MySQL, SQLite3) and NoSQL (MongoDB, Redis, Cassandra) variations.

To be fair to my previous larger employers and my software-selling friends, most of these tools/applications do have significant catches. Many operate on a ‘freemium’ model. This means that for individuals and small organizations with relatively few users, the service is free (or next to free), but costs quickly rise when you need larger numbers of users and/or want access to additional features, typically the types of features larger organizations need. Many of the above also provide no tech support or guarantees, meaning that executives have no one to blame if the software blows up. If you are responsible for maintaining the personal data of millions of clients, that may not be a risk you are willing to take.

For small business owners and entrepreneurs however, these tools are great news. They bring down barriers to entry for small businesses and make their survival more dependent on the quality of the product rather than how much money they have. That is surely only a good thing.

4. Blogging is a Full Time Job

Speaking of starting a business, a common dream these days is semi-retiring somewhere warm and writing a blog. My realization this year from running a blog (if only part time) is just how difficult it is to get any traction. Aside from being able to write reasonably well, there are two main hurdles that anyone planning to become a full time blogger needs to overcome – note that I have not come close to accomplishing either of these:

  1. You have to generate large amounts of good quality content – at least 2-3 longer form pieces a week if you want to maintain a consistent audience. That may seem easy, but after you have quickly bashed out the 5-10 article ideas you have been mulling over, the grind begins. You will often be writing things that are not super interesting to you. You will often not be happy with what you have written. You will quickly realize that your favorite time is the time immediately after you have finished an article and your least favorite is when you need to start a new piece.
  2. You will spend more time marketing your blog than writing. Yep, if you want a big audience (big enough to generate cash to live on) you will need to spend an inordinate amount of time:
    • cold emailing other blogs and websites, asking them to link to your blog (‘generating back links’ in blogspeak)
    • ensuring everything on your blog is geared towards your blog showing up in peoples’ Google search results (Search Engine Optimization or SEO)
    • promoting yourself on Facebook
    • building a following on Twitter
    • contributing to discussions on Reddit and LinkedIn to show people you are someone worth listening to, and
    • writing guest blogs for other sites.

None of this is easy. Begging strangers for links, incorporating ‘focus words’ into your page titles and headings, posting links on Facebook to something you spend days writing, only to find you get one like (thanks Mum!). Meanwhile, some auto-generated, barely readable click-bait trash from ‘viralnova’ or ‘quandly’ (yes, I am deliberately not linking to those sites) is clocking up likes in the 5 figures. It can be downright depressing.

Of course, there are an almost infinite number of people out there offering their services to help with these things (I should know, they regularly comment on my articles telling me how one weird trick can improve my ‘on page SEO’). The problem is, the only real help they can give you is adding more things to the list above. On the other hand, if you are thinking about paid promotion (buying like’s or a similar strategy) I’d recommend watching this video:

Still want to be a blogger? You’re welcome.

5. Do not be Afraid to Try New Things

One of the things that struck me in 2015 is how attached people get to doing things a certain way. To a large degree this makes sense, the more often you use/do something, the better you get at it. I am very good at writing SQL and using Excel – I have spent most of the last 10 years using those two things. As a result, I will often try to use those tools to solve problems because I feel most comfortable using them.

Where this becomes a problem is when you start trying to shoehorn problems into tools not just because you are comfortable with the tool, but to avoid using something you are less comfortable with. As you have seen above, two of the best things I learned this year were two concepts that were completely foreign to a SQL/Excel guy like me. But that is part of what made learning them so rewarding. I gained a completely new perspective on how data can be structured and manipulated and, even though I am far from an expert in those new skills, I now know they are available and which sorts of problems they are useful for.

So, do not be afraid to try new things, even if the usefulness of that experience is not immediately apparent. You never know when that skill might come in handy.


Happy New Year to everyone, I hope you have a great 2016!


[1] Or ‘Autonomous Province’ depending on your political views

[2] R is another programming language designed specifically for statistical analysis, data manipulation and data mining.

Climbing Mount Delusion – The Path from Beginner to Expert

In our careers there are various skillsets that we will be required to develop over time. Whether that is carrying multiple plates at a time, while working in a restaurant, or something more technically challenging, such as learning a programming language or learning to write good. Regardless of the skillset, there is always a learning curve that must be conquered.

It is tempting to think of this learning curve as a steady slope where knowledge is accumulated over time, or perhaps a steep initial slope that flattens out. In my experience though, this is rarely the case. I believe there is a reoccurring pattern in the way most people move from beginner to expert in a given subject, with distinct phases. What’s more, I believe many others will identify with these phases.

If you do identify with these phases, you will also realize there are risks that emerge at different times, and that being aware of those risks can help you avoid them. These risks typically occur where a persons’ belief in their mastery of a given subject diverges from their actual abilities. Sometimes it will be a lack of confidence that causes more experienced people to not speak up when they should. Other times, the person will exhibit there far too much confidence relative to their knowledge. The latter case is so common it has become cliché: A little bit of knowledge is a dangerous thing.

To help illustrate the various phases of the journey from beginner to expert, I am going to tell the story through a fictional character, Fred, who is learning Economics.

1. Initial Optimism


When Fred first begins to learn economics, he has a large burst of excitement. Everything is new and interesting, he is learning new ways to think about problems, and he can’t seem to get enough. He knows very little about the subject but is enthralled with how quickly he is absorbing all this new information, and how quickly the pieces seem to be fitting together.

For Fred though, the best part is that he can clearly see the point at which he believes he will be an expert.

2. The Summit of Mount Delusion


Finally, after months/years of working hard, Fred reaches the peak of Mount Delusion. He finishes his degree and he can feel the knowledge coursing through him. Fred loves spending hours enlightening his friends and family about the intricacies of interest rate policy and why minimum wage increases are wrong headed. He feels great. He set out to master something and did it. Already his mind is turning to what is next on the list of topics to master.

The problem with standing on the summit of Mount Delusion is the fog often blocks the view.

Fred, like many who have stood on the summit of Mount Delusion, espouses advice without realizing the risks of that advice. He provides clear, unambiguous recommendations because he lacks the experience and/or knowledge to realize what caveats are needed. Ironically, this often makes Fred all the more convincing to his colleagues. While the true experts are hedging their responses, Fred is completely convinced option 1 is the best. People like decisiveness and, as a result, they like and trust Fred.

3. The Clearing of the Fog of Ignorance


For Fred (and most people), a moment comes when the fog clears. Someone who is much further along this journey than Fred clears the fog completely unintentionally. With an innocuous comment and a simple question, this person – who does not even regard themselves as an expert – completely shakes Fred’s confidence to the core. For a horrible moment, Fred is left looking out over the vast expanse of knowledge and concepts he had not even known existed until 30 seconds ago. All the knowledge and experience accumulated to that point only seems to highlight how little he really knows. From here, it is a long way down …

It should be noted at this point that, for some people, the fog never clears. They simply lack the level of self-reflection required to ever critically review their performance and continue their development. They will go through life claiming they are an advanced user of X or an expert on Y without ever realizing just how misguided they are. To be frank, these people are often some of the most dangerous in the workplace.

4. The Valley of Self-Pity


After that horrifying moment when the fog cleared, our former expert Fred was left in a depressed state. His mind continually racing through all the times he fearlessly dispensed his advice, advice he now realizes was off base or often just completely wrong. What’s worse, he now realizes that anyone with any real knowledge could have identified him as a fraud based purely on that misguided advice. In short, he feels amazingly stupid.

He revises his resume, removes all words like “advanced” and “expert” and prays his ill formed advice doesn’t come back to haunt him. People who used to rely on Fred for unambiguous advice are completely mystified as to what happened. Where did his confidence go? They will speculate about what happened but most will never really realize the truth.

At this point in the learning process, there are two main risks. The first is that Fred gives up on economics altogether. In his depressed state, he feels like he is back at square one. He views his own skills as trivial and meaningless, while over valuing the skills of others. Many people will never exit the valley of self-pity for this reason.

The second risk is that, in this state, Fred (and people like him) begins to significantly undersell his expertise. He defers decision making to those around him, even though in many instances he will be much better placed to make decisions.

5. Exiting the Valley

After what feels like the world’s longest meal of humble pie, some strange things start happening to Fred.

Firstly, he will start bumping into people who are still standing on the peak of Mount Delusion. He will identify them, because, despite their claims of being experts, he knows significantly more than them. He will realize that they do not even realize what they do not know yet, exactly like he did, not so long ago. This provides comfort because he realizes he is unlikely to be the first or last person to fall from the top of Mount Delusion. In fact, compared to some of the people he is now meeting, he was amazingly restrained.

Secondly, Fred will meet people who didn’t study economics and realize that skills and knowledge that, in the Valley of Self-Pity, he assumed everyone had are, in fact, exceedingly rare. Fred will realize that many of the basic skills he has are actually not so basic and are quite valuable.

With each of these encounters, Fred’s confidence begins to recover. He will remain painfully conscious of how much he still has to learn, but for the first time since this journey began, his actual knowledge level and his assumed knowledge level will come into alignment.

6. The Never-ending Slope of True Mastery


Fred is finally on a sustainable path. He has acquired a large amount of knowledge and experience, but is fully aware of the limits of his knowledge. He has revised his resume again to include words like “advanced” and “expert”, but now seeks to play these down.

He continues to run into many people standing on the summit of Mount Delusion, but mostly just feels sorry for them – most have a large and embarrassing fall coming, and many will not recover from it. He attempts to coach these people where possible, to help lessen the pain from their fall. Some take his advice, some do not.

How I Can Relate to Fred

In my own life, I have taken the journey to the summit of Mount Delusion several times. With each subsequent visit I have learned to be more cautious, to pay more attention to people who have more experience than I do, but the scars of previous falls remain.

From SQL to Excel, writing blogs to learning Spanish, there has always been a specific depressing moment when the illusion of expertise disappeared and only a sense of inadequacy remained. I would always recover and continue to build knowledge (I have a reputation for being a little stubborn), but to this day, the words “expert” and “advanced user” continue to stick in the throat, the fear of being exposed as a fraud (again) always present.

So far I have been fortunate. Even my most reckless declarations and advice have only served to cause personal embarrassment rather than any significant damage to my career. It could have been so much worse.

To those that are beginning the journey, my only advice is to remain humble. To those that have already endured a fall or two, don’t give up. The world will be a better place for your continued contributions.

Why You Probably Don’t Need a Financial Advisor

I recently had an interesting series of conversations with my parents around investing and the world of financial advice, which encouraged me to outlay some thoughts on the subject. First things first – I am not a financial advisor, and none of this should be taken as specific investment advice. My only aim is to highlight some common mistakes people make when it comes investing. If after reading this you feel you may not be getting the best advice, your next step should be to do your own further research and/or have a more informed conversation with your current financial advisor.

Why You Probably Don’t Need a Financial Advisor

Generally speaking, financial advisors are people who provide a service to investors, helping them build, balance, manage and adjust a portfolio of assets, taking into account the prevailing economic conditions and the expectations and needs of the client. There are a multitude of scenarios where financial advisors and asset managers provide a valuable service to their clients, unfortunately, however, as Matt Yglesias points out, this is almost never for small “retail investors”, like you or me.

If this is the case, why are financial advisors still in such demand when it comes to retail investors? I believe it comes down to four factors:

  1. Unrealistic expectations on the part of retail investors
  2. Fear of complexity
  3. The belief that your financial advisor’s incentives align completely with your own, and/or
  4. A lack of understanding of compounding.

Adjusting Expectations

Your expectations may seem pretty straightforward – you want to maximize the returns on your assets. But let’s dig a little deeper – what level of returns are you looking for? For most people, the prospect of 5-7% per annum returns seems underwhelming – that’s pretty much the average for the market right? What if I want to beat the market – half the investing world is beating the market average in a given year – surely I can be one of those guys?

Unfortunately, there is no shortage of people who will tell you they can help you do exactly that. From stock pickers in your local newspaper to highly paid active fund managers on Wall St, there is an endless line of people who want to help you discover the secret of obtaining above average returns. More often than not, it is even sold as quite a reasonable thing, an opportunity that others have overlooked for some plausible sounding reason, or something that is only available to a tiny subset of investors that you happen to a part of.

The problem is not the salesmen – salesmen are gonna sell – the problem is we keep buying into the promises of above average returns, despite our own better judgment. The reality is beating the market is extremely difficult to do, particularly over any multi-year period. Even if there are advisors and money managers that have found a way to consistently beat the market, they are running a hedge fund with billions in assets, or providing advice to people and/or companies with a lot more money than you or I. They are almost certainly not working 9 to 5 at your local branch of ABC Bank.

To start thinking like an investor, instead of a gambler, the first step is to readjust your expectations. There are no shortcuts to wealth – average market returns should be your expectation. Once that is your expectation, your view on the best way to invest your money fundamentally changes. Now the question changes from “How big is the return I can get?” to “What is the lowest cost way to match the average market returns?” That is the right question to be asking.

Fear of Complexity

Fear is a tool that has been used by professionals in most fields essentially since the beginning of people doing things for money. From lawyers to auto mechanics to management consultants, they have a vested interest in making any job they do seem more complicated than it is to ensure that a) you don’t learn how to do it yourself; and b) they can charge as much as possible for their services. People working in the financial industry are no different.

That’s not to say there aren’t extremely complicated products and concepts in the financial world, but now that we have adjusted our expectations – all we want is to match average market returns – why do we need to understand these complicated products? Do you have a large exposure to Chinese Yuan that you need to hedge? Have you been creating short positions on European junk bonds that you need to cover? You can probably stop reading this if you do.

If we accept that matching the average market return is in fact a perfectly acceptable result, then there are a range of simple, understandable options available to retail investors that are only a regular brokerage account away.

The Incentive Misalignment

Despite the platitudes, a financial advisor’s primary incentive is to maximize the amount of money they make from you as a client. There is some alignment in incentives in the sense that their fees increase as your assets grow (fees are typically structured as a percentage of total assets), but the difference to your advisor between your assets growing at 5% or 7% is minimal. The real money is in finding additional pools of assets to manage. Because of this, it is much more economical for them to spend their time finding additional clients than it is for them to spend that time trying to squeeze an extra percent or two out of your portfolio. Confused? Similar incentives apply for Real Estate agents (I’m just picking fights with everyone today) as is explained very well in the following short clip.

The above misalignment is actually one of the more innocent ways in which an advisor’s incentives can diverge from your best interests. The more disturbing divergence occurs due to the opaque world of incentives and commissions. This varies widely between countries, states and even the specific type of advisor you have, but these payments can lead an advisor to recommend products and strategies that aren’t actually the best option for you[1]. This could include recommending unnecessarily complex portfolio structures, advising you to take on too little or too much risk, or even recommending funds and/or securities that the advisor actually receives commission for selling. Most people have heard about this happening in the US, but don’t believe it doesn’t happen elsewhere – take this excerpt from the Financial Services Guide for Colonial First State, an Australian Wealth Management Group (emphasis mine):

“You may receive advice in relation to the products we offer from financial advisers who do not work for Colonial First State or may be representatives of other licensees in the Bank. These advisers may receive some benefits from us. The adviser’s remuneration is included in the fees you pay when investing in our products.”

The issue here isn’t that these products are being marketed, but there is a blurring of the lines between advisor and salesman that is particularly bad in the financial industry. Again referring to Matt Yglesias – compare buying securities recommended by your advisor to buying a car: “we understand that the car salesman works for the dealership — he’s not your car advisor.”

The key point is that the only person who really cares about your money is you and you should spend as much time researching how you invest your money as you would on any other major purchase. Fortunately, there has never been a better time for investing novices to learn some of the basic concepts of investing – CNN, ASIC, Yahoo Finance and many others have beginner’s guides to investing. For those looking for something more in depth, Coursera is a fantastic resource of free courses offered by some of the worlds leading Universities. Two excellent beginners’ finance courses are currently being offered by the University of Michigan and Yale.

So instead of spending all your time online looking up Joe Pesci trivia, watching John Stewart clips on racial inequality, or researching the best toothbrush to buy, invest some time building your financial knowledge. Start with important concepts like the risk-return tradeoff and diversification, and move onto the different types of securities. Let your curiosity take you where you want… after you watch the Joe Pesci clip of course.

Underestimating Compounding

One of the big reasons so many of the injustices in the financial markets occur is because people consistently underestimate the effects of compounding. Let’s look at a simple example – the bank provides you with an asset worth $0.01, but it doubles in value every day (i.e. it would be worth $0.02 on day two, $0.04 on day three and so on) for an entire 31-day month. How much would that asset be worth at the end of the month?

If your guess had less than 7 figures, you are way off. By the end of the month, that asset would be worth over $10 million. That is the impact of compounding. Let’s look at a more relevant example for investors. Anecdotally, you will often hear people say something along the lines of the following:

X was a great investment – it doubled in price over the last 10 years.

What is the average rate of return that would cause an asset to double in value in 10 years? 7.18% per annum. Consistent 7.18% returns is nothing to sneeze at, but it is a lot less impressive than the returns sought by a lot of investors. It is also lower than the long run average return of the S&P500, which is over 9% (see Chart 1).

Chart 1 – Value of $100 Invested in the S&P500 in 1928


Ok, so leaving relatively small amounts of money invested at low rates results in a lot bigger returns than you might expect. If that is the case, it shouldn’t matter if my advisor is charging me 0.15% or 1.5%, as long as I leave it accumulating for long enough, right? Unfortunately the opposite is true, when it comes to fees, compounding works against you. Those seemingly small fees that financial advisors and intermediaries charge you for their services end up having a much bigger impact than you might expect.

Just as compounding works by exponentially increasing a value by giving us returns on our returns, the money lost through fees grows exponentially by taking away money each year that would be compounded in future years. Chart 2 shows a comparison of two $100,000 investments over 30 years assuming the long run returns of the S&P500 (9%). One investment is made in a low cost market index fund (cost 0.1% of assets) and the other in a high cost managed fund (cost 1.5% of assets).

Chart 2 – $100,000 Investment: High Cost vs. Low Cost Management


Within 5 years, the high cost fund has cost you over $10,000 more in fees and lost returns than the low cost fund – that’s over 10% of the value of your initial investment gone. The cost reaches over $30,000 by the 10-year mark, and over $135,000 by year 20.

The worst part of this, going back to the first point, is there is almost no chance that your high cost fund managed to outperform the market index fund over the course of those 20 years, and a pretty good shot it did significantly worse. At best you probably just paid $135,000 to match the average market returns… on the plus side, maybe they will take you out on their new yacht for your generosity.

What To Do?

If you understand and agree with the points made above, and if you are currently investing or are planning to invest any significant money, then what you should be looking for is something that will allow you to reproduce the market average performance at a very low cost. There is a growing number of ways to do this, but low cost managed funds and ETFs are the most accessible to most investors.

However, do not simply substitute this advice for your old financial advice. Do your own research – there is so much information out there, and the best advice is often free. Understand what the product options are, what the fees and costs are, and what returns are expected and why. Don’t be afraid to ask questions – the only dumb question is the one asked after you have lost a stack of money.


[1] The option recommended might simply be less beneficial than the best option as opposed to an option that is not in your interest at all, which would be a breach of fiduciary obligations.


Disagree with any of the above? Feel free to leave a comment below.

© 2024 Brett Romero

Theme by Anders NorenUp ↑