Brett Romero

Data Inspired Insights

7 traits of the best Data Scientists

Over the course of 10 years of working with data you tend to pick up a few things. You learn (and re-learn) languages and syntax. You make mistakes, some of them bad ones. You get better at spotting bullshit. You also start to observe some patterns, both in the work and in your colleagues.

Seven is an arbitrary number of course but I believe the following seven traits are things I have consistently observed across the best Data Scientists I have had the pleasure of working with. They are not things you can put on a resume. They probably won’t help you in that coding test or job interview. But they are traits which mean they are able to make more meaningful contributions, have a bigger impact, and get along better with their colleagues.

So, given this article is already long enough, let’s skip to the part we are all here for: 7 traits of the best Data Scientists.

Continue reading

Pandas: Where and Mask

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

In this guide we are going to look at three ways to handle a scenario where you want to update the values in a column based on whether some condition is true or not. This condition could be applied based on the same column you want to update, a different column, or a combination of columns.

The three methods (or two methods and a function we will look at are:

  1. where – a method of the pandas.DataFrame class.
  2. mask – a method of the pandas.DataFrame class and inverse of where
  3. numpy.where – a function in the numpy library.
Continue reading

Pandas: Append and Concat

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

In this guide we will look at a few methods we can use to add pandas DataFrames together vertically, stacking them one on top of the other. This will include two pandas methods concat and append, and a third way where we make use of some simple python methods. This last method can often be much faster than working with DataFrames directly, especially if we want to repeatedly append one row at a time to a DataFrame.

If you are looking at joining tables, or adding two tables together horizontally, try the guide on joining tables.

Continue reading

D3.js: Adding a colored background to a text element

There is no language I find quite so humbling as D3.js. Just when I think I understand it, I get stumped for a whole day trying to do something as simple as adding a colored background to a text element so the letters are readable against the rainbow-colored monstrosity sitting behind it. Well, if you found this post amongst the sea of woefully inadequate advice that I had to wade through, hopefully you have finally found your answer. In short, we are going to look at how to use D3.js to:

  1. Add a rect element for each text element in a dataset.
  2. Set the size of that rect element to match the corresponding text element.
  3. Ensure the rect element sits behind the text element, not on top of it.

Note: for this explainer we are working with D3.js version 6.

Continue reading

Pandas: Joining tables

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

One of the most fundamental concepts in data science and data work in general is joining two tables together based on some shared column or index. In SQL it is a JOIN. In Excel it is INDEX-MATCH or VLOOKUP. In pandas, two methods are available to join tables together: merge and join. We will look at both of those methods in this guide.

Continue reading

Pandas: How to Pivot data

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

When I was starting out with pandas, I was coming from an Excel and SQL background. Having spent a solid 8 years with Excel as my primary data munging and modeling tool, I was very comfortable using pivot tables, a tool I found extremely powerful and later discovered are strangely controversial. My workflow started to involve pivot tables so regularly that my SQL queries were often written to extract data in a format that would make it simpler to aggregate in a pivot table.

Naturally, when I started learning pandas, one of the first things I wanted to learn was “how can I recreate the functionality of an Excel pivot table in pandas”? In this guide we will look at several ways to do just that.

Continue reading

Pandas: Advanced Aggregation

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

Building on the basic aggregation guide, in this guide we will look at some more advanced ways we can aggregate data using pandas. We are going to cover three techniques:

  1. Aggregating using different methods at the same time, for example, summing one column and taking the average of another.
  2. Defining and using custom aggregation functions which we can use to calculate aggregates that are not available “out of the box”.
  3. The transform method which can be used to do some very useful things with aggregated values.
Continue reading

Pandas: Aggregation

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

A fundamental tool for working in pandas and with tabular data more generally is the ability to aggregate data across rows. Thankfully pandas gives us some easy-to-use methods for aggregation, which includes a range of summary statistics such as sums, min and max values, means and medians, variances and standard deviations, or even quantiles. In this guide we will walk through the basics of aggregation in pandas, hopefully giving you the basic building blocks to go on to more complex aggregations.

Continue reading

Pandas: SettingWithCopyWarning

This article is part of a series of practical guides for using the Python data processing library pandas. To see view all the available parts, click here.

For many users starting out with pandas, a common and frustrating warning that pops up sooner or later is the following:

SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.

Try using .loc[row_indexer,col_indexer] = value instead

To the uninitiated, it can be hard to know what it means or if it even matters. In this guide, we’ll walk through what the warning means, why you are seeing it, and what you can do to avoid it.

Continue reading
« Older posts

© 2021 Brett Romero

Theme by Anders NorenUp ↑