Super-charge your Data Science method - the importance of a hypothesis

It's been some time since I was in an academic setting and even longer since I dusted off my lab book and documented the hypothesis, method, and conclusion of the prescribed experiment of the day. I never excelled at lab classes and yet now, years later and in quite a different setting I'm being drawn to the scientific method. In fact, as machine learning has come to dominate the world of data science over the last few years it feels like the scientific method has had a resurgence in popularity.

Recently I was fortunate to be invited to a nearby university to view some of the projects their analytics students had been working on over the past semester. As well as being struck by the enthusiasm and creativity of their projects I was pleasantly surprised by the balance of scientific method and the applied nature of their work. Hypotheses formed  from problems that companies everywhere are trying to solve for their  own customers:

• Interpreting review text to provide better recommendations;

• Identifying objects from space to make life at ground level safer;

• Using government data to optimize capacity issues for public bodies.

Each presentation started with a hypothesis and while the method in each varied and was defined and reinvented over the course of the project, each ended with a conclusion on the results observed and I couldn't help but form my own follow-up hypotheses and questions for the students. Challenging the source data and exploring the context of the problem for new ideas.

In the day-to-day, it's easy to get lost in the data and subsequently misinterpret results. Exploratory analysis can be wonderfully liberating and prompt ideas and lines of inquiry that would never have existed before, however in isolation, this method risks wandering aimlessly through the data without any course of direction. The hypothesis in each presentation helped the students to stay focused on a clear objective and measure success, regardless of the outcome. 

Measuring success bears repeating. One of the things I've seen analytics teams struggle with is how to demonstrate their contribution to the business. Sometimes this contribution gets lost when a path of investigation doesn't yield positive results. In actual fact, there is often still a lot of value in the work conducted; Being able to prove that the original direction wasn't the optimal one and a course correction is needed or a new approach can be just as important as an analysis that proves you got things right first time. Setting the question out in a hypothesis and documenting the results can help demonstrate where time and effort were spent and what was learned over the course of the project.

Having a hypothesis can be the difference between performing exploratory analysis and conducting data science. Both are important in understanding the data landscape and deriving insight, but they are very different ways of looking at the data.

In a recent seminar Cassie Kozyrkov reminded the audience that if we’re conducting real data science then being able to independently test the hypothesis is key. Sometimes that means withholding not only a test set of data, but also a secret test set that even the scientists are not allowed to see - something to prevent them from gaming the experiment to provide optimal outcomes. It's a common trick used by Kaggle and other data science competitions to give participants enough data to experiment but not so much that they can cheat in the contest.

Whatever your motives, take a step back and ask ‘how will we know when we're done?’

How do you track value when performing your analysis? Do hypotheses feature in your approach or do you use a different technique? Let me know in the comments below, or on Twitter @eageranalyst.

The rise of the machines - not all automations were created equal

One thing I’ve been hearing a lot of the last year is Robotics, Automation and RPA. With the rise of companies like Automation Anywhere and Blue Prism the marketing machine has been hard at work trying to convince business leaders everywhere that their employees could either be more efficient or replaced altogether if they invest in automation. In the context of Data Analytics, I see it increasingly called upon to help collate data from difficult sources and in the context of Audit and Risk, to build automated testing solutions that are able to apply simple analytics to highlight exceptions and reduce the amount of manual testing that teams perform. Automation isn’t a new concept, automation as a general principle has been with us a long time, but popularized by these companies is a new form of automation that is somewhat different from traditional approaches - Robotics. Specifically, Robotic Process Automation or RPA for short.

Why is an RPA approach to automation so different? In a traditional technology process automation project, the design team may start by examining the existing process, mapping out their target process, and then identifying ways to change the underlying systems to bridge the perceived gaps. For example, you might have an exception process that flags mismatches between systems, but those exceptions still need to be reported, assigned to the right people, actioned and closed out. All of that might happen outside of the system making the management of that information very manually intensive and prone to error. If a portion of the exceptions could be automatically processed by the system or fed to another system for onward processing that could cut down the manual work portion substantially reducing the likelihood of errors and freeing up employees to perform other tasks.

One of the key principles behind RPA solutions is that you don’t need to make any changes to your underlying systems in order to automate manual tasks like this. If a process can be executed by a human right now, it can be automated using RPA techniques without uplifting the underlying technologies. This is particularly attractive in domains where there are limited opportunities to upgrade or change existing systems, for instance legacy technology platforms or external systems over which the business has little control. It also makes quantifying the benefit to the business easier as automation can be more easily expressed as a fraction of headcount. Take the time it would have taken to perform the task and the number of times it’s typically performed and multiply by the fraction of an employee’s salary you just replaced.

The rub, so to speak, is that RPA is often a tactical solution. By not updating the underlying business processes, the automation that’s put in place is more like a band-aid than a permanent solution. If the underlying process changes, that band-aid is going to come off. If you don’t have control over the underlying systems, it might rip off unexpectedly and at any moment.

So, when should you think about using RPA?

• When you can’t make changes to the underlying systems, either because they are outside of your control or they are no longer being developed.

• When the task does not require human reasoning in order to complete it. It may be completed by following rule based logic or there is a reliable ML model that can be used to determine course of action (true data-driven decision making!)

• When tasks can be completed using applications on a single workstation - just like a human user.

• When the underlying systems change infrequently. If underlying systems are updated your RPA may be brittle and require fixing each time a change is made. 

If you’re not meeting at least some of these requirements, then it may be better to think about RPA as a tactical solution and start investing elsewhere. However, if you’re sinking a significant amount of human time into a standard but disjointed process, it could be worth the investment, however tactical it might be.

Outside of RPA, the world of technology also continues to reinvent itself around business processes with increasingly more focus on the users that will interact with these systems. If you’re seeing large inefficiencies in user workload, it could be time to think about how you create a more user-centric environment.  Business Process Modeling is one approach to streamline user interaction with multiple systems, flowing information between systems and bringing elements of applications together into a single tool for common work tasks such as approvals, assignments and review. A key strength of this approach is the ability to centralize disjointed user tasks into a single application where users take action on smaller pieces of work and so it can be leveraged to build human-in-the-loop systems and user tasks are used to escalate work processes that require human input to proceed.

For analytics, the benefit of investing in RPA will depend on the type of challenges your business is facing, the level of investment you have to solve those problems, and perhaps the maturity of you technology stack in general. If you don’t have a robust technology stack that allows you to develop automated solutions, RPA can be a way to get your foot in the door.

As always, I’d love to hear your thoughts and the experiences you’ve had in automating your analytics solutions. Have you tried RPA? Do you have experience to share on the RPA journey, the tools chosen or the pitfalls to avoid? I’d love to hear from you.

Leave a comment below or send me a message @eageranalyst

Going Beyond Accuracy and Completeness - A deeper Dive into Data Quality

You will never have perfect data.

It’s a sad but inevitable truth. Rare enough are the cases where can find the data you need for your project let alone data without a variety of errors that could compromise its quality.

The silver lining is that you probably don’t need perfect data to begin with. As long as you can effectively measure and test the quality of what you do have, you can develop an understanding of your data’s limitations and develop new and creative ways to address them.

I often hear people talk about measuring the ‘accuracy’ and ‘completeness’ of their data, and while these are certainly important concepts, I wanted to go beyond that to discuss some of the other aspects of data quality that I think are equally important. After all, what does it mean for your data to be accurate and complete if it isn’t available until after you’ve made a decision you needed it for? So, let me summarize the six measures I consider for data quality - Accuracy, Completeness, Integrity, Precision, Validity and Timeliness - and share some thoughts on how to test for them in practice.

Accuracy

Data is accurate when it can be verified against an accepted version of the truth, a trusted authoritative source. Without an external validation source, it is very difficult to prove your data is accurate.

Data can be inaccurate at capture or become inaccurate due to corruption over time. An example of inaccurate data at capture would be if my address was captured as 132 Rand Street when I actually lived at 123 Rand Street. Both are valid addresses, but only one is my actual home. Over time I might move home, but not have my new address updated in my records.

How can I test it?

• Find an external trusted data source to validate against, if one is not available, find a proxy that you can compare against. For example, you might not be able to compare individual transactions against another source, but maybe you can compare them to the aggregate total or end of day values.

Completeness

Data is complete when each entity is counted exactly once. This means that you’re not missing any records or columns in the data that you need, and that you don’t have any extra records or values either. Nothing is being under or over reported.

Completeness issues often occur when data are transmitted between systems and records are either sent more than once or are missed altogether. In some cases, data is filtered before being shared with another system and so unknowingly, records are lost.

How can I test it?

When testing for completeness you will first need to decide on the things you are trying to measure. This might sound obvious, but consider the case when a customer signs up for an account on your website. If they sign up 5 times, how many records will you hold for that customer? What if they signed up using different email addresses? You will need to think about whether to consider them as individual accounts or as a single person.

• If you have a unique ID or reference key, check if you have more than one record for that key to identify duplicate values.

• If you have an external source of truth, reconcile your ID/reference key against that source to identify missing values or extra ones.

• If you have a checksum or control total, confirm that your dataset has that many records or computes to the same checksum.

• For column completeness, identify which columns should not be missing values and analyze any records that are missing data in those columns.

• If you don’t have a reference number, consider other things you could use as a proxy to indicate if your data is over or under reported. For example, the volume of transactions per day, the aggregate value of transactions in a month or per user.

• If you don’t have a reference number, consider what combination of attributes you could use to identify duplicates in your data. For example, email address, physical address or phone number paired with a name or date of birth.

Integrity

For users to be able to leverage data across multiple systems and reports it's important that the data is recorded in a consistent manner so that comparisons are meaningful.  If two sources disagree on a particular value or calculate the same value in a different way, it can lead to inconsistencies in the data that make it hard to focus on bigger issues.

Integrity is a big data risk when information flows between systems and could either be miscommunicated or become stale over time.

How can I test it?

• Try to obtain a data dictionary for any data provided so that there is a clear explanation of what each column is expected to contain. Compare these definitions across sources.

• Where you have two sources, compare attributes between the sources to identify potential omissions or inaccuracies and define which you consider to be the authoritative source of that data.

Precision

“I can report my height as 47.239476 meters – very precise, and utterly false”

Concepts such as precision measure how granular a result is. Precision is an important concept to consider when considering the reliability of measurements in their greater context and should be communicated using units and a tolerance – for example, height being measured in meters +/- 1cm.

Precision can lead to data quality risk if it is not maintained when flowing data between systems or there is a lack of clarity on the recorded precision (or the reasoning behind a particular level of precision). In low latency trading, for instance, decisions are made in milliseconds. If the time of a trade is recorded in seconds or minutes, the information shared would be meaningless for future decisions.

Note that precision is not the same as accuracy; a result can be emphatically precise and completely inaccurate (for example, I can report my height as 47.239476 meters – very precise, and utterly false).

How can I test it?

• Examine the values you have - do they cluster around specific values that may indicate they are being rounded to a less granular level?

• Be clear about the reasons why you have a particular level of precision for a recorded value. What level of precision do you need for your calculations?

Validity

Validity measures whether a data attribute falls within certain predefined validation criteria. These rules typically reflect business requirements and can be valuable in stopping bad data from entering a system in the first place. For example, birth dates should [probably] never be in the future and a person’s weight should not be a negative number.

If the data in our system don’t obey the rules we plan for them, they may break any information flows or systems that depend on that data.

How can I test it?

• Where you’re expecting categorical values, check that your data falls within those categories.

• If you require certain fields to be present, check they are populated.

• For free text fields, analyze field lengths to look for instances where dummy data has been recorded.

Timeliness

Timeliness measures how relevant the information provided is in the timeframe you’re using it. Information quality may deteriorate over time and become less meaningful. For example, a business may prepare accounts every month to provide a picture of the company’s finances. If they did this only on an annual basis, it may be harder for the company to make financial decisions come the end of the year.

Depending on the type of decision or action being performed and the nature of the data, there may be risk if data is not being updated or provided in a timely manner.

How can I test it?

• Look for expiration dates or last updated dates on key information fields. Things like contact information will inevitably change over time.

• Check when a report was last refreshed or prepared.

Round-up

Accuracy, Completeness, Integrity, Precision, Validity and Timeliness - These are my top six data quality considerations and concepts I routinely rely on to help test the quality of data I encounter. If your responsibilities include information management more generally, you may also want to consider things like privacy and security of your data.

Let me know if you have any other things you watch out for when performing your analysis in the comments below, or on Twitter @eageranalyst.

More data tips!

Little by little I've been continuing to share some of my top data tips on Instagram and even branched out from visualizations into agile and data prep. Now that I've got to number 10, it’s time for another recap.

If you have any suggestions on topics or have your own tips to share, please let me know and I'll try to work them into the series.

• MISSING VALUES - Check your data for missing values before performing your analysis.

• CONSISTENT ORDERING - Be consistent when ordering. Either order by value, or by label.

• ASK YOURSELF - What are the numbers you’re not hearing about?

• MINIMUM VALUABLE PRODUCT - Even your minimum viable product needs to demonstrate value to your users.

• HIGHLIGHT THE STORY - Use color and shade to highlight the story in the data.

Sketch-notes from Strata

Last week I had the good fortune to attend O’Reilly’s Strata conference where I heard from a number of leaders in the industry on how they were tackling both the technical and non-technical challenges of working with data. While listening to some of the talks I decided to try my hand at capturing some sketch-notes and I wanted to share these for others to see. I’ll add to this post over the next few weeks as I finish tidying-up my notes.

Stories Beat Statistics - Brent Dykes

The first sketch-note is for Brent Dykes (@analyticshero) talk Stories Beat Statistics where he shared the idea that by combining data and visuals with a narrative we can more successfully deliver key messages and influence change.

Download the letter-size image

Agile for Data Science Teams - Jennifer Prendki

The second sketch-note is for Jennifer Prendki’s (@jlprendki) talk Agile for Data Science Teams where she talked about how Figure Eight were applying agile practices in their data science teams. It was super useful to hear how other data teams are using Sprint and Kanban for different types of projects.

Download the letter-size image

Data Bytes: Things I loved - March 2018

Here are some of the data-related posts I loved this month.

• Citymapper's GoBot tackles MTA announcements - New York's Metropolitan Transit Area (MTA) announcements are so difficult to understand that Citymapper have written a bot to decipher their meaning and capture them into their app.
• Introducing spaCy - If you're solving NLP problems, you should know about SpaCy. 
• Visualizing supermarket checkouts - The author thought they’d stumbled onto a facinating insight until they realized the data wasn’t quite what they thought. Beautiful visuals despite the flawed analysis.
• Are You Setting Your Data Scientists Up to Fail? - Harvard Business Review comment on the challenges of integrating data science teams into an organization