Making better decisions – Preparing your data (Part II)

In one of our recent reading lists, we spoke about an article from Towards Data Science that looked at Data Visualization in Music.

As we collated the rest of our reading list, we found ourselves asking: what do John Coltrane and Clinical Pharmacy Software have in common? Watching “Giant Steps” (below), a visual representation of the John Coltrane song of the same name, the answer became apparent. Data analytics and visualisation provides us with a perspective that we might have otherwise missed.

Whether the data presented represents notes in a song or workload in a clinical pharmacy environment, the outcome is the same; we are presented with insights previously unseen.

Over the past few weeks, we’ve been considering how data can improve decision-making in organisations. We’ve dealt with first principles, business objectives and culture and how to prepare the data appropriately.

In this blogpost, we look more closely at the preparation phase, this time from the perspective of the data scientist and consider the unseen perspectives data analytics and visualisations can provide.

Data Science

So where does data science fit in the data journey? Broadly speaking, the software engineer builds software to handle a problem, the data engineer builds the infrastructure and the data scientist asks the questions and tests what’s present in the data.

In respect of our own approach, we don’t see visualisation and machine learning as just simply the outputs of data analysis but as integral tools to be used throughout the data analysis process.

Ingest and Cleansing

According to Analytics Engines’ Data Scientist Liam Brannigan two very important areas where value can be added lie in the data ingest and cleansing.

“We now see ingest and cleansing as areas where AI, machine learning and visualisation are key tools,” he said.

“Our team has recently been working on data ingest with a client that has a very large dataset. Our plan was to build on this data using machine learning. However, in order to design the optimum database structure, we really needed to understand the structure of the dataset.”

The process of understanding was complicated by the hierarchical structure of the data. The answer lay in building an interactive visualisation to provide both ourselves and the client with a new perspective on the data.

“From there we were able to develop our Database Optimisation Strategy and it allowed our client to understand what we were proposing and also provide their own domain expertise into the plan,” he said.

Efficiency gains

Data cleansing can be challenging. In another of Liam’s recent projects, he dealt with a dataset where people recorded their shopping habits.

“Inevitably there were errors and inconsistencies in the data – people had shopped at Tesco, but also at Tescos, Tesco’s, Tescos’, Yesco and so on. To clean this up, I built a simple machine learning model to do text classification,” he said.

“This model took in the raw entries and outputted the name of the shop they had actually been to with a high degree of success. The value of this approach is that you don’t simply have a one-off clean dataset, you instead have a method that can instantly be applied to any new data that arrives resulting in a major efficiency gain.”

In these instances, the data scientist was able to create a visualisation that invited co-creation and collaboration with the client, and a new tool capable of releasing additional value for future projects.

“Perspective is important. Data visualisations can provide us with insights we might have otherwise missed, or they can help identify unseen avenues for progress,” said Liam.

Bottomline

Data analytics, visualisations and machine learning can help remove the noise and provide clarity in order for the human to draw deeper insight and make better decisions. (We discussed the relationship between technology and the user in our intro to the blog series here).

If you’d like to find out more about effectively using your data, click HERE to get in touch or call us on +44 2890 669 022 to discuss.