How a modernized data architecture decreased survey analysis time from weeks to hours so our client could quickly respond to changing customer needs in the era of COVID-19.

While no better way to know what your customers want than directly obtaining this from the source, customer surveys will only help an organization if they are able to quickly analyze results, understand the feedback, and translate it into meaningful action that best serves their customer base. If you don’t take action on your insights, your customers won’t feel heard and your credibility is at risk.

One of our clients is a large medical membership association with members across the country operating their own clinical practices. Our client needed a clear understanding how their members were being impacted by COVID-19 and what approaches they were taking to deal with the drastic change in business operations. They created a 50-question survey with questions about operational changes, client numbers, PPE, and financial impact to be sent to more than 2,000 members.

Aware of the limitations of their existing survey structure, they knew a survey of this magnitude would take weeks to clean and aggregate the data into a format from which they could draw actionable insights.

Modernizing the Data Architecture For Faster, More Insightful Survey Analysis 

Instead of investing in new technology and learning new tools, we took advantage of managed services offered in their Microsoft Azure environment.

With a more flexible and agile data architecture, the process to analyze their survey data was cut down from weeks to under one hour.

Here’s how we did it:

  • Azure Data Lake Gen 2 became our landing zone for the raw files output from the vendor.
  • The creation of the blob file in the Data Lake kicked off a Trigger in Azure Data Factory which ran an Azure Databricks notebook that used Python to clean the data into an acceptable format before landing it in another Data Lake Gen 2 container.
  • Once the file had been manually reviewed by a member of the client’s Economics team, it was loaded into a new container which started another Azure Databricks notebook. This one aggregated and manipulated the data into a new format which was then loaded into a SQL Database.
  • We then employed our standard three-tier ETL architecture to move the data through the database and store it in the data warehouse.

Not only did this infrastructure significantly improve the speed by which our client could begin to analyze the data, it also served as a central repository of all survey information. By loading other types of surveys in the same format, we created a central data store which can be used by the client’s data scientists to perform more advanced analysis.

Translating Results into Action

The real value of survey results comes when your organization makes positive changes from gleaned insights. Our client is using the survey data to help their members better plan, strategize, and succeed during COVID-19 and beyond:

  • Identified and shared strategies working most successfully for some practices that can be replicated by others
  • Guided the development of tools and resources to support practices
  • Provided economic forecasts to prepare for what’s ahead
  • Provided survey data at the state level to help guide local support efforts

Through efficient and informed use of their data, our client is providing value to their customer base during a difficult time and helping shape effective policy moving into the next phase of recovery.

Learn more about responding to customer needs


In this on-demand webinar, data experts participated in an interactive panel discussion and discussed in more detail about how Customer Behavior, Supply Chain, and HR/People Analytics can help you handle new challenges during the recession.

Callum McCann Former accountant turned consultant, Callum utilizes his experience and skillset to help clients better use their data. From creating serverless ETL pipelines to designing efficient data warehouses in multiple cloud providers, he builds, designs, and implements modern data architectures that promote the usage of large volumes of data. Outside of work, Callum enjoys reading fantasy/science fiction, rock climbing, and biking around the city of Chicago.
Subscribe to

The Insider

Sign up to receive our monthly newsletter, and get the latest insights, tips, and advice.

Thank You!