5 Ways to Optimize Data & Analytics Spend: Expert Insights for Data Leaders

In recent years, enterprise organizations have increasingly invested in cloud technologies and platforms to enhance their use of data and analytics. However, many now find themselves grappling with higher-than-anticipated costs and are seeking ways to maximize their investments.

In this blog, Managing Director Tony Dahlager answers questions about five key focus areas you should consider for optimizing data and analytics spend. He provides insights on ways to maximize the value of your investment in data and analytics, but also how to avoid pitfalls that can lead to overspending or compromised quality. He discusses:

1. Managing public cloud spend effectively ↵
2. Navigating data transportation costs in complex environments ↵
3. Optimizing spend on centralized cloud data platforms ↵
4. Balancing technology costs in a modern data stack↵
5. Aligning data and analytics spend with talent strategy ↵

There are 5 key areas to focus on — public cloud, data transportation, cloud data platforms, tech costs, and talent strategy — for data and analytics spend optimization.

1. How can we manage public cloud spend (AWS, Azure, GCP) more effectively and optimize our existing environment?

Tony: Managing public cloud spend (AWS, Azure, GCP) effectively and optimizing your existing environment can be challenging, especially when running analytics workloads in the cloud. However, with the right strategies in place, you can achieve better cost management and resource allocation.

Start by assigning a steward for your cloud budget. This person should be responsible for regularly reviewing cloud provider bills, ensuring there are no unexpected or unjustified costs, and understanding the purpose and usage of each service. They should also work closely with the budget owner and be familiar with the cloud services and applications in use. A dedicated steward proactively creates granular budgets, policies, and alerts to monitor and control spend. Not sure if you have the right person for this? Public cloud providers often offer coursework that can help your steward learn the basics of each service and how to create policies and alerts.

Second, focus on rightsizing your cloud services. As most cloud services are elastic, you can (and should) scale them up or down based on your needs. Consistently adjusting your services is essential for managing costs effectively. To optimize your cloud spend, ask:

• Are services running when they’re not in use?
  Tip: Try to limit them to run only during working days and hours.
• Are services running at full capacity?
  Tip: Experiment with lower compute, storage, or memory requirements to find the most cost-effective solution.
• Are we taking advantage of reserved instances or committing to long-term agreements with cloud providers?
  Tip: Moving from on-demand to longer-term commitments can result in significant cost savings.
• Are we staying up to date with new enhancements to services?
  Tip: Regularly review your cloud infrastructure to identify areas where you can adopt more serverless technologies or find more optimized solutions to reduce total cost of ownership.

These two fundamental strategies — clear stewardship and rightsizing — will enable you to manage your public cloud spend more effectively and optimize your existing environment.

“In the rush to embrace cloud’s elasticity and high availability, many organizations simply lifted and shifted workloads without rearchitecting. But the beauty of the public cloud is its flexibility. Even small changes, like adjusting data analytics workloads to not run 24/7, can lead to significant cost reductions. It’s these low hanging fruits that not only save money but also foster a mindset focused on efficiency and cost-effectiveness.”

2. What factors should we consider when managing data transportation costs?

Tony: Managing data transportation costs (data extraction, data ingestion, data replication, data streaming, etc.) can be a complex task, especially as organizations increasingly adopt centralized cloud data platforms like Snowflake, Databricks, and BigQuery. But there are ways to optimize your data transportation processes for price and performance, ensuring you’re making the most of your resources. Start by looking at your total cost of ownership (TCO), the complexity of your data sources, tools and approaches, and your overall data usage.

• TCO: While building your own data replication tools might seem cheaper in the short-term, it’s essential to consider the time and effort your team spends on programming, monitoring, and maintaining them. Evaluate the people costs, skills required, and maintenance efforts before choosing a seemingly inexpensive option.
• Complexity of data sources: If your organization relies on complex enterprise applications — particularly legacy systems — consider using pre-built connectors rather than building your own. Leverage the expertise of others to lower your TCO and avoid unnecessary complications.
• Mixed models: Be open to using multiple tools and approaches for managing data transportation costs. Some commercial models charge by connector, others by consumption, capacity, or a flat fee. Assess each data source individually and consider adopting a multi-tool model to optimize costs.
• Data usage: Analyze data flows end-to-end to ensure the data you’re transporting is actually being used downstream. If it’s not, discontinue syncing that data. Focus on meeting the demand for data and eliminating unnecessary replication.

Carefully analyzing these four areas will enable you to make informed decisions about whether to build and manage your own data transportation pipelines or invest in tools that can help automate the process. By striking the right balance, you’ll be able to optimize your data transportation costs and maintain an efficient, cost-effective data management infrastructure.

“By balancing TCO and managing our data sources wisely, you can create a more cost-effective and pragmatic approach to rising costs. Remember, there isn’t a one-size-fits-all solution; sometimes multiple patterns are needed to transport data within a single organization. It is best to engage with your tech and service providers, explain your challenges, and explore creative solutions together for better cost predictability.”

3. How can my organization optimize spend on centralized cloud data platforms?

Tony: While cloud data platforms like Snowflake, Databricks, and BigQuery have taken enterprise data services to the next level, their consumption-based commercial models can lead to unexpected costs.

Here are several key areas to consider when evaluating opportunities to reduce your cloud data platform spend:

• Understand your organization’s minimum viable service model by assessing your data needs and determine the necessary level of service to support them.
  Tip: Consider whether real-time or near real-time data access is required, or if nightly batch loading is sufficient. Identify blackout times when services can be shut down without impacting critical processes.
• Address inefficiencies in your data architecture by examining your data models and looking for opportunities to improve efficiency.
  Tip: Remember that compute costs are the new cost paradigm, so optimizing for reduced compute usage can have a significant impact on your overall spend.
• Focus on high-impact users by analyzing consumption by user, role, tables, schemas, or individual queries to identify the biggest cost drivers.
  Tip: Work with those teams to find an optimal price-performance strategy that meets their needs without breaking the bank.
• Implement consumption policies and limits by setting guardrails for groups in your organization to avoid unintentional overspending.
  Tip: Leverage auto-suspend settings, if available, to help manage costs.
• Evaluate the necessity of auxiliary tools by reviewing your tech stack and ensuring that any additional tools, such as testing, data observability, data quality monitoring, ML model training, data profiling, or BI software, are genuinely essential for your organization.
  Tip: Keep your tech stack focused on the minimum viable components needed to support your operations.

By examining these areas and taking action where appropriate, you can optimize spend on centralized cloud data platforms, ensuring you’re making the most of your resources without sacrificing performance or essential functionality.

“Identifying your minimum viable data stack isn’t about cutting corners; it’s about understanding the actual stack you need to support the demand for data. Be practical about what you need now, and as your complexity and needs grow, so can your data stack.”

4. What recommendations do you have for managing technology costs in data and analytics?

Tony: There is a lot to consider when it comes to managing tech costs, especially for organizations that have opted for selecting individual technologies to build a modular modern data stack — also known as a “best-of-breed” approach. Although there are benefits to this approach, it’s difficult to identify the individual strengths of each vendor or to know where capabilities overlap. Start by determining your “minimum viable data stack”.

You need to:

• Identify your current methods and use cases of consuming data by assessing your analytical maturity and current use cases. This will help you figure out which technologies are critical for your data consumption and which ones are just nice-to-have or not needed at all.
• Identify what technology is necessary to deliver data to active consumers. This will help you to avoid overspending. For instance, if your current needs revolve around batch loading of BI dashboards and standard reporting, your minimum viable data stack will look very different from a setup that’s geared toward pushing ML models to production.
• Eliminate idle technology or anything that is aspirational (i.e., anything not part of — or without a clear path to — an active data consumption use case). An example is having a data streaming tool when your current data sources don’t support streaming. Scenarios like this can lead to unnecessary costs for services that aren’t actively adding value presently.
• Roadmap tech updates or implementations when it makes sense to phase it back into your tech stack at the right time. Consider what is critical to data flowing end-to-end in your demand cycle and use that to guide when tech updates or implementation can actually aid active demand for data.

Be honest about your organization’s maturity and readiness for change. Align your aspirational data stack with your actual capabilities and needs, rather than trying to appear more advanced than you are — it’s the best way to optimize your tech costs while getting the most value out of what you have.

“Aim for clarity in purpose and align your tech stack with your actual needs rather than aspirations. If a tool remains unused, it may be time to renegotiate or even end the contract. Always be realistic about your journey and open to discussions with vendors.”

5. How can I optimize my data analytics spend with my talent strategy?

Tony: When it comes to working with contractors, consultants, or managed services providers, there are a few key considerations to ensure you’re getting the most value, including:

• Choose strategy consultants who can demonstrate that they’ve been part of successful strategy implementations with quick value realization.
  Tip: Ask for references and inquire about the expected outcomes versus the actual results achieved.
• Be mindful of communication and time zones. Global teams can be great, but moving roles and resources offshore can cause delays and risks if there’s a dependency on synchronous communication.
  Tip: Design your global team models around work that can be done asynchronously and align teams with time zones where collaboration is crucial.
• When selecting consultants and contractors for implementation teams, don’t focus solely on their individual skills. By hiring a consultancy, you should get the benefit of the whole organization’s knowledge, not just one person’s expertise.
  Tip: Evaluate the processes, methodologies, and support networks they bring to the table.
• Pay attention to the sales process because it can be indicative of what it’ll be like working with the consultancy during an engagement.
  Tip: Consider the following questions:
  • Who do you get to talk to during the sales process?
  • Are actual consultants involved, or is it only sales representatives?
  • How responsive are they?
  • Do they listen to your needs and provide tailored proposals?
  • Are their cost estimates and models clear?
  • Do they show interest in the business benefits of your project?

Following these best practices can help you avoid potential issues down the line, such as having to bring in a new team of consultants for a “second pass.”

“Good consultants are worth their hourly rates; they bring value, experience, and even stability to your organization. They offer a broader perspective, can help avoid missteps, and can maintain continuity in an ever-changing environment. Even in cost-cutting, the focus should be on the total value of services provided, not blindly reducing expenses by the seemingly highest vendor rate. Sometimes the seemingly cheapest option can be the most expensive when you take into account lost time and rework.”

Watch Tony Answer Questions from Your Colleagues About Spend Optimization