Review the latest updates for key data and analytics technologies and platforms including Snowflake, Databricks, dbt, BigQuery, Looker, Qlik, Tableau, and Power BI.

We know the product release notes from the vendors can be very detailed and overwhelming. So, we have outlined the major product updates you need to know about for Q3 2022, how these updates can impact you, how they can be applied, and other major news for key technologies within the data and analytics space.

Snowflake

Q3 Product Release Updates

Snowflake continues to extend itself beyond a cloud data warehouse into a full-fledged cloud data platform. Its ever-expanding functionality for Python, Java, and Scala make it easier and more convenient to shift data workloads entirely onto Snowflake.

Release 6.28 (8/16): Snowpark Stored Procedures in Java and Scala – General Availability

Allows for users to write stored procedures in Java and Scala (AWS and Azure, preview feature on GCP). In your stored procedure, you can use the Snowpark API for Java/Scala to host your data pipelines in Snowflake. For example, you can write stored procedures in cases where you need to execute your Snowpark code without running a client application (e.g., from a task). 

  • With the addition of stored procedures, Snowflake tasks call these procedures, which can perform data engineering and data science functions.
  • Snowflake continues to ramp up competition with Databricks and other popular data engineering tools. Databricks was developed with Python, Java, and Scala in mind, whereas Snowflake was originally released with support for SQL. By augmenting its language support, it appeals to more organizations that already use those languages for their data pipelines.
  • Snowflake’s Snowpark is trying to take on Databricks’ data engineering capabilities. Now that it supports Java and Scala stored procedures, Snowpark code can be executed via Snowflake tasks.

Release 6.32 (9/22): DAG support for Tasks – General Availability

Snowflake Tasks can now be ordered by dependencies. Previously, users were limited to task trees, in which each task had at most a single predecessor (parent) task. In a DAG, each non-root task can have dependencies on multiple predecessor tasks, as well as multiple subsequent (child) tasks that depend on it. With the addition of DAG support for tasks, Snowflake:

  • Is beginning to mirror functionality of tools like Apache Airflow or Dagster.
  • Can now fully emulate the orchestration functionalities of other tools. This not only allows for consolidation of technologies involved in your data pipelines, but it also simplifies total cost of ownership to a single platform.
  • Pipelines can be migrated entirely to Snowflake. This means less administration of other tools and better visibility into the total cost of ownership.

Q3 Other News

  • Snowflake Snowday 2022—a virtual event taking place on Nov. 7, 2022—will be focused on semi-annual product announcements.
  • Snowflake is hosting Data Cloud World Tour—21 in-person events globally and 5 in North America.
See Q2 Updates for Snowflake

Q2 Product Release Updates

Unistore

  • Unistore’s hybrid table functionality allows fast single-row operations needed to support transactional business applications directly on Snowflake.
  • The new transaction-optimized tables (which support database constraints and row-level locking) can be combined with Snowflake’s existing table format which is optimized for analytics. Transactional business applications can now be built directly on Snowflake rather than shuttling data back and forth between Snowflake and transactional systems.

Native Application Framework

  • Snowflake’s Data Marketplace has been renamed Snowflake Marketplace because it now enables applications to be distributed, not just datasets.
  • Snowflake now offers a platform for building, monetizing, and deploying data-intensive applications in the cloud. This framework is made possible by prior feature releases such as stored procedures, user-defined functions (UDFs), and user-defined table functions (UDTFs) which are core functionalities used to build Snowflake native applications.

Snowpark API for Python — Preview (announced 6/22)

  • Snowflake API for python has entered public preview. Java & Scala support have entered GA (general availability) as of May.
  • According to Snowflake: “We are pleased to announce the preview of the Snowpark API for Python.Snowpark is a new developer experience that provides an intuitive API for querying and processing data in a data pipeline. Using this library, you can build applications that process data in Snowflake without moving data to the system where your application code runs.”

Process Unstructured Data Using Java UDFs — Preview (announced 4/2022)

  • Reduces the need for additional tools alongside Snowflake to manage and utilize unstructured data.
  • According to Snowflake: “Unstructured data is data that lacks a predefined structure. It is often textual, such as open-ended survey responses and social media conversations, but can also be non-textual, including images, video, and audio. Java UDFs enable you to perform custom operations using the Java programming language to manipulate data and return either scalar or tabular results. Call your custom UDFs and compiled code to extract text, process images, and perform other operations on unstructured data for analysis. You can either include the Java code inline in the function definition or package the code in a JAR file and copy the file to an internal or external stage. Call the UDF with the input as a scoped URL, file URL, or the string file path for one or more files located in an internal or external stage. The new SnowflakeFile class enables you to easily pass additional file attributes when calling the UDF, such as file size, to filter the results. Previously, Snowflake customers were limited to processing unstructured files using external functions and remote API services.”

Directed Acyclic Graph (DAG) Support for Tasks — Preview (announced 6/2022)

  • Enhances ability to build and orchestrate complex data pipelines directly in Snowflake through use of tasks. DAGs enable parallel processing, for example to update a set of dimension tables concurrently before aggregating facts for a dashboard.
  • According to Snowflake: “We are pleased to announce preview DAG support for tasks. A DAG is a series of tasks composed of a single root task and additional tasks, organized by their dependencies. Previously, users were limited to task trees, in which each task had at most a single predecessor (parent) task. In a DAG, each non-root task can have dependencies on multiple predecessor tasks, as well as multiple subsequent (child) tasks that depend on it.”

Object Dependencies — GA (April 2022)

  • With this release, Snowflake is pleased to announce the general availability of object dependencies in the OBJECT_DEPENDENCIES view (Account Usage).
  • This update provides data stewards and data engineers a unified picture of the relationships between referencing objects and referenced objects. For example, when a table owner plans to modify a column, querying the OBJECT_DEPENDENCIES view based on the table name returns all the objects (e.g., views) that will be affected by the modification.

Q2 Other News

  • In a recent JP Morgan Massive- Scale CIO Survey which polled 142 CIOs controlling more than $100B of IT spend, Analyst Mark Murphy notes that Snowflake has excellent standing from its customers. He says “Snowflake ranked No. 1 in installed base spending intentions, beating out Microsoft, Alphabet-owned Google Cloud Platform, and CrowdStrike. Snowflake also ranked No. 1 among emerging companies whose vision most impressed respondents……Snowflake enjoys excellent standing among customers as evident in customer interviews and a recently laid out clear long-term vision at its Investor Day toward cementing its position as a critical emerging platform layer of the enterprise software stack.”
Blue and white graph from JP Morgan illustrating Snowflake's standing in install base spending in 2022.

In a JP Morgan survey from June 2022, Snowflake is surging into elite territory.

See Q1 Updates for Snowflake

Q1 Product Release Updates

  • Release 6.0 (1/19): Snowflake Scripting – Preview: “Snowflake Scripting is an extension to Snowflake SQL that adds support for procedural logic. You can use Snowflake Scripting to write stored procedures in SQL.”
    • SQL Scripting is a step forward in usability and capability. It allows SQL logic to be executed directly, no longer requiring use of stored procedures wrapped in JavaScript. This increases the number of projects that can be performed entirely in Snowflake without third-party tools.
  • Release 6.1 (1/24): Unstructured Data Support – General Availability: “Enables users to access, load, govern, and share unstructured files of data types for which Snowflake has no native support, including some industry-specific types. Support for unstructured files adds to the existing robust support for structured and semi-structured data.”
    • This update reduces the need for additional tools alongside Snowflake to manage and utilize unstructured data.
  • Release 6.5 (2/8): External Table Support for Delta Lake — Preview: “With this release, we are pleased to announce preview support for Delta Lake in external tables. Delta Lake is a table format on your data lake that supports ACID (atomicity, consistency, isolation, durability) transactions among other features. All data in Delta Lake is stored in Apache Parquet format. Query the Parquet files in a Delta Lake by creating external tables that reference your cloud storage locations enhanced with Delta Lake.”
    • This improves Snowflake’s flexibility to be deployed in architectures alongside Databricks.

Q1 Other News 

  • Snowflake released new industry-specific data clouds. These are solutions to accelerate capabilities of organizations in financial services; healthcare and life sciences; retail and CPG; advertising, media, and entertainment; public sector; education; and technology.

Databricks

Q3 Product Release Updates

Databricks continues to pump out new features left and right, nearing the gap to Snowflake in some areas and widening the gap to Snowflake in others, its biggest competitor. The biggest feature where it catches up to Snowflake is Delta Sharing, which Snowflake already had with Snowflake sharing. The enhanced orchestration for dbt and Databricks SQL is also a big plus, as it simplifies native scheduling, which has been one of Databricks’ weaknesses up until now. 

  • Improved Notebook Visualizations (July 2022): In the past, you could only see visualizations within the Databricks SQL view, however with this release, visualizations can be created in individual notebooks within the Data Engineer view. This will allow developers to generate visualizations on the fly within their notebooks for enhanced data analysis and ad-hoc reporting.
  • Unity Catalog is generally available (August 2022): The Unity Catalog is a new data governance tool within Databricks that allows admins to better manage users’ access to data across all Databricks workspaces.
  • Delta Sharing is generally available (August 2022): Delta sharing is now generally available and means that data can be shared across different organizations that are using Databricks. This can greatly simplify the steps taken to get data from one organization to another as previously data would have to be exported by the sender, sent across via FTP, REST, or some other mechanism, ingested by the receiver, and then transformed. With Delta Sharing, data can simply flow from the sender to the receiver in one step.
  • Orchestrate dbt tasks in Databricks workflows (August 2022): dbt Core projects can now be orchestrated directly within Databricks workflows, meaning an entire project can be scheduled within Databricks without ever having to leave the platform and orchestrate dbt-specific tasks within dbt. This can save developers time from having to switch back and forth between platforms and keeps everything orchestrated in one unified place.
See Q2 Updates for Databricks

Q2 Product Release Updates

  • Delta Lake 2.0: Open sourcing all Delta Lake enhancements by Databricks, including those that were previously available only to Databricks customers. This update is aimed at allowing customers to maintain a fully open data architecture, one of the core principles of building a data lakehouse.
  • Spark Connect: Offering Apache SparkTM whenever and wherever, decoupling the client and server so it can be embedded everywhere, from application servers, IDEs, notebooks, and all programming languages.
  • Project Lightspeed: Bringing the next generation of Spark Structured Streaming.
  • Databricks SQL Serverless: Available in preview on AWS, providing instant, secure, and fully managed elastic compute for improved performance at a lower cost. Photon, the record-setting query engine for lakehouse systems, will be generally available on Databricks Workspaces in the coming weeks.
  • Unity Catalog: Generally available on AWS and Azure in the coming weeks, Unity Catalog offers a centralized governance solution for all data and AI assets, with built-in search and discovery, automated lineage for all workloads, with performance and scalability for a lakehouse on any cloud.
  • Databricks Marketplace: Provides an open marketplace to package and distribute data sets and a host of associated analytics assets like notebooks, sample code, and dashboards.
  • Cleanrooms: Available in the coming months, Cleanrooms will provide a way to share and join data across organizations with a secure, hosted environment and no data replication required.
  • Databricks Terraform provider (GA, June 22): One of the coolest updates to Databricks this quarter has been the addition of a Terraform provider to manage all Databricks workspaces and the underlying architecture setup. It works with the Databricks REST APIs to allow for complete automation of a Databricks deployment using Terraform scripting. This can automate architecture setup, cluster management, job scheduling, provision workspaces, and setup user access, all without having to use the GUI!
  • Delta Live Tables support SCD type 2 (GA, June 21): Delta Live Tables (DLT) now support type 2 slowly changing dimensions (previously only supported type 1). This means developers do not have to write custom code to handle changes to slowly changing dimensions in cases where we need to track history. Instead, by using the apply changes function, we can simply identify which table and column(s) we want to track history for, and Databricks will handle it automatically.
  • Parameter values can be passed between Databricks jobs (GA, June 13): Values can now be passed from the output of one job to downstream tasks. This is useful if we need to put a filter on a query, but the filter needs to be dynamic depending on changes in the data and is calculated in an upstream task. This essentially lets us save variables from a task and have it persist in-memory, meaning we do not have to create a table and save it to refer back to in a future task. This allows for efficiency and time-savings.
  • Jobs matrix view (GA, April 27): You can now view Databricks jobs in a matrix view in the jobs user interface. Overview of job and task run details, including start time, duration, and status of each run. This isn’t a major update, but provides a nicer visual for seeing how a job has executed over time (pic below).
    Screenshot of Log Query page on Databricks.

    Job Matrix view gives users an improved visual for seeing how a job has executed over time.

  • Only re-run unsuccessful tasks (GA, April 25): If a job fails, the new repair and re-run feature allows you to re-run only the unsuccessful subset of tasks that failed instead of running the entire job over again. This means that if a job fails 25 minutes into a 30-minute run, instead of having to re-run from the start and wait another 30 minutes, the developer can re-run it starting from the 25-minute mark, saving valuable time when debugging errors.
See Q1 Updates for Databricks

Q1 Product Release Updates

  • Version 3.6 – Syntax highlighting and autocomplete for SQL commands in Python cells: Syntax highlighting and SQL autocomplete are available when you use SQL inside a Python command, such as in a spark.sql command.
    • These features help to reduce developer errors and increase speed of development of teams working in Databricks.
  • Public Preview – Unity Catalog: The Unity Catalog is a cross-workspace metastore which will help improve centralized data governance for the Lakehouse. One of the main features of the Unity Catalog is the Delta Lineage view, which will auto-capture and map the flow of data into the Lakehouse (across all languages down to the table and column level).
    • This will bring a level of governance and oversight to Databricks that has previously been missing.
  • Public Preview – Delta Sharing: Delta Sharing is a feature of the new Unity Catalog and will allow for Databricks-to-Databricks verified data sharing. It will allow incremental data changes to be shared using Delta Change Data Feed and has a simple UI to create and modify shares and recipients.
    • This feature adds capabilities of data sharing features similar to existing solutions offered by competitors like Snowflake.
  • Public Preview (AWS only) – Serverless DBSQL: Serverless DBSQL sets aside managed server instances which are always running, patched, and upgraded automatically to allow customers to access instant compute Power.
    • This reduces wait times for clusters to spin up. Databricks is also working to improve first query performance up to 3x using serverless DBSQL.
  • Public Preview – Databricks SQL: Databricks SQL is undergoing new updates in public preview, including a tabbed editor, advanced autocomplete and syntax error highlighting, allowing for multiple concurrent executions per tab, and sharing past executions to other users.
    • The same queries can be run from notebooks using SQL, however the Databricks SQL view is a more traditional SQL query editor which will allow Databricks to compete more fully with Snowflake and other cloud DWH competitors as it provides a simpler experience for querying directly against the Lakehouse.

Q1 Other News 

  • You can now create Databricks workspaces in your VPC hosted in the AWS ap-northeast-2 region. You can use a customer-managed VPC to exercise more control over your network configurations to comply with specific cloud security and governance standards your organization may require.

dbt

Q3 Product Release Updates

August 2022- Python meets dbt!

Python models are available in beta through dbt Core v1.3. These will have the same testing and lineage capabilities as SQL models in your dbt projects. Instead of needing separate infrastructure to support python, you can now host this all one place.

  • Python models will be a critical addition to solve use cases that go beyond SQL. Specifically, we see this as a major benefit for data teams with advanced analytics use cases that want to leverage cutting-edge data science and statistics packages while keeping work centralized.
  • We are very excited to see how this impacts strategies on dbt projects going forward for transformation use cases as well. For more information, we recommend checking out dbt Labs’ Python documentation.

August 2022- New dbt Cloud UI!

dbt Labs has invested a lot of time into creating a new dbt Cloud user interface. In addition to a much cleaner and more intuitive interface, this comes with massive performance enhancements as they scale for their rapidly growing customer base. At Analytics8, our consultants have opted into the Beta group to test this out and have thoroughly been impressed with the updates.

  • Most significantly, the performance and load times have been massively improved, which has been very helpful for our projects. Although this is currently in Beta, in the coming months, dbt will do a full transition to the new IDE for all users, so stay tuned on that.

Q3 Other News

  • Coalesce 2022: At Coalesce 2022 in New Orleans, two of our dbt experts, Tony Dahlager and John Barcheski, presented on Data Modeling in the Modern Data Stack Era. This is available to watch online.
  • Best-in-Breed Modern Data Stack: BigQuery, dbt, and Looker (A8 blog): This blog highlights the core advantages of using dbt, Google BigQuery, and Looker in tandem. As the name implies, this is a best-in-breed combination that a lot of people are excited about. If you are interested in learning about a data stack that is dependable, cost-effective, and future proof, this blog is for you.
  • dbt Certification: In July, dbt Labs released their official Analytics Engineering Certification. Initial consensus on the exam is that it is fair, but tough. This is a great way to showcase an advanced skillset using dbt, and we highly recommend checking it out. If you are interested in more details, check out the information page on dbt Labs’ website.
  • dbt Semantic Layer: There has been a ton of interest in doing more with metrics, and the dbt Semantic Layer is coming soon. This will be an amazing addition to further integrate your data platform with your BI tools. A lot of information regarding this will be released during the Coalesce conference, but for a preview, we would recommend you check out this post by Callum McCann from dbt Labs.
See Q2 Updates for dbt

Q2 Product Release Updates

April 2022 – Audit Log 
To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt Cloud audit log lists events triggered in your organization within the last 90 days. The audit log includes details such as who performed the action, what the action was, and when it was performed. 

  • The audit log adds an additional layer of data governance for organizations. This helps ensure stability and history without having to bring more complex tools into your stack

April 2022 – Rebuilt Cloud Scheduler 
The dbt cloud scheduler became the most popular discrete product in dbt Cloud with over 75% of users engaging with the product monthly. dbt greatly improved performance in Q2, especially around customer prep time. A great tip for your scheduling workloads is to move them off the top of the hour to after the 15-minute or 45-minute mark each hour of the day to limit concurrency. 

Q2 Other Updates

  • dbt + Databricks partnership: We recently wrote this blog which showcases the value of using databricks with dbt to unify your BI and Data Science teams. Traditionally, these teams have worked in separate tools for compute and storage, which adds unnecessary complexity to data stacks. Now, by integrating dbt with Databricks, there is now a well-designed framework for engineers to transform data for both analytics and data science initiatives without having to extract the data into any additional places. A common frustration in the industry is that there are too many microservices that have been unbundled for almost every small use case. The dbt and databricks partnership is a substantial step toward simplifying data teams’ stacks. 
  • dbt now mainstream? Snowflake conference show out: Mentioned at over 10 customer-led sessions, booth was constantly crowded, and best-practices session had a line out the door.
    dbt was once considered a niche tool but became hard to ignore with the incredible growth over the last few years. It is safe to say dbt has become one of the most popular transformation tools to use in the modern cloud data warehouse ecosystem, with Snowflake leading the charge. Many large companies have adopted dbt and found unmatched success. 

Q2 Other News

  • Join Analytics8 at Coalesce 2022 this October in New Orleans! Let us know if you’re coming to the conference in-person, we’d love to spend time with customers.
See Q1 Updates for dbt

Q1 Product Release Updates

  • Version 1.0 (stable release): dbt version 1.0 offers ease of upgrades and is backward compatible. It serves as the foundation for new functionality to be built upon in future releases. This marks a significant milestone in product maturity as ideas that previously seemed ambitious—including artifacts, advanced testing, Slim CI, and more—have materialized as incredible features in dbt v0.21 and earlier releases.
    • As product manager Jeremy Cohen mentioned at Coalesce 2021, this means the dbt core features have reached a point where they are fast, stable, intuitive, extensible, and maintainable. After upgrading to dbt 1.0, you can be certain that your future upgrades will be simple and intuitive from that point on. A few future ideas on the roadmap for these upgrades are column level lineage, support for sharded tables, built-in linting, column level Slim CI, and eventually dbt-SQL.

Q1 Other News 

BigQuery

Q3 Product Release Updates

In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available. 

  • This allows you to combine the power of BigQuery with the simplicity of Google Sheets—interact with data without having to know any SQL. Finance, marketing, and operations teams stand to benefit from ability to easily filter, create charts, pivot tables, and leverage familiar formulas and functions.
  • Video: Intro to Connected Sheets and BigQuery 

In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API’s concurrent connections and throughput quotas. This feature is now generally available. 

  • “The BigQuery Storage Write API is a unified data-ingestion API for BigQuery. It combines streaming ingestion and batch loading into a single high-performance API. You can use the Storage Write API to stream records into BigQuery in real time or to batch process an arbitrarily large number of records and commit them in a single atomic operation.”  

The slot recommender creates recommendations for customers using on-demand billing and is now generally available. 

  • You can compare slot values against on-demand charges over the same period to determine whether you can reduce costs by switching from on-demand pricing to flat-rate pricing.For example, it will provide insights like “Based on this recommendation, if you switch to a monthly commitment of 2500 slots, you would save $X, with no performance impact 99% of the time, assuming no change in usage from the previous month. The other 1% of the time, you might see reduced query performance.

You can now set default configurations at a project or organization level. This feature is now generally available. Default configurations will help organizations better manage settings across projects.

Querying Google Cloud Bigtable external data sources is now generally available. 

  • Bigtable is Google’s sparsely populated NoSQL database which can scale to billions of rows, thousands of columns, and petabytes of data. Bigtable has a data model similar to Apache HBase and provides an HBase-compatible client library.

BigLake is now generally available. You can now create BigQuery ML models using data in Cloud Storage by using BigLake and publish BigLake tables as Analytics Hub listings.

  • BigLake extends BigQuery’s fine-grained row- and column- level security (including dynamic data masking) to tables on data resident object stores such as Amazon S3, Azure Data Lake Storage Gen2, and Cloud Storage.
See Q2 Updates for BigQuery

Q2 Product Release Updates

Column-level data masking, combined with column level access control, gives access to columns while obscuring the data

  • More control on who sees which column, but also what in each column.

Time Travel Window Configuration – Specify duration from 2-7 days (or longer if needed)

  • Having snapshots of what your data was, how it was being used, or ability to roll-back to a point is essential to maintaining your data. Having the ability to configure how and when is a very important tool to utilize.

Query Queues – For on-demand and flat-rate customers, automatically determine the query concurrency.

  • By having a queue, no longer do queries that go above the 100 current threshold return an error.

External Table Feature in Google Sheets – Connect Google Sheets to your BigQuery data for direct access to your data.

  • Give your end users direct access to your data in BigQuery via Google Sheets. Control is still maintained in BigQuery.
See Q1 Updates for BigQuery

Q1 Product Release Updates

  • General Availability – Qualify Clause: The qualify clause is key to make code more concise. This has eliminated redundancies in code while executing window functions for transformations.
    • One of our clients (Driven Brands) will use the Qualify clause to automate deduplication of critical source data in a view within BigQuery.
  • Preview – Table Clones: Cloning tables allows a cost effective, light-weight way to store copies of tables and test changes to a table. You are only billed for storing the cloned table once it differs from the base table.

Q1 Other News 

  • A BigQuery Reliability Guide was published in Q1 2022 to help customers create solutions that correctly contemplate reliability requirements for specific use cases. This helps customers determine the best service within GCP for their solution rather than trying to shoehorn BigQuery for everything when it is not the optimal service. They included areas of planning for import reliability, query reliability, read reliability, and disaster planning.

 

Looker

Q3 Product Release Updates

New Performance Recommendations dashboard in System Activity: The Performance Recommendations dashboard is an interactive dashboard with recommendations for improving Looker performance based on best practices.

  • This is a great resource for admins to catch issues with dashboards or explores before they become pain points for business users. Prior to this release, these recommendations would have required custom benchmarking logic and an expert Looker admin.
  • One of the major pitfalls of a self-service BI tool like Looker is that, without a defined set of best practices to follow, the instance can become sluggish. Users can create Dashboards that are not performant, or database configurations can be set incorrectly. Small problems can lead to a frustrating user experience. With the release of the Performance Recommendations dashboard, Looker transfers the depth of knowledge of a seasoned Looker admin to your fingertips. Admins can now see where their instance is not performing well, the severity of the performance issue, and the path to correcting the issue.

New Embedded Content Navigation lab feature: When navigating through folders in an embedded Dashboard, Look, or Explore, the experience matches Looker’s updated design.

  • The aesthetics of Looker Dashboards and folder navigation within the Looker app were updated over the past two quarters, but embedded content retained the original folder navigation appearance. This created a confusing user experience when viewing embedded content outside of the Looker app. With the release of the new embedded content navigation, users will have a seamless experience while viewing embedded content.
  • With this update, Looker makes a huge step forward for its out-of-the-box embedded experience. Looker is already a leader in the embedded analytics space from the back-end development perspective, but users now have a unified look and feel to embedded content that secures their position as a leader in the embedded analytics space.
  • Customers will be able to immediately review and address performance issues on their instance, which will have an impact on their users’ experience and perception of Looker. And for any customer using embedded analytics, the new embedded content navigation feature is sure to be a hit with their end users.

Q3 Other News

This quarter saw many incremental updates to how LookML code is processed, which leads to more thorough LookML validation. This points to general improvements for the LookML developer experience.

See Q2 Updates for Looker

Q2 Product Release Updates

Forecasting Labs Feature: Add forecasting to your new or existing Explore queries to help predict and monitor specific data points.

  • Looker has improved its native forecasting capability by incorporating seasonality, prediction interval, as well as length of forecast (forecast over X period). This allows for enhanced insights and control into how forecasts are constructed within Looker.

New LookML Runtime: More performant loading of Explores and dashboards, and faster SQL writing.

  • Continual improvement of how LookML is handled, served up, and utilized allows for users to get the latest and greatest without having to rebuild.

Q2 Other News

Hundreds of knowledge articles previously for internal use only are now available for public use. These “Knowledge Drops” were originally created for Looker by Looker and are now available for all to use. Most have already been published, but check it out while it lasts!

See Q1 Updates for Looker

Q1 Product Release Updates

  • API 4.0 is generally available: Introduces new functionality that was previously inaccessible via the Looker API, like copying dashboards.
    • Copying dashboards previously could not be done via the API without some heavy coding. This turns what used to be a 50-60 line script into about 2-3 lines of code. We immediately began using this for one of our customers, but could see it being implemented any time a regular set of dashboards needs to be copied into a new folder.
    • The ability to use Looker’s API to automate administrative and embedding use cases sets it apart from its competitors. Tedious management of user permissions and authentication can be automated down to a few keystrokes or scripts.
    • From an administration standpoint, Looker is an InfoSec dream. Permissions can be set up via a script and repeated multiple times, and pipelines for copying dashboards/checking user access can be established. The sky’s the limit with the API, but it blows the competition out of the water when it comes to administration.
  • LookML Dashboards can be moved to any content folder: Allows for more logical storage of LookML dashboards, especially for Looker instances where multiple companies are logging into the same central instance.
    • The LookML Dashboards update is a step toward fully version-controlled dashboard creation, where changes to a dashboard can be tracked over time, so users can always be confident that the dashboard they’re seeing is the correct, working version.
    • Before this update, LookML Dashboards all had to stay in one folder, which was not manageable if working on an instance with hundreds of LookML dashboards. The concept of connecting a Looker dashboard to a Git service is something that sets Looker apart from the competition, and benefits users and developers alike, knowing that changes can be tracked and rolled back, if needed.

Q1 Other News 

  • Looker is sunsetting its certification program to align more with Google’s ‘badge’ system.

 

Qlik

Q3 Product Release Updates

The new Q3 features position Qlik SaaS better in the market through improving feature parity as well as expanding the abilities of ‘nice-to-haves’.

Qlik AutoML is now fully included in Qlik Cloud Hub

Create automated machine learning experiments in Qlik Sense. Easily train and deploy models using best-in-class machine learning techniques, in a simple, code-free experience. This seamlessly integrates the best-in-class AutoML pipeline with your existing front facing applications and data in Qlik SaaS.

  • AutoML’s release directly within the Qlik Cloud UI enables data science teams to quickly prototype analysis directly within Qlik. The ability to leverage user-ready data sources already within the Qlik platform without needing to extract or connect an additional tool is a powerful option for organizations looking at advanced analytics use cases. Also, being able to easily embed this analysis into the data visualization eco-system helps to remove barriers for data science teams to make their work more readily available and consumable by other within an organization.

Qlik Data Gateway – Direct Access

The new Direct Access gateway makes it possible for Qlik Cloud applications to securely access data behind a firewall over a strictly outbound, encrypted, and mutually authenticated connection. Qlik Data Gateway – Direct Access operates behind your organization’s firewall, eliminating the need to open inbound firewall ports to access your data from the cloud for analytics, whether that data is on-premises or in a virtual private cloud. Direct Access should allow organizations who have been held back by IT security concerns to once again look at Qlik Cloud as an option.

  • Having Direct Access through the data gateway opens conversations with clients who had previous concerns with security and firewall rules. Being able to simplify the migration and IT setup of Qlik SaaS is a major consideration when recommending a new tool to our clients.

New Authoring Experience

The new UI gives visibility to fields and data usually hidden in the data model viewer. This allows for a more seamless chart creation and layout. It will make sheet design simpler for new developers while still allowing for the fine-grained experience of the old, advanced layout. This should reduce barriers to entry for developers coming from other BI tools and improve abilities for self-service app creation.

  • The New Authoring UI improves the user experience for new developers and self-service users of Qlik Cloud. It provides another development friendly interface which is familiar to users of other platforms. Having all the information of an app in one place decreases the need for a disjointed workflow and is more approachable for less experienced users.

Q3 Other News for Developers:

  • Inner and Outer Set expressions – You can now pass set analysis filters across an entire expression without needing to apply it within every aggregation. This makes set analysis logic much more concise and simplistic to write for developers.
  • Application Chaining – Ability to pass filters from one app to another via button navigation. This improves the user experience for navigation between apps, as a user no longer needs to repeat the filtering when hopping across applications and should decrease response time across applications.
  • Schedule App reloads based on data service events – You can now have applications refresh based on source data being refreshed. This will reduce the need for complex timing or issues in the case of reload failures further upstream.
  • View App reload log – You can now see multiple reloads to help debug scheduled reload issues.
  • Increase to maximum file size – Data files of up to 100GB can now be uploaded (previously, 6GB was the limit).

Q3 Other News for Users:

  • UI improvements in the Qlik Cloud Hub. There are now ways to fine tune notifications and favorites in the UI. You can also now see your alerts and subscription in the navigation pane.
  • Field-level Lineage and impact analysis – Now available to all users, you can now track the lineage of fields within Dimensions and Measures across the environment to their source. Impact analysis shows the impact a field level change will have downstream from a source. This allows for greater visibility and trust within sources and KPIs within an organization.
  • Improvements to Mobile App – Improvements to the UI and performance of the mobile app. Also, setup has been simplified for the tenant via a QR code.
  • Font styling for charts and maps – This allows more granular changes to the look and feel of applications.
See Q2 Updates for Qlik

Q2 Product Release Updates

Chart level scripting 

Chart level scripting is a powerful feature that allows you to modify the dynamic dataset behind a chart using a subset of the Qlik scripting language, with techniques such as variables and loops. You can add or modify rows and columns that were not in the original dataset. This allows for calculations in chart expressions that were previously not possible, such as simulations or goal-seeking.

Chart level scripting adds a new more precise level of customization within chart building. Although Qlik is known to be flexible with what can be achieved via backend scripting and frontend set analysis, there are still some limitations. Using chart scripting, you will have more control over how your visualization looks and behaves and should reduce the need for work arounds or alternative methods of displaying information in your apps. Developers should be cautious in using heavy chart scripting as it may affect performance vs. simpler, built-in chart functionality.

Support for field-level lineage 

As part of delivering explainable BI, new catalog capabilities are now available. Explainable BI is the concept that users will gain more confidence in using data and making decisions based on it, if they have a good understanding, or explanation, of the information’s origin. 

Lineage capabilities have expanded to not only visually show the data history by table but also by the specific field within a table—starting with applications all the way back to the original source. Field-level lineage helps you establish trust in the data. When you explore an app, you can now quickly access a data lineage summary that traces back any dimensions and measures in a chart to the original sources. This makes it easy for any user to understand where the data came from within a chart.

There is also now a distinction between Lineage and Impact Analysis. 

  • Lineage shows you a detailed visual representation of the history of a field or dataset, back through the applications and interim datasets to its original source.
  • Impact Analysis shows you a downstream view of the dependencies for a data element, including the databases, apps, files, or links that will be directly or indirectly impacted if the value of that particular field changes.

In data cataloging and managements, more information and detail into the lineage of your data is always needed. With field level lineage, you can track the source of a field throughout your pipelines and apps. This should the reduce the need to track a field’s origin by physically opening apps and data sources and provide additional clarity to data consumers.

See Q1 Updates for Qlik

Q1 Product Release Updates

In Q1 2022 the Qlik SaaS product has continued to add additional functionality and integration with their recent acquisitions.

  • Qlik Forts: Allows organizations to keep their data and application on premise, while taking advantage of SaaS functionality and upgrades.
    • Qlik Forts will provide an option for those clients that are hesitant to keep all of their data and applications on a cloud environment. This opens the tool to some clients that have security or regulatory concerns.
  • Qlik Automations: Qlik automations has become a more robust feature in the SaaS environment. Templates have been created to provide a library of ways that automation can be utilized to take analytics outside the SaaS environment and greater control within the SaaS environment.
    • Qlik Automations/templates is going to provide customers with a wide range of ways to create API code that will allow the organization to take analytics outside of simple dashboards.

 

Tableau

Q3 Product Release Updates

Tableau did not release major updates during this quarter. Instead, we will be focusing on one of the major updates in the Q2 release with the greatest value add when it comes to time savings on administrative tasks for developers and admins.

Tableau Cloud: Turning to cloud solutions have proven to help organizations reduce operational costs and take control of their digital transformations. When your organization is faced with this challenge, Tableau is here to help with Tableau Cloud. Prior to the Q2 release, Tableau’s SaaS platform was known as Tableau Online and this update included some feature updates which we highlighted in our previous tech update blog post. In this update, we want to cover the key impacts Tableau Cloud provides for customers.

  • Empowers organizations to easily scale and maximize efficiency without the restrictions of self-managing their application
  • Distributes analytic autonomy to everyone from analysts to end users
  • Cuts overhead costs encompassing analytics deployment
    • No investing in hardware setup
    • No need for hiring and enabling server administrators

Q3 Other News

Tableau has released some information on their 2022.3 release on their “Coming Soon” page, but we are still waiting for the official release notes.

See Q2 Updates for Tableau

Q2 Product Release Updates

Data Stories: Tableau wants to tell you stories in their 2022.2 release. The new feature Data Stories builds a “story” based on a data visualization within your dashboard. These plain-language explanations will help any user confidently access, understand, and communicate with data. Developers are given the ability to add story points on distinct visualization to add additional context. Data Stories are also fully customizable so you can tailor the stories based on your audience. 

Autosave: This feature lets you edit an existing workbook in a draft until you’re ready to publish so you won’t lose your changes or share them prematurely with other workbook users (note: this is for web authoring only).  

Quick Search: In Tableau Server, turn to the all-new Quick Search feature to view past searches and suggested content. Quick Search can be accessed from any page and allows you to review results within the search box.  Pairing Quick Search with Tableau’s enhanced Ask Data phrase builder ensures you can spend less time searching for your data and more time engaging with it! 

Tableau Cloud: Tableau Online is now Tableau Cloud! This product is the same easy-to-use self-service platform designed to streamline the power of data to people, just with a new name. 

  • Tableau Cloud will now autosave updates to an unpublished draft, which will allow users less familiar with Tableau to make changes without fear of disrupting production, sharing results prematurely, or losing unsaved work. Meanwhile, the new search experience for Tableau Server will save business users time and, hopefully, frustration by quickly returning suggestions based on popular search terms and past searches.

Ask Data: The Ask Data features for Tableau Server and Cloud help make Tableau’s software more accessible to business users as a self-service platform for obtaining data insights. Ask Data can return manipulable visualizations in response to common business questions, allowing users without technical Tableau knowledge to build views to fit their specific needs. Introduced in May, this enhancement is intended to enable business users to create self-service visualizations based upon questions they pose of their data.

Q2 Other News

  • If you missed the 2022 Tableau Conference, you could catch up on the keynotes and top sessions on-demand.
  • Tableau will enable Multi-Factor Authentication for site administrators on existing sites and all roles on new Tableau Cloud sites.
See Q1 Updates for Tableau

Q1 Product Release Updates

As of March 21, 2022 Tableau 2021.4 is the latest version of Tableau.

  • Virtual Connections: A content type in Tableau Server/Online that lets you create and manage access to data sources within Tableau Online. Unlike a standard published data source, virtual connections allow you to securely embed service account credentials and define data policies for row-level security, all within the Tableau Online platform. This greatly simplifies data management and access, and makes it much easier to implement and manage row-level security; previously you needed to create custom logic within a Tableau workbook, publish an entirely separate workbook data source, and manage control offline.
    • Virtual Connections is an important new feature because it automates and simplifies data management and security processes. This feature bridges the gap between the ‘core’ Tableau offering (Tableau Desktop, where you make your dashboards) and Tableau Online/Server, where you share reports. Having this single connection ensures all users are accessing fresh data sourced centrally from managed extract refresh schedules. Administrators save time by making database changes only once in the virtual connection rather than in every connection per Tableau content (data source, workbook, flow).
  • Copy and Paste: Added functionality to Tableau Desktop that allows you to copy and paste workbook elements from one page to another, even across workbooks.
    • This feature accelerates development of reports and dashboards, and it encourages and simplifies design consistency.

Q1 Other News 

  • Tableau for Slack: allows users to share a vizualization generated in Ask Data directly to Slack, putting data at the center of every conversation. Streamline communication and collaboration on one platform to move business forward.

 

PowerBI

Q3 Product Release Updates

Cross-Tenant Data Sharing: In the October Power BI update, Microsoft introduced the ability to share datasets across Power BI tenants. This new capability allows organizations (providers) to share their Power BI datasets with external users (consumers), allowing them to access the shared dataset from within their own tenant.

  • As this is in-place sharing, the data is not moved to another Power BI tenant. As a provider, the dataset resides in your tenant. Consumers query the shared datasets directly in the source data systems. In addition, consumers can also build composite datasets using these Cross-Tenant datasets, eliminating the need to manually transfer data between organizations.
  • With Cross-Tenant Sharing, providers can share datasets with consumer tenants within the Power BI Service without the need to manually transfer data or provide access directly to source systems. Consumers are also able to build reports using these shared datasets without ever leaving their tenant. Consumers can enhance these datasets by using them in composite models.
  • This update allows organizations to monetize or share datasets with their customers more easily and all within the Power BI service.
  • For our customers that are sharing data with other organizations, Cross-Tenant Sharing allows them flexibility in sharing curated datasets directly from Power BI service to their customers/consumers.
  • It can potentially remove automated or manual data sharing pipelines that our customers currently maintain provided they’re sharing their data with customers that are reporting within Power BI.

Q3 Other News

There have been several enhancements to existing visualizations in Power BI, providing capabilities found in other BI Tools. To name a few:

See Q2 Updates for Power BI

Q2 Product Release Updates

Feature Release: Power BI Datamarts 

  • Released May 2022 at MSFT Build, this feature allows the ability for business users to create data warehouses for custom, self-service, departmental reporting. Power BI Datamarts also unlock the ability for analysts to use SQL queries—one of the most requested features of Power BI. Work that may typically take weeks for a data engineer, including creating data pipelines, a SQL Datawarehouse, provisioning access, or adding any additional code can now be facilitated much faster through the creation of Datamarts.
  • How it Works: The entire ETL process can happen inside the Datamart by creating a fully managed Azure SQL database automatically—all done directly within the Power BI web service. Data pipelines and models are built using low-code or no-code PowerQuery tasks online—enabling analysts to do more with their data without all the traditional steps required by data engineers. This enables organizations to help bridge gaps between IT users and business users and reduce time and bottlenecks for self-service and custom departmental reporting.
  • Why it’s a Differentiator: While PBI Datamarts are similar to PBI dataflows—they provide an additional value-add by supporting more than just single table use cases—ideal for combining and working with data across multiple sources in a single place. It also provides a great option for Mac users to engage with data in the PBI ecosystem. Data modeling capabilities are enabled by hosting PowerQuery and model tools directly from the web service as opposed to locally developing.

Q2 Other News

  • Keep in tune with the Power BI Roadmap for all new and generally available feature releases
  • Microsoft Inspire— July 19-20 2022— partner event focused on MSFT cloud partner programs.
See Q1 Updates for Power BI

Q1 Product Release Updates

  • In the February 2022 Power BI update, Microsoft released the ability to create, modify, and execute Power BI Deployment Pipelines directly from Azure DevOps using the new Power BI automation tools extension. This open-source Azure DevOps extensions removes the need to use API or scripts to manage these pipelines.
    • The new Power BI automation tools extension for Azure DevOps allows for organizations to quickly incorporate Power BI dataset and report management into their existing Azure DevOps pipelines. Enabling continuous integration and continuous delivery (CI/CD) for Power BI. Automating this management process for Power BI datasets and reports provides decisions makers the latest information more quickly and accurately—removing error introduced by manual processes.
    • With the simplicity of incorporating this capability, Power BI developers can automate delivery of these different reports and datasets to users, unlike other BI tools. Managing workspaces, user access and promotion processes of these Power BI objects further enhances the amazing list of features and capabilities of Power BI.
    • Incorporating the Power BI automation tools extension into existing Azure DevOps pipelines is fast and easy. There is no longer a need for someone to manual promote datasets and reports, instead freeing them up for other tasks. This also significantly reduces potential error caused by manual promotions.

Q1 Other News 


These updates are current as of Q3 2022. Keep an eye out for quarterly updates on technologies within the data and analytics space.

Patrick Vinton Patrick oversees R&D and is responsible for the technical direction of Analytics8. When he's not working, he's probably playing with his 2 sons. If the kids are with the babysitter, he's sharing a bottle of wine with his wife while binging on Netflix - probably a documentary or historical drama.
Tony Dahlager Tony is Analytics8’s Managing Director of Data Management, leading sales, marketing, partnerships, and consulting enablement for our data management service line.
Kevin Lobo Kevin is our Managing Director of Analytics and is based out of our Chicago office. He leads our Analytics service line, overseeing its strategic direction and messaging, while ensuring delivery of high impact analytics solutions for our clients. Outside of work, Kevin enjoys spending time with his wife and daughter, as well as running and live music.
Subscribe to

The Insider

Sign up to receive our monthly newsletter, and get the latest insights, tips, advice.

Thank You!