The Data Analytics Technology Updates You Need to Know About

Q4 2022 Product Release Updates

Extended Search Optimization: This preview feature allows for the search optimization service to use columns in join predicates that use string patterns, semi-structured data, and geospatial functions.

• This is useful for very large datasets that contain ‘messy’ data, and could be especially useful for data science applications that regularly use raw and semi-structured data.
• The search optimization service can now use these kinds of columns and join predicates to speed up queries without having to increase warehouse size.
• There is also support for tables with masking and row-access policies, which previously returned an error.

EXECUTE TASK Command: Tasks are SQL statements that, until now, ran on a defined schedule. Now, tasks can be executed manually via the EXECUTE TASK command.

• This is useful for testing tasks before deploying them to a production system, especially now that tasks support DAGs (neural networks for deep learning).

As Snowflake’s data pipeline features expand, it’s becoming more important to be able to manually execute tasks.

Database Replication Support for Streams and Tasks: Snowflake offers database replication to support Disaster Recovery (DR) and High Availability (HA). However, there are several limitations to this functionality, including:

• Many objects cannot be replicated or can only be replicated under certain conditions.
• Replicating objects to a secondary account may limit some features as well.
• Replicating roles prohibits the creation of roles in the secondary account.
• Two features that were not previously supported for replication included tasks and streams. This was to avoid disruptions to the stream offset coming from two sources. The primary account could mistakenly have its streams’ offsets moved based on reads from the secondary account.

With this new feature, streams can now be replicated. Tasks can also be replicated but there are specific considerations that must be considered to ensure continuity in a DR/HA scenario.

Snowpark API, Stored Procedures for Python: The Snowpark API for Python is now generally available.

• This allows developers to augment their pipelines with APIs to query and process Snowflake data, while using Snowflake compute and without having to move data out of Snowflake.

Stored procedure support allows you to write stored procedures in python using the Snowpark API to host stored procedures in Snowflake.

• This augments Snowflake’s data engineering functionality and centralizes compute, storage, and code entirely within Snowflake.

Python UDFs/UDTF: Snowflake now supports user-defined functions and table functions that can be used as if they were bult-in functions.

• This means you can use python to define UDFS and UDTFs, instead of SQL. This will be a real quality of life improvement for those doing python work with Snowflake already.

SELECT*: Excluding and Renaming Specific Columns: This feature allows you to select all columns and exclude specific columns without having to explicitly name all the desired columns. You can also rename specific columns without having to explicitly select all desired columns.

• This is a quality-of-life update that will benefit just about everyone who queries Snowflake.

Database Roles: This is a preview feature of a new entity: database roles.

• This will allow you to define, grant, and revoke privileges within the scope of a database, instead of using account-level roles.
• These work the same as account-level roles, meaning all privileges on securable objects can be applied, but at a database level.
• This further extends Snowflake’s Role Based Access Control model to be more fine-tuned at a database level.

Q4 2022 Other News

• Snowflake Summit announced for June 2023 — we’ll be there! Let us know if you’re planning to come.

Q3 2022 Product Release Updates

Snowflake continues to extend itself beyond a cloud data warehouse into a full-fledged cloud data platform. Its ever-expanding functionality for Python, Java, and Scala make it easier and more convenient to shift data workloads entirely onto Snowflake.

Release 6.28 (8/16): Snowpark Stored Procedures in Java and Scala – General Availability

Allows for users to write stored procedures in Java and Scala (AWS and Azure, preview feature on GCP). In your stored procedure, you can use the Snowpark API for Java/Scala to host your data pipelines in Snowflake. For example, you can write stored procedures in cases where you need to execute your Snowpark code without running a client application (e.g., from a task). 

• With the addition of stored procedures, Snowflake tasks call these procedures, which can perform data engineering and data science functions.
• Snowflake continues to ramp up competition with Databricks and other popular data engineering tools. Databricks was developed with Python, Java, and Scala in mind, whereas Snowflake was originally released with support for SQL. By augmenting its language support, it appeals to more organizations that already use those languages for their data pipelines.
• Snowflake’s Snowpark is trying to take on Databricks’ data engineering capabilities. Now that it supports Java and Scala stored procedures, Snowpark code can be executed via Snowflake tasks.

Release 6.32 (9/22): DAG support for Tasks – General Availability

Snowflake Tasks can now be ordered by dependencies. Previously, users were limited to task trees, in which each task had at most a single predecessor (parent) task. In a DAG, each non-root task can have dependencies on multiple predecessor tasks, as well as multiple subsequent (child) tasks that depend on it. With the addition of DAG support for tasks, Snowflake:

• Is beginning to mirror functionality of tools like Apache Airflow or Dagster.
• Can now fully emulate the orchestration functionalities of other tools. This not only allows for consolidation of technologies involved in your data pipelines, but it also simplifies total cost of ownership to a single platform.
• Pipelines can be migrated entirely to Snowflake. This means less administration of other tools and better visibility into the total cost of ownership.

Q3 2022 Other News

• Snowflake Snowday 2022—a virtual event taking place on Nov. 7, 2022—will be focused on semi-annual product announcements.
• Snowflake is hosting Data Cloud World Tour—21 in-person events globally and 5 in North America.

Q2 2022 Product Release Updates

Unistore

• Unistore’s hybrid table functionality allows fast single-row operations needed to support transactional business applications directly on Snowflake.
• The new transaction-optimized tables (which support database constraints and row-level locking) can be combined with Snowflake’s existing table format which is optimized for analytics. Transactional business applications can now be built directly on Snowflake rather than shuttling data back and forth between Snowflake and transactional systems.

Native Application Framework

• Snowflake’s Data Marketplace has been renamed Snowflake Marketplace because it now enables applications to be distributed, not just datasets.
• Snowflake now offers a platform for building, monetizing, and deploying data-intensive applications in the cloud. This framework is made possible by prior feature releases such as stored procedures, user-defined functions (UDFs), and user-defined table functions (UDTFs) which are core functionalities used to build Snowflake native applications.

Snowpark API for Python — Preview (announced 6/22)

• Snowflake API for python has entered public preview. Java & Scala support have entered GA (general availability) as of May.
• According to Snowflake: “We are pleased to announce the preview of the Snowpark API for Python.Snowpark is a new developer experience that provides an intuitive API for querying and processing data in a data pipeline. Using this library, you can build applications that process data in Snowflake without moving data to the system where your application code runs.”

Process Unstructured Data Using Java UDFs — Preview (announced 4/2022)

• Reduces the need for additional tools alongside Snowflake to manage and utilize unstructured data.
• According to Snowflake: “Unstructured data is data that lacks a predefined structure. It is often textual, such as open-ended survey responses and social media conversations, but can also be non-textual, including images, video, and audio. Java UDFs enable you to perform custom operations using the Java programming language to manipulate data and return either scalar or tabular results. Call your custom UDFs and compiled code to extract text, process images, and perform other operations on unstructured data for analysis. You can either include the Java code inline in the function definition or package the code in a JAR file and copy the file to an internal or external stage. Call the UDF with the input as a scoped URL, file URL, or the string file path for one or more files located in an internal or external stage. The new SnowflakeFile class enables you to easily pass additional file attributes when calling the UDF, such as file size, to filter the results. Previously, Snowflake customers were limited to processing unstructured files using external functions and remote API services.”

Directed Acyclic Graph (DAG) Support for Tasks — Preview (announced 6/2022)

• Enhances ability to build and orchestrate complex data pipelines directly in Snowflake through use of tasks. DAGs enable parallel processing, for example to update a set of dimension tables concurrently before aggregating facts for a dashboard.
• According to Snowflake: “We are pleased to announce preview DAG support for tasks. A DAG is a series of tasks composed of a single root task and additional tasks, organized by their dependencies. Previously, users were limited to task trees, in which each task had at most a single predecessor (parent) task. In a DAG, each non-root task can have dependencies on multiple predecessor tasks, as well as multiple subsequent (child) tasks that depend on it.”

Object Dependencies — GA (April 2022)

• With this release, Snowflake is pleased to announce the general availability of object dependencies in the OBJECT_DEPENDENCIES view (Account Usage).
• This update provides data stewards and data engineers a unified picture of the relationships between referencing objects and referenced objects. For example, when a table owner plans to modify a column, querying the OBJECT_DEPENDENCIES view based on the table name returns all the objects (e.g., views) that will be affected by the modification.

Q2 2022 Other News

• In a recent JP Morgan Massive- Scale CIO Survey which polled 142 CIOs controlling more than $100B of IT spend, Analyst Mark Murphy notes that Snowflake has excellent standing from its customers. He says “Snowflake ranked No. 1 in installed base spending intentions, beating out Microsoft, Alphabet-owned Google Cloud Platform, and CrowdStrike. Snowflake also ranked No. 1 among emerging companies whose vision most impressed respondents……Snowflake enjoys excellent standing among customers as evident in customer interviews and a recently laid out clear long-term vision at its Investor Day toward cementing its position as a critical emerging platform layer of the enterprise software stack.”

In a JP Morgan survey from June 2022, Snowflake is surging into elite territory.

Q4 2022 Product Release Updates

After several quarters with large new features becoming generally available, this quarter Databricks focused on making improvements to the UI and Unity Catalog. They rolled out a new query engine Photon in July, which is gaining more traction as users upgrade to Databricks runtime 11 on their clusters. Besides the many small but useful changes like updates to the Add Data UI, table search and Privilege Inheritance in Unity Catalog, here are some updates that stand out:

Data lineage for Unity Catalog managed tables (December 2022):

• Lineage data (data and where it has moved overtime) can now be captured for Unity Catalog managed tables and will be retained for 30 days. The captured lineage data can be queried across workspaces and visualized in Data Explorer.

Repos support for non-notebook files (October 2022):

• It is now possible to edit non-notebook files in your remote git Repository without leaving Databricks. This prevents developers from having to switch to their Git provider to make quick edits to Markdown or Text files.

Enhanced notifications for Databricks Jobs (November 2022):

• Databricks added native Slack and webhook notification options to keep the organization aware when critical Job run start, complete, or fail. Notifications have also become more customizable.

Photon (July 2022):

• The new query engine Photon was built from the ground up to speed up queries and reduce costs. It requires no code changes and will work simply by upgrading clusters to runtime 11.1+ and enabling Photon on them.

Smaller UI and quality of life enhancements (Q4 2022):

• The new “Add Data” UI makes connecting to data sources and uploading files easier and can be controlled by administrators.
• It is now possible to search for tables in Unity Catalog, access recent objects, and search for Jobs from the top bar in a workspace.
• Privilege Inheritance in Unity Catalog makes it possible to grant privileges at a catalog or schema level, which will automatically cascade down to all current and future objects within that catalog or schema.

Q4 2022 Other News

• Databricks Data and AI Summit has been announced for June 2023. We’ll be there—hope to see you there too!

Q3 2022 Product Release Updates

Databricks continues to pump out new features left and right, nearing the gap to Snowflake in some areas and widening the gap to Snowflake in others, its biggest competitor. The biggest feature where it catches up to Snowflake is Delta Sharing, which Snowflake already had with Snowflake sharing. The enhanced orchestration for dbt and Databricks SQL is also a big plus, as it simplifies native scheduling, which has been one of Databricks’ weaknesses up until now. 

• Improved Notebook Visualizations (July 2022): In the past, you could only see visualizations within the Databricks SQL view, however with this release, visualizations can be created in individual notebooks within the Data Engineer view. This will allow developers to generate visualizations on the fly within their notebooks for enhanced data analysis and ad-hoc reporting.
• Unity Catalog is generally available (August 2022): The Unity Catalog is a new data governance tool within Databricks that allows admins to better manage users’ access to data across all Databricks workspaces.
• Delta Sharing is generally available (August 2022): Delta sharing is now generally available and means that data can be shared across different organizations that are using Databricks. This can greatly simplify the steps taken to get data from one organization to another as previously data would have to be exported by the sender, sent across via FTP, REST, or some other mechanism, ingested by the receiver, and then transformed. With Delta Sharing, data can simply flow from the sender to the receiver in one step.
• Orchestrate dbt tasks in Databricks workflows (August 2022): dbt Core projects can now be orchestrated directly within Databricks workflows, meaning an entire project can be scheduled within Databricks without ever having to leave the platform and orchestrate dbt-specific tasks within dbt. This can save developers time from having to switch back and forth between platforms and keeps everything orchestrated in one unified place.

Q2 2022 Product Release Updates

• Delta Lake 2.0: Open sourcing all Delta Lake enhancements by Databricks, including those that were previously available only to Databricks customers. This update is aimed at allowing customers to maintain a fully open data architecture, one of the core principles of building a data lakehouse.
• Spark Connect: Offering Apache SparkTM whenever and wherever, decoupling the client and server so it can be embedded everywhere, from application servers, IDEs, notebooks, and all programming languages.
• Project Lightspeed: Bringing the next generation of Spark Structured Streaming.
• Databricks SQL Serverless: Available in preview on AWS, providing instant, secure, and fully managed elastic compute for improved performance at a lower cost. Photon, the record-setting query engine for lakehouse systems, will be generally available on Databricks Workspaces in the coming weeks.
• Unity Catalog: Generally available on AWS and Azure in the coming weeks, Unity Catalog offers a centralized governance solution for all data and AI assets, with built-in search and discovery, automated lineage for all workloads, with performance and scalability for a lakehouse on any cloud.
• Databricks Marketplace: Provides an open marketplace to package and distribute data sets and a host of associated analytics assets like notebooks, sample code, and dashboards.
• Cleanrooms: Available in the coming months, Cleanrooms will provide a way to share and join data across organizations with a secure, hosted environment and no data replication required.
• Databricks Terraform provider (GA, June 22): One of the coolest updates to Databricks this quarter has been the addition of a Terraform provider to manage all Databricks workspaces and the underlying architecture setup. It works with the Databricks REST APIs to allow for complete automation of a Databricks deployment using Terraform scripting. This can automate architecture setup, cluster management, job scheduling, provision workspaces, and setup user access, all without having to use the GUI!
• Delta Live Tables support SCD type 2 (GA, June 21): Delta Live Tables (DLT) now support type 2 slowly changing dimensions (previously only supported type 1). This means developers do not have to write custom code to handle changes to slowly changing dimensions in cases where we need to track history. Instead, by using the apply changes function, we can simply identify which table and column(s) we want to track history for, and Databricks will handle it automatically.
• Parameter values can be passed between Databricks jobs (GA, June 13): Values can now be passed from the output of one job to downstream tasks. This is useful if we need to put a filter on a query, but the filter needs to be dynamic depending on changes in the data and is calculated in an upstream task. This essentially lets us save variables from a task and have it persist in-memory, meaning we do not have to create a table and save it to refer back to in a future task. This allows for efficiency and time-savings.
• Jobs matrix view (GA, April 27): You can now view Databricks jobs in a matrix view in the jobs user interface. Overview of job and task run details, including start time, duration, and status of each run. This isn’t a major update, but provides a nicer visual for seeing how a job has executed over time (pic below).
  

  Job Matrix view gives users an improved visual for seeing how a job has executed over time.

• Only re-run unsuccessful tasks (GA, April 25): If a job fails, the new repair and re-run feature allows you to re-run only the unsuccessful subset of tasks that failed instead of running the entire job over again. This means that if a job fails 25 minutes into a 30-minute run, instead of having to re-run from the start and wait another 30 minutes, the developer can re-run it starting from the 25-minute mark, saving valuable time when debugging errors.

Q4 2022 Product Release Updates

This quarter, dbt made changes to their Cloud Teams Version and introduced a semantic layer. Here are a few impactful changes to be aware of:

Beginning in February 2023, the price for dbt Cloud Teams licensing will be $100 per user and each account will be limited to eight total developer seats. Additionally, all new teams accounts will be limited to one project effective immediately.

• This is a move to more clearly define the licensing that is appropriate for small vs large data teams as many large organizations have adopted the transformational tool in the past few years.
• The restrictions on projects/developer seats will impact the licensing for larger data teams. If you are impacted by either restriction, we recommend looking into dbt Cloud Enterprise, which is still considered an affordable option.

A new dbt semantic layer is currently available in Public Preview for multi-tenant dbt Cloud accounts hosted in North America.

• The semantic layer provides an interface within dbt to define consistent metrics for supported downstream use cases. It is an incredible way to extend data beyond the data team by integrating with your downstream tools. In other words, you can now leverage this layer to tightly integrate your data stack with consistent definitions throughout.
• In the era of cloud microservices, it’s been difficult for many to bridge gaps between all your different tools, and this is massive step in filling this void. This is especially beneficial for dbt projects that have multiple downstream destinations.
• It currently integrates with Atlan, Deepnote, Hex, Houseware, Lightdash, and Mode. Since it is free of charge in public preview, we highly recommend checking this out and seeing the value it can add to your data stack.

Q4 2022 Other News

• Learn more about dbt’s Cloud pricing change.
• This quarter, Analytics8 Managing Director, Tony Dahlager, along with Analytics8 Senior Consultant and dbt Practice Leader, John Barcheski, presented at Coalesce, dbt Lab’s Annual Analytics Engineering Conference, where they discussed why dimensional modeling is still relevant in the Modern Data Stack. Check it out here.
• Analytics8 continued hosting the dbt Chicago Meetup, which offers a chance to both network with local data professionals and learn advanced dbt concepts through hands-on presentations. Our next meetup is taking place in mid-February. We are always looking for presenters at the Chicago meetup. Contact us if you’re interested in speaking. Check out the #local-chicago channel in dbt Slack or our LinkedIn for updates closer to February.

Q3 2022 Product Release Updates

August 2022- Python meets dbt!

Python models are available in beta through dbt Core v1.3. These will have the same testing and lineage capabilities as SQL models in your dbt projects. Instead of needing separate infrastructure to support python, you can now host this all one place.

• Python models will be a critical addition to solve use cases that go beyond SQL. Specifically, we see this as a major benefit for data teams with advanced analytics use cases that want to leverage cutting-edge data science and statistics packages while keeping work centralized.
• We are very excited to see how this impacts strategies on dbt projects going forward for transformation use cases as well. For more information, we recommend checking out dbt Labs’ Python documentation.

August 2022- New dbt Cloud UI!

dbt Labs has invested a lot of time into creating a new dbt Cloud user interface. In addition to a much cleaner and more intuitive interface, this comes with massive performance enhancements as they scale for their rapidly growing customer base. At Analytics8, our consultants have opted into the Beta group to test this out and have thoroughly been impressed with the updates.

• Most significantly, the performance and load times have been massively improved, which has been very helpful for our projects. Although this is currently in Beta, in the coming months, dbt will do a full transition to the new IDE for all users, so stay tuned on that.

Q3 2022 Other News

• Coalesce 2022: At Coalesce 2022 in New Orleans, two of our dbt experts, Tony Dahlager and John Barcheski, presented on Data Modeling in the Modern Data Stack Era. This is available to watch online.
• Best-in-Breed Modern Data Stack: BigQuery, dbt, and Looker (A8 blog): This blog highlights the core advantages of using dbt, Google BigQuery, and Looker in tandem. As the name implies, this is a best-in-breed combination that a lot of people are excited about. If you are interested in learning about a data stack that is dependable, cost-effective, and future proof, this blog is for you.
• dbt Certification: In July, dbt Labs released their official Analytics Engineering Certification. Initial consensus on the exam is that it is fair, but tough. This is a great way to showcase an advanced skillset using dbt, and we highly recommend checking it out. If you are interested in more details, check out the information page on dbt Labs’ website.
• dbt Semantic Layer: There has been a ton of interest in doing more with metrics, and the dbt Semantic Layer is coming soon. This will be an amazing addition to further integrate your data platform with your BI tools. A lot of information regarding this will be released during the Coalesce conference, but for a preview, we would recommend you check out this post by Callum McCann from dbt Labs.

Q2 2022 Product Release Updates

April 2022 – Audit Log 
To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt Cloud audit log lists events triggered in your organization within the last 90 days. The audit log includes details such as who performed the action, what the action was, and when it was performed. 

• The audit log adds an additional layer of data governance for organizations. This helps ensure stability and history without having to bring more complex tools into your stack

April 2022 – Rebuilt Cloud Scheduler 
The dbt cloud scheduler became the most popular discrete product in dbt Cloud with over 75% of users engaging with the product monthly. dbt greatly improved performance in Q2, especially around customer prep time. A great tip for your scheduling workloads is to move them off the top of the hour to after the 15-minute or 45-minute mark each hour of the day to limit concurrency. 

• https://www.getdbt.com/blog/a-good-problem-to-have/
• https://www.getdbt.com/blog/rebuilding-dbt-scheduler/
• The Cloud Scheduler improvement means that more customers can reliably trust dbt Cloud to orchestrate their largest workloads. dbt Cloud is an extremely affordable method for orchestration compared to hosting and managing your own process.

Q2 2022 Other Updates

• dbt + Databricks partnership: We recently wrote this blog which showcases the value of using databricks with dbt to unify your BI and Data Science teams. Traditionally, these teams have worked in separate tools for compute and storage, which adds unnecessary complexity to data stacks. Now, by integrating dbt with Databricks, there is now a well-designed framework for engineers to transform data for both analytics and data science initiatives without having to extract the data into any additional places. A common frustration in the industry is that there are too many microservices that have been unbundled for almost every small use case. The dbt and databricks partnership is a substantial step toward simplifying data teams’ stacks. 
• dbt now mainstream? Snowflake conference show out: Mentioned at over 10 customer-led sessions, booth was constantly crowded, and best-practices session had a line out the door.
  dbt was once considered a niche tool but became hard to ignore with the incredible growth over the last few years. It is safe to say dbt has become one of the most popular transformation tools to use in the modern cloud data warehouse ecosystem, with Snowflake leading the charge. Many large companies have adopted dbt and found unmatched success. 

Q2 2022 Other News

• Join Analytics8 at Coalesce 2022 this October in New Orleans! Let us know if you’re coming to the conference in-person, we’d love to spend time with customers.

Q4 2022 Product Release Updates

BigQuery has added several minor new features. These include:

• Customers can now transfer data from Amazon S3 and Azure Blob Storage to BigQuery using a LOAD DATA statement, making cross-cloud data transfers much simpler than they previously were.
• Additional data loading features such as the ability to create a reference file with the expected tables schema for external tables, and ASCII control characters for CSV files. These features make loading bad and unorthodox data files much easier.
• Read more about their feature updates here.

Q3 2022 Product Release Updates

In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available. 

• This allows you to combine the power of BigQuery with the simplicity of Google Sheets—interact with data without having to know any SQL. Finance, marketing, and operations teams stand to benefit from ability to easily filter, create charts, pivot tables, and leverage familiar formulas and functions.
• Video: Intro to Connected Sheets and BigQuery 

In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API’s concurrent connections and throughput quotas. This feature is now generally available. 

• “The BigQuery Storage Write API is a unified data-ingestion API for BigQuery. It combines streaming ingestion and batch loading into a single high-performance API. You can use the Storage Write API to stream records into BigQuery in real time or to batch process an arbitrarily large number of records and commit them in a single atomic operation.”  

The slot recommender creates recommendations for customers using on-demand billing and is now generally available. 

• You can compare slot values against on-demand charges over the same period to determine whether you can reduce costs by switching from on-demand pricing to flat-rate pricing.For example, it will provide insights like “Based on this recommendation, if you switch to a monthly commitment of 2500 slots, you would save $X, with no performance impact 99% of the time, assuming no change in usage from the previous month. The other 1% of the time, you might see reduced query performance.

You can now set default configurations at a project or organization level. This feature is now generally available. Default configurations will help organizations better manage settings across projects.

Querying Google Cloud Bigtable external data sources is now generally available. 

• Bigtable is Google’s sparsely populated NoSQL database which can scale to billions of rows, thousands of columns, and petabytes of data. Bigtable has a data model similar to Apache HBase and provides an HBase-compatible client library.

BigLake is now generally available. You can now create BigQuery ML models using data in Cloud Storage by using BigLake and publish BigLake tables as Analytics Hub listings.

• BigLake extends BigQuery’s fine-grained row- and column- level security (including dynamic data masking) to tables on data resident object stores such as Amazon S3, Azure Data Lake Storage Gen2, and Cloud Storage.

Q2 2022 Product Release Updates

Column-level data masking, combined with column level access control, gives access to columns while obscuring the data

• More control on who sees which column, but also what in each column.

Time Travel Window Configuration – Specify duration from 2-7 days (or longer if needed)

• Having snapshots of what your data was, how it was being used, or ability to roll-back to a point is essential to maintaining your data. Having the ability to configure how and when is a very important tool to utilize.

Query Queues – For on-demand and flat-rate customers, automatically determine the query concurrency.

• By having a queue, no longer do queries that go above the 100 current threshold return an error.

External Table Feature in Google Sheets – Connect Google Sheets to your BigQuery data for direct access to your data.

• Give your end users direct access to your data in BigQuery via Google Sheets. Control is still maintained in BigQuery.

Q4 2022 Product Release Updates

Looker Connections to Looker Studio (formerly Data Studio): This new feature allows customers with Google Cloud hosted instances to easily move between Looker and Looker Studio where they can explore and connect directly. This integration is the one of many that Google is planning between Looker and the other Google Cloud Platform (GCP) products. As more integrations are added, it will allow users to leverage the right tool for the job with minimal effort.

• Looker’s focus on integrating with other GCP products makes a “full Google stack” more viable for a data ecosystem.

New updates to the user experience when navigating dashboards and creating visualizations.

• Data history playback for visualization (labs feature) which allows users to explore data changes over time for visualizations within dashboards.
• Two new scatterplot chart visualization config options: Cluster Points and Plot Quadrants
• Ability to hide individual pivots from visualizations.
• Ability to zoom in on Cartesian charts.
• Ability to view visualizations in full screen and expanded views within a dashboard.
• WYSIWYG Text Tiles: Users will no longer need to use markdown code to do standard formatting in text tiles, but still have the option to use markdown, if needed.

These updates make Looker’s dashboards easier to build, easier to navigate/see, and should allow for excellent visualizations to be created by users earlier in their Looker enablement journey.

Read more for a deeper dive on changes in Q4.

Q4 2022 Other News

• Looker is starting up Hackathons again. These multi-day events have historically led to interesting new third-party extensions and developer tools. We look forward to more community events like this and encourage you to join upcoming events.

Q3 2022 Product Release Updates

New Performance Recommendations dashboard in System Activity: The Performance Recommendations dashboard is an interactive dashboard with recommendations for improving Looker performance based on best practices.

• This is a great resource for admins to catch issues with dashboards or explores before they become pain points for business users. Prior to this release, these recommendations would have required custom benchmarking logic and an expert Looker admin.
• One of the major pitfalls of a self-service BI tool like Looker is that, without a defined set of best practices to follow, the instance can become sluggish. Users can create Dashboards that are not performant, or database configurations can be set incorrectly. Small problems can lead to a frustrating user experience. With the release of the Performance Recommendations dashboard, Looker transfers the depth of knowledge of a seasoned Looker admin to your fingertips. Admins can now see where their instance is not performing well, the severity of the performance issue, and the path to correcting the issue.

New Embedded Content Navigation lab feature: When navigating through folders in an embedded Dashboard, Look, or Explore, the experience matches Looker’s updated design.

• The aesthetics of Looker Dashboards and folder navigation within the Looker app were updated over the past two quarters, but embedded content retained the original folder navigation appearance. This created a confusing user experience when viewing embedded content outside of the Looker app. With the release of the new embedded content navigation, users will have a seamless experience while viewing embedded content.
• With this update, Looker makes a huge step forward for its out-of-the-box embedded experience. Looker is already a leader in the embedded analytics space from the back-end development perspective, but users now have a unified look and feel to embedded content that secures their position as a leader in the embedded analytics space.
• Customers will be able to immediately review and address performance issues on their instance, which will have an impact on their users’ experience and perception of Looker. And for any customer using embedded analytics, the new embedded content navigation feature is sure to be a hit with their end users.

Q3 2022 Other News

This quarter saw many incremental updates to how LookML code is processed, which leads to more thorough LookML validation. This points to general improvements for the LookML developer experience.

Q2 2022 Product Release Updates

Forecasting Labs Feature: Add forecasting to your new or existing Explore queries to help predict and monitor specific data points.

• Looker has improved its native forecasting capability by incorporating seasonality, prediction interval, as well as length of forecast (forecast over X period). This allows for enhanced insights and control into how forecasts are constructed within Looker.

New LookML Runtime: More performant loading of Explores and dashboards, and faster SQL writing.

• Continual improvement of how LookML is handled, served up, and utilized allows for users to get the latest and greatest without having to rebuild.

Q2 2022 Other News

Hundreds of knowledge articles previously for internal use only are now available for public use. These “Knowledge Drops” were originally created for Looker by Looker and are now available for all to use. Most have already been published, but check it out while it lasts!

Q4 2022 Product Release Updates

Qlik reached an agreement on January 5, 2023 to acquire Talend. The acquisition significantly bolsters the data integration capabilities of the Qlik platform and continues the emphasis on Qlik building out its overall data management product offerings.

Qlik Cloud Data Integration: A platform to ingest, stage, and transform data into governed data marts through the Qlik Cloud platform.

• This allows for loading source data and preparing it for analytics use cases in a managed and unified way within the Qlik Cloud platform. This feature is a good fit for existing Qlik customers or customers looking for new data platforms with broad features.
• This feature of the platform works to bring data into the Qlik ecosystem while keeping the best practices of the modern data stack and cloud databases. Being able to ingest and manipulate data with Change-Data-Capture as well as batch scheduling can drastically simplify data warehouse buildouts for customers who have simplistic modeling needs for their data. It also opens up Qlik power users to self-service their back-end data much like they would within Qlik analytics apps.
• Having the data on a single platform will be a huge benefit to removing data silos and governing data infrastructure. It will be great to watch the features of the platform expand to more sources and targets for customers who have Data Integration needs within their Qlik Cloud platform.

HIPAA compliance/Customer Managed Keys:

• HIPAA compliance enables customers with sensitive HIPAA privacy concerns to utilize the Qlik Cloud Platform as they would any other HIPAA data service. This opens up Qlik to new customers who may have had security concerns holding them back from adoption.
• Customer Managed Keys are another feature that is often required by many data security standards. This is another available functionality that opens the Qlik Cloud platform to new customers or customers who may have had limitations on their reporting.
• HIPAA compliance and Customer Managed Keys are very important data security functionalities for organizations with PII and heightened security standards. These protections alleviate Infosec concerns within their broader IT organizations and among executives concerned with privacy.

Data Load Editor History: Script changes are now tracked and viewable back in time through the editor.

Qlik Cloud Reporting with PowerPoint: PowerPoint can now be used as an export type in the reporting tool.

Font Styling Enhancements: Allow users to change fonts in visualizations.

Multi-Tenant Provisioning: Allows multi distinct tenants under one license.

There are also various performance updates to connections and exports, new application automation templates, new connectors for direct query. Read more.

Q3 2022 Product Release Updates

The new Q3 features position Qlik SaaS better in the market through improving feature parity as well as expanding the abilities of ‘nice-to-haves’.

Qlik AutoML is now fully included in Qlik Cloud Hub

Create automated machine learning experiments in Qlik Sense. Easily train and deploy models using best-in-class machine learning techniques, in a simple, code-free experience. This seamlessly integrates the best-in-class AutoML pipeline with your existing front facing applications and data in Qlik SaaS.

• AutoML’s release directly within the Qlik Cloud UI enables data science teams to quickly prototype analysis directly within Qlik. The ability to leverage user-ready data sources already within the Qlik platform without needing to extract or connect an additional tool is a powerful option for organizations looking at advanced analytics use cases. Also, being able to easily embed this analysis into the data visualization eco-system helps to remove barriers for data science teams to make their work more readily available and consumable by other within an organization.

Qlik Data Gateway – Direct Access

The new Direct Access gateway makes it possible for Qlik Cloud applications to securely access data behind a firewall over a strictly outbound, encrypted, and mutually authenticated connection. Qlik Data Gateway – Direct Access operates behind your organization’s firewall, eliminating the need to open inbound firewall ports to access your data from the cloud for analytics, whether that data is on-premises or in a virtual private cloud. Direct Access should allow organizations who have been held back by IT security concerns to once again look at Qlik Cloud as an option.

• Having Direct Access through the data gateway opens conversations with clients who had previous concerns with security and firewall rules. Being able to simplify the migration and IT setup of Qlik SaaS is a major consideration when recommending a new tool to our clients.

New Authoring Experience

The new UI gives visibility to fields and data usually hidden in the data model viewer. This allows for a more seamless chart creation and layout. It will make sheet design simpler for new developers while still allowing for the fine-grained experience of the old, advanced layout. This should reduce barriers to entry for developers coming from other BI tools and improve abilities for self-service app creation.

• The New Authoring UI improves the user experience for new developers and self-service users of Qlik Cloud. It provides another development friendly interface which is familiar to users of other platforms. Having all the information of an app in one place decreases the need for a disjointed workflow and is more approachable for less experienced users.

Q3 2022 Other News for Developers:

• Inner and Outer Set expressions – You can now pass set analysis filters across an entire expression without needing to apply it within every aggregation. This makes set analysis logic much more concise and simplistic to write for developers.
• Application Chaining – Ability to pass filters from one app to another via button navigation. This improves the user experience for navigation between apps, as a user no longer needs to repeat the filtering when hopping across applications and should decrease response time across applications.
• Schedule App reloads based on data service events – You can now have applications refresh based on source data being refreshed. This will reduce the need for complex timing or issues in the case of reload failures further upstream.
• View App reload log – You can now see multiple reloads to help debug scheduled reload issues.
• Increase to maximum file size – Data files of up to 100GB can now be uploaded (previously, 6GB was the limit).

Q3 2022 Other News for Users:

• UI improvements in the Qlik Cloud Hub. There are now ways to fine tune notifications and favorites in the UI. You can also now see your alerts and subscription in the navigation pane.
• Field-level Lineage and impact analysis – Now available to all users, you can now track the lineage of fields within Dimensions and Measures across the environment to their source. Impact analysis shows the impact a field level change will have downstream from a source. This allows for greater visibility and trust within sources and KPIs within an organization.
• Improvements to Mobile App – Improvements to the UI and performance of the mobile app. Also, setup has been simplified for the tenant via a QR code.
• Font styling for charts and maps – This allows more granular changes to the look and feel of applications.

Q2 2022 Product Release Updates

Chart level scripting 

Chart level scripting is a powerful feature that allows you to modify the dynamic dataset behind a chart using a subset of the Qlik scripting language, with techniques such as variables and loops. You can add or modify rows and columns that were not in the original dataset. This allows for calculations in chart expressions that were previously not possible, such as simulations or goal-seeking.

• Using chart level scripting in visualizations
• Chart level scripting

Chart level scripting adds a new more precise level of customization within chart building. Although Qlik is known to be flexible with what can be achieved via backend scripting and frontend set analysis, there are still some limitations. Using chart scripting, you will have more control over how your visualization looks and behaves and should reduce the need for work arounds or alternative methods of displaying information in your apps. Developers should be cautious in using heavy chart scripting as it may affect performance vs. simpler, built-in chart functionality.

Support for field-level lineage 

As part of delivering explainable BI, new catalog capabilities are now available. Explainable BI is the concept that users will gain more confidence in using data and making decisions based on it, if they have a good understanding, or explanation, of the information’s origin. 

Lineage capabilities have expanded to not only visually show the data history by table but also by the specific field within a table—starting with applications all the way back to the original source. Field-level lineage helps you establish trust in the data. When you explore an app, you can now quickly access a data lineage summary that traces back any dimensions and measures in a chart to the original sources. This makes it easy for any user to understand where the data came from within a chart.

There is also now a distinction between Lineage and Impact Analysis. 

• Lineage shows you a detailed visual representation of the history of a field or dataset, back through the applications and interim datasets to its original source.
• Impact Analysis shows you a downstream view of the dependencies for a data element, including the databases, apps, files, or links that will be directly or indirectly impacted if the value of that particular field changes.

In data cataloging and managements, more information and detail into the lineage of your data is always needed. With field level lineage, you can track the source of a field throughout your pipelines and apps. This should the reduce the need to track a field’s origin by physically opening apps and data sources and provide additional clarity to data consumers.

Q4 2022 Product Release Updates

Tableau unveiled two major releases this past quarter that focus on stronger data management and incremental quality-of-life updates for developers. These updates further emphasize the interoperability between Salesforce and Tableau.

Tableau 2022.4 (December 2022):

• Tableau External Actions: Native integration with salesforce flow to help automate business process by connecting Tableau dashboards to Salesforce so business users can stay in the “flow” by making decisions in context (escalating a case, send an invoice, or any number of CRM workflows). Analytics can be baked into the business process to streamline complex processes and decision making that benefit from analysis and visualization.
• Image Role: An easy, natively supported way to view images as rows or column headers in dashboards. Use cases include fast moving consumer goods, where products are numerous and change frequently. You can now view the product in an analytics setting by mapping the asset location to elements in your data without having to store or maintain images in your dashboards. This feature requires that the image assets be publicly available or accessible on an internal network from Tableau server. Learn more.

Tableau 2022.3 (November 2022):

• Data Guide: A new information pane that provides information about a dashboard and the underlying data behind it. This feature allows you to give descriptions and commentary about the background of the data at a visualization level.
• Tableau Extension: Adds new depth to seeing and understanding data by bringing deep analytics and data shaping with code and APIs to the core of Tableau.
• Dynamic Zone Visibility: Allows developers to tailor experiences for end-users so they only see the dashboard elements relevant to them. This feature allows you to create sophisticated and interactive dashboards that previously required workarounds while introducing additional points of development and maintenance to achieve. Learn more.

Q3 2022 Product Release Updates

Tableau did not release major updates during this quarter. Instead, we will be focusing on one of the major updates in the Q2 release with the greatest value add when it comes to time savings on administrative tasks for developers and admins.

Tableau Cloud: Turning to cloud solutions have proven to help organizations reduce operational costs and take control of their digital transformations. When your organization is faced with this challenge, Tableau is here to help with Tableau Cloud. Prior to the Q2 release, Tableau’s SaaS platform was known as Tableau Online and this update included some feature updates which we highlighted in our previous tech update blog post. In this update, we want to cover the key impacts Tableau Cloud provides for customers.

• Empowers organizations to easily scale and maximize efficiency without the restrictions of self-managing their application
• Distributes analytic autonomy to everyone from analysts to end users
• Cuts overhead costs encompassing analytics deployment
• No investing in hardware setup
• No need for hiring and enabling server administrators

Q3 2022 Other News

Tableau has released some information on their 2022.3 release on their “Coming Soon” page, but we are still waiting for the official release notes.

Q2 2022 Product Release Updates

Data Stories: Tableau wants to tell you stories in their 2022.2 release. The new feature Data Stories builds a “story” based on a data visualization within your dashboard. These plain-language explanations will help any user confidently access, understand, and communicate with data. Developers are given the ability to add story points on distinct visualization to add additional context. Data Stories are also fully customizable so you can tailor the stories based on your audience. 

Autosave: This feature lets you edit an existing workbook in a draft until you’re ready to publish so you won’t lose your changes or share them prematurely with other workbook users (note: this is for web authoring only).  

Quick Search: In Tableau Server, turn to the all-new Quick Search feature to view past searches and suggested content. Quick Search can be accessed from any page and allows you to review results within the search box.  Pairing Quick Search with Tableau’s enhanced Ask Data phrase builder ensures you can spend less time searching for your data and more time engaging with it! 

Tableau Cloud: Tableau Online is now Tableau Cloud! This product is the same easy-to-use self-service platform designed to streamline the power of data to people, just with a new name. 

• Tableau Cloud will now autosave updates to an unpublished draft, which will allow users less familiar with Tableau to make changes without fear of disrupting production, sharing results prematurely, or losing unsaved work. Meanwhile, the new search experience for Tableau Server will save business users time and, hopefully, frustration by quickly returning suggestions based on popular search terms and past searches.

Ask Data: The Ask Data features for Tableau Server and Cloud help make Tableau’s software more accessible to business users as a self-service platform for obtaining data insights. Ask Data can return manipulable visualizations in response to common business questions, allowing users without technical Tableau knowledge to build views to fit their specific needs. Introduced in May, this enhancement is intended to enable business users to create self-service visualizations based upon questions they pose of their data.

Q2 2022 Other News

• If you missed the 2022 Tableau Conference, you could catch up on the keynotes and top sessions on-demand.
• Tableau will enable Multi-Factor Authentication for site administrators on existing sites and all roles on new Tableau Cloud sites.

Q4 2022 Product Release Updates

Filter-Based Subscriptions in Power BI Service: Power BI Service now allows end users to filter their data in an App and then apply a subscription on the filtered results so that many users can specialize their ideal views of the reports.

• This can lead to effective time savings for end users and grants them access to email alerts to their desired outcome of data. The ability to subscribe to a filtered down view of data gives users more power to cultivate their desired viewpoint into their data.

Google BigQuery Data Connection: Power BI now enables users to be able to connect to their Google BigQuery datasets for analysis.

Multiple Audience Groups in Power BI Service: Apps in Power BI Service can now be tweaked to make a custom user experience based on audience groups. In essence, one universal app can be designed so that different departments of an organization can be subjected to different user flows based on security groups.

• This creates a simpler way to build out an organizational user flow of an enterprise-wide app, so that users of all departments and levels can consume their data as they are intended without having to maintain multiple different apps for specific business units.

Slicer Type is now available on the Format pane.

Controlling/Customizing Data Labels in Azure Maps visual.

Improved DirectQuery performance when using TopN filters.

Q4 2022 Other News

Learn more about:

• Dynamic Slicers in Field Parameters (November)
• Quick Measure Suggestions (October)
• Relationship Editing in the Properties Menu (October)
• Deployment History in Data Pipelines (October)

Q3 2022 Product Release Updates

Cross-Tenant Data Sharing: In the October Power BI update, Microsoft introduced the ability to share datasets across Power BI tenants. This new capability allows organizations (providers) to share their Power BI datasets with external users (consumers), allowing them to access the shared dataset from within their own tenant.

• As this is in-place sharing, the data is not moved to another Power BI tenant. As a provider, the dataset resides in your tenant. Consumers query the shared datasets directly in the source data systems. In addition, consumers can also build composite datasets using these Cross-Tenant datasets, eliminating the need to manually transfer data between organizations.
• With Cross-Tenant Sharing, providers can share datasets with consumer tenants within the Power BI Service without the need to manually transfer data or provide access directly to source systems. Consumers are also able to build reports using these shared datasets without ever leaving their tenant. Consumers can enhance these datasets by using them in composite models.
• This update allows organizations to monetize or share datasets with their customers more easily and all within the Power BI service.
• For our customers that are sharing data with other organizations, Cross-Tenant Sharing allows them flexibility in sharing curated datasets directly from Power BI service to their customers/consumers.
• It can potentially remove automated or manual data sharing pipelines that our customers currently maintain provided they’re sharing their data with customers that are reporting within Power BI.

Q3 2022 Other News

There have been several enhancements to existing visualizations in Power BI, providing capabilities found in other BI Tools. To name a few:

• Default Hierarchical Axis
• Filled Map for Azure Map
• New Metric Visualization: This visual provides a huge level of flexibility in integrating metrics into your Power BI reporting solutions. You can now include individual metrics instead of the entire scorecard and create report pages showcasing metrics alongside other visuals, in the context of the rest of the report data.
• This Quarter Power BI also introduced the ability to auto-create a report from an existing dataset. This capability can provide a launching point for report creation.

Q2 2022 Product Release Updates

Feature Release: Power BI Datamarts 

• Released May 2022 at MSFT Build, this feature allows the ability for business users to create data warehouses for custom, self-service, departmental reporting. Power BI Datamarts also unlock the ability for analysts to use SQL queries—one of the most requested features of Power BI. Work that may typically take weeks for a data engineer, including creating data pipelines, a SQL Datawarehouse, provisioning access, or adding any additional code can now be facilitated much faster through the creation of Datamarts.
• How it Works: The entire ETL process can happen inside the Datamart by creating a fully managed Azure SQL database automatically—all done directly within the Power BI web service. Data pipelines and models are built using low-code or no-code PowerQuery tasks online—enabling analysts to do more with their data without all the traditional steps required by data engineers. This enables organizations to help bridge gaps between IT users and business users and reduce time and bottlenecks for self-service and custom departmental reporting.
• Why it’s a Differentiator: While PBI Datamarts are similar to PBI dataflows—they provide an additional value-add by supporting more than just single table use cases—ideal for combining and working with data across multiple sources in a single place. It also provides a great option for Mac users to engage with data in the PBI ecosystem. Data modeling capabilities are enabled by hosting PowerQuery and model tools directly from the web service as opposed to locally developing.

Q2 2022 Other News

• Keep in tune with the Power BI Roadmap for all new and generally available feature releases
• Microsoft Inspire— July 19-20 2022— partner event focused on MSFT cloud partner programs.