Dataedo version ``24.1` is out! Take a look at this release's highlights:

🚀 24.1 Highlights

• 🟢 New Statuses System Track progress of manual objects with draft/published statuses.

• 🧭 Steward Hub (Beta) Smart suggestions for terms, lookups, links, and data completion.

• 🔁 Stored Procedure Lineage Automatic lineage from T-SQL stored procedures (SQL Server, Azure SQL, Synapse).

• 🧩 New Connector Databricks Unity Catalog support added.

• 📥 Copy & Paste Imports Import data directly into interface tables via grid paste.

• 📈 Connector Enhancements

  • Power BI: Paginated report lineage
  • SSAS, SSIS, Dynamics365, ADF: Expanded lineage & metadata support
  • Jobs/sequences import for Oracle & SQL Server
  • Neo4j: Support for v5.13

• 🌐 Portal Enhancements

  • Refreshed Lineage UI with on-demand loading
  • New license-aware features (e.g., statuses, AI, domains)
  • New login options: AzureAD, Google OAuth, one-time access code

• 🖥️ Desktop Enhancements

  • Spellcheck in multiple languages
  • Admin Console now built into the app
  • Improved UI and performance in profiling & lineage

Bugfixes and improvements:

• Azure SQL Database connector didn't detect changes at column level during import changes

• Steward Hub crashed for SQL Server 2016 and older

• Data lineage was not updated with CMD import

• Data Lineage diagram improvements

• Saving the refresh profiling file crashed when the database didn't have saved password

• Power BI - import fix for reports with external dataset

• SSIS Multidimensional - changes in object names to better represent source objects

• Improvements in detecting linked sources for objects and building data lineage based on connections from those linked sources

Bugfixes:

• Fix for the PowerBI connector

• Fix for the SSRS connector

• Fixes for ERD diagrams in Desktop

• Fix for an issue in upgrade script (for SQL Server 2014 and older versions

I am proud to present to you the new long-awaited Dataedo 23.2! This is probably our biggest release so far. We have a lot of exciting new features and updates for you.

Domains and hierarchy of Subject Areas​

The most groundbreaking new feature of Dataedo, something many of our users have been waiting for, is the new domains. With the new domains, we replaced the old subject areas and met the following requirements:

1. Added the ability to group assets into domains (now, domains are separate from the sources, unlike subject areas),

2. Added the hierarchy of Subject Areas (domains can have a hierarchy of subject areas up to 3 levels),

3. It is now possible to define access to groups of data assets (each domain/subject area has its own permissions, and objects linked inherit those permissions),

4. Data teams can build a wiki-like portal for business users with the hierarchy of business domains.

We built not one, but two types (hierarchies) of domains: Business and Data Domains.

Business Domains​

Business Domains, and the hierarchy of Subject Areas, allow you to break down your organization's business processes and act like a wiki-like portal to the data and concepts in your organization. This can be an easy start with data for business users or even be the place where they learn about various areas and processes of your business.

Data Domains​

Data Domains, with the hierarchy of data areas, are meant to represent the Data Team's/IT view and organization of the data and applications. You can use it to represent Data Mesh domains or different departments, teams, applications, modules, or even data products.

Permissions​

From 23.2, objects inherit permissions from domains and areas (business and data). Users that have access to a specific domain/area

When you assign an object to one of the that user has access to he will gain access to all the linked objects. Therefore, domains can be a way of controlling access on the object level.

Auto documentation with AI​

One of the most exciting features of 23.2 is AI Autodocumentation. Now, you can plug in Open AI's GPT subscription into Dataedo and ask it to generate descriptions for you.

Keyword Explorer​

Keyword Explorer is our own unique approach to browsing, searching and discovering metadata found in your databases and other sources. Dataedo analyzes different keywords that occur in table and column names. You can then explore keywords, finding their occurrences in data set names, finding glossary terms that exist with that name, and finding similar keywords.

Import improvements​

Performance​

In 23.2 we redesigned our import engine and now imports should be much faster.

There are still some suboptimal elements in current import mechanism that we will be correcting in subsequent versions. We are planning to remove first step (counting objects) and continue to improve loading time.

User Experience (UX)​

We built better import status tracking and logging.

Interface tables import improvements​

Interface tables are set of tables in Dataedo repository database that you can write to raw metadata that you would like to import to repository safely. To load data from interface tables you need to run Interface tables connector.

Changes:

1. Import of reports

2. Import of linked sources

3. Additional fields for importing data lineage (including custom fields)

4. Ability to build a lineage between different databases

New connectors​

In 23.2 we added the following connectors:

1. Custom (SQL) connectors

2. MS Fabric

3. Azure Synapse Pipelines

4. Power BI Report Server

5. SSAS Multidimensional (Cubes)

6. Local disk, FTP/SFTP connector

7. MySQL Core connector

Custom (SQL) connectors​

To ship connectors faster and without the need to release new version Desktop, we build a "custom (SQL) connector" functionality. Custom connectors are saved and distributed in proprietary .dataedocon files (XML format). Connectors are built and delivered by Dataedo team or can be build by users (they currently require signing by Dataedo team).

We built and shipped with 23.2 custom connectors to:

1. ClickHouse

2. IBM Informix

3. IBM iSeries

4. SAP Advantage Server (Sybase ADS)

5. Starburst/Trino

Learn more about custom SQL connectors.

MS Fabric​

We created connectors to MS Fabric - unified analytics platform. It allows to connect to Power BI and Data Warehouse inside the Fabric Workspace.

Power BI

Metadata imported with this connector is same as for the conventional Power BI connector.

Learn more about Microsoft Fabric - Power BI.

Data Warehouse

MS Fabrics Data Warehouse will import same metadata as the Azure Synapse connector

Learn more about Microsoft Fabric - Data Warehouse.

Learn more about Microsoft Fabric connector.

Azure Synapse Pipelines​

We have created a separate connector for Pipelines in Azure Synapse Analytics (Azure Synapse Pipelines) that imports pipelines and builds lineage automatically.

What is imported:

1. Pipelines

2. Activities

3. Source and sink datasets

4. Linked services

5. Lineage between pipelines, activities, datasets, and external data sources

Learn more about Azure Synapse Pipelines connector.

Power BI Report Server​

We built a new connector to Power BI - Power BI Report Server. It is a on-prem version of Power BI service.

We import the following objects:

1. Power BI Reports

2. Paginated Reports

3. Paginated Report Datasets

We haven't figure out how to import Power BI Datasets and build lineage for Power BI reports due to limitations of what's provided by the platform.

Learn more about Power BI Report Server connector.

SSAS Multidimensional (Cubes)​

Dataedo 23.2 comes with brand new connector to SQL Server Analysis Services (SSAS) Multidimensional model (cubes, measures and dimensions). Previously you could connect to SSAS Tabular, that is completely different technology.

We import the following objects:

1. Cubes

2. Measures

3. Dimensions

4. Data Source View

5. Data Sources (as Linked Sources)

6. Column level lineage!

Learn more about SSAS Multidimensional connector.

Local disk, FTP/SFTP connector​

In previous versions you could import big data files from Amazon S3, Azure Blob Storage and Azure Data Lake Storage cloud storages. Now we extended this list with local disk and FPT and SFTP connections.

Learn more about FTP/SFTP connector.

MySQL Core connector​

There are many databases derived from MySQL. Many of them are not compatible with most recent version of MySQL. We created a MySQL Core connector that scans just the most basic metadata therefore giving you the most chance of successful connection to all those other types of databases using this connector.

Learn more about MySQL Core connector.

Connector improvements​

In 23.2 we made improvements to the following connectors:

1. SQL Server Polybase

2. Azure Synapse

3. SSIS

4. SSRS

5. Tableau

6. Power BI

7. Azure Data Factory

8. Snowflake

9. Amazon Redshift

10. Oracle

11. CSV/Delimited text files

12. Dbt

SQL Server Polybase​

In 23.2 we added import of object-level lineage for external tables in SQL Server Polybase. Supported sources of external tables are the following: SQL Server, Oracle, MongoDB, Excel, S3, Azure Blob Storage, Azure Data Lake Gen2.

Azure Synapse​

Lineage based on COPY INTO history - Synapse Dedicated instances

Dataedo will create lineage from source file to destination table of COPY INTO command. It requires documenting source file before importing Synapse Dedicated metadata.

Lineage for external tables - Synapse Dedicated and Synapse Serverless instances

External tables will have full lineage from source file to external table in Dataedo 23.2.

Learn more about Azure Synapse Analytics connector.

SSIS​

From 23.2 Dataedo imports the following object types:

1. Packages (now moved to Packages folder)

2. Sources

3. Destinations

4. Connections (as Linked Sources)

More detailed lineage

For some types of SSIS processes such as Balanced Data Distributor, Conditional Split, Lookup, Multicast Transform and many more (more details in SSIS documentation) Dataedo can extract column data lineage

Learn more about SSIS connector.

SSRS​

Learn more about SSRS connector.

Tableau (Prep Builder data flows)​

In 23.2 we now read Tableau Prep Builder data flows into repository.

Upgrades include:

1. Importing of flows with lineage

2. Filtering out system tables during import

3. Building of column level lineage from Custom SQL (represented as datasets) with SQL parsing

4. Importing of fields hierarchy in Data sources

Learn more about Tableau connector.

Power BI​

We upgraded our Power BI connector:

1. Added support for column-level data lineage for live connection

2. Support for Datasets that have been created as a Live connection

3. We added Linked Sources for better control over automatic lineage building (see Data Lineage improvements section)

4. Importing Dashboards as "Dashboard" not a "Report"

Learn more about Power BI connector.

Azure Data Factory​

1. Importing of Linked Services (as Linked Sources)

2. Importing of Data sources and destinations into Datasets

3. Better way of fetching dataset columns

Learn more about Azure Data Factory connector.

Snowflake​

23.2 Snowflake connector includes following upgrades:

1. Loading lineage from data loading history of COPY INTO <table>

2. Loading of lineage for external tables

3. Distinguished external functions as a separate type

Learn more about Snowflake connector.

Amazon Redshift​

We upgraded automated lineage creation for Amazon Redshift connector. Upgrade includes:

1. Import of external tables with lineage

2. import of COPY commands (not COPY Jobs)

3. Import of Data Ingestion (as Linked Sources)

Learn more about Amazon Redshift connector.

Oracle​

In 23.2 we moved Oracle packages and their stored procedures and functions into Packages folder.

Learn more about Oracle connector.

CSV/Delimited text files​

Dataedo 23.1 and older supported import of CSV files (comma separated files). CSVs store values separated by comma (","). Some files use this format with other separators (";", "|", "tab"). Now during the import you can provide a separator character.

Dbt​

Added option to add manifest.json and catalog.json files manually.

Learn more about Dbt connector.

Data Lineage improvements​

Lineage from SQL parsing​

1. Snowflake dialect support

2. CTE support for PostgreSQL, MySQL & Oracle

Linked Sources​

In 23.2 we introduced a new object type - Linked Source. Linked Source represents a connection in one data source to another data source. In BI and ETL tools those are simply connections. This allows you identify manually links across various sources in Dataedo repository in case Dataedo is not able to detect it automatically (for instance, in one source address is identified by url and in another one by IP). This gives you more control over automated lineage creation and increases scope of what is created automatically.

Selectable SQL dialect​

Now, you can direct parser and lineage builder on what parser/SQL dialect should be used. This gives you more control over parsing and lineage automation and allows you to use one of the existing parsers on sources we don't provide dedicated SQL parsers for.

You can assign SQL dialect to:

1. Data source

2. Linked source - link to data source from another data source

3. Data Lineage process

Column level lineage fields​

From 23.2 you can add additional metadata to column mapping:

1. Description - you can provide a description for each column mapping

2. Transformation (as text)

3. Custom fields

Process level fields​

We added a number of fields to process level of the lineage

1. Description - you can describe each process

2. Script - script is imported automatically or added manually

3. Linked Source

Marking as deleted​

Automated data lineage process now marks processes and object and column mapping as deleted, rather than removing it. This is because now processes and column mapping have extra manual metadata that would be lost otherwise.

Data Profiling improvements​

Refreshed UI​

In 23.2 we refreshed the UI of data profiling window:

1. Ribbon options have been grouped

2. Sparklines and progress icons have been updated

3. Context menu have been updated

Blocking classified fields​

Now, Dataedo will not profile and save data for columns that have classification.

You can define what classification labels disable what profiling steps.

Filtering tables​

Now Dataedo allows you to add where clause to tables (and views) to limit rows that will be profiled.

Value distribution chart​

Dataedo now builds value distribution chart for numbers columns, and length distribution for text columns.

Sample data as a tab​

We added Sample data tab directly to the table form so it is easier to preview table sample rows. This information is stored only in Desktop memory and is not saved in the repository or shared with other users.

Re-run profiling​

Now you can re-run profiling for the same set of tables, columns and settings of profiling scope with one button.

Running profiling from command line​

You can also re-run profiling using command line which means you can schedule profiling operation.

Other improvements​

1. You can now also profile saved SQL Queries

2. Profiling skips empty tables

Dataedo Portal​

This section lists changes in Dataedo Portal, formerly known as Dataedo Web Catalog.

Preview pane​

Dataedo 23.2 Portal has a convenient preview pane, that presents basic information about selected object, in the example below a column in the table, without the need for opening of a new page. Click arrow next to the name to go to the object page.

History of changes in Portal​

Since 10.2 and 10.3 we are saving changes to descriptions, titles and custom fields, both from Desktop and Portal. There is also an option to view this history for each field in Desktop. Now, we added this capability to Portal.

Customization​

From now on, you can customize the following elements of Dataedo Portal :

1. Logo

2. Login page background

3. Login page additional information box

4. Portal name

5. Home page additional information box

To customize portal, log in as an admin, go to System Settings > Customization.

Search redesigned​

New search is faster and groups different types of objects in separate tabs.

Open in Desktop​

In 23.1 we added in Dataedo Desktop an "Open in Web" button, that allows you to quickly open the same page in Dataedo Portal (formerly Dataedo Web). Now, we added a similar button in Portal that allows you to open the same object in Dataedo Desktop - "Open in Desktop".

Business Glossary types configuration​

Now, you can add/edit glossary entry types directly in the Portal.

Improved code diff​

In schema change tracking report we improved:

1. Before and after code are aligned to the same lines

2. Marking removed and new fragments in red/green

Dataedo Desktop​

This section lists changes in Dataedo Desktop.

New navigation and double click​

In 23.2 Desktop got a complete overhaul of the navigation. The repository explorer now has two layers. Items got grouped into categories what makes it easier to navigate.

Now, you have to double click on the object to open it.

Breadcrumbs, history and Back button​

A long awaited new feature - option to go back to the last visited object. We have made a full package - Windows-like breadcrumbs (that in next version will allow copy and paste), back button and history of viewed objects.

You can navigate repository in the breadcrumb:

Browser-like stacked history:

Full history (for current session):

Secure repository connection​

To make it more secure to host Dataedo Portal and repository in the cloud Desktop now allows you to connect to the repository over SSH tunnel.

Before you upgrade​

Before you upgrade consider the following:

1. Upgrade requirements - this release requires upgrade of the following:

   • Repository

   • Dataedo Portal (formerly Web Catalog)

   • Dataedo Desktop

2. Check your plan - not all the features added in 23.2 are available in all the plans. Check the overview of plans and check your plan in Dataedo Account.

Bug fix release.

Bug fixes:

1. Fixed metadata import error for SQL Server 2016 and older

I am proud to present you a new Dataedo: 23.1. This is the first release of the next generation of our product. We have a number of new features, improvements, and new metadata connectors and automations.

From this release, we introduced a new numbering system - year and release number. "23" stands for 2023, ".1" is the first major release this year. As usual, we are determined to deliver 3-4 new releases every year, each bringing new and exciting capabilities.

Before you upgrade​

Before you upgrade, please make sure to see the following:

1. Upgrade requirements - this release requires an upgrade of the following:

   • Repository

   • Web Catalog

   • Desktop

2. End of support for file repositories - from 23.1, Dataedo will only support server repositories. This is due to our focus on the development of a collaborative data catalog, rather than a standalone application. See the section below for details.

3. End of support for NetSuite - due to technical constraints, we had to close support for the NetSuite connector in Desktop. See the section below for details and workarounds.

New connectors and improvements​

Tableau connector [new capabilities]​

In Dataedo 23.1, we are adding a new type of connector for Tableau, one of the most popular BI tools.

Dataedo imports the following structures:

• Workbooks and Sheets

• Database Tables

• Custom SQL Tables

• Virtual Connection Tables

• Embedded and Published Data Sources

Dataedo builds column-level data lineage automatically for:

• Flows between Workbooks and Data Sources

• Flows between Data Sources and Tables

• Flows between Tables and source databases that have been imported into Dataedo

Users can migrate the database imported by the Tableau Data Model connector to the Reporting type without losing data.

Export of descriptions from Dataedo to Tableau!​

For users who have the Tableau Data Management Addon, there is also the option to export descriptions from Dataedo with a list of linked terms for columns in Database Tables to Tableau. Exported descriptions are inherited by Data Source fields. In the Tableau Web editor, they are always visible to the user, while in the Tableau Desktop editor, exported descriptions are visible only for Published Data Sources.

Learn more about Tableau connector

Power BI [new capabilities]​

We overhauled our Power BI connector significantly. Improvements include:

1. Creation of column-level lineage between reports, datasets, and sources

2. Importing of dataflows

3. Premium workspace and XMLA Endpoint are no longer required; now you can connect to Power BI Pro

Learn more about Power BI connector

SSRS connector [new]​

We created a new BI connector for SQL Server Reporting Services.

Connector imports the following objects:

1. Reports

2. Datasets - embedded and shared

Learn more about SSRS connector

Snowflake [new capabilities]​

We extended the Snowflake connector. Now it is capable of importing:

1. Snowpipes (added in 10.4)

2. Stages

3. Snowpipe data lineage

Learn more about Snowflake connector

Redash connector [new]​

Dataedo 23.1 can connect to Redash and import Dashboards, Visualizations, Queries, and create lineage automatically.

Learn more about Redash connector

Automated column-level lineage of views from SQL parsing​

In Dataedo 10.4, we automated data lineage creation for SQL views using our own parsing technology.

In 23.1, we extended support for the following dialects:

• MySQL (and forks, such as MariaDB)

• PostgreSQL (supports also Redshift)

• PL/SQL (Oracle)

Since 10.4, Dataedo also supports:

• Transact-SQL (SQL Server, Azure SQL, Azure Synapse)

We are working on support for:

• Snowflake

Data Lineage improvements​

Apart from extraction and automation of data lineage documentation mentioned above, we also

1. Dropped limitations of manual lineage creation - now all object types can read and write data (including tables) and have unlimited number of processes

Data Profiling improvements​

Data Profiling feature got an overhaul. We redesigned UI to make it easier to run, save, refresh and clear profiling, describe column, and improved security by disabling of profiling of classified columns.

Edit descriptions and classifications directly in profiling window​

When you are browsing column profile - its distribution, top or sample values - and you learn about the column, you can now describe and classify column directly in the same window.

Profiling of classified columns is disabled​

To improve security, profiling data, other than distribution, of classified columns is disabled.

Definition of which classification labels should be disabled for profiling is configurable in new Settings option.

Web Catalog​

Web Catalog new design, once again​

We are constantly working on improving Web Catalog design and user experience. We present you the next version of our catalog:

Notifications​

Dataedo 23.1 has a new notification system that aggregates notifications from community and edits. User gets notifications from comments/edits of object that he/she follows, or when they get mentioned.

Mentions​

Now, in community posts you can mention other users:

They will receive a notification of mention:

Your profile​

Now, you can edit your profile name and picture. It will be visible in the community and on User type fields.

Web Catalog improvements​

1. Performance improvements - we did refactoring of some of Web Catalog code and now some of the pages will open much, much faster

2. Security improvements:

   • Secured Stored XSS security vulnerability

New custom field types​

In 23.1 we added a few types of custom fields:

• Tag

• User

• Date

• Color

Tag​

Tag custom field works just like you'd expect it to - users can create and assign labels to various elements of the catalog.

Then you can use those tags for searching.

User​

This is a useful new custom field type that allows you to assign a person to any asset or element. You could create fields such as Owner, Steward, SME, etc.

You can then use to filter by specific person.

Date​

Date custom field allows you to add restrict users to provide valid dates in specific fields.

Color​

Color field allows you to categorize assets and elements in the catalog to predefined color categories, similarly to MS Outlook.

New object types​

Till now, Dataedo had the following main object types in the catalog:

• Reports

  • Reports

• Data assets

  • Tables

  • Views

  • Dataset

  • SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts

• Code

  • Procedures

  • Functions

In 23.1 we made the types more granular, to better represent objects that we import and catalog, and we added some new types:

• Reports

  • Reports

  • Dashboards

  • Visualizations

• Data assets

  • Tables

  • Views

  • Dataset

  • SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts

  • Graph Nodes - taken out of the Tables folder, for documentation of graph databases

  • Graph Edges - taken out of the Tables folder, for documentation of graph databases

  • Search Indexes - taken out of the Tables folder, used to document Search Indexes, i.e. from Elasticsearch

  • Cubes - taken out of the Tables folder, OLAP cubes

  • Dimensions - taken out of the Tables folder, Dimensions from OLAP databases

  • Data Types - might be used to document user-defined data types

  • Sequences - will be used to extract and document database sequences

  • Topics - in the future used to document Kafka topics

  • Application Lists - allow you to document lists in your applications, such as CRM

  • Application Forms - allow you to document forms in your applications, such as CRM

  • Entities

  • Sources - will be used for ETL documentation

  • Destinations - will be used for ETL documentation

• Code

  • Procedures

  • Functions

  • Packages - taken out of the Procedures folder, will be used to document Oracle packages

  • ETL Programs - new object type used to document ETL packages and data pipelines

  • Scripts/Programs - new object type that represents generic script or program

  • Tasks/Jobs - new object type that will be used to import and represent database tasks/jobs

  • SQL Scripts - new object type, similar to SQL Query, that will allow you to document SQL code or snippet (that does not return a dataset)

  • APIs - new object type that represents API method

Note: Your data and imports will stay in the folders as previously. In our next release we will migrate all the data to new folders.

FK Relationship tester - more sources​

In Dataedo 10.4 we created a feature that helps you validate foreign key relationship in Desktop.

In 23.1 we extended support for the following connectors:

• PostgreSQL

• MySQL

• Oracle

• Redshift

• Aurora

Since 10.4 Dataedo supports also:

• SQL Server

• Azure SQL

• Snowflake

We are working on support for:

• Azure Synapse

AD Kit: Active Directory Integration​

With Dataedo 23.1 we're introducing ADKit - command line application that allows to synchronize users between Active Directory and Web Catalog. It can be setup to automatically add, delete, and update users from AD group to chosen group in Web Catalog. It should help with users management (especially in larger organizations), as administrators won't have to create users manually.

AD Kit installation

End of support for NetSuite connector​

Starting with Dataedo version 23.1, Dataedo will no longer support the NetSuite connector. It will not be possible to add NetSuite as a new data source and update changes.

Workaround:

Leave one copy of Dataedo Desktop 10.4 and 10.4 repository, that you will be using to import NetSuite schema. After you import, copy data into interfacing tables in the 23.x repository and use interface tables import.

End of support for file repositories​

From 23.1 Dataedo will only support server repository. This change is due to our focus on development of collaborative data catalog, rather than a stand alone application.

Migrating to server repository​

It is no longer possible to migrate file repository to a server repository. If you're still using a file repository please make sure you've migrated your repository to a server one before upgrading Dataedo application.

Merging repositories​

It is now possible to merge two repositories into one and merge its contents.

You can select one of the chosen repositories and click Merge repositories button.

After selecting the second repository to be merged you will be informed on the amount of objects that are going to appear after merging. Please double check if these numbers are correct and make sure you made a backup.

After merging the target repository will contain of all the objects from both repositories.