I am proud to present to you the new long-awaited Dataedo 23.2! This is probably our biggest release so far. We have a lot of exciting new features and updates for you.
The most groundbreaking new feature of Dataedo, something many of our users have been waiting for, is the new domains. With the new domains, we replaced the old subject areas and met the following requirements:
Added the ability to group assets into domains (now, domains are separate from the sources, unlike subject areas),
Added the hierarchy of Subject Areas (domains can have a hierarchy of subject areas up to 3 levels),
It is now possible to define access to groups of data assets (each domain/subject area has its own permissions, and objects linked inherit those permissions),
Data teams can build a wiki-like portal for business users with the hierarchy of business domains.
We built not one, but two types (hierarchies) of domains: Business and Data Domains.
Business Domains, and the hierarchy of Subject Areas, allow you to break down your organization's business processes and act like a wiki-like portal to the data and concepts in your organization. This can be an easy start with data for business users or even be the place where they learn about various areas and processes of your business.
Data Domains, with the hierarchy of data areas, are meant to represent the Data Team's/IT view and organization of the data and applications. You can use it to represent Data Mesh domains or different departments, teams, applications, modules, or even data products.
From 23.2, objects inherit permissions from domains and areas (business and data). Users that have access to a specific domain/area
When you assign an object to one of the that user has access to he will gain access to all the linked objects. Therefore, domains can be a way of controlling access on the object level.
One of the most exciting features of 23.2 is AI Autodocumentation. Now, you can plug in Open AI's GPT subscription into Dataedo and ask it to generate descriptions for you.
Keyword Explorer is our own unique approach to browsing, searching and discovering metadata found in your databases and other sources. Dataedo analyzes different keywords that occur in table and column names. You can then explore keywords, finding their occurrences in data set names, finding glossary terms that exist with that name, and finding similar keywords.
In 23.2 we redesigned our import engine and now imports should be much faster.
There are still some suboptimal elements in current import mechanism that we will be correcting in subsequent versions. We are planning to remove first step (counting objects) and continue to improve loading time.
Interface tables are set of tables in Dataedo repository database that you can write to raw metadata that you would like to import to repository safely. To load data from interface tables you need to run Interface tables connector.
Changes:
Import of reports
Import of linked sources
Additional fields for importing data lineage (including custom fields)
Ability to build a lineage between different databases
To ship connectors faster and without the need to release new version Desktop, we build a "custom (SQL) connector" functionality. Custom connectors are saved and distributed in proprietary .dataedocon files (XML format). Connectors are built and delivered by Dataedo team or can be build by users (they currently require signing by Dataedo team).
We built and shipped with 23.2 custom connectors to:
We have created a separate connector for Pipelines in Azure Synapse Analytics (Azure Synapse Pipelines) that imports pipelines and builds lineage automatically.
What is imported:
Pipelines
Activities
Source and sink datasets
Linked services
Lineage between pipelines, activities, datasets, and external data sources
Dataedo 23.2 comes with brand new connector to SQL Server Analysis Services (SSAS) Multidimensional model (cubes, measures and dimensions). Previously you could connect to SSAS Tabular, that is completely different technology.
In previous versions you could import big data files from Amazon S3, Azure Blob Storage and Azure Data Lake Storage cloud storages. Now we extended this list with local disk and FPT and SFTP connections.
There are many databases derived from MySQL. Many of them are not compatible with most recent version of MySQL. We created a MySQL Core connector that scans just the most basic metadata therefore giving you the most chance of successful connection to all those other types of databases using this connector.
In 23.2 we added import of object-level lineage for external tables in SQL Server Polybase. Supported sources of external tables are the following: SQL Server, Oracle, MongoDB, Excel, S3, Azure Blob Storage, Azure Data Lake Gen2.
Lineage based on COPY INTO history - Synapse Dedicated instances
Dataedo will create lineage from source file to destination table of COPY INTO command. It requires documenting source file before importing Synapse Dedicated metadata.
Lineage for external tables - Synapse Dedicated and Synapse Serverless instances
External tables will have full lineage from source file to external table in Dataedo 23.2.
From 23.2 Dataedo imports the following object types:
Packages (now moved to Packages folder)
Sources
Destinations
Connections (as Linked Sources)
More detailed lineage
For some types of SSIS processes such as Balanced Data Distributor, Conditional Split, Lookup, Multicast Transform and many more (more details in SSIS documentation) Dataedo can extract column data lineage
Dataedo 23.1 and older supported import of CSV files (comma separated files). CSVs store values separated by comma (","). Some files use this format with other separators (";", "|", "tab"). Now during the import you can provide a separator character.
In 23.2 we introduced a new object type - Linked Source. Linked Source represents a connection in one data source to another data source. In BI and ETL tools those are simply connections. This allows you identify manually links across various sources in Dataedo repository in case Dataedo is not able to detect it automatically (for instance, in one source address is identified by url and in another one by IP). This gives you more control over automated lineage creation and increases scope of what is created automatically.
Now, you can direct parser and lineage builder on what parser/SQL dialect should be used. This gives you more control over parsing and lineage automation and allows you to use one of the existing parsers on sources we don't provide dedicated SQL parsers for.
You can assign SQL dialect to:
Data source
Linked source - link to data source from another data source
Automated data lineage process now marks processes and object and column mapping as deleted, rather than removing it. This is because now processes and column mapping have extra manual metadata that would be lost otherwise.
We added Sample data tab directly to the table form so it is easier to preview table sample rows. This information is stored only in Desktop memory and is not saved in the repository or shared with other users.
Dataedo 23.2 Portal has a convenient preview pane, that presents basic information about selected object, in the example below a column in the table, without the need for opening of a new page. Click arrow next to the name to go to the object page.
Since 10.2 and 10.3 we are saving changes to descriptions, titles and custom fields, both from Desktop and Portal. There is also an option to view this history for each field in Desktop. Now, we added this capability to Portal.
In 23.1 we added in Dataedo Desktop an "Open in Web" button, that allows you to quickly open the same page in Dataedo Portal (formerly Dataedo Web). Now, we added a similar button in Portal that allows you to open the same object in Dataedo Desktop - "Open in Desktop".
In 23.2 Desktop got a complete overhaul of the navigation. The repository explorer now has two layers. Items got grouped into categories what makes it easier to navigate.
Now, you have to double click on the object to open it.
A long awaited new feature - option to go back to the last visited object. We have made a full package - Windows-like breadcrumbs (that in next version will allow copy and paste), back button and history of viewed objects.
I am proud to present you a new Dataedo: 23.1. This is the first release of the next generation of our product. We have a number of new features, improvements, and new metadata connectors and automations.
From this release, we introduced a new numbering system - year and release number. "23" stands for 2023, ".1" is the first major release this year. As usual, we are determined to deliver 3-4 new releases every year, each bringing new and exciting capabilities.
Before you upgrade, please make sure to see the following:
Upgrade requirements - this release requires an upgrade of the following:
Repository
Web Catalog
Desktop
End of support for file repositories - from 23.1, Dataedo will only support server repositories. This is due to our focus on the development of a collaborative data catalog, rather than a standalone application. See the section below for details.
End of support for NetSuite - due to technical constraints, we had to close support for the NetSuite connector in Desktop. See the section below for details and workarounds.
For users who have the Tableau Data Management Addon, there is also the option to export descriptions from Dataedo with a list of linked terms for columns in Database Tables to Tableau. Exported descriptions are inherited by Data Source fields. In the Tableau Web editor, they are always visible to the user, while in the Tableau Desktop editor, exported descriptions are visible only for Published Data Sources.
Apart from extraction and automation of data lineage documentation mentioned above, we also
Dropped limitations of manual lineage creation - now all object types can read and write data (including tables) and have unlimited number of processes
Data Profiling feature got an overhaul. We redesigned UI to make it easier to run, save, refresh and clear profiling, describe column, and improved security by disabling of profiling of classified columns.
Edit descriptions and classifications directly in profiling window
When you are browsing column profile - its distribution, top or sample values - and you learn about the column, you can now describe and classify column directly in the same window.
Dataedo 23.1 has a new notification system that aggregates notifications from community and edits. User gets notifications from comments/edits of object that he/she follows, or when they get mentioned.
This is a useful new custom field type that allows you to assign a person to any asset or element. You could create fields such as Owner, Steward, SME, etc.
With Dataedo 23.1 we're introducing ADKit - command line application that allows to synchronize users between Active Directory and Web Catalog. It can be setup to automatically add, delete, and update users from AD group to chosen group in Web Catalog. It should help with users management (especially in larger organizations), as administrators won't have to create users manually.
Starting with Dataedo version 23.1, Dataedo will no longer support the NetSuite connector. It will not be possible to add NetSuite as a new data source and update changes.
Workaround:
Leave one copy of Dataedo Desktop 10.4 and 10.4 repository, that you will be using to import NetSuite schema. After you import, copy data into interfacing tables in the 23.x repository and use interface tables import.
From 23.1 Dataedo will only support server repository. This change is due to our focus on development of collaborative data catalog, rather than a stand alone application.
It is no longer possible to migrate file repository to a server repository. If you're still using a file repository please make sure you've migrated your repository to a server one before upgrading Dataedo application.
It is now possible to merge two repositories into one and merge its contents.
You can select one of the chosen repositories and click Merge repositories button.
After selecting the second repository to be merged you will be informed on the amount of objects that are going to appear after merging. Please double check if these numbers are correct and make sure you made a backup.
After merging the target repository will contain of all the objects from both repositories.
For repositories created with old versions of Dataedo (5.0 and 7.3), in cases where Web Catalog was never connected, we were missing configuration parameters in the repository. The issue was fixed by validating those edge cases. A similar problem occurred for repositories manually created with scripts from 10.3.2 and was also fixed.
The report is a new type of object next to the Business Glossary and Database. Reporting is an object that contains data sets and visualizations. Check out more about existing object types in Dataedo.
In version 10.4 this feature is coming with the support of the Power BI connector so that you can automatically extract metadata of and document your reports.
Reports and visualizations details
Report preview
Automatic object-level lineage
Column-level lineage will be developed in future versions.
We created a new type of object – Lookup. Lookups allow you to describe values stored in objects’ columns.
They are also known as reference data, lists of values, or dictionaries. You can manually create new lookups or create them automatically based on the values in your database.
You can now drag and drop tabs on the objects overview and organize them however you like. You can even move some tabs outside of the Dataedo and utilize more screens!
Open in Web feature allows you to open objects viewed in Dataedo Desktop in the Web Catalog interface without the need to navigate to them. Find out more in Open in Web - feature article
We introduced an option to disable selected login methods in Web Catalog. You can now access settings and decide which login method should be allowed in your organization.
If you are not using Active Directory login method, you should disable it through settings to ensure security as it is common that default environment is available already, and that can lead to giving everyone who has an AD account an access to the Web Catalog. If you are using Active Directory, you may want to remove or limit permissions assigned to default groups.
Read more about accessing Web Catalog in this article.
We added a view in which one can check permissions from a data source perspective. View lists of all users with some level of access to the data source.