Skip to main content

Release notes 26.4.1

ยท 13 min read

Dataedo 26.4.1 introduces the option to set up your own encryption key, to ensure more secure way of handling secrets within your Dataedo repository. This release also modifies docker setup. Additionally, we improved the import logger, and made an array of UI and UX changes to make using the Portal a more seamless experience. As always, this patch also comes with a range of bugfixes and performance improvements to our metadata connectors. Read the whole article for more info!

Breaking docker changes:

  • The Portal UI in Docker is now exposed on port 8080 instead of 80, so port mappings need to be updated accordingly.
  • The Docker agent now requires write permissions to a custom log directory, so if one is configured, these permissions need to be granted.

Security:

  • Your own encryption key: Self-hosted installations can now encrypt the secrets stored in the repository with an encryption key that only you hold. This covers data source passwords and credentials, SSO secrets, the SMTP password, and Teams, Slack, and AI engine tokens. Secrets are encrypted with AES-GCM, a modern authenticated encryption standard. Setting a key is optional; without one, Dataedo works as before.
    • Setting the key: Generate a key in System Settings > Encryption. On IIS, Portal saves it and starts using it right away. In Docker, set it as the Encryption__SecretKey variable. Secrets saved before you set the key are re-encrypted with it automatically.
    • Agents: Every Agent needs the same key as Portal. Enter it in the new Encryption key field of the Agent Installer, or set the same variable in the Agent's container. The Agents list shows whether each Agent's key matches, and an Agent without the matching key doesn't run tasks.
    • Desktop: Dataedo Desktop can't connect to a repository that uses your own key. If your team still works in Desktop, don't set a key yet.
    • Rotating and losing the key: You can rotate the key at any time without re-entering any secrets. Keep the key in a safe place: if it's lost, you can reset encryption, but all secrets then have to be entered again.
  • Changing thresholds of Data Quality rule instances in bulk now requires the Manage External Queries permission for custom SQL instances, as other changes to these instances already do. Before, a request sent directly to the API could change their thresholds without it.
  • The PK/FK tester now checks the user's permissions to selected assets before performing a test. If a user doesn't have sufficient permissions to access table data, they will see a generic error message that doesn't disclose any extra information.

Portal:

  • ERD diagrams:

    • ERD diagrams for Data Sources, Domains, and Data Products can now be included in HTML Portal Exports.
    • When adding objects to a data source's ERD diagram, the Choose object window now applies the Sources filter, so you can find and add objects from other sources.
    • The Relationships tab of data sources with tens of thousands of tables, such as large Databricks catalogs, now opens without an error on SQL Server repositories, and ERD diagrams load faster. In the diagram's search, tables marked Unrelated are now found only by their name and schema, not by their columns.
  • Bulk import:

    • Domains, Areas, and Data Products can now be added in bulk using an Excel spreadsheet.
    • In the Bulk import preview of glossary terms and Data Products, you can now clear the checkbox of any row to skip it when saving, without editing the file. Clearing a new parent term also clears the terms under it.
  • Connection wizard:

    • When extended properties can't be recognized when establishing a connection to an external data source, the wizard no longer skips the Extended properties step; instead, you will see an error message prompting you to check your database.
    • The Linked Sources step now downloads the list of available linked sources every time it is opened. This ensures that when you return to previous steps to add new databases to your import, their linked sources are properly imported.
    • When you see the Wrong connector selected popup, clicking switch connector now opens the detected connector's wizard right away, without any interim prompts.
  • Import summary:

    • Object categories with issues now show Failed or Partially imported instead of Success.
    • In the Data lineage tab, the No data tooltip in the Parser row now explains why nothing was parsed: either no view, procedure, or other object with SQL changed since the last import (lineage from earlier imports is kept), or the import found no such objects.
  • Data Profiling and Data Quality:

    • Arithmetic overflow warnings from Data Profiling on SQL Server, Azure SQL Database, and Synapse now explain the cause and what to do: switch the column's Profiling mode to Statistical or None, or fix the column's definition on the source.
    • Data Profiling now shortens values in Top values and Random values to 256 characters, so columns with long texts no longer produce results of tens of MB or run the Agent out of memory. JSON values stay valid JSON with the same keys and nesting (only texts and arrays are shortened), so JSON detection on import still creates the same nested columns.
    • In Data Quality > Insights, the Missing integer values grid now shows the actual Value min and Value max for MySQL and MariaDB columns, instead of values like 1E+28. This happened when the Agent ran with a decimal comma in its regional settings (for example, Polish or German). Values from earlier runs stay empty until you profile the column again.
    • The Medium filter now properly filters for medium severity data quality results.
    • After a failed or manually canceled Classification run, the next successful run now applies semantic type rule changes and large row count changes made before the failed run, instead of skipping them.
  • Scheduler:

    • When viewing tasks from the Calendar level, the menu of a profiling run with warnings or errors now properly displays the Check warning message or Check error message button.
    • When you trigger an import task manually, the Log level field is replaced with a Collect detailed logs checkbox. Select it to log that single run at the Debug level; otherwise, the run uses the log level configured for the Agent.
  • Data Lineage:

    • When exporting Data Lineage as a PNG, all the node labels are now shown properly on the resulting image.
    • On data source and area lineage diagrams, the Data loop badge no longer marks objects that a processor writes to in one process and reads from in another. A loop is now shown only when data enters and leaves the processor in the same process.
  • Back and Forward buttons in Search now restore the search phrase, filters, tab, and tab title to match the URL.

  • Sessions of users with editing permissions are now recorded correctly, so usage dashboards count editors' activity from this version on.

  • After a repository upgrade, Steward Hub suggestions are now refreshed correctly. Before, the refresh could start before the upgrade finished and fail, so suggestions weren't recalculated.

  • A network error in a single widget or dashboard request (for example, blocked by a VPN, proxy, or browser extension) no longer sends Dataedo into a loop with the Service is offline page.

  • UI fixes:

    • You can now close dialog and popup windows using the Escape key on your keyboard, and reopen them afterwards.
    • The Ctrl + K (or Cmd on Mac) shortcut now opens a search bar.
    • Mentions in the community section now initially load only up to 20 users; other users can still be found by typing after the @ symbol. This change improves performance in Dataedo instances with a significant user base.
    • This version unifies the maximum title length for all objects in the Dataedo repository: 256 characters.
    • In the glossary's sidebar navigation, Admin users now see proper name labels of unpublished terms, rather than the restricted access label. This matches the expected glossary behavior for that user role.
    • Removed visual bugs related to elements rendering in incorrect places, e.g., buttons overlapping with grid columns, warning icons for overlong descriptions hugging text, popup frames cutting content.
    • The Linked Terms section of report sheets and nested columns now also lists terms linked to their fields or child columns.
    • In multi-choice lists, destructive actions (like permanently deleting an asset) are now always listed last, and separated from the other options with a thicker line. Buttons used to confirm destructive actions are now red.
    • The relationship grid in the Relationships section no longer vanishes when entering or exiting full screen mode.
    • Shortened names in navigation sidebars now look neater.
    • On the Catalog Assets list, sorting by a custom field column now orders the sources by its values, instead of emptying the list.
    • Improved the interactivity of the Data Quality rule instances grid.
    • DBMS column content is now displayed properly
    • Data Lineage diagrams:
      • Improved the diagram layout and unified the spacing between nodes.
      • Highlighting logic is more intuitive now.
      • The Ctrl (or Cmd on Mac) + mouse wheel shortcut now lets you zoom in and out.
      • Node inflow counters are hidden if the total number of inflows is equal to one.
      • Processors got a visual update and are now grouped just like other object types.
      • The Base frame is now collapsed into a single node. Expanding it manually lets you view the entire frame as before.
      • On a table's diagram, switching to All columns now shows classifications and data quality of the added columns right away, without refreshing the page.

Agent:

  • Agent now properly connects to a data source through the SSH tunnel if it was set up when scheduling tasks. If a tunnel can't be opened, the task ends with an error message.
  • When upgrading a running Agent, the installer now shows a Checking running tasks... progress screen while it checks the repository for tasks in progress.
  • With a PostgreSQL repository, regular imports no longer skip object changes made shortly after the object's previous modification when the Agent runs outside the UTC time zone.
  • Agent now always recalculates the permissions cache after a metadata import task.
  • Agent now properly cleans old notification events on large repositories.
  • From version 26.4.1 onwards, the console installation of Agent is no longer supported.

Connectors:

  • Data lineage from SQL parsing:
    • Table and view names used without a schema in views and procedures now link to the object in the same schema, rather than a random object with the same name in another schema or database. If several schemas match, no lineage is created and a warning appears in the Import summary.
    • Column lineage for SQL with long chains of CTEs, derived tables, or temporary tables is now generated in milliseconds instead of minutes or hours, so such procedures no longer stall or fail the import.
    • Object-level lineage now shows as sources only the tables that supply data to the result. Tables used only to filter rows (for example, in JOIN conditions or WHERE) and queries returning only constants or COUNT(*) no longer create inflows.
  • Snowflake:
    • Lineage is now created for SQL that references columns by position ($1, $2, t.$1), including INSERT ALL/INSERT FIRST, INSERT ... SELECT, SELECT * and SELECT a.*, b.* (also over joins), temporary tables, derived tables, and CTEs.
    • Subqueries in INSERT ALL/INSERT FIRST values no longer produce circular lineage or wrong column sources. This also applies to Oracle.
    • Fixed lineage between stages and tables loaded with COPY INTO:
      • A table is no longer linked to every stage that shares the same location; lineage is created only when exactly one stage matches.
      • Scripts that are no longer found are marked as deleted on import with Data lineage from query history enabled.
      • Lineage from GET_LINEAGE (Enterprise edition and higher) no longer requires access to SNOWFLAKE.ORGANIZATION_USAGE.ACCOUNTS. If the import role lacks the VIEW LINEAGE privilege, a warning now appears in the Import summary.
  • Power BI:
    • Gen2 dataflow issues, such as a definition the import account can't read, are now reported under Dataflow tables in the Import summary, and lineage from datasets and dataflows is no longer labeled Unknown.
    • When the import account's workspace role (for example, Viewer) doesn't allow reading semantic model or Gen2 dataflow definitions, the Import summary now shows a warning about missing permissions, instead of an authentication failure or an error with Contact support. Tables kept from a previous import are no longer described as missing. Reading definitions requires at least the Contributor role.
    • Semantic model tables that use a native SQL query in the connector's Query option, as saved by Power Query Online, now get table and column lineage.
    • When a semantic model and a dataflow read the same database with different letter case in its name, the import now reuses the existing linked source instead of creating a duplicate. Duplicates created by earlier versions are not merged.
    • Paginated report datasets now get lineage from views and from tables named without a schema. Tables missing from the documentation and data sources that can't be linked are now listed in the Import summary.
    • Paginated report data sources with the same name but different servers or databases now get separate linked sources, and their datasets' lineage follows manual linked source mapping. Existing mappings are kept.
  • Databricks Unity Catalog: Large catalogs (tens of thousands of objects) no longer show Request timed out on the Objects to Import and Metadata Sync tabs of a data source, or in the object selection step of the connection wizard. Dataedo now lists tables in far fewer requests to Databricks.
  • dbt Cloud: Jobs running on dbt Fusion can now be selected when connecting to dbt Cloud. Each job in Portal now shows whether its files can be imported, and the run they come from. Job's Files come from the most recent run (up to 3 back) that has both manifest.json and catalog.json, so a run that stopped before generating docs no longer fails the import.
  • MongoDB: Error logs when a wrong credential was used are no longer blank.
  • PostgreSQL: Data-based classification of JSON columns no longer uses server memory in proportion to the number of JSON fields, which could run the database out of memory on large documents. JSON fields inside arrays are now classified by name only, instead of getting a 0% data match.
  • SSIS: Attempting to connect to SSIS Server using SQL Server Authentication now fails with a clear message to switch to Windows Authentication, as SQL Server doesn't allow downloading packages with this authentication.

Desktop:

  • When Desktop crashes, all of its background processes are terminated as well.
  • Importing changes to documentation whose custom connector was removed, is missing, or can't be verified no longer crashes Desktop. The import now stops with a message explaining the cause. Learn how to fix it.
  • Custom connectors used by documentation can no longer be removed in Manage Custom Connectors. Desktop now lists the documentation that uses the connector and explains how to install its new version instead. Learn more.