Metadata scanning overview
Types of connectors
Native connectors
Dataedo is shipped with a number of native connectors that can connect to specific technologies: databases, applications, BI tools, ETL tools, etc., and extract metadata.
- Database connectors: SQL Server, Snowflake, PostgreSQL, and more
- BI connectors: Power BI, Tableau, SSRS, and more
- Application connectors: Microsoft Dynamics 365, Salesforce
- Metadata repositories connectors: AWS Glue Data Catalog, Apache Hive Metastore, Microsoft Dataverse
- File connectors: Apache Parquet, Delta Lake, CSV, JSON, and more
- ETL connectors: Azure Data Factory, SSIS, dbt
- Storage access: Local disk, FTP/SFTP, Amazon S3, Azure Blob Storage/Data Lake Storage
Browse full list of connectors
ODBC connector
ODBC (Open Database Connectivity) is a standard way of accessing databases. Most databases, applications, and other sources provide ODBC drivers that you install and configure on your workstation. Once configuration is set, you can then use the Dataedo ODBC connector, select the connector, and metadata will be imported from your source. Please note that metadata available through most ODBC drivers is limited.
Learn more about ODBC connector
Custom SQL Connectors (from 23.2)
In version 23.2, we are unveiling a mechanism that allows for the creation of custom connectors for data sources that support SQL and extract metadata with data dictionary tables (supported by most relational databases). These custom connectors are delineated through a series of predefined SQL queries that extract metadata.
Furthermore, these connectors have the capability to support data profiling, reference data, and primary key/foreign key tester functionalities.
As of now, the custom connectors are crafted exclusively by the Dataedo team. However, we are working towards empowering users to create their own custom connectors in forthcoming updates.
Learn more about custom SQL connectors
Metadata import with interface tables
Dataedo provides a powerful feature that allows users to import various metadata into the catalog using interface tables - a set of predefined tables where users can upload metadata extracted on their own and run an import that loads it safely into the repository.
Learn more about interface tables
SQL DDL imports
Sometimes you cannot allow third-party software to connect to your database, but you can dump the database structure with a set of SQL DDL scripts (CREATE statements). Dataedo can import schema from DDL scripts in selected SQL dialects.
Learn more about importing metadata from DDL scripts
Metadata extraction techniques
Dataedo uses the following techniques to extract metadata from your source:
Scanning data dictionary tables
Most databases and data platforms provide System Catalog / Data Dictionary tables. Dataedo scans those tables to identify tables, columns, and other structures.
Calling APIs
Some data sources provide APIs that deliver metadata.
Sampling documents
Sources like document stores (MongoDB, for instance) require sampling of documents to identify their structure.
Parsing report/ETL code
To extract metadata from BI or ETL tools, Dataedo needs to parse the code of reports and packages.
SQL Parsing
To build lineage for SQL code, Dataedo parses SQL to identify objects used, their columns, and data movement.
Data profiling
To extract information about column data profiles (min-max, average, number of nulls, etc.), Dataedo executes a number of predefined data profiling SQL queries.
