v4.36

Data Catalog

This module allows the consultation and enrichment of metadata obtained from the organization's systems. For this, search, filtering and browsing options are provided through the catalog. Additionally, this module connects with both the glossary of concepts and the data quality glossary to allow navigation between the different modules.

When accessing the Catalog we will obtain a list of the available systems, showing for each one of them the volume of structures that have been imported as well as the count by type of structures when hovering over the number.

From this point on we will have several options to find the structure that we want to consult.

  • Discover your system metadata: by clicking on the system we will find a tree-like navigation that you can navigate in order to find your metadata. In this type of navigation we will also have a search box that will allow us to filter the list of current structures to easily find the one that we are looking for.

  • Filtering: You will be able to filter using information discovered from the data source and aditional information documented in your installation using the data catalog templates. Some filter examples: structure type, last changed date, linked to business concept, has aditional information.

  • Search: We will be able to search inside the complete matadata, including any manual fields, structure path and description.

  • Save filters: User will have the option to save his most used filters. In order to do that user will need to provide a name to the combination of filters and will be provided with this as an aditional quickfilter every time he/she enters the data catalog.

Structures

When viewing the detail of a structure we can see the following information:

  • Domain: the data domain to which the structure belongs.

  • System: is the metadata repository from which the information must be obtained

  • Imported: date the structure was included in the catalog for the first time.

  • Updated: date the structure information was last modified in the catalog.

The detail of structures and fields will make a series of tabs available to us, showing or not depending on the casuistry.

  • Fields: Those structures that are composed of fields will have a tab that shows them. If it is a structure that does not have fields, this tab will not be displayed.

  • Notes: In the event that the type of structure has an associated template (see the corresponding administration chapter), a tab will be displayed to manage the notes information. Notes are manually managed metadata for a data structure. We include a workflow in case that this information needs to be reviewed before approval.

  • Grants: Display the permission granted to different users to the corresponding structure

To populate grants local integration need to be included

  • Profiling: It will be shown if we have profiling information for the children fields of the structure or data distribution information for the structure we are viewing.

  • Quality: The list of implementations / quality rules in which the structure we are consulting is being validated will be displayed.

  • Linkage: Tab that will always be shown showing existing links and giving the option to authorized users to generate new links.

  • Versions: The different versions available for the structure will be displayed with the option of being able to browse to consult them.

  • Audit: Details of all manual changes that have been made to the structure are displayed.

Domain modification

When metadata are loaded into truedat a domain is assigned by the connector as specified in the data source. In the data catalog you will be able to reassign the domain of a specific data structure. This is specially important if you are planning to manage diferent permissions to data structures depending on the data domain the are assigned.

Fields

It shows a list of the fields of the application if you have any. For these fields, it shows the name of the field, links to business concepts, link to the traceability of the field and other metadata columns that may have been loaded when obtaining the information from the system.

From this field listing screen we can perform the following actions:

  • Navigate to the detail of a field by clicking on the field name.

  • Navigate to the traceability of a field if you have traceability for that field available.

  • Navigate to the concept associated with our field if there is any (Terms Column)

Notes

This tab will be available in those structures whose type of structure is has been assigend a template. It will allow us to view the metadata entered in our tool to complement the metadata obtained automatically.

There are some permissions linked to this functionality which will allow the users to complete different steps of the process in order to publish a note. In case that you do not need an approval workflow you can just setupo a permission to directly publish for roles that are allowed to modify the notes.

Users will be able to perform different actions depending on permission and note status using following workflow to manage structures notes and depending on user permissions

Notes download

You will be able to download a file with al the notes information for the structures filtered on your search. This file will download in the same format that you need to upload the information and it is used to easily modify notes information in a bulk way before uploading it again.

Bulk load of notes

For users with permission to create notes it will be possible to upload this information with a bulk file. In order to do that there needs to exist a template linked to the data structure type to which you want to add that information. This is the template where the note layout is defined.

The file to be uploaded needs to be a text file with fields separated by semicolon. The first column needs to be "external_id" and the rest of columns correspond to the fields defined in the corresponding template.

You can get the external ids corresponding to your structures by downloading the list of structures from the data catalog screen.

File example:

external_id;extended_description;gdpr
glue:/all_glue/default/table_without_columns;This an example table without any columns;Yes

When creating the file to be uploaded you need to keep attention to the following points:

  • Columns headers need to have a corresponding value in the template of the structure to be updated. In case that they do not match, those columns will be ignored.

  • In case that you include a field which has a list of values defined in the template, the values given need to match with one value in the list. Otherwise the process will fail.

Using the option to download current notes will also help. Using this you can just download, modify and upload again.

Besides when loading the information you will get different options based on your permission level:

  • If you can create notes you will get the option "Upload notes". This will create a new version of the notes in draft status.

You will not be allowed to upload note information for notes which are currently "pending approval"

  • If you have permission to publish notes from draft status you will get the option to "Upload and publish". This will create a new version of the note and publish it directly without having to go through the approval process.

Bulk update

From the main screen of the data catalog, administrator users will have the option of performing a massive update of the additional information associated with the structures. In order to carry out this massive update, the following must be taken into account:

  1. Additional information can only be massively updated for structures that have this option active, that is, have an associated template. Furthermore, only one type of structure can be updated in each action.

  2. All fields are optional and only those for which a value is entered will be updated on the structure. All the fields of the template / type by which it has been filtered will appear.

Once the fields to be updated are filled in and when we press the "Save" button we will be asked for confirmation to carry out the update, informing us of the number of structures to be updated.

Profiling

The profiling of a structure allows us to better understand our data, in such a way that we can better understand the structure that we have and the uses that can be given to it to obtain information within the organization. It will also help us to identify quality problems that we are having with our data in the first instance. In order to have the profiling information loaded, it will be necessary that the corresponding integration tasks have been carried out in order to obtain said profiling data from those systems for which we have such a need.

Two types of profiling display are currently available:

Outlined Summary of Fields of a Structure

The list of fields in the structure that have profiling information and the following information will be displayed, if available:

    • % unique values

    • % null values

    • Lowest data

    • Highest data

    • Fashion (most repeated value)

    • If there is a distribution of values, an icon will be displayed that will link to the outlined field.

Distribution of values ​​for a field

In case of having information on the distribution of values, the profiling tab will be displayed, containing a graph with the different values ​​that we have for said field.

Value patterns

If we have loaded pattern information using the profiling connector, we can also consult this distribution helping us to better understand our data.

This feature is not available for all data sources

Lineage and Impact

If traceability information is available for the selected structure, lineage and impact tabs will be displayed where both diagrams can be viewed quickly. It will also allow us to navigate through the graph as explained in the Data Lineage section

Lineage: It will inform the origin of the data. What information are we using to generate the data?

Impact: It will be shown in which other data structures the data we are consulting is being used. Where would it impact if we make a change in this data?

Grants

Using integration pieces we will be able to document in Truedat the users that have access a certain structures. This will allow a user to see to which structures he/she has been granted access.

Users with permission to view grants of a structure will get a tab in which all the grants that apply to this structure will be displayed. This will include all grants given directly to this structure or inherited from any ascendant.

Quality

A list is shown with the quality implementations where the structure we are consulting is being referenced. Additionally, information on the rule to which this implementation belongs is displayed and navigation is allowed both for the quality rule and for the implementation to which the list refers. The last quality result will also be displayed if we have it.

Linkage

This tab will allow us to consult and manage the links that are available for a structure / field.

Versions

In case of having several versions loaded in the application, it will allow us to navigate between them. This navigation between versions will allow us to consult the fields that made up the structure in previous versions.

When consulting previous versions we will see marked those fields of our structure that have undergone modifications with respect to the current version. They will be marked in the event that these fields have been deleted or modified.

Audit

The data structure audit will show us all the manual actions carried out with said structures, being able to view the changes that have been made in the additional information of said structure as well as the people who have made said changes.

Structure confidentiality

For those users with the appropriate permissions, there is the option of marking a structure as confidential. A structure that is marked confidential will only be visible to people who have permission to manage confidential structures and view confidential structures about the domain in which the structure is located. If you do not want to use this functionality, you simply should not activate the permission in any of the defined roles.

Structure tagging

If your administrator has created structure tags and permissions have been assigned, you will be able to link a tag to structure including a description on why this tag has been assigned to the structure.

Once a tag has been linked to a structure any user with permission to see the given structure will also be able to see the tag and the description (making hover over the tag).

Sharing structures

You will be able to share your structures with others using the corresponding action on the right upper corner of the structure information. This will produce a notification and an email for the users specified in the form.

You will need to include SMTP server configuration in your installation in order to receive notifications via email

Catalog information export

Carrying out any search or filtering within the data catalog, we will have the option of exporting the metadata information contained in the catalog in csv format, so that these data can be processed in a third Excel type application.

Last updated