K Platform Architecture
Last updated: 19/07/2021
Overview
Introduction to K
K is a Data Knowledge platform for discovering, profiling and understanding how data products (data sets, analysis, reports etc) across an Enterprise is used.
K focuses on identifying and storing how users work with data; leveraging this information to enable data producers to improve their products; data owners to take accountability for the proper use of their data; and to scale hidden knowledge to all data workers. The product vision is to become the central platform for all Enterprise data users to easily discover, understand and use data .
K Architecture
...
Services
Component | Description |
Ingestion | The service is used for loading metadata and logs from data sources and tools. |
Profiler | The service is used to identify and profile data assets and their usage. A set of proprietary algorithms are used to automatically match and analyse data assets over their lifecycle. |
Usage | The service is used to monitor and track data assets over time. |
Identity | The service is used to integrate with the Enterprise Identity Management service to provide single sign on. |
Search | The service provides fast, accurate and contextual search for all assets within K. |
Applications | The service is used to access dedicated applications built to solve specific data problems. E.g. migration assessment, impact assessment etc. |
Inventory | The service manages the hierarchical structure for all assets within. |
Scheduler | The service manages the integration and scheduling of ingestion of metadata and logs into K. |
...
Component | Description |
Data Sources | Data sources (e.g. Teradata, Hadoop, Snowflake, SQL Server etc.) where data is stored and used by the Enterprise data teams. K has integrators for many on-premise and cloud data sources and can also ingest custom data sources through the K ingestion framework. |
Data Tools | Reporting and Analytics applications (e.g. Tableau, Power BI etc.) used by the Enterprise data teams to create, manage and distribute content. K has integrators for common data tools and can also ingest custom data tools through the K ingestion framework. |
Identity / SSO | Identity provider and user management sources (e.g. LDAP, SAML, OpenID Connect) that can provide single sign on and user and team data. |
...
Deploying into the Enterprise
Kubernetes
K is deployed using Kubernetes on infrastructure that is managed by the Client. This can be on premise or in the Client’s cloud. A SaaS offering is also available.
Deploying on Kubernetes
Typical Kubernetes services used to deploy K include OpenShift, AWS’s Elastic Kubernetes Service (EKS), Azure Kubernetes Services (AKS) and Google Kubernetes Engine (GKE). The following diagram outlines how K is deployed in a typical Enterprise environment.
...
Components | Details |
KADA Repository | KADA provides a repository for clients to quickly and easily download the K product and updates. |
Considerations
There a several considerations that should be checked prior to setting up K on your Kubernetes environment.
...