Quick Start

Get a full ODIN stack running locally in under five minutes using Docker Compose.

1. Clone and configure

git clone https://github.com/ODIN-Data-Intelligence/odin.git
cd odin
cp .env.example .env          # review and edit credentials

2. Start the stack

make up
# or: docker compose up -d

# Watch services come healthy:
docker compose ps

Services start in dependency order. Allow ~60 seconds for Kafka, PostgreSQL, and OpenSearch to initialise before the Spring Boot services become healthy.

3. Create your first dataset

curl -s -X POST http://localhost:8001/api/v1/datasets \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "X-Tenant-Id: 00000000-0000-0000-0000-000000000001" \
  -d '{
    "title": "Trade Blotter",
    "description": "Intraday trade records from the front-office OMS.",
    "keywords": ["trading", "blotter", "positions"],
    "accrualPeriodicity": "daily"
  }' | jq .

4. Open the frontend

5. Load sample data

make seed        # loads financial services sample data
make reindex     # pushes all datasets into OpenSearch

The seed script creates 12 financial datasets, 5 data products, logical models with FIBO vocabulary mappings, and OpenLineage pipeline events for a BCBS 239 risk aggregation scenario.

Prerequisites

Runtime requirements

Development requirements

Optional — AI features

# Install Ollama for local LLM inference
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull nomic-embed-text   # embedding model (768 dimensions)
ollama pull llama3             # chat model

# Then start the AI profile:
docker compose --profile ai up -d

Without Ollama, the ai-service will not start. All other services function normally. You can also configure an OpenAI key in .env instead.

Configuration

All runtime configuration is driven by environment variables. Copy .env.example to .env and edit before running make up.

Core variables

AI variables

Architecture Overview

ODIN follows Domain-Driven Design with a database-per-service pattern. Seven Spring Boot 3.3 microservices communicate via Kafka events. Traefik routes external HTTP traffic.

Design principles

• API-first — every capability is a versioned REST endpoint before any UI is built on top.
• Database-per-service — no service shares a database with another. Cross-service reads go through REST or Kafka events.
• Event-driven — state changes publish Kafka events on log-compacted topics. Downstream services maintain their own read models.
• Standards-based exports — the catalog exports DCAT 3.0 JSON-LD via Apache Jena; the lineage service accepts OpenLineage JSON.

Services

Event Topology

All inter-service communication uses Kafka with an envelope schema that carries tenant, event type, and schema version on every message.

Topics

Event envelope

{
  "eventId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "eventType": "DatasetCreated",
  "schemaVersion": "1.0",
  "producerService": "inventory-service",
  "tenantId": "00000000-0000-0000-0000-000000000001",
  "timestamp": "2026-05-18T10:23:00Z",
  "payload": { ... }
}

Security Model

Authentication methods

Tenant isolation

Every resource row carries a tenant_id UUID. When using Keycloak JWT tokens, the tenant is extracted from the tenant_id claim automatically. When using API keys, the key's associated tenant is used. Rows from other tenants are never returned.

See Roles & Login for the four defined roles, producer UI login flow, and how to add users in Keycloak.

Metamodel Overview

ODIN's metamodel has three tiers: conceptual (business), logical (semantic), and physical (technical).

Layer responsibilities

DCAT Datasets & Distributions

ODIN models datasets and distributions using the DCAT 3.0 vocabulary. The full catalog can be exported as DCAT JSON-LD.

Dataset fields

DCAT export

# Export full catalog as DCAT 3.0 JSON-LD
curl http://localhost:8001/api/v1/catalogs/{id}/export \
  -H "Accept: application/ld+json" \
  -H "X-API-Key: dev-local"

DPROD Data Products

Data products are modelled using the OMG DPROD standard. A data product represents a business-owned, governed unit of data with a defined lifecycle.

Lifecycle stages

Create a data product

curl -X POST http://localhost:8001/api/v1/data-products \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "title": "Trade Risk Data Product",
    "description": "Aggregated risk metrics for regulatory reporting.",
    "lifecycleStatus": "Consume",
    "keywords": ["risk", "trading", "BCBS239"],
    "informationSensitivity": "Internal"
  }'

CSV-W Physical Schema

The physical layer is modelled using CSV on the Web (CSV-W). Each distribution that is harvested from a source system produces a CSVWTable with a CSVWSchema containing one CSVWColumn per field.

Column fields

Physical columns are created automatically during harvest. Each CSVWColumn carries an optional logicalDataElementId FK that, when set, creates the logical–physical binding. A single LogicalDataElement may be bound by multiple physical columns across different distributions or schema versions.

Logical Models

A LogicalModel belongs to a Dataset and provides the business-oriented view of its structure. It contains LogicalDataElements — each representing a named business concept with an optional binding to a physical column and zero or more vocabulary mappings.

LogicalDataElement fields

Bind a physical column

curl -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/bind \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{ "physicalColumnId": "b3f1a2e4-..." }'

AI description recommendation

For each element in a model the AI service can generate a plain-English business description grounded in the element's name, label, logical type, and vocabulary mappings. The recommendation is stored in a recommendedDescription field and surfaced as an inline suggestion in the Model tab — a data steward reviews and accepts or dismisses it.

# Request a description recommendation for one element
curl -s -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/recommend-description \
  -H "X-API-Key: dev-local"
# → { "elementId": "...", "recommendedDescription": "The gross notional value of the trade,
#      expressed in settlement currency before any netting adjustment." }

# Accept the recommendation (writes to description field)
curl -s -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/accept-description \
  -H "X-API-Key: dev-local"

# Bulk — request descriptions for all elements in a model
curl -s -X POST \
  http://localhost:8001/api/v1/logical-models/{modelId}/recommend-descriptions \
  -H "X-API-Key: dev-local"

Auto-scaffold from harvest

When a harvest run discovers columns for a dataset that has no published LogicalModel, ODIN automatically generates a draft LogicalModel with one LogicalDataElement per CSVWColumn. Each harvested column has its logicalDataElementId set to the newly created element, and its logicalType is inferred from the source datatype. You can then enrich the draft with business names and vocabulary mappings.

Vocabulary & FIBO

ODIN ships with seven system vocabularies pre-loaded. You can register additional RDF vocabularies at any time.

Pre-loaded vocabularies

SKOS match types

Add a vocabulary mapping

curl -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/vocab-mappings \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "vocabularyId": "...",
    "conceptIri": "https://spec.edmcouncil.org/fibo/ontology/FND/Accounting/CurrencyAmount/MonetaryAmount",
    "conceptLabel": "MonetaryAmount",
    "matchType": "exactMatch"
  }'

Vocabulary & AI

Semantic vocabularies are the missing layer between your data and AI. This page explains why standard concept IRIs — schema.org and FIBO — transform how language models reason over catalog metadata, and how ODIN makes that connection operational.

Ambiguity is the root cause of AI failure

RAG pipelines retrieve chunks of text. Without semantic grounding, a question about "settlement amount" returns every table that mentions the word "amount." A SKOS exactMatch binding to fibo-fnd-acc-cur:MonetaryAmount makes retrieval precise — the model finds the right element, not the most popular one.

Vocabulary mappings give the search layer a signal that survives synonym drift, column renaming, and schema evolution. The concept IRI is stable even when the column name is not.

Standard IRIs are native to foundation models

schema.org and FIBO IRIs appear extensively in the training corpora of every major LLM. Annotating a data element with https://schema.org/price or fibo-md-temx-ex:MarketPrice puts it in semantic proximity to everything the model already knows about that concept — zero prompt engineering required.

The model does not need to be told what a LegalEntityIdentifier is. It learned from the FIBO specification itself. Your vocabulary mapping surfaces that latent knowledge at query time.

Agents need contracts, not descriptions

As AI moves from answering questions to taking actions — writing pipelines, generating reports, triggering workflows — it needs to know exactly what data it is handling. A vocabulary mapping is a contract:

ODIN exposes these contracts through the logical model API. AI agents can read the full vocabulary profile for any dataset before writing a pipeline that touches it.

Your metadata becomes a knowledge graph

ODIN's vocabulary mappings, logical models, and lineage edges form a traversable knowledge graph stored in Apache AGE. AI agents don't just search it — they walk it.

-- Walk from a regulatory report upstream through lineage,
-- then sideways through vocabulary to equivalent columns in other datasets
MATCH (report:Dataset {name: 'RISK_AGGREGATION'})
      <-[:DERIVED_FROM*1..4]-(src:Dataset)
RETURN src.namespace, src.name, src.fibo_concepts

From a regulatory report, upstream through lineage to source systems, sideways through vocabulary to equivalent concepts in other datasets. That kind of reasoning is only possible when meaning is explicit.

FIBO: regulatory-grade semantics, pre-loaded

The Financial Industry Business Ontology is the only semantic vocabulary built specifically for financial data with regulatory intent. When an AI model encounters a FIBO-annotated dataset, it has access to the same ontological structure that regulators, risk managers, and auditors use.

Cross-system equivalence without ETL

Different source systems use different column names for the same concept: trade_ccy, SETTL_CURR, SettlementCurrency. All three mapped to fibo-fnd-acc-cur:Currency with exactMatch become interchangeable to any AI agent — without moving a byte of data.

# Find all datasets with a column mapped to fibo Currency concept
curl "http://localhost:8004/api/v1/search?fibo_concept=fibo-fnd-acc-cur%3ACurrency" \
  -H "X-API-Key: dev-local"

Semantic equivalence is free once vocabulary mappings exist. The search index stores fibo_concepts as a keyword facet — no text matching, no NLP, just exact IRI lookup across every dataset in the catalog.

IRI → label translation

Raw concept IRIs are never shown in the UI. The inventory-service resolves them to human-readable labels via GET /api/v1/vocabularies/translate?iri=<iri> (single) and POST /api/v1/vocabularies/translate (a JSON array of IRIs → {“translations”:{“iri”:”label”,...}}). The frontend batches lookups and caches results in localStorage, so a mapping like fibo-fnd-acc-cur:Currency renders as “Currency” everywhere.

DPV & DPV-PD: privacy vocabulary for AI PII detection

ODIN pre-loads the W3C Data Privacy Vocabulary (DPV) and its personal-data extension (DPV-PD) alongside FIBO. DPV gives the AI service a precise, machine-readable vocabulary for classifying data elements as personal information or direct identifiers — the same vocabulary referenced in GDPR guidance and privacy impact assessments.

When you request PII recommendations for a logical data element, the AI service maps each element to a DPV-PD concept IRI (e.g. https://w3id.org/dpv/pd#Name, https://w3id.org/dpv/pd#EmailAddress) and sets isPersonalInformation / isDirectIdentifier flags. Accepted recommendations are stored as vocabulary mappings under the dpv prefix and propagate into the ODRL terms-of-use derivation — a column mapped to any DPV-PD direct-identifier concept automatically elevates the dataset's access level toward HIGHLY_RESTRICTED.

OpenLineage Integration

ODIN's lineage-service exposes an OpenLineage-compatible HTTP endpoint. Any tool that emits OpenLineage events (Spark, dbt, Airflow, Flink) can send lineage directly to ODIN.

Send a lineage event

curl -X POST http://localhost:8003/api/v1/lineage \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "eventType": "COMPLETE",
    "eventTime": "2026-05-18T10:00:00Z",
    "producer": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark",
    "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent",
    "run": { "runId": "3fa85f64-5717-4562-b3fc-2c963f66afa6" },
    "job": { "namespace": "TRADING_DB", "name": "risk_aggregation_job" },
    "inputs":  [{ "namespace": "TRADING_DB.BLOTTER", "name": "TRADE_BLOTTER" }],
    "outputs": [{ "namespace": "REGULATORY_DB.BCBS239", "name": "RISK_AGGREGATION" }]
  }'

Query lineage graph

# Upstream lineage, 4 hops
curl "http://localhost:8003/api/v1/datasets/REGULATORY_DB.BCBS239/RISK_AGGREGATION/lineage?direction=upstream&depth=4" \
  -H "X-API-Key: dev-local"

# Downstream impact analysis
curl "http://localhost:8003/api/v1/datasets/TRADING_DB.BLOTTER/TRADE_BLOTTER/impact" \
  -H "X-API-Key: dev-local"

Lineage is stored in an Apache AGE property graph on PostgreSQL. Cypher queries traverse DERIVED_FROM, READ_BY, and WRITES_TO edges. Column-level lineage uses COLUMN_LINEAGE edges.

Semantic Types

A semantic type is the business domain type a dataset contains — e.g. Customer, DebitCardAccount, LoanOrCredit. Types are derived automatically from the vocabulary mappings on a dataset's published logical models, so they need no curation beyond mapping elements to FIBO / schema.org / SKOS concepts.

How a type is derived

• Only exactMatch and closeMatch mappings count — stronger signals than broad, related, or narrow matches.
• The type is the terminal fragment of the concept IRI: everything after the last / or #. For example …/ClientsAndAccounts/Customer → Customer, and https://schema.org/LoanOrCredit → LoanOrCredit.
• Types are de-duplicated across all elements of the dataset's published logical model(s).

Semantic context

The full picture is returned by GET /api/v1/datasets/{id}/semantic-context:

curl -s http://localhost:8001/api/v1/datasets/$ID/semantic-context \
  -H "X-API-Key: dev-local"
# → { "semanticTypes": ["Customer", "DebitCardAccount"],
#     "vocabConceptLabels": [...], "fiboConcepts": [...],
#     "logicalElementNames": [...], "logicalTypes": [...],
#     "acceptedTags": [ { "id": "...", "type": "Customer", "vocabularyIri": "..." } ] }

Where types surface

• Search — indexed as the OpenSearch semanticTypes keyword field and exposed as a facet (filter with ?semanticType=Customer).
• Consumer UI — rendered as colour-coded badges on the dataset detail view (blue for FIBO, green for schema.org).
• AI chat — types are embedded into the dataset's vector chunks, so questions like "which datasets contain Customer data?" surface the right datasets.
• Manual tags — stewards can accept AI-recommended or hand-entered types via POST /api/v1/datasets/{id}/semantic-tags; see the AI Service for recommendations.

Governance & Audit

Every dataset has an optional data owner and an immutable audit history. Ownership only changes hands through a proposal workflow, so a transfer is never unilateral.

Assigning an owner

An unowned dataset can be claimed with PUT /api/v1/datasets/{id}/owner (body {"userId":"..."}). If the dataset already has an owner the call is rejected with 409 — use the transfer workflow instead.

Ownership transfer workflow

1. Propose — POST /api/v1/datasets/{id}/ownership-proposals with {"proposedOwnerId":"..."} creates a PENDING proposal.
2. Approve / reject — only the current owner (or a catalog admin) may call …/ownership-proposals/{proposalId}/approve or …/reject. Approval atomically updates the dataset's ownerId.
3. Pending queue — GET …/ownership-proposals/pending returns the open proposal, or 204 if none.

A proposal carries status (PENDING / APPROVED / REJECTED), proposedOwnerId, proposedById, and timestamps.

Audit history

GET /api/v1/datasets/{id}/history returns a reverse-chronological, paginated log. Each entry records the eventType, who made the change (changedById / changedByEmail), and JSON snapshots payloadBefore / payloadAfter.

Tracked events: CREATED, UPDATED, DELETED, OWNER_ASSIGNED, OWNER_TRANSFER_PROPOSED, OWNER_TRANSFER_APPROVED, OWNER_TRANSFER_REJECTED.

AI action gating

Accepting or rejecting an AI-generated classification recommendation is gated to the dataset's current data owner. The producer UI enforces this in the Model tab — the Accept / Reject buttons are shown only when the logged-in user is the dataset owner or an administrator. The same restriction applies to description recommendations. This ensures that AI suggestions never modify metadata without an accountable human approving the change.

Governance dashboard

The producer UI dashboard (http://localhost:3000/{tenant}) provides a personal governance view for every logged-in user:

• Stat cards — count of datasets and data products the user owns.
• Outstanding Tasks — pending ownership transfer proposals directed at the user (nomination or transfer awaiting their acceptance).
• My Proposals — ownership proposals the user has submitted or been nominated for, with current status (Pending / Approved / Rejected) and resolution notes.
• My Changes — a chronological feed of dataset events the user performed: Created, Updated, Assigned Owner, Proposed Transfer, Approved/Rejected Transfer.

The dashboard data is served by GET /api/v1/dashboard/summary (stat counts) and GET /api/v1/dashboard/activity (proposals + changes feed) on the inventory-service.

Terms of Use (ODRL)

ODIN derives a machine-readable ODRL terms-of-use policy for every dataset automatically — no manual policy authoring required. The policy is inferred from two signals already present in the catalog: element classifications and vocabulary concept mappings.

How derivation works

1. Effective classification — the most restrictive accepted classification across all published logical model elements determines the access level. Order: HIGH_CONFIDENTIAL > CONFIDENTIAL > INTERNAL > PUBLIC.
2. Regulatory signals — FIBO vocabulary IRI prefixes (fibo-fbc, fibo-sec, fibo-md) and dataset keywords (mifid, emir, gdpr, basel, finrep) identify applicable regulatory frameworks and add corresponding obligations (e.g. "Comply with market data vendor licence terms" when fibo-md concepts are mapped).

Access levels

API

Accept pre-condition

The Accept Policy action requires that every element in the published logical model has both an accepted classification and at least one vocabulary concept mapping. While unmet, the producer UI shows per-element readiness hints — e.g. "3 elements still need classification" — rather than disabling the button silently.

Policy source

The policySource field in the response indicates provenance:

• derived — computed live from current classifications and vocabulary mappings.
• explicit — a data owner has accepted and locked the derived policy via POST .../accept. The stored ODRL JSON is returned as-is.
• fallback — no element classifications found; terms fall back to the dataset's declared license / accessRights / rightsStatement fields.

Consumer UI

Every dataset in the consumer discovery drawer has a Terms tab showing the access level badge (colour-coded), Permitted Uses, Restrictions, Obligations, Applicable Regulations (as pills), and a collapsible ODRL JSON block for technical consumers.

Producer UI

The producer Governance tab shows a Terms of Use Policy panel with derivation details (classified element count, vocabulary concept count, matched signals). The data owner can Accept Policy to lock the derived terms, or Reset to Derived to clear a locked policy.

Database ERDs

Six databases, one per service. All primary keys are UUID. Foreign keys are intra-database only; cross-service references are soft UUID columns without a database-level FK constraint.

Use the zoom controls or scroll wheel to navigate large diagrams. Click and drag to pan.

Inventory Service

The inventory-service is the primary metadata store. It owns all DCAT, DPROD, CSV-W, logical model, and vocabulary resources. All other services treat it as the source of truth.

Key responsibilities

• Persist and version DCAT Datasets, Distributions, DataServices, Catalogs
• Persist DPROD DataProducts, Ports, and lifecycle transitions
• Store CSV-W tables and columns (populated by harvest events)
• Manage LogicalModels and LogicalDataElements with physical column bindings
• Maintain the vocabulary registry and per-dataset vocabulary profiles
• Export the full catalog as DCAT 3.0 JSON-LD via Apache Jena
• Publish catalog.*.changes Kafka events on all mutations

Database

PostgreSQL 16 on port 5433 (Docker). Migrations managed by Flyway. Key tables: resources, datasets, distributions, data_products, csvw_columns, logical_models, logical_data_elements, vocabularies.

Harvest Service

The harvest-service crawls external data sources, normalises their metadata, and publishes it to Kafka for the inventory-service to ingest. Jobs are scheduled with Quartz and executed as Spring Batch jobs.

Supported connectors

Configure a source

curl -X POST http://localhost:8002/api/v1/sources \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "name": "Snowflake Production",
    "sourceType": "snowflake",
    "baseUrl": "orgname-accountname.snowflakecomputing.com",
    "databaseName": "TRADING_DB",
    "schemaFilter": ["BLOTTER", "RISK"],
    "credentialRef": "vault://snowflake/prod"
  }'

Lineage Service

The lineage-service ingests OpenLineage events and DDL, persists them to PostgreSQL, and builds a property graph in Apache AGE for multi-hop Cypher traversal.

DDL lineage

Submit raw DDL to extract lineage without running a pipeline:

curl -X POST http://localhost:8003/api/v1/ddl/submit \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "dialect": "SNOWFLAKE",
    "ddl": "CREATE VIEW RISK_DB.MARKET_RISK.DAILY_POSITIONS AS SELECT t.*, p.close_price FROM TRADING_DB.BLOTTER.TRADE_BLOTTER t JOIN kafka://prices-realtime p ON t.instrument_id = p.instrument_id"
  }'

Apache Calcite parses the DDL across Snowflake, Teradata, and Hive dialects. A DERIVED_FROM edge is created in the AGE graph between each source table and the view.

Search Service

The search-service maintains an OpenSearch index that is enriched with logical model data, vocabulary concept labels, and FIBO IRIs. It consumes Kafka events to stay in sync with the catalog.

Search query

# Full-text search with filters
curl "http://localhost:8004/api/v1/search?q=trade&type=dataset&domain=Finance&hasLineage=true" \
  -H "X-API-Key: dev-local"

# FIBO concept facet search
curl "http://localhost:8004/api/v1/search?fibo_concept=MonetaryAmount" \
  -H "X-API-Key: dev-local"

# Semantic type facet search
curl "http://localhost:8004/api/v1/search?semanticType=Customer" \
  -H "X-API-Key: dev-local"

# Autocomplete suggestions
curl "http://localhost:8004/api/v1/search/suggest?q=trad" \
  -H "X-API-Key: dev-local"

Datasets are indexed with a semanticTypes keyword field derived from their vocabulary mappings, exposed as both a filter and an aggregation facet. See Semantic Types.

Reindex

curl -X POST http://localhost:8004/api/v1/admin/reindex \
  -H "X-API-Key: dev-local"
# → {"datasetsIndexed":142,"dataProductsIndexed":31,"distributionsIndexed":287}

AI Service

The ai-service provides a RAG (Retrieval-Augmented Generation) pipeline over your metadata corpus using Spring AI. It can run fully on-premises with Ollama or use the OpenAI API.

Start a conversation

# Create a conversation
CONV=$(curl -s -X POST http://localhost:8005/api/v1/conversations \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"title": "My session"}' | jq -r .id)

# Ask a question (streaming SSE response)
curl -N -X POST http://localhost:8005/api/v1/conversations/$CONV/messages \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "Accept: text/event-stream" \
  -d '{"content": "Which datasets contain monetary amounts mapped to FIBO?"}'

# Multi-dataset SQL query — schema for both datasets loaded; join hints derived from shared vocab IRIs;
# platform conflict detected if datasets are on different systems (Snowflake vs Delta Lake, etc.)
curl -N -X POST http://localhost:8005/api/v1/conversations/$CONV/messages \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "Accept: text/event-stream" \
  -d '{
    "content": "Write a Snowflake SQL join between trade positions and counterparty master",
    "focusDatasetIds": ["<trade-positions-id>", "<counterparty-master-id>"]
  }'

Embedding pipeline

The ai-service listens on inventory.datasets.changes. On each event it builds four pgvector chunks per dataset and upserts them:

• Chunk 0 — title + description
• Chunk 1 — keywords + themes
• Chunk 2 — semantic types + vocabulary concept labels + logical element names
• Chunk 3 — physical column names and SQL datatypes (used to ground multi-dataset queries)

Element classification & description

The ai-service provides two metadata enrichment operations for logical data elements:

• Classification — infers a data sensitivity level (PUBLIC, INTERNAL, CONFIDENTIAL, or HIGH_CONFIDENTIAL) from the element's name, logical type, and FIBO / schema.org vocabulary mappings. Returns a one-sentence reasoning.
• Description — generates a plain-English business description grounded in the element's context. Descriptions are surfaced in the Model tab's Description column as inline suggestions pending owner review.

Both operations are triggered via the inventory-service proxy (which owns persistence). Results are stored as recommendedClassification / recommendedDescription on the element and cleared when accepted or rejected. Only the dataset's data owner may accept recommendations.

curl -s -X POST http://localhost:8005/api/v1/classify/elements \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{
    "elements": [
      { "elementId": "e1", "name": "ssn", "label": "Social Security Number",
        "logicalType": "string", "vocabConceptLabels": ["Person Identifier"] }
    ]
  }'
# → { "results": [ { "elementId": "e1",
#       "classification": "HIGH_CONFIDENTIAL", "reasoning": "..." } ] }

Element description endpoint

curl -s -X POST http://localhost:8005/api/v1/describe/elements \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{
    "elements": [
      { "elementId": "e1", "name": "trade_amt", "label": "Trade Amount",
        "logicalType": "MonetaryAmount",
        "vocabConceptLabels": ["MonetaryAmount", "Currency"] }
    ]
  }'
# → { "results": [ { "elementId": "e1",
#       "description": "The gross notional value of the trade in settlement currency,
#                       recorded pre-netting." } ] }

Semantic recommendations

Analyses a dataset's metadata (title, description, keywords, element names, logical types, and current vocabulary mappings) and recommends additional business domain types and vocabulary concepts to improve semantic coverage. The producer UI calls this through the inventory-service proxy POST /api/v1/datasets/{id}/recommend-semantic-context.

curl -s -X POST http://localhost:8005/api/v1/recommend-semantic-context \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{
    "datasetId": "...", "title": "Retail Customer Accounts",
    "keywords": ["customer", "account"],
    "elementNames": ["customer_id", "balance"],
    "currentVocabLabels": ["Customer"]
  }'
# → { "types": [ { "type": "DebitCardAccount",
#       "rationale": "...", "vocabularyHint": "FIBO" } ], "rationale": "..." }

Agentic review (proposer / reviewer)

Beyond one-shot recommendations, the ai-service runs a two-agent proposer/reviewer loop over a logical model to produce higher-quality, self-critiqued enrichment. A proposer drafts per-element descriptions, classifications, vocabulary concept mappings, and PII / direct-identifier flags; a reviewer then audits that draft against the dataset's full DCAT context and returns a verdict (APPROVE / REJECT) with per-issue comments. The proposer revises on each REJECT. The loop is capped at 10 iterations, and a long-term review memory carries lessons from past reviews into new runs to speed convergence. On convergence (or the cap) the final recommendation is persisted to the model's elements for a data owner to accept or reject.

Progress streams over Server-Sent Events — each data: line is a JSON AgenticEvent: phase markers (CONTEXT, MEMORY, PROPOSING, REVIEWING, LOCKED), the proposer's PROPOSAL, the reviewer's REVIEW (verdict + comments + summary), and a terminal DONE / MAX_REACHED / ERROR.

# Run the agentic review over one logical model and stream progress (SSE)
curl -N -X POST http://localhost:8005/api/v1/agentic-review \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "Accept: text/event-stream" \
  -d '{"datasetId": "<dataset-id>", "modelId": "<logical-model-id>"}'

Identity Service

Port: 8006  |  Database: PostgreSQL 16 (port 5436) + Keycloak (port 8180)

The identity-service manages organisations, users, roles, and access policies. It integrates with Keycloak 24 for OIDC token issuance and validation. All other backend services validate JWTs issued by Keycloak.

Responsibilities

• User provisioning (invite, list, enable/disable) backed by the Keycloak Admin REST API — changes made in the producer Admin › Users UI are written directly to Keycloak and synced to the local catalog_users table
• Bidirectional Keycloak sync — on startup the service imports any existing Keycloak users into the local catalog database so that user references (e.g. ownerId) resolve correctly
• Organisation and domain management
• Long-lived API key issuance (stored as SHA-256 hashes)
• ABAC policy evaluation
• JWT issuer — all services trust http://keycloak:8180/realms/datacatalog

Keycloak realm

The datacatalog realm is auto-imported from infra/keycloak/datacatalog-realm.json on first startup. Subsequent changes must be made via the Keycloak Admin Console or REST API — the file is only read against a fresh database.

For roles, default users, and how to invite new users see Roles & Login →

Policy Service

Port: 8007  |  Database: PostgreSQL 16 (port 5438)

The policy-service is the platform's Policy Decision Point (PDP). It holds the ODRL policy registry for all datasets and evaluates them on demand using an internal implementation of the ODRE enforcement algorithm (Cimmino et al., Computers & Security, 2025) — a concrete enforcement layer on top of ODRL that produces machine-readable UsageDecision tuples at request time.

Responsibilities

• Maintain a policy_records registry keyed by (dataset_id, tenant_id)
• Auto-sync policies from inventory.datasets.changes Kafka events when a data owner accepts terms
• Evaluate ODRL policies at request time via POST /api/v1/policies/{datasetId}/evaluate
• Persist an evaluation_log for every evaluation call
• Publish PolicyEvaluationResultPayload to policy.evaluations.completed

Policy levels

Policy composition (component pieces)

A dataset's effective policy is assembled from reusable, typed fragments called policy pieces. Each piece has a type of CLASSIFICATION (data sensitivity), REGULATION (e.g. FCRA-aware rules), or CONTRACTUAL (terms-of-use obligations), keyed by a dimension value. dataset_policy_links records which pieces apply to which dataset, and the registry composes them into the effective policy_records document. Inspect the breakdown via GET /api/v1/policies/{datasetId}/components.

UsageDecision semantics

Quick examples

# Register a policy (upsert)
curl -X PUT http://localhost:8007/api/v1/policies/{datasetId} \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"policyJson":"{...odrl...}","policyLevel":"A"}'

# Evaluate access (A-Level — no M variables needed)
curl -X POST http://localhost:8007/api/v1/policies/{datasetId}/evaluate \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"M":{},"F":{}}'
# → {"granted":true,"policyLevel":"A","decisions":[{"action":"read","result":"true","delegated":false}]}

# Evaluate access (B1-Level — inject callerRole at runtime)
curl -X POST http://localhost:8007/api/v1/policies/{datasetId}/evaluate \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"M":{"callerRole":"DATA_OWNER"},"F":{}}'

See Policy API → for the full endpoint reference.

Roles & Login

The producer UI (http://localhost:3000) requires login before any content is shown. The consumer UI (http://localhost:3001) is read-only and does not require authentication.

Login flow

1. User visits the producer UI → redirected to the Keycloak login page automatically
2. User logs in with email and password
3. Keycloak issues an OIDC access token (JWT) via Authorization Code + PKCE
4. The producer app stores the token in memory and sends it as Authorization: Bearer <token> on every API call
5. Backend services validate the JWT against the Keycloak JWKS endpoint
6. The token is refreshed automatically every 30 seconds before it expires
7. Click Sign out in the sidebar to terminate the Keycloak session

Roles

Role → backend permissions

Default users

Inviting Users

Users are managed through the Keycloak Admin Console. The identity-service user API (POST /api/v1/users/invite) is the programmatic path for the same workflow.

Via Keycloak Admin Console

1. Open the admin console and log in as admin / admin — Docker Compose: http://localhost:8180; MicroK8s: http://keycloak.catalog.local/admin (add the host to /etc/hosts as printed by deploy.sh)
2. Select the datacatalog realm from the top-left dropdown
3. Go to Users → Add user
4. Set Email, First name, Last name; enable Email verified; click Create
5. On the Credentials tab → set a temporary password and click Save password
6. On the Role mapping tab → click Assign role → select one of the four catalog roles
7. On the Attributes tab → add the following key/value pairs:

Permissions attribute by role

Force logout / session revocation

To expire all active sessions for a user (e.g. after a role change takes effect):

1. Keycloak Admin Console → Users → select the user
2. Sessions tab → Sign out all sessions

The user will be redirected to the login page on their next API call.

Keycloak Setup

Realm

Clients

Protocol mappers (catalog-scopes)

Applying realm changes

The realm JSON is only imported on a fresh database. For an already-running stack, use the Keycloak Admin Console or the Admin REST API to apply changes:

# Get an admin token
TOKEN=$(curl -s -X POST http://localhost:8180/realms/master/protocol/openid-connect/token \
  -d "client_id=admin-cli&username=admin&password=admin&grant_type=password" \
  | jq -r .access_token)

# List current roles
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:8180/admin/realms/datacatalog/roles | jq '.[].name'

# Create a new role
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"my-role","description":"Custom role"}' \
  http://localhost:8180/admin/realms/datacatalog/roles

Authentication

Request headers

Dev API key (local development only)

Any key starting with dev- bypasses Keycloak entirely and grants catalog:admin scope with tenant 00000000-0000-0000-0000-000000000001. Use it for curl smoke tests and CI pipelines that don't need a real login.

curl -H "X-API-Key: dev-local" http://localhost:8001/api/v1/datasets

Producer UI authentication

The producer UI uses the Keycloak catalog-frontend public client (Authorization Code + PKCE). Visiting http://localhost:3000 redirects unauthenticated users to the Keycloak login page automatically. See Roles & Login for details.

Inventory API :8001

Catalogs

Datasets

Dataset semantic types

Dataset ownership & audit

Terms of use

Distributions

Physical Schema (CSV-W)

Data Products

Logical Models

Logical Elements

AI enrichment — single element

AI enrichment — bulk (model-wide async)

Vocabularies

Vocabulary Profiles

Dashboard

Terms Policies (ODRL)

Harvest API :8002

Sources

Jobs

Lineage API :8003

OpenLineage ingest

Dataset lineage

DDL parser

Search API :8004

AI API :8005

Conversations

AI recommendations (element-level)

Agentic review

Policy API :8007

B1-Level example

# Store a B1-Level policy (role check via variable injection)
curl -X PUT http://localhost:8007/api/v1/policies/{datasetId} \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{
    "policyLevel": "B1",
    "policyJson": "{\"@context\":\"http://www.w3.org/ns/odrl.jsonld\",\"@type\":\"Set\",\"uid\":\"urn:b1\",\"permission\":[{\"target\":\"dataset:x\",\"action\":\"read\",\"constraint\":[{\"leftOperand\":{\"@value\":\"[=callerRole]\",\"@type\":\"xsd:string\"},\"operator\":\"eq\",\"rightOperand\":{\"@value\":\"DATA_OWNER\",\"@type\":\"xsd:string\"}}]}]}"
  }'

# Evaluate — DATA_STEWARD → denied
curl -X POST http://localhost:8007/api/v1/policies/{datasetId}/evaluate \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"M":{"callerRole":"DATA_STEWARD"},"F":{}}'
# → {"granted":false,"decisions":[]}

# Evaluate — DATA_OWNER → granted
curl -X POST http://localhost:8007/api/v1/policies/{datasetId}/evaluate \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"M":{"callerRole":"DATA_OWNER"},"F":{}}'
# → {"granted":true,"decisions":[{"action":"read","result":"true","delegated":false}]}

Identity API :8006

Users

Bookmarks

Bookmark Collections

Docker Compose

The repository ships a docker-compose.yml that starts the full stack and a docker-compose.override.yml for development hot-reload overrides.

Makefile targets

Profiles

# Start including AI features
docker compose --profile ai up -d

Environment Variables

Full reference

Kubernetes (Raw Manifests)

Raw Kubernetes manifests live under infra/kubernetes/. Fourteen numbered YAML files cover everything from namespace creation to ingress rules. deploy.sh applies them in dependency order via kubectl and envsubst.

Prerequisites

1. Build and push images

export IMAGE_REGISTRY=localhost:32000/   # MicroK8s built-in registry

for svc in inventory-service harvest-service lineage-service \
           search-service ai-service identity-service; do
  docker build -t ${IMAGE_REGISTRY}odin/${svc}:latest services/${svc}/
  docker push ${IMAGE_REGISTRY}odin/${svc}:latest
done

2. Deploy

IMAGE_REGISTRY=localhost:32000/ ./infra/kubernetes/deploy.sh

The script applies all manifests in order, then waits for StatefulSets, Jobs, and Deployments to reach ready state before exiting.

3. Update /etc/hosts

NODE_IP=$(kubectl get nodes \
  -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
echo "$NODE_IP  catalog.local manage.catalog.local api.catalog.local" \
  | sudo tee -a /etc/hosts

4. Load sample data

./infra/kubernetes/seed.sh --namespace odin-catalog

The seed script establishes kubectl port-forward tunnels to each service, waits for readiness, then delegates to infra/seed/seed.sh — the same Meridian Capital financial dataset scenario used with Docker Compose.

5. Access

Manifest inventory

Kubernetes (Helm / MicroK8s)

A Helm umbrella chart is provided under infra/helm/ for clusters managed with Helm 3. A MicroK8s-specific deploy.sh under infra/microk8s/ wraps the Helm install with sensible single-node defaults.

MicroK8s quick deploy

# Full resources
./infra/microk8s/deploy.sh

# Reduced resources for machines with < 16 GB RAM
./infra/microk8s/deploy.sh --reduced-resources

Standard Helm install / upgrade

# Install
helm install odin infra/helm \
  --namespace odin-catalog --create-namespace \
  -f infra/microk8s/values.yaml

# Upgrade after image rebuild
helm upgrade odin infra/helm \
  --namespace odin-catalog \
  -f infra/microk8s/values.yaml

Local Development

Build all services

./gradlew build               # compile + test all services
./gradlew :services:inventory-service:bootRun  # run one service

Build the frontends

cd frontend/shared && pnpm install && pnpm build
cd ../producer  && pnpm install && pnpm dev   # http://localhost:3000
cd ../consumer  && pnpm install && pnpm dev   # http://localhost:3001

Rebuild a single Docker image

# Always build then up — `restart` reuses the old image
docker compose build inventory-service
docker compose up -d inventory-service

Remote debugging

All Java services support JDWP remote debugging via the standard JAVA_TOOL_OPTIONS env var — no rebuild required. Enable it in .env:

JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

Then restart the services you want to debug (docker compose up -d). Each service listens on port 5005 inside its container, exposed on a unique host port:

In IntelliJ IDEA: Run → Edit Configurations → + → Remote JVM Debug, set Host to localhost and Port to the service's debug host port above. In VS Code, add a launch.json entry with type java, request: attach, and the corresponding port.

Contribution Guide

Before you start

• Open an issue describing the bug or feature before submitting a PR.
• One logical change per PR — keep diffs reviewable.
• All new API endpoints need an integration test that runs against a real database (no mocks).

Code conventions

• Java: hexagonal architecture — domain classes have no Spring annotations; all infrastructure concerns live in the infrastructure/ package.
• TypeScript: no any; shared API types live in frontend/shared/src/types/.
• SQL: all schema changes via numbered Flyway migrations (V{n}__description.sql).

License

ODIN Catalog is released under the Apache 2.0 License. By contributing you agree your changes will be licensed under the same terms.