Quick Start

Get a full ODIN stack running locally in under five minutes using Docker Compose.

1. Clone and configure

git clone https://github.com/odin-catalog/odin.git
cd odin
cp .env.example .env          # review and edit credentials

2. Start the stack

make up
# or: docker compose up -d

# Watch services come healthy:
docker compose ps

Services start in dependency order. Allow ~60 seconds for Kafka, PostgreSQL, and OpenSearch to initialise before the Spring Boot services become healthy.

3. Create your first dataset

curl -s -X POST http://localhost:8001/api/v1/datasets \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "X-Tenant-Id: 00000000-0000-0000-0000-000000000001" \
  -d '{
    "title": "Trade Blotter",
    "description": "Intraday trade records from the front-office OMS.",
    "keywords": ["trading", "blotter", "positions"],
    "accrualPeriodicity": "daily"
  }' | jq .

4. Open the frontend

5. Load sample data

make seed        # loads financial services sample data
make reindex     # pushes all datasets into OpenSearch

The seed script creates 12 financial datasets, 5 data products, logical models with FIBO vocabulary mappings, and OpenLineage pipeline events for a BCBS 239 risk aggregation scenario.

Prerequisites

Runtime requirements

Development requirements

Optional — AI features

# Install Ollama for local LLM inference
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull nomic-embed-text   # embedding model (768 dimensions)
ollama pull llama3             # chat model

# Then start the AI profile:
docker compose --profile ai up -d

Without Ollama, the ai-service will not start. All other services function normally. You can also configure an OpenAI key in .env instead.

Configuration

All runtime configuration is driven by environment variables. Copy .env.example to .env and edit before running make up.

Core variables

AI variables

Architecture Overview

ODIN follows Domain-Driven Design with a database-per-service pattern. Six Spring Boot 3.3 microservices communicate via Kafka events. Traefik routes external HTTP traffic.

Browser / CLI
    │
    ▼
Traefik (port 80/443)
    ├── catalog.local/          → consumer-frontend (nginx, port 3001)
    ├── manage.catalog.local/   → producer-frontend (nginx, port 3000)
    └── api.catalog.local/      → services (ports 8001–8006)
            │
            ├── catalog-service  :8001  PostgreSQL :5433
            ├── harvest-service  :8002  PostgreSQL :5434 + MinIO :9000
            ├── lineage-service  :8003  PostgreSQL+AGE :5435
            ├── search-service   :8004  OpenSearch :9200
            ├── ai-service       :8005  PostgreSQL+pgvector :5437
            └── identity-service :8006  PostgreSQL :5436 + Keycloak :8180
                    │
                    └─── Apache Kafka :9092 (KRaft, no ZooKeeper)

Design principles

• API-first — every capability is a versioned REST endpoint before any UI is built on top.
• Database-per-service — no service shares a database with another. Cross-service reads go through REST or Kafka events.
• Event-driven — state changes publish Kafka events on log-compacted topics. Downstream services maintain their own read models.
• Standards-based exports — the catalog exports DCAT 3.0 JSON-LD via Apache Jena; the lineage service accepts OpenLineage JSON.

Services

Event Topology

All inter-service communication uses Kafka with an envelope schema that carries tenant, event type, and schema version on every message.

Topics

Event envelope

{
  "eventId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "eventType": "DatasetCreated",
  "schemaVersion": "1.0",
  "producerService": "catalog-service",
  "tenantId": "00000000-0000-0000-0000-000000000001",
  "timestamp": "2026-05-18T10:23:00Z",
  "payload": { ... }
}

Security Model

Authentication methods

Tenant isolation

Every resource row carries a tenant_id UUID. The X-Tenant-Id header (set by Traefik or the frontend nginx) scopes all queries. Rows from other tenants are never returned.

Metamodel Overview

ODIN's metamodel has three tiers: conceptual (business), logical (semantic), and physical (technical).

Conceptual   DataProduct (DPROD)
               ├── InputPort  → DataService
               └── OutputPort → DataService → Distribution

Logical      Dataset (DCAT)
               ├── VocabularyProfile → Vocabulary (FIBO / schema.org)
               └── LogicalModel
                     └── LogicalDataElement
                           ├── physicalColumnId ──FK──┐
                           └── VocabularyMapping      │ (SKOS)
                                                      │
Physical     Distribution (DCAT)                      │
               └── CSVWTable → CSVWSchema → CSVWColumn ◄┘

Lineage      OpenLineage Job ─[READS_FROM/WRITES_TO]→ Dataset
             (stored in Apache AGE graph, linked to DCAT Dataset)

Layer responsibilities

DCAT Datasets & Distributions

ODIN models datasets and distributions using the DCAT 3.0 vocabulary. The full catalog can be exported as DCAT JSON-LD.

Dataset fields

DCAT export

# Export full catalog as DCAT 3.0 JSON-LD
curl http://localhost:8001/api/v1/catalogs/{id}/export \
  -H "Accept: application/ld+json" \
  -H "X-API-Key: dev-local"

DPROD Data Products

Data products are modelled using the OMG DPROD standard. A data product represents a business-owned, governed unit of data with a defined lifecycle.

Lifecycle stages

Create a data product

curl -X POST http://localhost:8001/api/v1/data-products \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "title": "Trade Risk Data Product",
    "description": "Aggregated risk metrics for regulatory reporting.",
    "lifecycleStatus": "Consume",
    "keywords": ["risk", "trading", "BCBS239"],
    "informationSensitivity": "Internal"
  }'

CSV-W Physical Schema

The physical layer is modelled using CSV on the Web (CSV-W). Each distribution that is harvested from a source system produces a CSVWTable with a CSVWSchema containing one CSVWColumn per field.

Column fields

Physical columns are created automatically during harvest. A physicalColumnId UUID is assigned to each and can be referenced from a LogicalDataElement to create the logical–physical binding.

Logical Models

A LogicalModel belongs to a Dataset and provides the business-oriented view of its structure. It contains LogicalDataElements — each representing a named business concept with an optional binding to a physical column and zero or more vocabulary mappings.

LogicalDataElement fields

Bind a physical column

curl -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/bind \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{ "physicalColumnId": "b3f1a2e4-..." }'

Auto-scaffold from harvest

When a harvest run discovers columns for a dataset that has no published LogicalModel, ODIN automatically generates a draft LogicalModel with one LogicalDataElement per CSVWColumn. Each element has its physicalColumnId pre-bound and its logicalType inferred from the source datatype. You can then enrich the draft with business names and vocabulary mappings.

Vocabulary & FIBO

ODIN ships with six system vocabularies pre-loaded. You can register additional RDF vocabularies at any time.

Pre-loaded vocabularies

SKOS match types

Add a vocabulary mapping

curl -X POST \
  http://localhost:8001/api/v1/logical-data-elements/{elementId}/vocab-mappings \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "vocabularyId": "...",
    "conceptIri": "https://spec.edmcouncil.org/fibo/ontology/FND/Accounting/CurrencyAmount/MonetaryAmount",
    "conceptLabel": "MonetaryAmount",
    "matchType": "exactMatch"
  }'

OpenLineage Integration

ODIN's lineage-service exposes an OpenLineage-compatible HTTP endpoint. Any tool that emits OpenLineage events (Spark, dbt, Airflow, Flink) can send lineage directly to ODIN.

Send a lineage event

curl -X POST http://localhost:8003/api/v1/lineage \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "eventType": "COMPLETE",
    "eventTime": "2026-05-18T10:00:00Z",
    "producer": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark",
    "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent",
    "run": { "runId": "3fa85f64-5717-4562-b3fc-2c963f66afa6" },
    "job": { "namespace": "TRADING_DB", "name": "risk_aggregation_job" },
    "inputs":  [{ "namespace": "TRADING_DB.BLOTTER", "name": "TRADE_BLOTTER" }],
    "outputs": [{ "namespace": "REGULATORY_DB.BCBS239", "name": "RISK_AGGREGATION" }]
  }'

Query lineage graph

# Upstream lineage, 4 hops
curl "http://localhost:8003/api/v1/datasets/REGULATORY_DB.BCBS239/RISK_AGGREGATION/lineage?direction=upstream&depth=4" \
  -H "X-API-Key: dev-local"

# Downstream impact analysis
curl "http://localhost:8003/api/v1/datasets/TRADING_DB.BLOTTER/TRADE_BLOTTER/impact" \
  -H "X-API-Key: dev-local"

Lineage is stored in an Apache AGE property graph on PostgreSQL. Cypher queries traverse DERIVED_FROM, READ_BY, and WRITES_TO edges. Column-level lineage uses COLUMN_LINEAGE edges.

Catalog Service

The catalog-service is the primary metadata store. It owns all DCAT, DPROD, CSV-W, logical model, and vocabulary resources. All other services treat it as the source of truth.

Key responsibilities

• Persist and version DCAT Datasets, Distributions, DataServices, Catalogs
• Persist DPROD DataProducts, Ports, and lifecycle transitions
• Store CSV-W tables and columns (populated by harvest events)
• Manage LogicalModels and LogicalDataElements with physical column bindings
• Maintain the vocabulary registry and per-dataset vocabulary profiles
• Export the full catalog as DCAT 3.0 JSON-LD via Apache Jena
• Publish catalog.*.changes Kafka events on all mutations

Database

PostgreSQL 16 on port 5433 (Docker). Migrations managed by Flyway. Key tables: resources, datasets, distributions, data_products, csvw_columns, logical_models, logical_data_elements, vocabularies.

Harvest Service

The harvest-service crawls external data sources, normalises their metadata, and publishes it to Kafka for the catalog-service to ingest. Jobs are scheduled with Quartz and executed as Spring Batch jobs.

Supported connectors

Configure a source

curl -X POST http://localhost:8002/api/v1/sources \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "name": "Snowflake Production",
    "sourceType": "snowflake",
    "baseUrl": "orgname-accountname.snowflakecomputing.com",
    "databaseName": "TRADING_DB",
    "schemaFilter": ["BLOTTER", "RISK"],
    "credentialRef": "vault://snowflake/prod"
  }'

Lineage Service

The lineage-service ingests OpenLineage events and DDL, persists them to PostgreSQL, and builds a property graph in Apache AGE for multi-hop Cypher traversal.

DDL lineage

Submit raw DDL to extract lineage without running a pipeline:

curl -X POST http://localhost:8003/api/v1/ddl/submit \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -d '{
    "dialect": "SNOWFLAKE",
    "ddl": "CREATE VIEW RISK_DB.MARKET_RISK.DAILY_POSITIONS AS SELECT t.*, p.close_price FROM TRADING_DB.BLOTTER.TRADE_BLOTTER t JOIN kafka://prices-realtime p ON t.instrument_id = p.instrument_id"
  }'

Apache Calcite parses the DDL across Snowflake, Teradata, and Hive dialects. A DERIVED_FROM edge is created in the AGE graph between each source table and the view.

Search Service

The search-service maintains an OpenSearch index that is enriched with logical model data, vocabulary concept labels, and FIBO IRIs. It consumes Kafka events to stay in sync with the catalog.

Search query

# Full-text search with filters
curl "http://localhost:8004/api/v1/search?q=trade&type=dataset&domain=Finance&hasLineage=true" \
  -H "X-API-Key: dev-local"

# FIBO concept facet search
curl "http://localhost:8004/api/v1/search?fibo_concept=MonetaryAmount" \
  -H "X-API-Key: dev-local"

# Autocomplete suggestions
curl "http://localhost:8004/api/v1/search/suggest?q=trad" \
  -H "X-API-Key: dev-local"

Reindex

curl -X POST http://localhost:8004/api/v1/admin/reindex \
  -H "X-API-Key: dev-local"

AI Service

The ai-service provides a RAG (Retrieval-Augmented Generation) pipeline over your metadata corpus using Spring AI. It can run fully on-premises with Ollama or use the OpenAI API.

Start a conversation

# Create a conversation
CONV=$(curl -s -X POST http://localhost:8005/api/v1/conversations \
  -H "X-API-Key: dev-local" -H "Content-Type: application/json" \
  -d '{"title": "My session"}' | jq -r .id)

# Ask a question (streaming SSE response)
curl -N -X POST http://localhost:8005/api/v1/conversations/$CONV/messages \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-local" \
  -H "Accept: text/event-stream" \
  -d '{"content": "Which datasets contain monetary amounts mapped to FIBO?"}'

Embedding pipeline

The ai-service listens on catalog.datasets.changes and catalog.data-products.changes. On each event it fetches the enriched entity from catalog-service, chunks the text (title + description + element names + vocabulary labels), embeds the chunks using the configured model, and upserts into the pgvector store.

Identity Service

The identity-service manages organisations, users, roles, and access policies. It integrates with Keycloak 24 for OIDC token issuance and validation.

Keycloak configuration

A realm export is included at infra/keycloak/realm-export.json. The realm is imported automatically on first startup. Keycloak runs on port 8180.

Authentication

Request headers

Dev API key

Any key starting with dev- bypasses authentication and grants catalog:admin scope. The tenant ID defaults to 00000000-0000-0000-0000-000000000001.

# These are equivalent for local development:
curl -H "X-API-Key: dev-anything" ...
curl -H "X-API-Key: dev-local" ...
curl -H "X-API-Key: dev-my-test-key" ...

Catalog API :8001

Datasets

Data Products

Logical Models

Vocabularies

Harvest API :8002

Sources

Jobs & Runs

Lineage API :8003

Search API :8004

AI API :8005

Docker Compose

The repository ships a docker-compose.yml that starts the full stack and a docker-compose.override.yml for development hot-reload overrides.

Makefile targets

Profiles

# Start including AI features
docker compose --profile ai up -d

Environment Variables

Full reference

Kubernetes (Helm)

Helm charts are provided under infra/helm/charts/ for each service. A top-level umbrella chart is planned for GA.

# Install catalog-service chart
helm install odin-catalog infra/helm/charts/catalog-service \
  --namespace odin --create-namespace \
  --set postgresql.password=changeme \
  --set kafka.brokers=kafka:9092

Local Development

Build all services

./gradlew build               # compile + test all services
./gradlew :services:catalog-service:bootRun  # run one service

Build the frontends

cd frontend/shared && pnpm install && pnpm build
cd ../producer  && pnpm install && pnpm dev   # http://localhost:3000
cd ../consumer  && pnpm install && pnpm dev   # http://localhost:3001

Rebuild a single Docker image

# Always build then up — `restart` reuses the old image
docker compose build catalog-service
docker compose up -d catalog-service

Contribution Guide

Before you start

• Open an issue describing the bug or feature before submitting a PR.
• One logical change per PR — keep diffs reviewable.
• All new API endpoints need an integration test that runs against a real database (no mocks).

Code conventions

• Java: hexagonal architecture — domain classes have no Spring annotations; all infrastructure concerns live in the infrastructure/ package.
• TypeScript: no any; shared API types live in frontend/shared/src/types/.
• SQL: all schema changes via numbered Flyway migrations (V{n}__description.sql).

License

ODIN Catalog is released under the Apache 2.0 License. By contributing you agree your changes will be licensed under the same terms.