Helm & Kubernetes

This guide explains how to deploy Rafiki using Helm charts on a Kubernetes cluster. Helm is a package manager for Kubernetes that allows you to define, install, and upgrade complex Kubernetes applications through Helm charts.

Rafiki uses the following key infrastructure components:

• PostgreSQL : Used for storing application data, grants, and Open Payments resources
• Redis : Used for caching and session management
• TigerBeetle (recommended): High-performance accounting database for financial transaction processing and ledger management

Prerequisites

Recommended software version

We recommend using the latest vendor supported version for each of the software dependencies listed in this section.

Before you begin, ensure you have the following:

• Kubernetes cluster deployed
• kubectl installed and configured to access your cluster
• Helm installed

Deploy PostgreSQL and Redis dependencies

Before installing Rafiki, ensure your target Kubernetes cluster already has Redis and PostgreSQL deployed.

• PostgreSQL must be initialized with two databases: one for the backend service and one for the auth service
• Each service must be configured with a connection URL that points to its corresponding database

Install Rafiki using Helm

Add the Interledger Helm repository

Add the official Interledger Helm repository which contains the Rafiki charts:

helm repo add interledger-helm https://interledger.github.io/charts/interledger
helm repo update

Verify available charts

You can list the available Rafiki charts:

helm search repo interledger-helm/rafiki

This returns available chart versions, including the main interledger-helm/rafiki chart.

Create values file

Create a single values.yaml file to customize your Rafiki deployment.

Configure the following required values:

• config.auth.redisUrl.value
• config.auth.databaseUrl.value
• config.backend.redisUrl.value
• config.backend.databaseUrl.value
• config.auth.tenancy.tenantId
• config.backend.tenancy.tenantId
• config.auth.identityServer.serverSecret.value
• config.backend.ilp.streamSecret.value
• config.backend.webhookSignatureSecret.value
• config.frontend.kratos.enabled

Base values file

For full configuration details, refer to the base values.yaml file on GitHub: interledger/charts charts/interledger/rafiki/values.yaml .

Configure chart-managed secrets

When using --set config.auth.shouldCreateSecrets=true and --set config.backend.shouldCreateSecrets=true, Helm creates and manages the auth and backend Kubernetes secrets as part of the chart installation.

In this mode, include the required secret values in values.yaml so the chart can populate those generated secrets during install/upgrade:

config:
  auth:
    identityServer:
      serverSecret:
        value: your-identity-server-secret
  backend:
    ilp:
      streamSecret:
        value: your-stream-secret
    webhookSignatureSecret:
      value: your-webhook-signature-secret

If you prefer to pre-create Kubernetes secrets manually, set both config.auth.shouldCreateSecrets and config.backend.shouldCreateSecrets to false and configure secretKeyRef values in values.yaml.

Install the service

Install Rafiki using the following command:

helm install rafiki interledger-helm/rafiki -f values.yaml \
  --set config.auth.shouldCreateSecrets=true \
  --set config.backend.shouldCreateSecrets=true

If you want to install to a specific namespace:

kubectl create namespace rafiki

helm install rafiki interledger-helm/rafiki -f values.yaml \
  --set config.auth.shouldCreateSecrets=true \
  --set config.backend.shouldCreateSecrets=true \
  -n rafiki

The namespace used for Rafiki must match the namespace used by your Redis and PostgreSQL services.

Verify the deployment

Check the status of your deployments:

# Check Helm releases
helm list

# Check running pods
kubectl get pods

# Check deployed services
kubectl get services

# Check logs for any issues
kubectl logs -l app.kubernetes.io/name=rafiki-auth
kubectl logs -l app.kubernetes.io/name=rafiki-backend
kubectl logs -l app.kubernetes.io/name=rafiki-frontend

Configure environment variables

Each Rafiki service can be configured via environment variables through Helm values. Below are the complete environment variable configurations for each service:

Auth service

The Rafiki auth service is responsible for handling Open Payments authorization and grant management. It implements the Open Payments authorization server specification and manages access tokens, grants, and interactions. The auth service connects to a PostgreSQL database to store grant-related data and uses Redis for session management.

Ports exposed:

• 3003 (admin) - Auth Admin API for managing grants and access tokens
• 3006 (auth) - Open Payments authorization server endpoint
• 3007 (introspection) - Access token introspection endpoint
• 3009 (interaction) - User interaction endpoint for grant flows

Running Rafiki behind a proxy

If you plan to run your Rafiki instance behind a proxy, you must set the trustProxy variable to true

Required values

Helm variable name	Default	Description
config.auth.databaseUrl.value	''	The URL of the PostgreSQL database storing your Open Payments grant data.
config.auth.redisUrl.value	''	The connection URL of Redis used for session/cache operations.
config.auth.authServerUrl	http://rafiki-auth:3006	The public endpoint for your Rafiki instance’s public Open Payments auth routes.
config.auth.cookieKey.value	"changeme"	The koa KeyGrip key that’s used to sign cookies for an interaction session.
config.auth.identityServer.domain	http://rafiki-backend/idp	The URL of your identity provider (IdP) server, used by the authorization server to tell an Open Payments client where to redirect the end-user to start interactions.
config.auth.identityServer.serverSecret.value	''	A shared secret between the authorization server and the identity provider (IdP); the authorization server uses the secret to secure its IdP-related endpoints. When the IdP sends requests to the authorization server, it must provide the secret via an x-idp-secret header.

Conditionally required values

Helm value name	Default	Description
config.auth.trustProxy	true	Must be set to true when running Rafiki behind a proxy. When true, the X-Forwarded-Proto header is used to determine if connections are secure.
config.auth.tenancy.tenantId	some-guid	The tenant identifier for the operator tenant. Should match config.backend.tenancy.tenantId.

Optional values

Helm value name	Default	Description
config.auth.shouldCreateSecrets	false	If true, chart-managed auth secrets are created from secretsMaps.auth.
config.auth.accessToken.deletionDays	30	The days until expired and/or revoked access tokens are deleted.
config.auth.accessToken.expirySeconds	600 (10 minutes)	The expiry time, in seconds, for access tokens.
config.auth.admin.signatureVersion	1	The version of the request signing algorithm used to validate signatures.
config.auth.admin.signatureTtlSeconds	30	The TTL, in seconds, for which a request signature is valid.
config.auth.port.admin	3003	The port of your Rafiki Auth Admin API server.
config.auth.port.auth	3006	The port of your Open Payments authorization server.
config.auth.port.interaction	3009	The port number of your Open Payments interaction-related APIs.
config.auth.port.introspection	3007	The port of your Open Payments access token introspection server.
config.auth.workers.cleanup	1	The number of workers processing expired or revoked access tokens.
config.auth.interaction.incomingPayment	"false"	When "true", incoming Open Payments grant requests are interactive.
config.auth.interaction.expirySeconds	600 (10 minutes)	The time, in seconds, for which a user can interact with a grant request before the request expires.
config.auth.interaction.quote	"false"	When "true", quote grants are interactive.
config.auth.logLevel	info	Pino log level.
config.auth.nodeEnv	production	The type of node environment: development, test, or production.
config.auth.grant.waitSeconds	5	The wait time, in seconds, included in a grant request response (grant.continue).

Backend service

The Rafiki backend service handles the core Open Payments capabilities and business logic. It exposes the Open Payments resource server APIs for wallet addresses, incoming payments, outgoing payments, and quotes. The backend also includes an ILP connector for sending and receiving Interledger packets. It connects to PostgreSQL for storing Open Payments resources, Redis for caching, and optionally TigerBeetle for high-performance accounting.

Ports exposed:

• 3000 (openPayments) - Open Payments resource server API
• 3001 (admin) - Backend Admin API and GraphQL endpoint
• 3002 (connector) - ILP connector for sending and receiving packets
• 3005 (autoPeering) - Autopeering service (when enabled)

TigerBeetle vs PostgreSQL for accounting

TigerBeetle is strongly recommended for production deployments due to its high-performance accounting capabilities. If you prefer to use PostgreSQL as the accounting database, set useTigerbeetle to false.

Required values

Helm value name	Default	Description
config.backend.auth.grantUrl	http://rafiki-auth.rafiki-auth:3006	The endpoint on your Open Payments authorization server to grant a request.
config.backend.auth.introspectionUrl	http://rafiki-auth.rafiki-auth:3007	The endpoint on your Open Payments authorization server to introspect an access token.
config.backend.databaseUrl.value	''	The PostgreSQL database URL of the database storing your resource data.
config.backend.redisUrl.value	''	The Redis URL used for queue, cache, and connector state.
config.backend.ilp.host	http://rafiki-backend:3000	The public endpoint of your Open Payments resource server.
config.backend.ilp.address	test.rafiki-backend	The ILP address of your Rafiki instance.
config.backend.ilp.connector	http://rafiki-backend:3002	The ILP connector address where ILP packets are received.
config.backend.key.id	rafiki-override-this-value	Your Rafiki instance’s client key ID.
config.backend.webhook.url	http://wallet/webhooks/rafiki	Your endpoint that consumes webhook events.
config.backend.ilp.streamSecret.value	''	The seed secret to generate shared STREAM secrets.
config.backend.webhookSignatureSecret.value	''	The secret used to generate webhook request signatures.

Conditionally required values

Helm value name	Default	Description
config.backend.instanceName	rafiki-backend-changeme	Your Rafiki instance name used for automatic peering and/or telemetry. Required when either feature is enabled.
config.backend.trustProxy	"true"	Must be set to true when running Rafiki behind a proxy. When true, the X-Forwarded-Proto header is used to determine if connections are secure.
config.backend.tenancy.tenantId	some-guid	The tenant identifier for the operator tenant. Should match config.auth.tenancy.tenantId.

Optional values

Helm value name	Default	Description
config.backend.shouldCreateSecrets	false	If true, chart-managed backend secrets are created from secretsMaps.backend.
config.backend.port.admin	3001	The port of your Backend Admin API server.
config.backend.port.connector	3002	The port of the ILP connector for sending packets via ILP over HTTP.
config.backend.port.openPayments	3000	The port of your Open Payments resource server.
config.backend.port.autoPeering	3005	If automatic peering is enabled, the server uses this port.
config.backend.autoPeering.enabled	"false"	When "true", automatic peering is enabled.
config.backend.telemetry.enabled	"false"	When "true", telemetry is enabled.
config.backend.useTigerbeetle	false	When true, TigerBeetle is used for accounting.
config.backend.rates.url	''	The endpoint your Rafiki instance uses to request exchange rates.
config.backend.webhook.timeout	200	The time, in milliseconds, for wallet-address worker webhook polling timeout behavior.
config.backend.ilp.slippage	0.01	The accepted ILP rate fluctuation.
config.backend.key.mount	/mnt/keys/pvk.pem	The mounted path to your Rafiki instance private key file.
config.backend.idempotency.keyTTL	86400000	The TTL, in milliseconds, for idempotency keys on Backend Admin API mutations.
config.backend.idempotency.keyLock	2000	The TTL, in milliseconds, for idempotency key lock concurrency control.
config.backend.workerIdle	200	The time, in milliseconds, workers wait before polling an empty queue again.
config.backend.workers.incomingPayment	1	The number of workers processing incoming payment requests.
config.backend.workers.outgoingPayment	1	The number of workers processing outgoing payment requests.
config.backend.workers.walletAddress	1	The number of workers processing wallet address requests.
config.backend.workers.webhook	1	The number of workers processing webhook events.
config.backend.lifetime.exchangeRate	15000	The time, in milliseconds, that exchange rates from config.backend.rates.url are considered valid.
config.backend.lifetime.quote	300000	The time, in milliseconds, an Open Payments quote is valid for.
config.backend.lifetime.webhook	200	The time, in milliseconds, to wait for a 200 response from your webhook endpoint before retry.
config.backend.withdrawalThrottleDelay	''	The delay in liquidity withdrawal processing.
config.backend.logLevel	info	Pino log level.
config.backend.nodeEnv	production	The type of node environment: development, test, or production.

Frontend service

The Rafiki frontend service provides an administrative web interface for managing your Rafiki instance. It offers a user-friendly dashboard for monitoring and managing wallet addresses, payments, peers, and other Open Payments resources. The frontend communicates with the Backend Admin API to perform administrative operations.

Ports exposed:

• 3010 (http) - Rafiki Admin web interface

Authentication

The frontend can optionally integrate with Ory Kratos for user authentication. When authentication is disabled (kratos.enabled: false), the admin interface is accessible without login credentials.

Required Values

Helm value name	Default	Description
config.frontend.serviceUrls.GRAPHQL_URL	http://rafiki-backend-service.rafiki:3001/graphql	URL of Rafiki’s GraphQL Admin API
config.frontend.serviceUrls.OPEN_PAYMENTS_URL	https://rafiki-backend-service/	Your Open Payments API endpoint
config.frontend.port	3010	Port from which to host the Rafiki Remix app

Conditionally required values

The following values are required only when config.frontend.kratos.enabled is set to true.

Helm value name	Default	Description
config.frontend.kratos.enabled	true	Enables Kratos-backed authentication for the frontend.
config.frontend.kratos.adminUrl	undefined	The admin endpoint/container address for Kratos.
config.frontend.kratos.containerPublicUrl	undefined	The URL for internal backend calls to the Kratos container from within the cluster network.
config.frontend.kratos.browserPublicUrl	undefined	The browser-facing URL for Kratos, used by the Rafiki Admin UI to complete auth flows.

Optional values

Helm value name	Default	Description
config.frontend.signatureVersion	1	The signature version number used for outgoing HTTP signatures to the Backend Admin API.
config.frontend.nodeEnv	production	The type of node environment: development, test, or production.
config.frontend.logLevel	info	Pino log level.

Configure ingress

To expose Rafiki services outside the cluster, you’ll need to configure ingress. Here’s an example using NGINX Ingress Controller:

Install NGINX Ingress Controller

If you don’t already have an ingress controller installed:

# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Install the ingress controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
  --set controller.publishService.enabled=true

Configure ingress in your values file

Add ingress configuration to your values.yaml file:

# In values.yaml
ingress:
  backend:
    enabled: true
    className: nginx
    hosts:
      - host: rafiki-backend.your-domain.com
        paths:
          - path: /
            pathType: Prefix
            service:
              name: rafiki-backend-service
              port: 3000
    tls:
      - secretName: rafiki-backend-tls
        hosts:
          - rafiki-backend.your-domain.com
  auth:
    enabled: true
    className: nginx
    hosts:
      - host: rafiki-auth.your-domain.com
        paths:
          - path: /
            pathType: Prefix
            service:
              name: rafiki-auth-service
              port: 3006
    tls:
      - secretName: rafiki-auth-tls
        hosts:
          - rafiki-auth.your-domain.com

Configure DNS

Create DNS records pointing to your ingress controller’s external IP:

• rafiki-backend.your-domain.com
• rafiki-auth.your-domain.com
• rafiki-frontend.your-domain.com

Configure dependencies

PostgreSQL

You can use an external PostgreSQL instance or deploy one using Helm:

# Add Bitnami repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install PostgreSQL for the backend database
helm install rafiki-postgres bitnami/postgresql \
  --set auth.postgresPassword=your-secure-password \
  --set auth.database=rafiki_backend \
  --set primary.persistence.size=20Gi

Then create the auth database:

kubectl exec -it rafiki-postgres-postgresql-0 -- psql -U postgres -d postgres -c "CREATE DATABASE rafiki_auth;"

Redis

Deploy Redis for caching and session management:

# Install Redis
helm install rafiki-redis bitnami/redis \
  --set auth.enabled=false \
  --set master.persistence.size=8Gi

TigerBeetle (recommended)

For high-performance accounting, deploy TigerBeetle:

# Add TigerBeetle repository
helm repo add tigerbeetle https://tigerbeetle.github.io/helm-charts

# Install TigerBeetle
helm install rafiki-tigerbeetle tigerbeetle/tigerbeetle \
  --set replicaCount=3 \
  --set persistence.size=20Gi

Upgrade Rafiki

To upgrade your Rafiki services to newer versions:

helm repo update
helm upgrade rafiki interledger-helm/rafiki -f values.yaml

Uninstall Rafiki

To uninstall Rafiki services:

helm uninstall rafiki

# Optionally uninstall dependencies
helm uninstall rafiki-postgres
helm uninstall rafiki-redis

Persistent data

Uninstalling doesn’t delete Persistent Volume Claims (PVCs) by default. To delete all data:

kubectl delete pvc -l app.kubernetes.io/instance=rafiki-postgres
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-redis
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-tigerbeetle

Troubleshooting

Check service health

Verify that all services are running correctly:

# Check pod status
kubectl get pods -l app.kubernetes.io/name=rafiki-auth
kubectl get pods -l app.kubernetes.io/name=rafiki-backend
kubectl get pods -l app.kubernetes.io/name=rafiki-frontend

# Check service endpoints
kubectl get endpoints

# Check for events
kubectl get events --sort-by=.metadata.creationTimestamp

Common issues and solutions

Database connection errors

1. Verify database connectivity:

   kubectl logs -l app.kubernetes.io/name=rafiki-backend | grep -i database

2. Check PostgreSQL status:

   kubectl get pods -l app.kubernetes.io/name=postgresql
   kubectl logs -l app.kubernetes.io/name=postgresql

3. Test database connection:

   kubectl run -it --rm debug --image=postgres:14 -- psql postgresql://username:password@rafiki-postgres-postgresql:5432/rafiki_backend

Redis connection issues

1. Check Redis status:

   kubectl get pods -l app.kubernetes.io/name=redis
   kubectl logs -l app.kubernetes.io/name=redis

2. Test Redis connectivity:

   kubectl run -it --rm debug --image=redis:7 -- redis-cli -h rafiki-redis ping

Service discovery problems

1. Verify service names and ports:

   kubectl get services
   kubectl describe service rafiki-backend-service

2. Check DNS resolution:

   kubectl run -it --rm debug --image=busybox -- nslookup rafiki-backend-service

Authentication and authorization issues

1. Check auth service logs:

   kubectl logs -l app.kubernetes.io/name=rafiki-auth

2. Verify secrets are correctly mounted:

   kubectl describe pod -l app.kubernetes.io/name=rafiki-auth

3. Test auth endpoints:

   kubectl port-forward svc/rafiki-auth-service 3007:3007
   curl http://localhost:3007/healthz

Ingress and networking issues

1. Check ingress controller status:

   kubectl get pods -n ingress-nginx
   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller

2. Verify ingress resources:

   kubectl get ingress
   kubectl describe ingress rafiki-backend-ingress

3. Check TLS certificates:

   kubectl get secrets
   kubectl describe secret rafiki-backend-tls

Resource and performance issues

1. Check resource usage:

   kubectl top pods
   kubectl top nodes

2. Monitor pod metrics:

   kubectl describe pod -l app.kubernetes.io/name=rafiki-backend

3. Scale services if needed:

   # Update replicaCount in values.yaml and upgrade
   helm upgrade rafiki interledger-helm/rafiki -f values.yaml

Production considerations

Security

• Use strong passwords: Replace all default passwords with cryptographically secure values
• Enable TLS: Configure HTTPS for all external communications
• Network policies: Implement Kubernetes network policies to restrict pod-to-pod communication
• RBAC: Use Kubernetes Role-Based Access Control to limit cluster access
• Secrets management: Consider using external secret management solutions like HashiCorp Vault
• Image security: Use specific image tags and scan images for vulnerabilities
• Pod security: Configure pod security standards and security contexts

High availability

• Multi-replica deployments: Run multiple replicas of critical services
• Pod disruption budgets: Configure disruption budgets to maintain availability during updates
• Resource limits: Set appropriate CPU and memory limits
• Health checks: Configure proper readiness and liveness probes
• Load balancing: Use multiple ingress controller replicas

Monitoring and observability

• Prometheus: Deploy Prometheus for metrics collection
• Grafana: Set up Grafana dashboards for visualization
• Logging: Implement centralized logging with ELK stack or similar
• Alerting: Configure alerting rules for critical issues
• Distributed tracing: Consider implementing distributed tracing for request flow analysis

Backup and disaster recovery

Database backups

PostgreSQL backup:

# Create a backup job
kubectl create job --from=cronjob/postgres-backup manual-backup-$(date +%Y%m%d-%H%M%S)

# Manual backup
kubectl exec -it rafiki-postgres-postgresql-0 -- pg_dump -U postgres rafiki_backend > backup.sql

TigerBeetle backup:

# Create volume snapshots
kubectl create -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: tigerbeetle-snapshot-$(date +%Y%m%d-%H%M%S)
spec:
  source:
    persistentVolumeClaimName: tigerbeetle-data-0
  volumeSnapshotClassName: your-snapshot-class
EOF

Performance optimization

• Resource allocation: Right-size CPU and memory requests/limits based on actual usage
• Connection pooling: Configure appropriate database connection pool sizes
• Caching: Optimize Redis configuration for your workload
• TigerBeetle tuning: Configure TigerBeetle cluster size based on expected transaction volume

Upgrade strategies

• Rolling updates: Use rolling update strategies to maintain availability
• Blue-green deployments: Consider blue-green deployments for zero-downtime upgrades
• Database migrations: Plan for database schema migrations
• Backup before upgrade: Always backup data before major upgrades
• Testing: Test upgrades in staging environments first