4. Managing profiles

Goal

Learn how to add new DataSHIELD profiles to your local Opal deployment from section 1. This builds directly on the simple local setup, showing you how to extend it with multiple profiles without losing data or adding production complexity.

What are profiles?

Profiles in Opal/DataSHIELD are named configurations that can include:

  • Specific sets of DataSHIELD packages
  • Different R package versions
  • Custom package repositories
  • Environment-specific configurations

Think of profiles as isolated R environments within your Opal deployment, similar to Python virtual environments, but for DataSHIELD research contexts.

Prerequisites

  • Working local Opal deployment from section 1
  • Basic understanding of Docker Compose
  • Your local deployment should be running and accessible at http://localhost:8080

Common use cases for profiles

  • Research-specific environments: Different studies requiring different package versions
  • Development vs. stable: Testing new packages before using them in research
  • Package version management: Maintaining stable versions while testing updates
  • Learning: Experimenting with different DataSHIELD packages safely

Architecture with profiles (local)

graph TB
  B["Browser<br/>http://localhost:8080"] --> O["Opal Server (8080)"]
  O --> R1["Rock Profile: default<br/>dsBase"]
  O --> R2["Rock Profile: genomics<br/>dsOmics, dsExposure"] 
  O --> R3["Rock Profile: survival<br/>dsSurvival"]
  subgraph Profiles["docker network: opalnet"]
    R1 --> P1["rock-default:8085"]
    R2 --> P2["rock-genomics:8085"] 
    R3 --> P3["rock-survival:8085"]
  end

  %% Define a lighter background for the subgraph
  classDef light fill:#f9f9f9,stroke:#aaa,stroke-width:1px;
  class Profiles light;

Starting point: Your local deployment

From section 1, you should have this structure:

opal-local/
├── .env
├── docker-compose.yml
├── data/
│   ├── opal/
│   └── mongo/
└── logs/

We’ll extend this by adding new Rock services for each profile.

Step 1: Update your .env file

Your .env file from section 1 only needs the password. No changes required:

OPAL_ADMINISTRATOR_PASSWORD=ChangeMe123!

Step 2: Extend docker-compose.yml with profiles

Replace your docker-compose.yml from section 1 with this extended version that adds a survival profile:

services:
  opal:
    image: obiba/opal:latest
    depends_on:
      - rock-default
      - rock-survival
      - mongo
    ports:
      - "8080:8080"
      - "8443:8443"
    environment:
      - OPAL_ADMINISTRATOR_PASSWORD=${OPAL_ADMINISTRATOR_PASSWORD}
      - MONGO_HOST=mongo
      - MONGO_PORT=27017
      # Multiple Rock hosts - comma separated
      - ROCK_HOSTS=rock-default:8085,rock-survival:8085
    volumes:
      - ./data/opal:/srv
      - ./logs:/var/log/opal

  # Default profile (same as section 1, just renamed)
  rock-default:
    image: datashield/rock-base:latest
    environment:
      - ROCK_ID=default

  # New survival analysis profile  
  rock-survival:
    image: datashield/rock-base:latest
    environment:
      - ROCK_ID=survival

  # MongoDB (unchanged from section 1)
  mongo:
    image: mongo:6.0
    volumes:
      - ./data/mongo:/data/db

networks:
  default:
    name: opalnet

Understanding the changes

Let’s break down what changed from the simple setup in section 1 and why:

1. Modified Opal Service Dependencies

depends_on:
  - rock-default
  - rock-survival  # NEW
  - mongo

This ensures Opal waits for all Rock profile containers to start before attempting to connect to them, preventing connection errors during startup.

2. Updated ROCK_HOSTS Environment Variable

- ROCK_HOSTS=rock-default:8085,rock-survival:8085

This comma-separated list tells Opal where to find all available Rock servers. Opal uses this to discover and manage multiple DataSHIELD computation environments simultaneously.

3. Renamed Original Rock Service

# From: rock (section 1) 
# To:   rock-default
rock-default:
  environment:
    - ROCK_ID=default  # Explicit profile name
  • Consistency: All profiles now follow the same naming pattern (rock-{profile-name})
  • Clarity: Makes it explicit that this is the “default” profile
  • ROCK_ID: Each Rock container needs a unique identifier for Opal to distinguish between profiles

4. Added New Profile Services

rock-survival:
  environment:
    - ROCK_ID=survival  # Unique profile identifier

Each profile runs in its own container, providing isolated R environments. The unique ROCK_ID allows Opal to route DataSHIELD operations to the correct profile.

5. What Stays the Same

  • MongoDB service: Data storage is independent of computation profiles
  • Data persistence: Local folders preserve your existing data during the transition
  • Network: Same opalnet network ensures all services can communicate

This transforms your architecture from a single computation environment to multiple isolated profiles:

Before: Opal ↔ Single Rock Container

After:  Opal ↔ Multiple Rock Containers (profiles)
        ├── rock-default (dsBase)
        └── rock-survival (dsBase,dsSurvival)

Step 3: Apply changes with maintenance window

Warning

Service Interruption: Adding or removing profiles requires stopping the Opal service, which will temporarily interrupt access for all users. Plan a maintenance window and notify users in advance.

The key to adding profiles without losing your existing data is using local folders that persist between container recreations:

# 1. Stop and remove all services (data in local folders is preserved)
docker-compose down

# 2. Update your docker-compose.yml file (copy the extended version above)

# 3. Start the new configuration
docker-compose up -d

# 4. Verify all services are healthy
docker-compose ps

Benefits of this approach: Using docker-compose down ensures clean container recreation, proper service registration, and profile discovery while preserving all data in local folders (./data/opal, ./data/mongo, ./logs).

Step 4: Verify your profiles are running

Check that all Rock containers are running:

# Check container status
docker-compose ps

# Check logs for each profile
docker-compose logs rock-default
docker-compose logs rock-genomics  
docker-compose logs rock-survival

You should see all three Rock containers running and healthy.

Adding a single profile (step-by-step)

If you want to add just one profile at a time, here’s the simplified process:

1) Update docker-compose.yml

Add the new Rock service to the services: section:

      rock-newprofile:
    image: datashield/rock-base:latest
    environment:
      - ROCK_ID=newprofile

And update the ROCK_HOSTS environment variable in the opal service:

- ROCK_HOSTS=rock-default:8085,rock-genomics:8085,rock-survival:8085,rock-newprofile:8085

2) Apply the changes (with maintenance window)

# Stop and remove all services (clean restart)
docker-compose down

# Start with new configuration
docker-compose up -d

Removing a profile

To remove a profile safely:

1) Update docker-compose.yml

  • Delete the entire rock-genomics: service block
  • Remove rock-genomics:8085 from the ROCK_HOSTS environment variable

2) Apply the changes (with maintenance window)

# Stop and remove all services (clean restart)
docker-compose down

# Start with updated configuration
docker-compose up -d
Note

Maintenance Planning: Profile changes require a brief service interruption. For production deployments, schedule these changes during low-usage periods and communicate the maintenance window to users.

Testing your profiles from R

Verify each profile works correctly:

library(DSI)
library(DSOpal)
library(httr)
# Disable SSL verification for local testing
set_config(config(ssl_verifyhost = 0L, ssl_verifypeer = 0L))

# Test genomics profile
builder <- DSI::newDSLoginBuilder()
builder$append(
  server = "genomics",
  url = "http://localhost:8080",
  user = "administrator", 
  password = "ChangeMe123!",
  driver = "Opal",
  profile = "genomics"  # Specify the profile
)

logins <- builder$build()
conns <- DSI::datashield.login(logins)

# Check available packages
DSI::datashield.pkg_status(conns)

DSI::datashield.logout(conns)

Troubleshooting profiles

Profile not appearing in Opal UI

# Check Rock container is running
docker-compose ps

# Check Rock container logs for errors
docker-compose logs rock-genomics

# Test network connectivity from Opal to Rock
docker-compose exec opal curl -f http://rock-genomics:8085/

Connection timeouts

  • Verify the ROCK_HOSTS environment variable includes all profiles
  • Check that service names match between docker-compose.yml and ROCK_HOSTS
  • Ensure all containers are on the same network (opalnet)
  • Try a clean restart: docker-compose down && docker-compose up -d

Memory issues

# Monitor container resource usage
docker stats

# If running low on memory, consider:
# 1. Reducing the number of active profiles
# 2. Stopping unused profiles temporarily
docker-compose stop rock-genomics

Next steps

With profiles working locally, you can:

  • Install different packages in each profile using the Opal UI
  • Create project-specific profile assignments
  • Test new DataSHIELD packages safely in development profiles
  • Scale up to production deployment (sections 2 and beyond) when ready

This local multi-profile setup gives you a powerful development environment for DataSHIELD research while keeping complexity manageable.

References