Skip to content

DIRAC for Mu3e

User Introduction & Setup Guide

This document introduces DIRAC and Mu3e and describes the initial client setup.

Warning

Status: Work in progress

Further sections (catalog access, job submission, data management) are intentionally incomplete and will be extended.

What is DIRAC?

DIRAC (Distributed Infrastructure with Remote Agent Control) is a distributed computing framework widely used in high-energy physics and other scientific communities. It provides:

  • Secure authentication using X.509 certificates
  • Workload and data management across multiple computing centers
  • A unified interface for grid, cloud, and HPC resources

DIRAC uses a central Configuration Service, a Proxy Manager, and multiple backend services to allow users to submit jobs and manage data securely.

What is Mu3e?

Mu3e is a particle physics experiment at the Paul Scherrer Institute (PSI) searching for the rare decay:

Text Only
μ⁺ → e⁺ e⁻ e⁺

Mu3e relies on distributed computing resources for simulation, reconstruction, and data analysis. DIRAC is used as the Distributed Data Management and Workload Management System that can be used by the Mu3e experiment.

Configuring DIRAC for the Mu3e experiment

Prerequisites

Before starting, you need:

  • A valid X.509 user certificate (and private key).
  • Access to the Mu3e DIRAC VO.

Install User Certificates

DIRAC relies on standard Grid certificates. This often comes in .p12 format, and the private key and user certificate should be retreived from it, as follows:

Bash
openssl pkcs12 -in my_personal_CERN_cert.p12 -clcerts -nokeys -out usercert.pem
openssl pkcs12 -in my_personal_CERN_cert.p12 -nocerts -nodes -out userkey.pem

Once retrieved, place your personal certificate and private key in:

Bash
~/.globus

Tip

userkey.pem must be chmod 600

Install DIRAC Client Environment (DIRACOS)

DIRAC provides a self-contained runtime called DIRACOS.

Bash
mkdir ~/dirac_ui
cd ~/dirac_ui

curl -LO https://github.com/DIRACGrid/DIRACOS2/releases/latest/download/DIRACOS-Linux-$(uname -m).sh
bash DIRACOS-Linux-$(uname -m).sh

rm DIRACOS-Linux-$(uname -m).sh

Active the environment

Bash
source ~/dirac_ui/diracos/diracosrc

Install the DIRAC Python client:

Bash
pip install DIRAC==8.0.74

Warning

Do not install the latest DIRAC release (v9 or newer).

In particular, do not run:

Bash
pip install DIRAC
as this will automatically install the most recent DIRAC version.

⚠️ Mu3e is currently compatible only with DIRAC v8. Using DIRAC v9 or newer is known to cause incompatibilities and will result in a non-functional setup.

Install Certificate Authorities (CAs)

DIRAC requires trusted CA certificates to validate remote services. Merlin7 does not provide CA certificates and users are responsible for managing and updating them.

Warning

Users are responsible to maintain the CAs up to date

Download and install the CA bundles relevant to your GRID certificates: - UK Science: required for accessing Mu3e.org services. - CERN: optional, required only if you use a CERN-issued GRID certificate. - Other CAs: required if your GRID certificate was issued by a different certification authority.

Bash
mkdir -p ~/Downloads
cd ~/Downloads

# UK Science
wget https://repository.egi.eu/sw/production/cas/1/current/tgz/ca_UKeScienceCA-2B-1.138.tar.gz
wget https://repository.egi.eu/sw/production/cas/1/current/tgz/ca_UKeScienceRoot-2007-1.138.tar.gz

# CERN
wget https://repository.egi.eu/sw/production/cas/1/current/tgz/ca_CERN-Root-2-1.138.tar.gz
wget https://repository.egi.eu/sw/production/cas/1/current/tgz/ca_CERN-GridCA-1.138.tar.gz

Install them into DIRACOS (please ensure that the target directory -C exists):

Bash
cd ~/dirac_ui

mkdir -p diracos/etc/grid-security/certificates

for ca_file in $(ls ~/Downloads/ca_*.tar.gz)
do
  tar xvzf ~/Downloads/$ca_file \
    --strip-components=1 \
    -C diracos/etc/grid-security/certificates/
done

Fetch Certificate Revocation Lists (CRLs)

CRLs are required to ensure certificates have not been revoked.

Warning

Users are responsible to maintain this list up to date.

Load fetch-crl and update CRLs:

Bash
module use unstable
module load fetch-crl

fetch-crl -l $HOME/dirac_ui/diracos/etc/grid-security/certificates/

Initialize a Local Proxy (No Configuration Service)

Create a local proxy without contacting the Configuration Service (useful for first setup and debugging):

Bash
dirac-proxy-init --nocs

Configure DIRAC for Mu3e

Configure the DIRAC client to use the GridPP setup and configuration server:

Bash
dirac-configure -F \
  -S GridPP \
  -C dips://diracweb.grid.hep.ph.ic.ac.uk:9135/Configuration/Server \
  -I

Set the certificate directory explicitly (important when using DIRACOS):

Bash
export X509_CERT_DIR=$HOME/dirac_ui/diracos/etc/grid-security/certificates

Initialize a Mu3e DIRAC Proxy

Create and upload a proxy for the Mu3e VO:

Bash
dirac-proxy-init -g mu3e.org_user -M

Verify:

Bash
dirac-proxy-info

Next Steps (Work in Progress)

Warning

WORK IN PROGRESS, not fully documented yet

As of today, no services are available. What's possible and how to set it up accordingly needs to be clarified with the experiment.

Once the proxy and configuration are in place, users can start interacting with Mu3e's DIRAC services.

Accessing the File Catalog (WIP)

DIRAC provides a File Catalog that tracks all Mu3e datasets and their replicas across storage elements.

Examples of operations that will be documented:

  • Listing directories and files
  • Querying logical file names (LFNs)
  • Checking replicas and storage locations
  • Uploading and downloading files

Tools involved include:

  • dirac-dms-filecatalog-cli
  • dirac-dms-lfn-replicas
  • dirac-dms-get-file

Submitting Jobs (WIP)

DIRAC supports job submission to distributed resources using:

  • Job Description Language (JDL)
  • Python-based DIRAC job APIs
  • Command-line submission tools

Planned topics:

  • Writing a minimal JDL
  • Submitting and monitoring jobs
  • Retrieving job outputs
  • Debugging failed jobs

Other DIRAC Tools (WIP)

Further DIRAC components to be documented:

  • Data Management System (DMS)
  • Workload Management System (WMS)
  • Production system
  • Monitoring and accounting tools