How to configure Xrootd for Third Party Copy (TPC)

When configurating Xrootd for 3rd Parth Copy, there are several things to consider:

  1. Is this a standard along Xrootd or a cluster of Xrootd ?
  2. Is the storage accessible to the Xrootd server as a file system? e.g. ext4, xfs, GPFS, Lustre?
  3. Is the storage accsssible to the Xrootd server as a remote object store?
  4. What data transfer protocol(s) will be used, xroot and/or https?
  5. What authentication/authoriaztion mechanism will be used? X509/VOMS, Token, sss key?

As an introduction, we will first cover a very simple case. After that, we will provide a more complicated configuration example to cover several use cases in WLCG. We will assume that TLS is always enabled (which means using Xrootd release 5.3.0 and above).

A simple TPC using xroot protocol and rendezvous key

Xrootd TPC using rendezvous key works this way:

  1. The client authenciates with both the data source and the destination (using GSI or others) and obtains authorization for read and write.
  2. The client supplies a rendezvous key to both the source and destination.
  3. The destination initiates a pull request using a pre-configured program or script.
  4. In the pull request, the destination authenticates against the source. It will then use the rendezvous key for authorization.
  5. The destination pulls the data file.

An example config file for TPC on a single Xrootd storage node like this the following:

all.adminpath /tmp/xrootd/var/spool
all.pidpath   /tmp/xrootd/var/run

oss.localroot /tmp
all.export /xrootd

# Enable checksum
xrootd.chksum adler32

# Config TLS
xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
xrootd.tls capable all -data

#sec.level all compatible

# Enable Security
xrootd.seclib libXrdSec.so

# Enable "gsi" security
sec.protparm gsi -vomsfun:libXrdVoms.so -vomsfunparms:dbg
sec.protocol gsi -ca:1 -crl:3 -gridmap:/dev/null

# Authorizaton
acc.audit deny
acc.authdb /etc/xrootd/auth_file
acc.authrefresh 60
ofs.authorize 1

# Xrootd TPC using rendezvous key
ofs.tpc logok autorm pgm /etc/xrootd/xrdcp-tpc.sh

# Env var needed by the above TPC script.
setenv X509_USER_CERT = /etc/grid-security/xrd/xrdcert.pem 
setenv X509_USER_KEY = /etc/grid-security/xrd/xrdkey.pem

We will need to enable TLS so that we can securely transport the rendezvous key from the client to both source and destination. This also means that the xrootd program needs to have access to the /etc/security/xrd/{xrdcert.pem, xrdkey.pem} (a copy of host cert and key).

The destination will run the TPC program (xrdcp-tpc.sh). The program authenticates againt the source (using GSI), it then uses the rendezvous key for authorization. After that, it pulls the data. The xrdcp-tpc.sh looks like the following.

#!/bin/sh

# To work with data source's GSI, a X509 proxy is needed. It can be any X509 
# proxy - the authorization uses the rendezvous key, not the X509 proxy.
export X509_USER_PROXY=/tmp/x509up_u$(id -u)

# Note that the main xrootd config file exported the location of 
# X509_USER_CERT and X509_USER_KEY
xrdgsiproxy init \
    -cert $X509_USER_CERT \
    -key $X509_USER_KEY \
    -bits 1024 \
    -f $X509_USER_PROXY > /dev/null 2>&1

unset X509_USER_CERT
unset X509_USER_KEY
xrdcp -f $@

Please make sure the above script has u+rx bits set.

Note that for xrootd TPC, both sides (source and destination) needs to be configured to support xrootd TPC, whereas for HTTP TPC, only one side (usually the desitination) needs to be configured to support HTTP TPC.

An example of WLCG TPC configuration with X509 authentication

The many different use cases

In WLCG, Xrootd is used in a wide range of cases to provide TPC services, with HTTP TPC being the primary, and xrootd TPC the secondary. The example configuration files below will try to covers several use cases. Sorting out these use cases and concepts behind them will help understand the example configuration file.

How xrootd TPC service will access the storage?

  • If storage is not mounted to the xrootd TPC service node, the TPC node functions as a Xrootd DTN (a.k.a xrootd proxy, or gateway to a backend storage - on LAN or WAN).
  • If storage is mounted to the xrootd TPC node as a posix file system, the TPC node functions as a regular xrootd storage server. However, there are two sub-categories:
    • The storage is a local posix file system (only mounted to that particular TPC node, e.g. XFS filesystem on a disk-array)
    • The storage is a shared posix file system (mount on many nodes, e.g Lustre or GPFS).

A single node for TPC services, or a cluster?

A cluster is how xrootd scales. In a cluster, there are

  • Several server nodes. Their roles in the cluster are either server or proxy server. The latter refers to the scenario where xrootd TPC service function as DTNs. Why this is important?
    • If server nodes mount local filesystems, then each of they have access to a different set of data files.
    • If server nodes mount a shared filesystem, then they all have access to the same set of files
    • If server nodes are proxy server, they usuall have access to the same set of files.
  • A redirector node. Its role is either a manager or proxy manager. A redirector sends a client to a server or proxy server, based on which one has the files wanted by the client. (A redirector is usually lightweight, require very little resource).

If there is only a single node, it is either a server or a proxy server.

So before moving forward, ask youself

  • Do I run DTN or do I have filesystem access to storage
  • Is this a local filesystem or a shared filesystem?
  • Do I run a single node, or a cluster.

An example WLCG TPC setup will cover the following:

  1. TPC services described in the above use case scenarios:
    • DTN vs local posix file system vs share posix file system
    • a single TPC node or a cluster.
  2. Use X509/VOMS for authentication and authorization
  3. Support HTTP TPC via the macaroon tokens and SciTokens
  4. Support Xrootd TPC via X509 proxy delegation

Requirements

  1. A copy of host certificate at /etc/grid-security/xrd/{xrdcert.pem, xrdkey.pem}. These files should be owned by user xrootd (or equivalent).
  2. CA certificates (default /etc/grid-security/certificates)
  3. voms certificates (default /etc/grid-security/vomsdir)
  4. RPMs: xrootd xrootd-voms voms libmacaroons scitokens-cpp (and their dependences). Other RPMs may also be required.
  5. (optional) TCmalloc (gperftools-libs) or JEmalloc (jemalloc) may provide better memory management for xrootd.

One may also need to set nproc >= 8192 and nofile >= 8192 in /etc/security/limits.d if the systems will be under high load.

Example Xrootd configuration file

We expect TLS be enabled on all nodes, include the redirectors.

# If there is a redirector
# "redirector" should be full qualified DNS name (e.g. hostname -f). 
set redirector = <myrdr_hostname>
all.manager $(redirector):1213

xrd.port 1094

all.adminpath /var/spool/xrootd/var/spool
all.pidpath   /var/run/xrootd/var/run

all.export /dir1
all.export /dir2

# For DTN only
# Expect the backend xrootd door to have the same all.export as the above
#pss.origin <mybackend.storage.xrootd.door>:<port>

# TLS:
xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
xrd.tlsca certdir /etc/grid-security/certificates
xrootd.tls capable all

if $(redirector)
    xrootd.chksum adler32

    # "manager" role: for a cluster
    #
    # Manager (rediector) role
    # For a 'proxy manager'
    #all.role proxy manager
    # For a 'manager'
    all.role manager

    # For Xrootd DTN, or Xrootd storage on a shared file system
    #cms.dfs lookup distrib redirect immed

    # For Xroodt DTN
    #pss.ckslib adler32 /usr/lib64/libXrdPss.so
else
    xrootd.chksum adler32

    # "server" role: for both single node and cluster
    #
    # config the Role: server / proxy server, standalone or cluster
    # For a standalone 'proxy server'
    #ofs.osslib /usr/lib64/libXrdPss.so
    # For a standalone 'server'
    #NOTHING
    # For a 'proxy server' in a cluster
    #all.role proxy server
    # For a 'server' in a cluster
    all.role server

    # Enable security
    xrootd.seclib /usr/lib64/libXrdSec.so

    # X509 VOMS security in xroot protocol
    sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:dbg
    sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null

    # Xrootd TPC using X509 proxy delegation
    # For 'proxy server'
    #ofs.tpc fcreds ?gsi =X509_USER_PROXY ttl 60 70 xfr 100 autorm pgm /etc/xrootd/xrdcp-tpc.sh
    # For 'server'
    ofs.tpc fcreds ?gsi =X509_USER_PROXY ttl 60 70 xfr 100 autorm pgm /usr/bin/xrdcp

    # Authorizatoin
    acc.audit deny
    acc.authdb /etc/xrootd/auth_file
    acc.authrefresh 60
    ofs.authorize
fi

# Config HTTP
if ($redirector)
    http.desthttps yes
fi

# HTTP Protocol
if exec xrootd
    xrd.protocol http libXrdHttp.so
fi
http.selfhttps2http no

# X509 VOMS in HTTP(s) protocol
http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
http.secxtractor libXrdVoms.so dbg
http.header2cgi Authorization authz

# Set all.sitename in order to use Macaroon
all.sitename MYSITE

http.exthandler xrdtpc libXrdHttpTPC.so
http.exthandler xrdmacaroons libXrdMacaroons.so
macaroons.secretkey /etc/xrootd/macaroon-secret
# order of the following two lines is important
ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
ofs.authlib ++ libXrdMacaroons.so

An example xrdcp-tpc.sh

Please set permission of this script to 755

#!/bin/sh
set -- `getopt S: -S 1 $*`
while [ $# -gt 0 ]
do
  case $1 in
  -S)
      ((nstreams=$2-1))
      [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams"
      shift 2
      ;;
  --)
      shift
      break
      ;;
  esac
done

src=$1
dst=$2
xrdcp --server $TCPstreamOpts -f $src root://$XRDXROOTD_PROXY/${dst}

SciToken config file /etc/xrootd/scitokens.cfg

The following example support tokens issued by the WLCG Indigo IAM. 

[Global]
onmissing = passthrough

[Issuer WLCG IAM]
issuer = https://wlcg.cloud.cnaf.infn.it/
base_path = /xrootd
map_subject = false
default_user = xrootd

Please refers to page Token-based AuthN/Z for WLCG for more info about this config file (and next config file below), VO specific IAMs and how to obtain a WLCG JWT token.

The access control file /etc/xrootd/auth_file:

= atlasprod o: atlas g: /atlas r: production
x atlasprod /dir1 rwildn
g /atlas /dir1 rl
g /cms /dir2 rl
g /dteam /dir1 rwild /dir2 rwild
# authz based on WLCG JWT token
= wlcgtknusr o: https://wlcg.cloud.cnaf.infn.it/ g: /wlcg
x wlcgtknusr /xrootd a
  1. line 1: define a special compound id "atlasprod" to identify all users within ATLAS VO with "/atlas/Role=production".
  2. line 2: grant this "atasprod" access permission 'rwildn'
  3. line 3: grant all other ATLAS users (all of those in VO group /atlas) permission 'rl'
  4. line 4: grant all /cms users (all users in VO group /cms) permission 'rl'
  5. line 5: grant all /dteam users (all users in VO group /dteam) permission 'rwild'
  6. default is deny access

The robots.txt

It is there to ask the internet search engines to stay away:

User-agent: *
Disallow: /

Generating the macaroon secret

Run command openssl rand -base64 -out macaroon-secret 64. All nodes, include redirectors should have the same macaoon secret.

Ports

For a standalone Xrootd, xroot and https can have different ports. For a cluster, xroot and https can use different ports only on the redirector. On data servers, xroot and https have to use the same port.

References:

  1. WLCG Xrootd 3rd Party Copy (TPC), https://twiki.cern.ch/twiki/bin/view/LCG/XrootdTpc
  2. HTTP TPC, https://twiki.cern.ch/twiki/bin/view/Main/XRootDoverHTTP#Enable_Third_Party_Copy
  3. Macaroons support, https://twiki.cern.ch/twiki/bin/view/Main/XRootDoverHTTP#Macaroons_Support

Using xrootd to support TPC for S3 object store

Xrootd can provide the following services to an S3 object store.

  • Object store functions such as Upload/Download/Deletion/Stat/DirListing/Checksum.
  • HTTP TPC and xrootd TPC services
  • X509/VOMS authentication/authorization (and JWT in the future).

In this case, Xrootd is used as a DTN to the S3 storage. It authenticates to the S3 storage using a pair of access key and secret key (aka, username/password). Configuring xrootd in this use case is largely based on the WLCG TPC configurat example, with the following additional configurations:

  • S3 object store uses the word bucket for what we usually called directory.

  • Change to the configuration file (the server / proxy serve section in the else clause)

    pss.origin https://s3store.xyz:port setenv AWS_ACCESS_KEY_ID = GOOGS4MV4PQTK3D4G0LB34AN setenv AWS_SECRET_ACCESS_KEY = QtUMprl2j07m4VfzygcjJQlVrvO9ZfU+eKDbR2sP

    (Note that to demostrate the idea, we leave the credentials in the xrootd config file. This is not a good security practice. setenv directive allows putting these credentials in files. The variables set by setenv are Unix environment variables so they can also be set accordingly before xrootd starts.)

  • Disabling Davix session caching to prevent memory leak and fragmentation setenv DAVIX_DISABLE_SESSION_CACHING = 1

  • xrootd will use scripts for checksum and xrootd TPC (examples below).

    • in the else clause of the configuration file, change the adler32 line to

      xrootd.chksum max 32 adler32 /etc/xrootd/xrdadler32.sh

  • A staging space. When writing to s3 storage, Davix temporarily stores a complete file to this staging area before it uploads the file to the s3 storage.
    • The default space is /tmp. It can be changed by unix environment variable DAVIX_STAGING_AREA
  • xrootd needs the XrdClHttp plugin and configuration to load the plugin. The rpm for this plugin (and the configuration file) is available in EPEL as xrdcl-http
  • Install the Davix rpm (available from EPEL).

Checksum and an example /etc/xrootd/xrdadler32.sh

#!/bin/sh

file=$1
davix-get --s3accesskey $AWS_ACCESS_KEY_ID \
          --s3secretkey $AWS_SECRET_ACCESS_KEY \
          --s3alternate https://${XRDXROOTD_PROXY}$file | xrdadler32 | awk '{print $1}'

As you see from the above script, the ckecksum is calculated by fetching (using davix-get) the file from the s3 storage. There are two drawbacks:

  • no place to store the ckecksum
  • may incure Egress charge if the s3 store is in a commercial cloud, e.g. Google (using HMAC key) or AWS, but xrootd runs outside of the cloud.

An example /etc/xrootd/xrdcp-tpc.sh

#!/bin/sh
set -- `getopt S: -S 1 $*`
while [ $# -gt 0 ]
do
  case $1 in
  -S)
      ((nstreams=$2-1))
      [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams"
      shift 2
      ;;
  --)
      shift
      break
      ;;
  esac
done

src=$1
dst=$2
xrdcp --server -s $TCPstreamOpts $src - | \
  aws --endpoint-url https://$XRDXROOTD_PROXY s3 cp - s3:/$dst 2>/dev/null

In the script, aws is the AWS Command Line Interface. Loading aws to run may take a second or two. If this is too slow, one can use davix-put. However, using davix-put requires the xrdcp to first save the file - davix-put does not read from STDIN.

The XrdClHttp plugin

This plugin is available from the EPEL repo as the 'xrdcl-http' rpm. The following is for those who needs the lastest updates that is not available in the rpm. This is no common.

If the latest feature in the XrdClHttp plugin (libXrdClHttp-5.so or libXrdClHttp.so) is not available in the above rpm, one will need to compile xrootd to create this plugin. After that copy the .so to /usr/lib64 and then run chrpath -d /usr/libi64/libXrdClHttp.so.

To instruction xrootd to load this plugin, create /etc/xrootd/client.plugins.d/xrdcl-http-plugin.conf with the following lines inside:

url = https://*;http://*
lib = /usr/lib64/libXrdClHttp.so
enable = true

Renaming

As a final note, renaming is poorly supported in s3 object store. For this reason, the XrdClHttp plugin disabled renaming.