GRID UK Data Access Plans

• Introduction
• Migrating dCache and NFS data to CASTOR
• Residual Data Structure
• Choosing a catalogue regime: DCM v LFC
• NFS Disk Usage Summaries

Introduction

This is contingent on an equivalent stable CASTOR allocation and time to migrate. We should take this opportunity to review and organise our data and possibly catalogue it under LFC

Migrating dCache and NFS data to CASTOR

Every file has to be migrated to a single structure within CASTOR or scrapped.

Caution: files written to CASTOR via GRID middleware are all owned by the same account (minos001), so we ALL have to be careful about deleting files!

Caution: CASTOR is not a suitable place to hold small files such as software and configuration files. Such files should be held at user home sites and where appropriate passed in via the input sandbox

Alex and I have decided on the following structure:-

raw/
  <detector>/                             (near/;far/)

mc/
  <detector>/                             (near/;far/)
    <MC s/w version>/                     (daikon_04/)
      <beam>/                             (L010185N/)

reco_data/
  <detector>/                             (near/;far/)
    <Reco s/w version >/                  (cedar_phy_bhcurv/;dogwood/)
      <N-tuple type>/                     (sntp_data/;mrnt_data/)

reco_mc/
  <detector>/                             (near/;far/)
    <MC s/w version>/                     (daikon_04/)
      <beam>/                             (L010185N/)
        <Reco s/w version >/              (cedar_phy_bhcurv/;dogwood/)
          <N-tuple type>/                 (sntp_data/;mrnt_data/)

caldet/                                         All CalDet related data

Other areas:

ana/
  <Analysis group>                        (e.g. nc/)

user/                    All files specific to a single user
  <user-name>/

tmp/                     Temporary files, expected only to last a few days at most
  <user-name>/             Users can put files here that are created at RAL but are
                           wanted locally.

flux/                    Flux data sets
  gnumi/
    v19/

archive/
  ADS/                   Data migrated from Atlas Data Store
    minos/
    soudan2/
      raw_pdkdata/
      processed_pdkdst/
      isis_test_data_pdkisis/

obsolete/                Files belonging to uncontactable users.  To be deleted after a few months    
  <user-name>/   

Residual Data Structure

  /stage/minos-data1/vo

  dbm_auto/                      Database maintenance
    work/
    scripts/

  grid/                          GRID operations support
    dcm_catalogue/               DCM catalogues
    dcm_resource_locks/          DCM resource locks
    rsd_work                     RSD work area
    sshkrb5/                     Kerberos 5 ssh and scp
    tests/                       GRID tests

  mc_production/                 Top level directory for all recent and current MC productions
    crashed/                     core dumps
    daikon_scripts/              Daikon scripts
    log/                         GBS logdir
    in_transit/                  GBS LOCAL_SAVE_DIR
    users/                       User specific production and testing.
      marta/                       
      rodriges/
      west/

  mc_production_obsolete/        Temporary location for files that we probably don't need.
                                 The plan will be to delete anything still in this area in early 2009

  minos_tools/                   $MINOS_TOOLS - a collection of utilities

  sl3_software/                  Obsolete SL3 code, to be removed once SL3 UIs go.

  software/                      SL4 software + Top level setup scripts 
    apps/                        RSD directory
    libs/                        RSD directory
    rsd/                         RSD directory
    rsd_home/                    RSD directory

  users/                         Private areas for VO and production managers.
    marta/
    rodriges/
    west/

NFS Disk Usage Summaries

  Disk usage summary

and files by each of the following usernames:-

      28343
      annah
      asousa
      hartnell
      kordosky
      marta
      minosmc
      nwest
      root

Choosing a catalogue regime: DCM v LFC