See also MINOS on the GRID
SL4 Migration:
Oxford and RAL have migrated to SL4 (Scientific Linux 4) and the
information about SL3 has been stripped from this document.
See the
old version of this page
for that SL3 information.
The basic organisation is:-
/data/minos/minos/software/
tools/ Tools for building/using the software.
OO/minossoft/ minossoft (Development Base Release only)
OO/minos_packs_sl4/ 3rd party libraries needed by above release
labyrinth/ Obsolete Fortran code (gminos and reco_minos)
misc/ Obsolete Extra libraries for Fortran code
/data/minos/software/grid Installed Base Releases at Oxford
October 2007: The farm is migrating to Scientific Linux 4 (SL4) and the GRID!
Requests for MINOS allocation of farm resources are discussed at GridPP User Board Meetings and are determined primarily according to the number of UK authors.
At the meeting on Tuesday 24 June 2008 MINOS requested, and were granted:-
| Year | CPU (kSpecInt2k) | Castor Disk (TB) | NFS Disk (TB) | Tape (TB) |
|---|---|---|---|---|
| 2009 | 125 | 8.7 | 0.2 | 15 |
| 2010 | 125 | 8.7 | 0.2 | 20 |
| 2011 | 125 (Q1) 0 (Q2/4) | 8.7 | 0.2 | 20 |
| 2012 | 0 | 8.7 | 0.2 | 20 |
| 2013 | 0 | 8.7 | 0.2 | 20 |
For the UB meeting in Spring 2009 we requested
| Year | CPU (kSpecInt2k) | Castor Disk (TB) | NFS Disk (TB) | Tape (TB) |
|---|---|---|---|---|
| 2009 | 250 (Q2) 125(Q3/4) | 8.7 | 1.0 | 15 |
| Year | CPU (kSpecInt2k) | Castor Disk (TB) | NFS Disk (TB) | Tape (TB) |
|---|---|---|---|---|
| 2009 | 225 (Q4) | 8.7 | 0.2 | 15 |
| 2010 | 125 | 8.7 | 0.2 | 15 |
| Year | CPU (kSpecInt2k) | Disk (TB) | Tape (TB) |
|---|---|---|---|
| 2011 | 125 | 10 | 20 |
| 2012 | 125 | 10 | 20 |
| 2013 | 25 | 10 | 20 |
| 2014 | 25 | 10 | 20 |
We currently have 2 NFS disks:-
/rutherford/minos-soft2/tools General support tools for building and using the software.
/stage/minos-data1/software/grid Installed Base Releases at RAL
We also have Castor disk space:-
At RAL the tool DCM (Data Cache Manager) is used to copy and share FNAL data files on these disks. For example:-
$MINOS_TOOLS/dcm.sh get C00110071_0000.mdaq.root C00110072_0000.mdaq.rootuses wget to copy the specified files and catalogues them. For more information see Data Cache Manager
sql.gridpp.rl.ac.ukThe names of the databases and user accounts follow the standard convention except that each is prefaced by minos_ e.g. minos_reader, minos_offline etc.
The database is not directly accessible from off-site but a connection can be made via an SSH tunnel. Proceed as follows:-
ssh -g -L 3306:sql.gridpp.rl.ac.uk:3306 user@csf.rl.ac.uk
where "user" is some valid account at RAL. This logs you into RAL
but also sets up a tunnel for the local port 3306 (MySQL) on
my-proxy-machine that appears on csf and gets forwarded to
the RAL MySQL server. The -L tells SSH to set up the tunnel and
the -g says to make the tunnel global i.e. visible to other machines.
setenv ENV_TSQL_URL mysql:odbc://my-proxy-machine.physics.my-site.ac.uk/minos_offline setenv ENV_TSQL_USER minos_reader setenv ENV_TSQL_PSWD=\0
We have now completed the process of migrating to the GRID with a few special cases which was based on MINOS input to UB Meeting on 04 Dec 2007
The key points for us are:-
User Name --- ---- rodriges Philip Rodrigues minosmc Generic Account. Used by Philip Rodrigues, Ruth Toner and Nick West nwest Nick West
gLite 3.1 UI tarball distribution
Tutorial: Accessing Storage ElementsFor people outside the UK I (Nick) am prepared to help out with data movement back to FNAL so long as it doesn't take too much time.
Installing a Test Release on a Worker Node
At Oxford it is used all Frozen and Snapshot releases but the Development release is maintained using the classic SRT setup. To simplify setup a wrapper script is provide to source the appropriate script. For a csh/tcsh shell:-
source /data/minos/software/setup_minos_oxford.csh {release|application}
Where release or application is an Installed Base Release
e.g.
source /data/minos/software/setup_minos_oxford.csh development_sl4
source /data/minos/software/setup_minos_oxford.csh minossoft:S07-10-22-R1-26-build_1-SL4
source /data/minos/software/setup_minos_oxford.csh
- the last of these just sets up support tools e.g. dcm, svn, SameWebClient.
For sh/bash use the same script replacing the extension .csh by .shBy default, when setting up any minossoft based application you will get the unoptimised version. To subsequently switch to the optimised one type:-
srt_setup SRT_QUAL=maxoptor to return to the unoptimised one:-
srt_setup SRT_QUAL=default
For a csh/tcsh shell:-
source /stage/minos-data1/software/grid/setup_minos_local-SL4.csh {release|application}
Where
<application> Required application e.g. minossoft:S07-09-20-R1-26-build_1-SL4
For a list see Installed Base Release at RAL - SL4
e.g.
source /stage/minos-data1/software/grid/setup_minos_local-SL4.csh minossoft:S07-10-22-R1-26-build_1-SL4
source /stage/minos-data1/software/grid/setup_minos_local-SL4.csh CedarDaikon:03-build_0-SL4
source /stage/minos-data1/software/grid/setup_minos_local-SL4.csh
- the last of these just sets up support tools e.g. dcm, svn, SameWebClient.
For sh/bash use the same script replacing the extension .csh by .shBy default, when setting up any minossoft based application you will get the unoptimised version. To subsequently switch to the optimised one type:-
srt_setup SRT_QUAL=maxoptor to return to the unoptimised one:-
srt_setup SRT_QUAL=default
cd /stage/minos-data1/allusers/tools/mtr export MTR_DIR=<my-working-directory> (the parent directory that will hold the Test Release) export MTR_NOTIFY=<email-address> (used to send confirmation email) - or the equivalent setenv commands if using a csh based shell.
Command syntax:-
./mtr <create|update|build> <test_release_name> \
<base_release_name> <package1> <package2> (...) <packageN>
Examples:-
./mtr create MyTest minossoft:S07-09-20-R1-26-build_1-SL4 NCUtils AnalysisNtuples
./mtr update MyTest CedarDaikon:03-build_0-SL4 (update and rebuild all existing packages)
./mtr update MyTest CedarDaikon:03-build_0-SL4 MCReweight (as above but also add new ones)
./mtr build MyTest CedarDaikon:03-build_0-SL4
Create in the usual way using any available SL3 version of minossoft:-
my_base_release_sl3=R1.25 my_base_release_sl4=S07-09-20-R1-26 my_test_release_parent=some-existing-directory my_test_release_name=some-test-release-name my_package=some-package source /stage/minos-data1/software/minossoft/setup/setup_minossoft_csf.sh $my_base_release_sl3 cd $my_test_release_parent newrel -t $my_base_release_sl3 $my_test_release_name cd $my_test_release_name srt_setup -a addpkg -h $my_packageNow comes the sneaky part, move the supporting release to the one that is needed on the SL4 back-end:-
rm .base_release echo $my_base_release_sl4 > .base_release
This works because SRT is smart enough to create the platform specific directories to hold binaries as and when it needs them. So even though the Test Release installed on SL3, it can still be built on SL4 - which is the whole point about SRT's ability to support multiple platforms sharing a single source tree.
my_base_release_sl4=S07-09-20-R1-26 my_application=minossoft:$my_base_release_sl4-build_1-SL4 my_test_release_parent=some-existing-directory my_test_release_name=some-test-release-name source /rutherford/minos-soft2/tools/GridTools/Scripts/setup/setup_minos_lcg_grid.sh $my_application cd $my_test_release_parent/$my_test_release_name gmake all
| Application | Last Updated | Minossoft Version | Neugen3 Version | ROOT Version | Notes |
|---|---|---|---|---|---|
| CedarDaikon04:071114-build_1-SL4 | 22 Nov 2007 | R1.24.3 | v3_5_5 | 5-12-00f | a.k.a. pro-SL4 |
| DogwoodDaikon04:build_0-SL4 | 23 Jul 2009 | R2.0.1 | v3_5_5 | 5-22-00a | 23 Jul High momentum track bug fix |
| DogwoodDaikon07:build_1-SL4 | 23 Jul 2009 | R2.0.1 | v3_5_5 | 5-22-00a | 23 Jul Daikon 07 +High momentum track bug fix |
| minossoft:R1.28-build_1-SL4 | 23 Jan 2008 | R1.28 | v3_5_5 | 5.18.00 | None |
| minossoft:R1.29-build_1-SL4 | 21 May 2008 | R1.29 | v3_5_5 | 5.18.00c | None |
| minossoft:R1.30-build_1-SL4 | 01 Sep 2008 | R1.30 | v3_5_5 | 5.20.00 | None |
| minossoft:S09-10-16-R2-00-build_1-SL4 | 19 Oct 2009 | S09-10-16-R2-00 | v3_5_5 | 5.24.00b | a.k.a. old-SL4 |
| minossoft:S10-01-19-R2-01-build_1-SL4 | 21 Jan 2010 | S10-01-19-R2-01 | v3_5_5 | 5.26.00 | a.k.a. new-SL4 and pro-SL4 |
To see applications are currently installed:-
ls -l /stage/minos-data1/software/grid/appsand then to see what versions of say minossoft are installed:-
ls -l /stage/minos-data1/software/grid/apps/minossoftand to see what libraries a given application contains type e.g.:-
cat /stage/minos-data1/software/grid/apps/minossoft/S07-10-22-R1-26-build_2-SL4/installed_librariesTo get the application name from the directory names replace the intervening "/" with a ":" e.g.:-
minossoft/S07-09-20-R1-26-build_1-SL4 -> minossoft:S07-09-20-R1-26-build_1-SL4To see in detail exactly what any application consists of, look it up in the current build_config_table.dat
| Application | Last Updated | Minossoft Version | Neugen3 Version | ROOT Version | Notes |
|---|---|---|---|---|---|
| CedarDaikon04:071114-build_1-SL4 | 15 Jan 2008 | R1.24.3 | v3_5_5 | 5-12-00f | a.k.a. pro-SL4 |
| minossoft:R1.28-build_1-SL4 | 18 Jan 2008 | R1.28 | v3_5_5 | 5.18.00 | None |
| minossoft:R1.29-build_1-SL4 | 21 May 2008 | R1.29 | v3_5_5 | 5.18.00c | None |
| minossoft:R1.30-build_1-SL4 | 01 Sep 2008 | R1.30 | v3_5_5 | 5.20.00 | None |
| minossoft:R2.2-build_1-SL4 | 02 Mar 2010 | R2.2 | v3_5_5 | 5.26.00b | None |
| minossoft:R2.3-build_1-SL4 | 09 Apr 2010 | R2.3 | v3_5_5 | 5.26.00b | None |
| minossoft:S08-03-20-R1-28-build_1-SL4 | 25 Mar 2008 | S08-03-20-R1-28 | v3_5_5 | 5.18.00a | Held by Justin |
| minossoft:S09-10-16-R2-00-build_1-SL4 | 16 Oct 2009 | S09-10-16-R2-00 | v3_5_5 | 5.24.00b | a.k.a. old-SL4 |
| minossoft:S10-01-19-R2-01-build_1-SL4 | 19 Jan 2010 | S10-01-19-R2-01 | v3_5_5 | 5.26.00 | a.k.a. pro-SL4 |
| minossoft:S10-04-02-R2-03-build_1-SL4 | 02 Apr 2010 | S10-04-02-R2-03 | v3_5_5 | 5.26.00b | a.k.a. new-SL4 |
| development Use for testing only! | Recently | HEAD | v3_5_5 | HEAD | None |
To check what applications are currently installed:-
ls -l /data/minos/software/grid/appsand then to see what versions of say minossoft are installed:-
ls -l /data/minos/software/grid/apps/minossoftand to see what libraries a given application contains type e.g.:-
cat /stage/minos-data1/software/grid/apps/minossoft/S07-10-22-R1-26-build_2-SL4/installed_librariesTo get the application name from the directory names replace the intervening "/" with a ":" e.g.:-
minossoft/S07-09-20-R1-26-build_1-SL4 -> minossoft:S07-09-20-R1-26-build_1-SL4To see in detail exactly what any application consists of, look it up in the current build_config_table.dat
For example, just to use the current production Snapshot:-
Oxford: source /data/minos/software/setup_minos_oxford.sh/csh minossoft:pro-SL4
RAL: source /stage/minos-data1/software/grid/setup_minos_local-SL4.sh/.csh minossoft:pro-SL4
The Development Release should not be used as the basis of production work.
Although it is constantly changes it is recommended for code development and should always work. In order to achieve this we have a double system. The basic idea is that there are two independent builds of minossoft called dev_1 and dev_2 and associated ROOT builds called root_cvs_1 and root_cvs_2. At any one time one version works at some level and is available for use. The other is updated and built and, should it appear to work, the system flips so that it becomes the current version. Flipping is achieved by soft links.
$SRT_DIST/releases:-
dev_1
dev_2
development -> dev_1 (or dev_2)
$INSTALLATION/:-
root_cvs_1
root_cvs_2
root_cvs -> root_cvs_1 (or root_cvs_2)
Updating and flipping works as follow:-
gmake cleanwhen you first come in if using development
addpkg XXXas that will attempt to check out the currently selected HEAD_* release and will fail with a message of the type
Release development uses XXX version HEAD_2, will check that out ... cvs [checkout aborted]: no such tag HEAD_2Instead do:-
addpkg XXX -h
setenv CVSROOT :pserver:anonymous@minos1.fnal.gov:/cvs/minoscvs/rep1 setenv CVS_RSH sshwhich tell cvs to user the anonymous server at minos1.fnal.gov, using SSH security. When you want to access the Repository you may have to login. If attempting to access results in a request to login then do so using:-
cvs -d $CVSROOT loginand will be asked for a password - ask your Software Librarian if you don't know it. If you need both read and write access see Acquiring access to the MINOS CVS repository.
$MINOS_TOOLS/dcm.sh disk_usageThe system can retrieve files from FNAl Dcache, e.g.:-
$MINOS_TOOLS/dcm.sh get C00070590_0000.tdaq.root C00070590_0001.tdaq.root ...and also read and write files to local SEs (Storage Elements) at RAL.
If moving many files to/from RAL its a good idea to log into one of the "Data Mover" front-ends:
csfmove01.rl.ac.uk or csfmove02.rl.ac.ukas they have higher bandwidth connectivity to the Internet.
For more details see Data Cache Manager
samTranslateDimensions \
--dim="run_type physics% \
and data_tier sntp-near \
and physical_datastream_name cosmic \
and start_time < to_date('2005-10-02','yyyy-mm-dd') \
and end_time > to_date('2005-10-01','yyyy-mm-dd')"
Notes about forming queries (the --dim="..." option above):-
run_type physics%means that run_type matches anything starting physics.
data_tier sntp-nearis equivalent to:-
data_tier like sntp-near.
samLocate \ --file=N00008695_0023.cosmic.sntp.R1_18.0.rootand retrieve the complete metadata entry for the file with:-
samGetMetadata \ --file=F00032684_0023.mdaq.rootThe script minosGetFiles.py is installed so to copy the files that satisfy a query into the current directory:-
minosGetFiles.py --dim=querye.g.:-
minosGetFiles.py --dim="file_name = N00008695_002%.cosmic.sntp.R1_18.0.root"although now that DCM uses SAM there should be no need to work directly with the SAM commands.