20 Jun I have given up the unequal struggle to move our VO over to FNAL. The show stoppers are:-
Derek: "The LCG Resource Broker isn't VOMS aware so it can only use the vo and not the voms part, so it tries to match against resources for "fermilab". The new GLite WMS can match using voms (but would also match with the vo) but that hasn't yet been certified for production use."
Derek: "The problem with the YAIM installation tools was that while trying to determine pool accounts it would fail if it the it couldn't find a match for "/VO=X/GROUP=/X" which we didn't have as we didn't want to support the fermilab "vo", plus other than a hardcoded special case which we have need to duplicate and maintain, the tools expected the "special users" - for software installation or production to be of the form "/VO=X/GROUP=/X/Y" or "/VO=X/GROUP=/X/Role=Z"
17 MayDerek has has tweak MinosL on lcgce01 to match my FNAL proxy and now glite-job-list-match matches plus a bunch of sites in the States. I run /usr/bin/whoami on them all and could look at the mapping:-
antaeus.hpcc.ttu.edu fermilab ce01.cmsaf.mit.edu fermilab cit-gatekeeper.ultralight.org fermilab cmsgrid02.hep.wisc.edu osg_fermilab fermigrid1.fnal.gov minos lcgce01.gridpp.rl.ac.uk minossgm osg-gw-2.t2.ucsd.edu fermilab osgce.hepgrid.uerj.br fermilab top.ucr.edu fermilabbut submitting a job,ended on top.ucr.edu (University of California, Riverside, aborted. I could run on lcgce01 but Derek said:-
"I suspect it won't as the CE has not been reconfigured to use the FNAL voms server, however if it falls back to checking the standard grid-map file it may work for you as you should be in there from the Manchester one, but not for someone who wasn't."
16 MayDerek has explained that the problem lies with installation tools that cannot cope with the experiment (minos) being different to the VO name (fermilab). He is trying to get the glite interface to work on lcgui0358 with configuration selecting dev261 (the development m/c for WMS), but the test he has asked me to do fails:-
**** Error: UI_NO_VO_CONF_INFO **** Unable to find configuration information for VO "fermilab"
4 MayDan and I figured out that I was incorrectly registered at FNAL (CA was still eScience, not eScienceCA), and now I can setup a proxy from FNAL. Now for the tricky bit ... persuading CEs to authenticate and correctly map accounts.
2 MayI have mailed dteam (gridpp-dteam@cern.ch) asking about VOMS migration and Jeremy Coles has replied. Being a subgroup of an FNAL VO is (possibly?) a name change and then it's more complicated that just reconfiguring sites to accept a different VOMS. In particular it could mess up the LFC - a reason to do it now. Also they need to find out more about what it means to host an FNAL subgroup. Apparently Steve Traylen had reservations.
Derek Ross: It turns out that we had a fermigrid cert on our UIs but it expired in March, I've put your updated ones on lcgui0358.gridpp.rl.ac.uk.
I try but:-
voms-proxy-init -voms fermilab:/fermilab/minos/ukminos --confile ~/.glite/vomses Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=nick west Enter GRID pass phrase: Creating temporary proxy ............................................. Done Contacting fermigrid2.fnal.gov:15001 [/DC=org/DC=doegrids/OU=Services/CN=fermigrid2.fnal.gov] "fermilab" Failed Error: Could not establish authenticated connection with the server. GSS Major Status: Unexpected Gatekeeper or Service Name GSS Minor Status Error Chain: an unknown error occurred
1 MayFor the past month or so I have been focussing on DCM development and not running GRID jobs but today I tried running install jobs on RAL T1 and T2. Both failed!!
RAL T1 - VO_MINOS_SW_DIR misconfigured.
It's /stage/sl3-lcg-exp/minos (but should be ../minossgm).
RAL T2 - Software disk full:-
Filesystem Size Used Avail Use% Mounted on
heplnx202:/raid/expt-sw
162G 154G 8.0K 100% /raid/expt-sw
The MINOS share is 5GB
- but both were fixed the same day.
21 MarBonny explained that she put the file into the default Service Class which was why I could not see it. If I look for my two files now:-
rfdir /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/ -rwxr-xr-x 1 nwest minos 11506790 Feb 27 12:41 F00034621_0000.mdaq.root -rw-r--r-- 1 nwest minos 403597 Feb 27 14:51 LVJ_F00034638_0000.mdaq.rootone is back on disk and the other must be off on tape:-
stager_qry -M /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/ Received 1 responses /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/F00034621_0000.mdaq.root 255464@castorns.ads.rl.ac.uk STAGEDSo I decided to run a test using TRFIOFile (i.e. rfiod) selecting the file that was on tape. The system worked perfectly: the stager status at once showed STAGEIN, and shortly thereafter STAGED and the job ran.
20 MarBonny Strong (19 Feb): Should be fixed now. When I check, rfdir claims there are two files:-
rfdir /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/ -rwxr-xr-x 1 nwest minos 11506790 Feb 27 12:41 F00034621_0000.mdaq.root -rw-r--r-- 1 nwest minos 403597 Feb 27 14:51 LVJ_F00034638_0000.mdaq.rootbut stager reports only one, and that's bad news:-
stager_qry -M /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/ Received 1 responses /castor/ads.rl.ac.uk/test/grid/hep/disk0tape1/minos/nwest/LVJ_F00034638_0000.mdaq.root 255548@castorns.ads.rl.ac.uk INVALID_STATUS
13 Mar With Sergey's help I find a typo in my .glite/vomses: there was a space between OU=Manchester/ and L=DHEP. Having fixed that I can switch accounts by switching roles:-
voms-proxy-init --voms minos --confile ~/.glite/vomses
maps to minos003,
voms-proxy-init --voms minos:/Role=lcgadmin --confile ~/.glite/vomses
maps to minossgm
11 MarNo report back from helpdesk yet re. CASTOR, but now the status is INVALID_STATUS.
I have been discussing with Sergey how to switch roles when creating a proxy as this is possible at FNAL. The same should work for EGEE using as ~.glite/vomes having an entry:-
"minos" "voms.gridpp.ac.uk" "15004" "/C=UK/O=eScience/OU=Manchester/ L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk" "minos"and the command:-
voms-proxy-init -voms minosand works for Sergey but fails for me:-
Contacting voms.gridpp.ac.uk:15003 [/C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk] "minos" Warning: t2k: Unable to satisfy G/minos Request! Error: VERR_SERVERCODE Failed. Failed to contact servers for minos.Note the 15003 not 15004! Changing the number in the vomes file doesn't effect it.
With Sergey's help I have found
/opt/glite/etc/vomses/minos-voms.gridpp.ac.ukwhich contains:-
"minos" "voms.gridpp.ac.uk" "15003" "/C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk" "minos"so that explains the 15003. Also Sergey says I should be using
/opt/glite/bin/voms-proxy-init
not /opt/edg/bin/voms-proxy-init
More examples (like CASTOR) where there are multiple versions of the
tools and by default you get the wrong ones! Sigh! Using the
alternative gets a bit further:-
/opt/glite/bin/voms-proxy-init --voms minos:Rlcgadmin Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=nick west Enter GRID pass phrase: Creating temporary proxy .................................................. Done Contacting voms.gridpp.ac.uk:15004 [/C=UK/O=eScience/OU=Manchester/ L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk] "minos" Failed Error: Could not establish authenticated connection with the server. GSS Major Status: Service or hostname could not be understood GSS Minor Status Error Chain: an unknown error occurred Trying next server for minos. Creating temporary proxy .................................. Done Contacting voms.gridpp.ac.uk:15003 [/C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk] "minos" Failed Error: t2k: Unable to satisfy Rlcgadmin Request! Failed to contact servers for minos.At least it's trying the local one on my ~/.glite/vomes before going onto the standard one, but it's still failing.
08 MarI wanted to try CASTOR to TCasterFile, but so far the CASTOR-SUPPORT list cannot tell me if there is a rootd server running. I also found that my test file had migrated to tape. When I tried to copy, stager_qry reported STAGED, but continued to report this even when the file had been restored (as confirmed by stager_qry in another window). I have ask CASTOR-SUPPORT why?. In fact I have now found that logging out and in clears the problem and it reports STAGED, but if I then do rfdir it goes back to STAGEIN, but if I logout and in again it goes back to and stays at STAGED. So now I try to do my TRFIOFile test (which worked before) and it hangs and when I check the status in anothe window it has gone back to STAGEIN! Ah, yet more quality software!
02 Mar Sergey cancels my registration, I can no loner run jobs and I re-register. Later I am back in.
01 Mar I discover that I am no longer a member of our VO when I attempt to check my details at: https://voms.gridpp.ac.uk:8443/voms/minos/ but can still run GRID jobs!
27 Feb Start testing CASTOR. Manage to read and write files including read access from RAL UI via TRFIOFile.
22 Feb Dan Yocum (FNAL VOMS) has sent me their new server and host certificates so I have asked Chris Brew (RAL T2 manager) about switching.
15 Feb We finally get our CASTOR allocation. Its disk0tape1. We will also need disk1tape0.
05 Jan I request CASTOR allocation
04 Jan At today's UB meeting we are told:
23 Nov The RAL Tier 2 CE is moving from heplnx201 to heplnx206 so I am publishing my software tags there simply by re-validating. I hit one problem: My R1.21 loon failed as the validation script that was installed did not accept a recently introduced --verbose switch. I have had to rebuild my R1.21 tarball so that it has the up to date version of the script.
I have recently been reminded that the registration of out VO at http://cic.in2p3.fr/ hasn't got an AUP so I have added a few line .txt one that just gives the URL in the web.
10 Nov With more help from Sergey and Alessandra I was able to set up my new user account and give it VO-Admin and lcgadmin roles and then with the new account, run jobs (that mapped to minossgm which I take to mean that the role was working) and remove the old one.
02 Nov Sergey confirms: I must use VOMS-Admin. I ask him if I can switch between certificates simply by exporting/importing into browser and whether getting a new certificate involves RA authentication.
30 Oct I tried to get a new CA certificate from:- http://ca.grid-support.ac.uk/pub/cacert/cacert.cer but was told I already had that certificate. I'll ask Sergey and Alessandra. I was going to the wrong (old) place.
27 OctAccording to Instruction for VO administratorsCA rollover the DN (Distinguished Name) of our CA has changed
from: /C=UK/O=eScience/OU=Authority/CN=CA to: /C=UK/O=eScienceCA/OU=Authority/CN=CAand for people who still have unexpired ticket need to go and register a new certificate at https://www.gridpp.ac.uk/wiki/Instruction_for_VO_administrators#CA_rollover but that seems only to apply to the dteam VO. Perhaps I shouldn't be trying to use VOMRS but VOMS-Admin.
18 Oct Problems with our VO registeration with EGEE (10 Oct) missing AUP and VOMS key although I thought I had supplied both.
17 Oct GGUS have transferred my query to ROC-UK and closed the ticket! Dan has made me a VO representative, group manager, group owner, member which leaves me very confused! I have asked for clarification and how this plays with the roles minossoft, root, VOMS-Query and the groups /fermilab/minos /fermilab/minos/ukminos and /fermilab/minos/usminos
13 Oct At yesterday's s/w meeting interest was expressed in running GRID code at FNAL so I have raised a GGUS ticket to push along the migration of our VOMS to FNAL. I have got a reply from Steve (now at CERN) that they hit a problem, reported on the 9 Sep, when trying to switch to FNAL at RAL. He will try to escalate it.
10 Oct Register our VO with EGEE.
21 Sep Having fixed TGAFFile fails: Communication error on send I had hoped TGFALFile might now work but with the latest ROOT (cvs) and my hacked dcap 1-2-39 tar file from /opt/d-cache/dcap/lib it fails SysError in <TGFALFile::TGFALFile>: file ... can not be opened for reading (Invalid argument)
28 Jul I have been thinking about, and discussing with Robert, LABYRINTH jobs. It looks possible that I can get away tarring up and installing just the binary versions of vegetable flavoured gminos_batch and reco_minos. The problem is the flux files. It's a lot of data (e.g. v19 is ~ 10GB for LE-10 ROOT + PAW). A few versions of different beams could easily get to ~ 50GB. Chris Brew advises that this is too much for the software disk (both capacity and possibly bandwidth to WN jobs), however the good news is that downloads across the Internet of this sort of size would be O.K.
14 Jul Another first! Today I successfully ran a loon R1.23 job on Ral Tier 2 with:-
dcap://dcache-head.gridpp.rl.ac.uk:22125/pnfs/gridpp.rl.ac.uk/tape/minos/nwest/test/LVJ_F00034638_0000.mdaq.rooti.e. reading directly from SE that isn't "close". So it looks like my recent problems reading with dcap were down to the fact that ROOT needs to be at least 5.12/00.
12 Jul I thought I would have a go at file relication, but that fails too!
11 Jul Still struggling with TGFALFile and dcap problems
30 Jun The TGFALFile problem has been passed to GGUS. In the meantime I am trying something less ambitious: dcache access using a TDCacheFile with:-
dcap://dcache-head.gridpp.rl.ac.uk:22125/pnfs/gridpp.rl.ac.uk/tape/minos/nwest/test/LVJ_F00034638_0000.mdaq.rootat least I though it was less ambitious but doesn't work so is now a current problem
30 Jun I am trying to use ROOT's TGFALFile to access a file given it's LFN. Under the covers TGFALFile does:-
Int_t ret = ::gfal_open64(pathname, flags, (Int_t) mode);
and in my case: pathname = lfn:/grid/minos/nwest/tape/test/LVJ_F00034638_0000.mdaq.root
flags = 0 RDONLY
mode = 0644 (and not used for read)
but it fails:-
file lfn:/grid/minos/nwest/tape/test/LVJ_F00034638_0000.mdaq.root can not be opened for reading (Communication error on send)
27 Jun Our VOMS is still Manchester, there have been problems trying to switch to FNAL, but Dan and Steve are working on it.
12 Jun I can now publish software tags on Ral Tier 2. I get the CE name I need to publish the tag from the .BrokerInfo file if it exists, or failing that, the environment variable EDG_WL_LOG_DESTINATION.
I am trying to follow Chris' suggestion and use TGFALFile which can take a LFN and means we can bypass all the messing about with TURLs. So far not much luck. I have build the ROOT library on Tier 1 with
--enable-gfalbut trying to open
gfal:lfn:/grid/minos/nwest/tape/test/LVJ_F00034638_0000.mdaq.root (or lfn/)fails