The DQ2 tools and web pages of this document (below) are outdated.
I submitted an error report on Savannah and they pointed me to the new web pages
(but the old web pages are still active and not updated).
I don't understand why the old tools are not disabled,
some tools work and others just crash with error messages.
I don't understand why the old documentation is not corrected or moved offline.
Purpose
The issue: I need to get some files from "DQ2".
Ayana is putting log files there.
Here are her instructions:
At CERN you might have to source some setup scripts and get your grid certificate.
The commands are something like this:
- source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.[c]sh
- voms-proxy-init -voms atlas
- source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.[c,z]sh.CERN
...
dq2_ls -f datasetName
... tells you avaliable files.
dq2_get datasetName file1 file2 ...
... will get files file1, file2 ...
You can try this out using a file from datasetName
user.AyanaTamuHolloway.inclJet.test: I will register the
logfiles the same way so if you can get some of those RDO you
should have no problem with the logs.
These instructions are helpful to get started,
but I am aparently missing some pieces of the puzzle so they
do not work off the bat (not unexpected).
I am not at CERN, I am at Oxford. I have not used my Grid certificate...
Have to get registered, certified, branded, initiated, ...
Here is the process of trial and error and hopefully resolution
(I get some files).
Steps to get files from DQ2
There is a lot of information out there on ATLAS and CERN web pages.
Problems are redundant information and missing critical steps
that more typically apply for users outside of CERN and new users
who don't have much working that other users forgot they did).
I include a few related links at the bottom of this page
where some relevent information was found.
No web page I found had anywhere near what I needed to make this work.
I was lucky Stefan was here to provide some missing pieces.
-- I am working on pplxgen.physics.ox.ac.uk.
-- I may need AFS file access at CERN (cern.ch token, cvs)
klog -tmp
Password:
Wrote ticket file to /tmp
-- Following the instructions at (similar to Ayana's instructions):
-- https://twiki.cern.ch/twiki/bin/view/Atlas/UsingDQ2
> source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh
-- okay I guess, no errors
> voms-proxy-init -voms atlas
Cannot find file or dir: /userdisk3/gallas/.glite/vomses
user certificate not found
Cert File=/userdisk3/gallas/.globus/usercert.pem
Function: proxy_init_cred
-- From the errors above,
-- I must need the usercert.pem and vomses file in new directories off my home area.
--
-- The usercert.pem file is some kind of Grid Certificate.
-- I need to find my Grid certificate on my windows machine
-- (which I have never used ... but I think I got it a few months ago).
--
-- I need to make a copy of it on the Linux machine I am working on.
-- Look in my email accounts folder.
-- I have an email from ca@ca.grid-support.ac.uk
-- which gives details of my CA certificates.
--
-- Import it into my Firefox browser by going to the URL:
-- https://ca.grid-support.ac.uk/cgi-bin/pub/pki?cmd=getcert&key=8719&type=CERTIFICATE
--
-- Then, in Firefox,
-- under Tools tab,
-- click on Options,
-- click on Advanced Options,
-- under Security tab, select View Certificates Button.
--
-- Under 'Your Certificates' select the certificate I see:
- eScienceCA
---- elizabeth gallas .... expires on 24/11/2007
-- Select the 'Backup' button.
-- Create the directory .globus under my login directory.
-- Name the backup file: escience_cert_070118.p12
-- with 'Backup certificate password' / pass##
-- In response, I receive a message that I
-- 'Successfuly backed up your security certificate(s) and private key(s).'
--
-- Great. what does it mean ?
-- try to run the voms command again.
> voms-proxy-init -voms atlas
Cannot find file or dir: /userdisk3/gallas/.glite/vomses
user certificate not found
Cert File=/userdisk3/gallas/.globus/usercert.pem
Function: proxy_init_cred
-- Stefan says I need to 'convert the certificate file' (from p12 to pem file):
--
-- PK12_FILE="${HOME}/.globus/escience_cert_070118.p12"
-- openssl pkcs12 -nocerts -in ${PK12_FILE} -out ${HOME}/.globus/userkey.pem
-- openssl pkcs12 -nokeys -clcerts -in ${PK12_FILE} -out ${HOME}/.globus/usercert.pem
--
> PK12_FILE="${HOME}/.globus/escience_cert_070118.p12"
> openssl pkcs12 -nocerts -in ${PK12_FILE} -out ${HOME}/.globus/userkey.pem
Enter Import Password:pass##
MAC verified OK
Enter PEM pass phrase:pass##
Verifying password - Enter PEM pass phrase:pass##
> openssl pkcs12 -nokeys -clcerts -in ${PK12_FILE} -out ${HOME}/.globus/usercert.pem
Enter Import Password:pass##
-- okay, so the .globus/userkey.pem file exists now so try this again:
> voms-proxy-init -voms atlas
Cannot find file or dir: /userdisk3/gallas/.glite/vomses
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
bad file system permissions on private key
key must only be readable by the user
File=/userdisk3/gallas/.globus/userkey.pem
Function: proxy_init_cred
-- Stefan says I need to create the .glite/vomses file:
--
-- The vomses file certifies that the user in on the experiment.
--
-- mkdir ${HOME}/.glite
-- cp /afs/cern.ch/project/gd/egee/glite/ui_PPS/glite/etc/vomses .glite/
mkdir ${HOME}/.glite
cp /afs/cern.ch/project/gd/egee/glite/ui_PPS/glite/etc/vomses .glite/
-- okay, so the .glite/vomses file exists now so try this again:
> voms-proxy-init -voms atlas
data format in file: /userdisk3/gallas/.glite/vomses incorrect!
Line: "ops" "lcg-voms.cern.ch" "15009
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
bad file system permissions on private key
key must only be readable by the user
File=/userdisk3/gallas/.globus/userkey.pem
Function: proxy_init_cred
-- all files in .globus should be readable only by me
> cd .globus/
> dir
-rwxrwxrwx 1 gallas atlas 3182 Jan 18 12:18 escience_cert_070118.p12*
-rw-r--r-- 1 gallas atlas 2182 Jan 18 12:29 usercert.pem
-rw-r--r-- 1 gallas atlas 1926 Jan 18 12:29 userkey.pem
> chmod 400 *.*
> dir
-r-------- 1 gallas atlas 3182 Jan 18 12:18 escience_cert_070118.p12
-r-------- 1 gallas atlas 2182 Jan 18 12:29 usercert.pem
-r-------- 1 gallas atlas 1926 Jan 18 12:29 userkey.pem
-- try again
> voms-proxy-init -voms atlas
data format in file: /userdisk3/gallas/.glite/vomses incorrect!
Line: "ops" "lcg-voms.cern.ch" "15009
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
-- xemacs the vomses file:
-- fixing the "ops" line to have a consistent format with the other entries:
-- "ops" "lcg-voms.cern.ch" "15009...
-- try again
> voms-proxy-init -voms atlas
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
Creating temporary proxy .................................... Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Failed
Error: atlas: User unknown to this VO.
Trying next server for atlas.
Creating temporary proxy ............................................. Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Failed
Error: atlas: User unknown to this VO.
Trying next server for atlas.
Creating temporary proxy ........................................................ Done
Contacting voms.cern.ch:15001 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "atlas" Failed
Error: atlas: User unknown to this VO.
Failed to contact servers for atlas.
-- Aparently I am not listed in the VOM for Atlas.
-- Follow a few dead ATLAS documentation links,
-- then I found:
-- http://www.usatlas.bnl.gov/twiki/bin/view/Support/HowToAtlasVO
-- Register through the ATLAS VOMRS (Virtual Organization Management Registration Service)
-- https://lcg-voms.cern.ch:8443/vo/atlas/vomrs?path=/RootNode&action=execute
-- Access to ATLAS grid resources is granted by joining the ATLAS VO
--
-- 1. I am registered with CERN HR
-- 2. I went to https://lcg-voms.cern.ch:8443/vo/atlas/vomrs to become a candidate.
-- I completed the application for candidate status in the ATLAS VO.
-- Wait for an email.
-- Received email within 10 minutes to go to the next phase (registration phase I)
-- 3. Complete the registration by choosing what 'Group' (and 'Group role')
-- I think I belong. Groups and Roles are not described in any way.
--
-- Registration (Phase II)
-- You have successfully submitted phase II of ATLAS VO registration!
-- You will receive an email notice from the VOMRS ATLAS Service
-- indicating that you've been approved (or denied) as a VO member.
-- This could take up to a few days; it depends on how soon
-- your representative completes this task.
-- ...
--
-- You are logged in as /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
-- /C=UK/O=eScienceCA/OU=Authority/CN=CA
--
-- The above process took less than 1 day (Jan 18, 2007).
-- I await an email approving VO membership.
--
-- 7pm:
-- I received nearly 50 emails telling me which groups and roles I am
-- assigned or denied. I have been assigned to the following groups:
-- /atlas/lcg1
-- and 46 denials
--
-- try the voms proxy init again...
> voms-proxy-init -voms atlas
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
Creating temporary proxy ............................. Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Done
Creating proxy ......................................................... Done
Your proxy is valid until Fri Jan 19 07:08:23 2007
-- now continue with the DQ2 stuff
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
-- In principle, https://twiki.cern.ch/twiki/bin/view/Atlas/UsingDQ2#2_DQ2_end_user_tools
-- now I can use the following DQ2 end-user tools
* dq2_get
* dq2_ls
* dq2_put
* dq2_poolFCjobO
* dq2_register
* dq2_cleanup
* dq2_sample
-- I am going home now
New Day
Let's see if I can get some files today.
> source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh
> voms-proxy-init -voms atlas
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
Creating temporary proxy ....................................................... Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Done
Creating proxy .............................................. Done
Your proxy is valid until Sat Jan 20 03:25:38 2007
> source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
> dq2_ls -f user.AyanaTamuHolloway.inclJet.test
user.AyanaTamuHolloway.inclJet.test Total: 148 - Local: 0
dq2_get user.AyanaTamuHolloway.inclJet.test
The following files are not found at CERN.
Use '-r' if you want to copy them over the grid
streamtest2.inclJet.digit.RDO._00108.pool.root
streamtest2.inclJet.digit.RDO._00040.pool.root
streamtest2.inclJet.digit.RDO._00003.pool.root
streamtest2.inclJet.digit.RDO._00027.pool.root
streamtest2.inclJet.digit.RDO._00018.pool.root
...
streamtest2.inclJet.digit.RDO._00074.pool.root
streamtest2.inclJet.digit.RDO._00060.pool.root
streamtest2.inclJet.digit.RDO._00128.pool.root
streamtest2.inclJet.digit.RDO._00145.pool.root
streamtest2.inclJet.digit.RDO._00042.pool.root
Done
Total:0 - Failed:0
I don't want these files anyway, they are root files.
-- WWW: Use the DQ2 browser now that the port restriction has been lifted
on my Windows machine:
http://panda.atlascomp.org/?overview=dslist&cat=csc&redirect=pandamon
Click on catagory: user 6318 users found
Click on user: AyanaTamuHolloway 15 datasets found
Click on user.AyanaTamuHolloway.streamtest03.mixer.log at BNLPANDA -
290 files found with names like:
Logical file name Click to check replica at site
streamtest_mixer_00001.log.tgz BNLPANDA
streamtest_mixer_00002.log.tgz BNLPANDA
streamtest_mixer_00003.log.tgz BNLPANDA
streamtest_mixer_00004.log.tgz BNLPANDA
streamtest_mixer_00005.log.tgz BNLPANDA
-- Okay, get one of these 'files' on disk.
-- See how much space there is on different disks: df -h
-- Ewan says to put them on /var/tmp/
-- Follow https://twiki.cern.ch/twiki/bin/view/Atlas/UsingDQ2#dq2_get
> dq2_get -h
-- get a remote file
> dq2_get -rv streamtest_mixer_00001.log.tgz
http://atlddmpro.cern.ch:8000/dq2/ws_repository/dataset?dsn=streamtest_mixer_00001.log.tgz&version=0
(0, {})
0 {}
ERROR : streamtest_mixer_00001.log.tgz was not found in the DQ2 catalog
-- Write email to Tadashi Maeno . he wrote back with the solution
-- Now it seems obvious !
dq2_get -rv user.AyanaTamuHolloway.streamtest03.mixer.log streamtest_mixer_00001.log.tgz
-- This was started about 19:30 pm friday night.
-- An empty file was created Jan 19 20:09 streamtest_mixer_00001.log.tgz
-- So the dq2 get failed.
-- It produced hundreds of messages on the screen that I do not understand.
DQ2_log1.txt
-- Try to sort through it....
Highlights include messages like:
kill 16742
Timeout
could not finish
retry 0
/afs/.cern.ch/project/gd/LCG-share/3.0.10-1/d-cache/srm/sbin/srm: line 161: /userdisk3/gallas/.srmconfig/config.xml: No such file or directory
mkdir: too few arguments
Try `mkdir --help' for more information.
configuration file not found, configuring srmcp
srm client error: /userdisk3/gallas/.srmconfig/config.xml (No such file or directory)
SRMClientV1 : getRequestStatus: try #0 failed with error
SRMClientV1 : org.dcache.srm.SRMAuthorizationException: user usatlas4 is not found
SRMClientV1 : getRequestStatus: try again
Look in .../Atlas/UsingDQ2#Copy_files_over_the_grid.
Found note to Change ~/.srmconfig/config.xml; from
<x509_user_trusted_certificates>
/etc/grid-security/certificates
</x509_user_trusted_certificates>
to, e.g.,
<x509_user_trusted_certificates>
/afs/cern.ch/project/gd/LCG-share/certificates
</x509_user_trusted_certificates>
Ask your local admin which directory you should use for x509_user_trusted_certificates.
I don't have a .srmconfig/config.xml directory or file,
so I will create the directory and file to contain
<x509_user_trusted_certificates>
/afs/cern.ch/project/gd/LCG-share/certificates
</x509_user_trusted_certificates>
on and try again.
> cd /var/tmp/gallas
> source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh
> voms-proxy-init -voms atlas
> source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
> dq2_get -rv user.AyanaTamuHolloway.streamtest03.mixer.log streamtest_mixer_00001.log.tgz
cd /var/tmp/gallas
source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh
voms-proxy-init -voms atlas
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -rv user.AyanaTamuHolloway.streamtest03.mixer.log streamtest_mixer_00001.log.tgz
-- now I get a seemingly infinite number of of messages like this:
-- "bash: unalias: dq2_get: not found"
-- Try again.
-- Get errors
-- "on error, root element "srm-configuration" is not found
-- srm client error: null"
DQ2_log2.txt
-- Less errors this time and did not take 30 minutes, but I still get
-- fatal errors an no files appear.
-- The most apparent error is:
"error, root element "srm-configuration" is not found
srm client error: null
could not finish 'srmcp -retry_num=1 -streams_num=10
-x509_user_trusted_certificates=/afs/cern.ch/project/gd/LCG-share/3.0.10-1/etc/grid-security/certificates
srm://dcsrm.usatlas.bnl.gov:8443/pnfs/usatlas.bnl.gov/u04/holloway/DataStreaming/logs//streamtest_mixer_00001.log.tgz
file:////userdisk3/gallas/./streamtest_mixer_00001.log.tgz' with 256
Done
Total:1 - Failed:1
streamtest_mixer_00001.log.tgz"
-- Tadashi says to "rm ~/.srmconfig/config.xml"
-- because... This is a wrong fix.
-- You don't need to create it because you already have X509_CERT_DIR.
> rm ~/.srmconfig/config.xml
> voms-proxy-init -voms atlas
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
Creating temporary proxy ..................................................... Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Done
Creating proxy ................................. Done
Your proxy is valid until Mon Jan 22 22:21:51 2007
> dq2_get -rv user.AyanaTamuHolloway.streamtest03.mixer.log streamtest_mixer_00001.log.tgz
-- After about an hour, I get fatal errors again (new errors)
DQ2_log3.txt
Semi - Success on Oxford PP linux cluster
Note: I am logged into ppslgen.
I downloaded Yves's attached file to my Oxford ~/bin/dq2-env-oxford.sh
(inserting the following text):
#!/bin/sh
#source /usr/local/bin/lcguisetup
export DQ2_URL_SERVER=http://atlddmpro.cern.ch:8000/dq2/
export DQ2_URL_SERVER_SSL=https://atlddmpro.cern.ch:8443/dq2/
export DQ2_LOCAL_ID=
export DQ2_LOCAL_PROTOCOL=unix
# Following variables could be empty, check in code what is done
# with them
export DQ2_STORAGE_ROOT="/dpm/physics.ox.ac.uk/home/atlas"
export DQ2_SRM_HOST="srm://t2se01.physics.ox.ac.uk:8443"
export DQ2_GSIFTP_HOST="gsiftp://t2se01.physics.ox.ac.uk:2811"
# Simplify people's life
export DQ2_USE_SRM=False
export LCG_CATALOG_TYPE=lfc
export DQ2_LFC_HOME=/grid/atlas
export DQ2_COPY_COMMAND='lcg-cp -v --vo atlas'
# prevent the wild-card expansion
_dq2_ls_helper () {
unalias dq2_ls
dq2_ls $*
set +f
alias dq2_ls='set -f; _dq2_ls_helper'
}
alias dq2_ls='set -f; _dq2_ls_helper'
_dq2_get_helper () {
unalias dq2_get
dq2_get $*
set +f
alias dq2_get='set -f; _dq2_get_helper'
}
alias dq2_get='set -f; _dq2_get_helper'
exit 0
---- I commented out the final 'exit 0' so I could run it without being logged out (?)
cd /var/tmp/gallas
source ~/bin/dq2-env-oxford.sh
---- I need a certificate...
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init --valid 99:59
---- I need some definition of dq2_get
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -h
---- get one of Ayana's new datasets rather than the previous test dataset
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399
(previously user.AyanaTamuHolloway.streamtest03.mixer.log)
---- This seems to be working !!!!!!!!!!!!!
---- I get some 'timeout' messages, then retry...
---- As I am watching the directory, I notice there seems to be no order of file retrieval.
---- Started 14-Mar 14:00 (first successful file)
---- 14-Mar 15:45 (last successful file)
---- Ended 14-Mar 16:39 (last file date, which failed)
... lots of kill... timeout... retry...
... lots of kill... timeout... retry...
... lots of kill... timeout... retry...
.... then:
Done
Total:300 - Failed:3
logfile_00132.tgz
logfile_00225.tgz
logfile_00257.tgz
----
alias dir='ls -xlF'
dir
---- order files by size
dir -sS
---- all files seem normal size except the 3:
atlas 0 Mar 14 15:06 logfile_00132.tgz
atlas 0 Mar 14 16:39 logfile_00225.tgz
atlas 0 Mar 14 15:34 logfile_00257.tgz
---- Try to retrieve the 3 files which failed (0 size)
cd /var/tmp/gallas/streamtest00000/
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00132.tgz
---- failed at Oxford and CERN
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00225.tgz
---- failed at Oxford
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00257.tgz
---- Here are the types of messages before the final failure:
...
srmcp -retry_num=1 -streams_num=10
-x509_user_trusted_certificates=/afs/cern.ch/project/gd/L
CG-share2/certificates
srm://dcsrm.usatlas.bnl.gov:8443/pnfs/usatlas.bnl.gov/u04/holloway/DataStreaming/MixingLogs/run0/logfile_00132.tgz
file:////afs/cern.ch/user/g/gallas/scratch0/streamtest00000/./logfile_00132.tgz
kill 23037
Timeout
could not finish 'srmcp -retry_num=1 -streams_num=10
-x509_user_trusted_certificates=/afs/cern.ch/project/gd/LCG-share2/certificates
srm://dcsrm.usatlas.bnl.gov:8443/pnfs/usatlas.bnl.gov/u04/holloway/DataStreaming/MixingLogs/run0/logfile_00132.tgz
file:////afs/cern.ch/user/g/gallas/scratch0/streamtest00000/./logfile_00132.tgz' with 255
retry 0
kill 27818
Timeout
could not finish 'srmcp -retry_num=1 -streams_num=10
-x509_user_trusted_certificates=/afs/cern.ch/project/gd/LCG-share2/certificates
srm://dcsrm.usatlas.bnl.gov:8443/pnfs/usatlas.bnl.gov/u04/holloway/DataStreaming/MixingLogs/run0/logfile_00132.tgz
file:////afs/cern.ch/user/g/gallas/scratch0/streamtest00000/./logfile_00132.tgz' with 255
retry 1
kill 14815
Timeout
could not finish 'srmcp -retry_num=1 -streams_num=10
-x509_user_trusted_certificates=/afs/cern.ch/project/gd/LCG-share2/certificates
srm://dcsrm.usatlas.bnl.gov:8443/pnfs/usatlas.bnl.gov/u04/holloway/DataStreaming/MixingLogs/run0/logfile_00132.tgz
file:////afs/cern.ch/user/g/gallas/scratch0/streamtest00000/./logfile_00132.tgz' with 255
Done
Total:1 - Failed:1
logfile_00132.tgz
Okay, so this is declared just as successful at Oxford and CERN
(cannot get the same files, so it is not site dependent).
Now, try getting just one file
(in fresh window to check what setups really needed,
for example is the setup from Yves really needed at Oxford ?).
cd /var/tmp/gallas
mkdir streamtest00003
cd streamtest00003/
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init --valid 99:59
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
phrase is too short, needs to be at least 4 chars
Enter GRID pass phrase:
Creating proxy ............................................. Done
Your proxy is valid until Sun Mar 18 20:30:02 2007
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -h
dq2_get -rv user.AyanaTamuHolloway.streamtest.000003.mixing.digit.log.v12000399 logfile_00900.tgz
Done
Total:1 - Failed:1
logfile_00900.tgz
---- try again with the setup from Yves Coppens
source ~/bin/dq2-env-oxford.sh
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -h
dq2_get -rv user.AyanaTamuHolloway.streamtest.000003.mixing.digit.log.v12000399 logfile_00900.tgz
Done
Total:1 - Failed:0
---- Okay, something in that setup from Yves Coppens is absolutely necessary
---- before the dq2_user_client/setup.sh.CERN !!!!!!!!!!!!!!!!!!!!!
---- See if I can get another dataset 'streamtest.000008'
mkdir streamtest00008
cd streamtest00008
dq2_get -rv user.AyanaTamuHolloway.streamtest.000008.mixing.digit.log.v12000399
---- See if I can get another dataset 'streamtest.000009'
mkdir streamtest00009
cd streamtest00009
dq2_get -rv user.AyanaTamuHolloway.streamtest.000009.mixing.digit.log.v12000399
Success on lxplus
---- See if I can get dataset user.AyanaTamuHolloway.streamtest03.mixer.log
on lxplus into my new scratch0 area today.
ssh gallas@lxplus007.cern.ch
cd
cd scratch0
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init --valid 99:59
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -rv user.AyanaTamuHolloway.streamtest03.mixer.log
---- yes ! I got all the tgz files ...
---- See if I can get another dataset 'streamtest.000001'
mkdir streamtest00001
cd streamtest00001
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init --valid 99:59
Cannot find file or dir: /afs/cern.ch/user/g/gallas/.glite/vomses
Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=elizabeth gallas
Enter GRID pass phrase:
Creating proxy ................................ Done
Your proxy is valid until Sun Mar 18 20:32:16 2007
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399
ERROR : No replica location for the dataset. You may be able to copy the dataset
using 'dq2_get -l', Or dataset registration may be incomplete.
Submit a report to https://savannah.cern.ch/projects/dq2-ddm-ops/
---- See if I can get another dataset 'streamtest.000002'
mkdir streamtest00002
cd streamtest00002
dq2_get -rv user.AyanaTamuHolloway.streamtest.000002.mixing.digit.log.v12000399
... (timing below)
Total:300 - Failed:0
---- Started 14-Mar 16:40 (first file date/time)
---- Ended 14-Mar 17:27 (last file date/time)
---- There are no abnormally small files.
---- Conclusion: dq2_get faster and more complete on lxplus ? ............
---- See if I can get another dataset 'streamtest.000004'
mkdir streamtest00004
cd streamtest00004
dq2_get -rv user.AyanaTamuHolloway.streamtest.000004.mixing.digit.log.v12000399
Done
Total:299 - Failed:0
[gallas@lxplus202]~/scratch0/streamtest00004% pwd
/afs/cern.ch/user/g/gallas/scratch0/streamtest00004
---- See if I can get another dataset 'streamtest.000005'
mkdir streamtest00005
cd streamtest00005
dq2_get -rv user.AyanaTamuHolloway.streamtest.000005.mixing.digit.log.v12000399
Done
Total:300 - Failed:0
---- See if I can get another dataset 'streamtest.000006'
mkdir streamtest00006
cd streamtest00006
dq2_get -rv user.AyanaTamuHolloway.streamtest.000006.mixing.digit.log.v12000399
---- See if I can get another dataset 'streamtest.000007'
mkdir streamtest00007
cd streamtest00007
dq2_get -rv user.AyanaTamuHolloway.streamtest.000007.mixing.digit.log.v12000399
---- try to retrieve the missing files (20-Mar)
---- setup on lxplus:
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init --valid 99:59
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
---- retrieve on lxplus:
cd ~scratch0/streamtest00001/
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399 logfile_00361.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399 logfile_00446.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399 logfile_00481.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399 logfile_00566.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000001.mixing.digit.log.v12000399 logfile_00588.tgz
---- setup on oxford:
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.CERN
voms-proxy-init --valid 99:59
cd /var/tmp/gallas/streamtest00000
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00132.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00225.tgz
dq2_get -rv user.AyanaTamuHolloway.streamtest.000000.mixing.digit.log.v12000399 logfile_00257.tgz
cd /var/tmp/gallas/streamtest00009
dq2_get -rv user.AyanaTamuHolloway.streamtest.000009.mixing.digit.log.v12000399 logfile_02997.tgz
| dataset
| on disk
| first/last file data/time
dir -rlt
| first/last file numbers
dir
| smallest/largest file size
dir -sS
| #File / #Failed
| Errors
|
| 000000
| /var/tmp/gallas/streamtest00000
| 14-Mar 14:00-15:45
16:39 (last failure)
| 00000 - 00299
| 64939 - 107371
| Total:300 - Failed:3
| logfile_00(132,225,257).tgz
DQ2 get failure
|
| 00001
|
ERROR on dataset streamtest00001: logfile_00300 - 00599.tgz.
No replica location for the dataset. You may be able to copy the dataset
using 'dq2_get -l', Or dataset registration may be incomplete.
Submit a report to
https://savannah.cern.ch/projects/dq2-ddm-ops/
Submitted bug report to Savannah
Resolved. See next entry.
|
| 00001
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00001
| Mar 16 15:12-20:14
| 00300 - 00599
| 69k - 105k
| Total 300 - Empty 5
| logfile_00(361,446,481,566,588).tgz
resolved Mar 20 11:41 - 11:46
|
| 00002
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00002
| Mar 14 16:40 - 17:27
| 00600 - 00899
| 67061 - 107153
| Total:300 - Failed:0
| Complete
|
| 00003
| /var/tmp/gallas/streamtest00003/
| Mar 14 17:48- 19:32
| 00900 - 01199
| 60124 - 106744
| Total:300 - Failed:0
| Complete
|
| 00004
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00004
| Mar 14 19:11 - 20:19
| 01200 - 01499
| 69568 - 108037
| Total:299 - Failed:0
| logfile_01302.tgz
missing from DQ2
|
| 00005
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00005
| Mar 15 17:28 - 18:17
| 01500 - 01799
| 64702 - 107466
| Total:300 - Failed:0
| Complete
|
| 00006
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00006
| Mar 15 17:52 - 18:46
| 01800 - 02099
| 66677 - 106656
| Total:299 - Failed:0
| logfile_1823.tgz
missing from DQ2
|
| 00007
| /afs/cern.ch/user/g/gallas/scratch0/streamtest00007
| Mar 15 18:28 - 19:25
| 02100 - 02399
| 77970 - 92587
| Total:299 - Failed:0
| logfile_02312.tgz
missing from DQ2
|
| 00008
| /var/tmp/gallas/streamtest00008
| Mar 15 19:40 - 20:52
| 02400 - 02699
| 65448 - 107174
| Total:300 - Failed:0
| Complete
|
| 00009
| /var/tmp/gallas/streamtest00009
| Mar 15 16:54 - 18:42
| 02700 - 02999
| 64732 - 106960
| Total:299 - Failed:1
| logfile_02997.tgz
resolved Mar 20 11:41 - 11:46
logfile_02721.tgz
missing from DQ2
|