[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference smurf::buildhelp

Title:USG buildhelp questions/answers
Moderator:SMURF::FILTER
Created:Mon Apr 26 1993
Last Modified:Mon Jan 20 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:2763
Total number of notes:5802

2596.0. "Re: I: QAQC RCS pool full - tarball submits to blame" by AOSG::FILTER (Automatic Posting Software - mail to flume::puck) Tue Oct 15 1996 18:51

Date Of Receipt: 	 1-OCT-1996 12:54:55.41
From: 	HUNCH::bella "Izabella Kantsepolsky USG  01-Oct-1996 1252"
To: 	aaron@DEC:.zko.hunch
CC: 	glumac@DEC:.zko.hunch, kdolan@DEC:.zko.hunch, qmg@DEC:.zko.hunch,
	bella@DEC:.zko.hunch, odehelp@DEC:.zko.hunch, tresvik@DEC:.zko.hunch,
	tomg@DEC:.zko.hunch, [email protected]
Subj: 	Re:  I: QAQC RCS pool full - tarball submits to blame

Large files may be unavoidable when Standards test suites start coming 
to ODE. We cannot (legally) unbandle the test suite tar/cpio files which 
tend to be really huge. Every change comes in a form of a new test suite 
version i.e., the entire tar/cpio file is going to be "duplicated" to
reflect a few updates in the test suite. We already discussed this 
fenomenon with Standards folks in UNX and they absolutely refuse to unpack
these files stating primarily legal reasons. Perhaps, we'll have to make
an exception for Standards files size in ODE.

Regards,
               - Bella -
 
------- Forwarded Message

Return-Path: aaron
Received: from alpha.zk3.dec.com by hunch.zk3.dec.com; 
(5.65v3.2/1.1.8.2/11Mar96-0342PM)
	id AA26779; Tue, 1 Oct 1996 11:38:00 -0400
Received: from pram.zk3.dec.com by falpha.zk3.dec.com; 
(5.65v3.2/1.1.8.2/20May95-1022AM)
	id AA02652; Tue, 1 Oct 1996 11:37:38 -0400
Received: from localhost by pram.zk3.dec.com (5.65v3.2/1.1.10.5/11Apr96-0916AM)
	id AA21966; Tue, 1 Oct 1996 11:37:50 -0400
Message-Id: <[email protected]>
To: Susan Schueller - Digital UNIX Systems Group <[email protected]>
Cc: aaron, nitika, mentch, hooton, glumac, mjvg, kdolan, qmg
Subject: I: QAQC RCS pool full - tarball submits to blame
In-Reply-To: Your message of "Tue, 01 Oct 96 10:28:38 EDT."
             <[email protected]> 
Date: Tue, 01 Oct 96 11:37:49 -0400
From: "One page shy of a working set" <aaron>
X-Mts: smtp

Hi, Sue,

I do not have access to buffer to play in the rcs pool.

% rlogin buffer -l aaron
Password:***MyPasswordHere***
Login incorrect
% 

I see by later messages the Grant was able to deal with the
problem.  I will forward this thread to QMG to try to raise folks'
awareness of the problems regarding targlop (== hugefile) submits.

Thanks,
=Aaron



- -------- Forwarded Messages
>Message-Id: <[email protected]>
>To: [email protected]
>Cc: snow, tea, odehelp, [email protected], [email protected], [email protected],
>        kdolan, tresvik
>Subject: re: Problem bci'ing a large file  (make that HUGE)
>Date: Thu, 26 Sep 96 17:34:06 -0400
>From: "Joshua M. Friedman, Digital UNIX, 381-1548" <jmf>

Rich, please pass this message along to the developer who's trying to
do this check-in.  I've cc'd the QA teams responsible for the pools.

My feeling is that a single file of 145MB is unacceptable use of ODE.

The QA pools should not be accumlating large tar files, with lots of
regression results (for example); these regression results should each
be bci'd and submitted individually, both for best usefulness, as well
as most economic use of resources.

To submit a large tar file into a configuration management system
really makes no sense.  Say there are 1000 test items in that tar file,
and at the next milestone or release stream there are changes to say 2
of them.  If each test is submitted separately, then only 2 need to be
changed, and if they are each 100KB, then to save the changes might
take an additional 200KB, say.  The pool "snapshot" would then reflect
the 2 items that changed.  To put these two changes into the .tar.Z
file would mean that the new .tar.Z binary would be completely
different, and would require another 145MB or whatever to store, and
now the RCS image in the pool is 290MB.  In addition, there's no
configuration management record of what really changed.

We have had some images as large as 5MB in the pool (gemc), and after
10 to 20 check-ins when the underlying RCS file is nearing 100MB, people
have found the performance hit to be unacceptable (like over 20 minutes
for any ODE operation, even completely locally in zk3).

I suggest that the QA teams not allow this sort of "dumping" into the
pools; more thought needs to be given to what is being submitted and
how it should best be structured for how it will be used.

- -josh


 ----------------------------------------------------------------------
 Joshua M. Friedman	 		Internet: [email protected]	
 Mailstop: ZKO3-3/W20			DECnet:   flume::friedman
 Digital UNIX Engineering		603-881-1548, dtn 381-1548
 110 Spitbrook Road 			Fax 603-881-2257, dtn 381-2257
 Nashua, NH  03062 			Office: zko3-3/z20
 ----------------------------------------------------------------------

- ------- Forwarded Message
> To: [email protected]
> Subject: Problem bci'ing a large file
> Date: Thu, 26 Sep 96 15:57:24 -0400
> From: Rich Larsen <[email protected]>
> 
> Someone here is trying to bci a large file ( around 145MB ) into the
> ptbqa pool. Here is the error:
> 
> $ bci -m"Initial submit" Shells_Utili_Regression.tar.Z
> 
> [ ./test/shells_utilities/su_regression/Shells_Utili_Regression.tar.Z ]
> [ using default log message ]
>         Initial submit
> bci: The following rcs command failed:
> ode_client -t2 6 ci -q -f -sExp -u1.1.1 -m      Initial submit
>  Shells_Utili_Regression.tar.Z 
> ./test/shells_utilities/su_regression/Shells_Utili_Regression.tar.Z,v
> 
> Has anyone else had a problem submitting a very large file ( > 100MB )?
> 
> Rich
> ================================================================
> Rich Larsen, M/S: UNX			TCP/IP: [email protected]
> USSG/User Env. & Std. Group		DECnet: UNXA::LARSEN
> Digital Equipment Corporation		FAX:	908-577-6003
> 200 Route 9 North			Voice:	908-577-6083	
> Manalapan, New Jersey 07726		DTN:	462 
> 
- -------- Forwarded Message 2
> Message-Id: <[email protected]>
> To: [email protected]
> Cc: [email protected], [email protected],
>         [email protected]
> From: Susan Schueller - Digital UNIX Systems Group <[email protected]>
> Subject: 
> Date: Tue, 01 Oct 96 10:28:38 -0400
> 
> Hi Aaron,
>   The "/usr/sde/osf1/rcs/qaqc" directory on buffer is 100% full. I can
> ask Release Engineering Ops to provide more disk space if necessary, but
> you should probably check and see if there's anything that can be deleted
> (log files, etc.)  
> 
> Please let me know if you have any questions or need any help.
> 
> Sue
> 
> ------- Forwarded Message
> Date: Tue, 1 Oct 1996 08:02:03 -0400
> From: system account <[email protected]>
> Message-Id: <[email protected]>
> To: [email protected], [email protected]
> Subject: ALERT: 1 Full Reng Filesystem(s)
> 
> Archived records are in buffer:/var/adm/resource.dated
> 
> 1 Full filesystem(s) (100% or more) detected during resource check:
> 
> buffer:/dev/rz29c      1991325     1792191           1   100%    
> /usr/sde/osf1/rcs/qaqc
> 
> ------- End of Forwarded Message
> -- 
> ------------------------------------------------------------
> Susan Schueller              Digital Equipment Corporation
> USG Release Engineering      MS ZKO3-3/W20
> (603) 881-6348               110 Spit Brook Road
> [email protected]         Nashua, NH 03062-2698
> ------------------------------------------------------------
- -------- Forwarded Message 3
Message-Id: <[email protected]>
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: 
In-Reply-To: Your message of "Tue, 01 Oct 96 10:28:38 EDT."
             <[email protected]> 
Date: Tue, 01 Oct 96 10:44:32 -0400
From: [email protected]
X-Mts: smtp

Thanks to who ever cleaned out the 200+ MB!

Before:
  /dev/rz29c      1991325     1792191           1   100%    
/usr/sde/osf1/rcs/qaqc

After:
  /dev/rz29c           1991325     1515617      276575    85%    
/usr/sde/osf1/rcs/qaqc

Mark
- -------- Forwarded Message 4
Message-Id: <[email protected]>
To: [email protected], [email protected]
Cc: [email protected]
Subject: apparently they've (QA) chosen to ignore Josh's advice ...
Date: Tue, 01 Oct 96 10:49:42 -0400
From: "Grant Van Dyck" <[email protected]>

and stuff those big tarballs into RCS anyway.  I was able to recover 276 MB by 
deleting all the leftover tmpfiles from their failed attempts. There are 3 
individuals involved: Nitika_Kohli, Deb_Mentch and David_Hooton  who are 
trying to submit respectively:

45249623 Sep 27 17:17 devarch_tar_22297.tar.Z
42923452 Sep 26 15:48 Shells_Utili_Regression.tar.Z

40288892 Sep 30 11:22 sve.ptb.bl2.tar.Z

30699500 Sep 27 14:27 bad.5690.trace.out

Top 10 directories in QARCS (size-wise) look like this in KB:

416480  perf
414006  dev_env
221478  Xtea
101803  CHART
86192   cde
53756   lat
47526   standards
31821   filesystems
25598   file_system
15717   shells_utilities

				-Grant
- -------- End of Forwarded Messages

------- End of Forwarded Message
T.RTitleUserPersonal
Name		DateLines
2596.1Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 18:5940
Date Of Receipt: 	 1-OCT-1996 22:43:13.03
From: 	ALPHA::hpsiegel "Howard Siegel USG"
To: 	aaron@DEC:.zko.alpha, bella@DEC:.zko.alpha
CC: 	[email protected], glumac@DEC:.zko.alpha, kdolan@DEC:.zko.alpha,
	odehelp@DEC:.zko.alpha, qmg@DEC:.zko.alpha, tomg@DEC:.zko.alpha,
	tresvik@DEC:.zko.alpha
Subj: 	Re:  I: QAQC RCS pool full - tarball submits to blame
> To: 	aaron
> Cc: 	glumac, kdolan, qmg, bella, odehelp, tresvik, tomg, [email protected]
> Subject: 	Re:  I: QAQC RCS pool full - tarball submits to blame
> Date: 	Tue, 01 Oct 96 12:52:11 -0400
> From: 	bella

> 
> Large files may be unavoidable when Standards test suites start coming 
> to ODE. We cannot (legally) unbandle the test suite tar/cpio files which 
> tend to be really huge. Every change comes in a form of a new test suite 
> version i.e., the entire tar/cpio file is going to be "duplicated" to
> reflect a few updates in the test suite. We already discussed this 
> fenomenon with Standards folks in UNX and they absolutely refuse to unpack
> these files stating primarily legal reasons. Perhaps, we'll have to make
> an exception for Standards files size in ODE.

Another alternative is to not put the Standards tarfiles into the
QAQC RCS tree at all.

At this point, they reside in the test-specific subdirectory of
the ~qatest/suites directory, and different versions are stored
there with appropriate labels.  If we have new standards tests, or
new tests that are like the standards tests in this respect, we
(I suppose I should say "you") can simply create new subdirectories
in ~qatest/suites.

I recognize that wrapping everything up into a single tape or CD
is thereby made somewhat more complicated, but the difference is
negligible compared to the existing complications, and should be
regarded as such.

Howard
2596.2Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:1564
Date Of Receipt: 	 3-OCT-1996 13:20:51.37
From: 	HUNCH::"[email protected]"
To: 	[email protected], [email protected], [email protected]
CC: 	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected]
Subj: 	Re:  I: QAQC RCS pool full - tarball submits to blame

Howard,

I was opposed to the proposal of putting those Standards test suite
under Source Control from the start, and I am still opposed. I could
understand putting the configuration files and installation scripts
under Source Control, but that's about it.

Regards,
Andre'

> From [email protected]  Tue Oct  1 22:47:45 1996
> Delivery-Date: Tue, 01 Oct 96 22:47:45 -0400
> Date: Tue, 1 Oct 1996 22:40:23 -0400
> From: Howard Siegel USG <[email protected]>
> To: [email protected], [email protected]
> Subject: Re:  I: QAQC RCS pool full - tarball submits to blame
> Cc: [email protected], [email protected], [email protected],
        [email protected], [email protected], [email protected],
        [email protected]
> Content-Length: 1434
> X-Status: 
> X-UID: 98
> 
> > To: aaron
> > Cc: glumac, kdolan, qmg, bella, odehelp, tresvik, tomg, [email protected]
> > Subject: Re:  I: QAQC RCS pool full - tarball submits to blame 
> > Date: Tue, 01 Oct 96 12:52:11 -0400
> > From: bella
> > 
> > Large files may be unavoidable when Standards test suites start coming 
> > to ODE. We cannot (legally) unbandle the test suite tar/cpio files which 
> > tend to be really huge. Every change comes in a form of a new test suite 
> > version i.e., the entire tar/cpio file is going to be "duplicated" to
> > reflect a few updates in the test suite. We already discussed this 
> > fenomenon with Standards folks in UNX and they absolutely refuse to unpack
> > these files stating primarily legal reasons. Perhaps, we'll have to make
> > an exception for Standards files size in ODE.
> 
> Another alternative is to not put the Standards tarfiles into the
> QAQC RCS tree at all.
> 
> At this point, they reside in the test-specific subdirectory of
> the ~qatest/suites directory, and different versions are stored
> there with appropriate labels.  If we have new standards tests, or
> new tests that are like the standards tests in this respect, we
> (I suppose I should say "you") can simply create new subdirectories
> in ~qatest/suites.
> 
> I recognize that wrapping everything up into a single tape or CD
> is thereby made somewhat more complicated, but the difference is
> negligible compared to the existing complications, and should be
> regarded as such.
> 
> Howard
>
2596.3Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:1688
Date Of Receipt: 	 3-OCT-1996 13:39:45.71
From: 	HUNCH::"[email protected]" "03-Oct-1996 1337"
To: 	Andre Belloti <[email protected]>
CC: 	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected]
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

I think I speak for all of release engineering and we concur with that.

I believe that we need to sit down (reng and qaqc) and setup some ground rules 
for what sort of things go in and what doesn't. If we can communicate how RCS 
works, then the choices will be more obvious.

Then for all the things that don't belong in RCS, you'll need some alternate 
approach to archive them. If disk space is a problem,  it's relatively easy to 
burn a CD of whatever (individual test suites, groups of test suites etc.) up 
to about 650MB. These could be stored and put on-line by anybody who needed it 
whenever you wanted it.


		-Grant



| Howard,
| 
| I was opposed to the proposal of putting those Standards test suite
| under Source Control from the start, and I am still opposed. I could
| understand putting the configuration files and installation scripts
| under Source Control, but that's about it.
| 
| Regards,
| Andre'
| 
| > From [email protected]  Tue Oct  1 22:47:45 1996
| > Delivery-Date: Tue, 01 Oct 96 22:47:45 -0400
| > Date: Tue, 1 Oct 1996 22:40:23 -0400
| > From: Howard Siegel USG <[email protected]>
| > To: [email protected], [email protected]
| > Subject: Re:  I: QAQC RCS pool full - tarball submits to blame
| > Cc: [email protected], [email protected], [email protected],
|         [email protected], [email protected], [email protected],
|         [email protected]
| > Content-Length: 1434
| > X-Status: 
| > X-UID: 98
| > 
| > > To: aaron
| > > Cc: glumac, kdolan, qmg, bella, odehelp, tresvik, tomg, [email protected]
| > > Subject: Re:  I: QAQC RCS pool full - tarball submits to blame 
| > > Date: Tue, 01 Oct 96 12:52:11 -0400
| > > From: bella
| > > 
| > > Large files may be unavoidable when Standards test suites start coming 
| > > to ODE. We cannot (legally) unbandle the test suite tar/cpio files which 
| > > tend to be really huge. Every change comes in a form of a new test suite 
| > > version i.e., the entire tar/cpio file is going to be "duplicated" to
| > > reflect a few updates in the test suite. We already discussed this 
| > > fenomenon with Standards folks in UNX and they absolutely refuse to unpac
k
| > > these files stating primarily legal reasons. Perhaps, we'll have to make
| > > an exception for Standards files size in ODE.
| > 
| > Another alternative is to not put the Standards tarfiles into the
| > QAQC RCS tree at all.
| > 
| > At this point, they reside in the test-specific subdirectory of
| > the ~qatest/suites directory, and different versions are stored
| > there with appropriate labels.  If we have new standards tests, or
| > new tests that are like the standards tests in this respect, we
| > (I suppose I should say "you") can simply create new subdirectories
| > in ~qatest/suites.
| > 
| > I recognize that wrapping everything up into a single tape or CD
| > is thereby made somewhat more complicated, but the difference is
| > negligible compared to the existing complications, and should be
| > regarded as such.
| > 
| > Howard
| > 

-- 

				-Grant
2596.4Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:1819
Date Of Receipt: 	 3-OCT-1996 14:02:50.14
From: 	ALPHA::kdolan "Kathy Dolan  03-Oct-1996 1400"
To: 	"Grant Van Dyck" <[email protected]>
CC: 	Andre Belloti    <[email protected]>, [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected]
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

Alex Glumac and I talked about this issue this morning. Alex will be
putting together a set of guidelines for what should / shouldn't go
into the pool. In the case of standards, it will probably be the unique
config scripts etc. with the standards linked in from somewhere else.
Bottom line, we will be addressing this.

thanks

Kathy
-
2596.5Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:3016
Date Of Receipt: 	 4-OCT-1996 14:07:59.81
From: 	HUNCH::jmf "Joshua M. Friedman Digital UNIX"
To: 	kdolan@DEC:.zko.hunch, [email protected]
CC: 	<[email protected]>, Andre@DEC:.zko.hunch, Belloti@DEC:.zko.hunch,
	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected], [email protected]
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

Kathy, also note that equally important is what pools should be used
to submit what and when.  For example, any submits into ptbqa now will
not be available in steelqa for quite some time, since steel is backing
currently to pta and will at some point change to ptc, as I understand it.

-josh
2596.6Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:3642
Date Of Receipt: 	 4-OCT-1996 17:50:30.45
From: 	ALPHA::aaron "Aaron Sawyer USG  04-Oct-1996 1747"
To: 	Joshua M. Friedman Digital UNIX <jmf@DEC:.zko.alpha>
CC: 	kdolan@DEC:.zko.alpha, aaron@DEC:.zko.alpha, vandyck@DEC:.zko.alpha,
	[email protected], bella@DEC:.zko.alpha, glumac@DEC:.zko.alpha,
	odehelp@DEC:.zko.alpha, qmg@DEC:.zko.alpha, tomg@DEC:.zko.alpha,
	tresvik@DEC:.zko.alpha
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

Hi, Josh,

Currently, the QA pools are backed as follows:

	STEELQA backs to PTAQA backs to PTQA
	  PTBQA backs to PTAQA backs to PTQA

When the Steel OS pool merges in PTB OS SSB, that is the time when
STEELQA should merge in the PTBQA pool entries and reback to PTBQA.
I think this will be some date in November.

PTCQA does not yet exist.  When created, it will back to the QA pool
that corresponds to the OS pool backing PTC OS:

% ls -l /usr/sde/osf1/build/ptcos/link
lrwxr-xr-x   1 root     staff         29 Sep 23 23:24 /usr/sde/osf1/build/ptcos/link -> /usr/sde/osf1/build/ptbos.bl2
% 

PTCQA will be backed to PTBQA.

STEELQA will merge in PTCQA when PTCOS goes to SSB.

Similarly for PTD, PTE, ...

=Aaron

> Kathy, also note that equally important is what pools should be used
> to submit what and when.  For example, any submits into ptbqa now will
> not be available in steelqa for quite some time, since steel is backing
> currently to pta and will at some point change to ptc, as I understand it.
> 
> -josh
2596.7Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:3730
Date Of Receipt: 	 5-OCT-1996 00:09:25.36
From: 	FLUME::"[email protected]"
To: 	[email protected] (One page shy of a working set)
CC: 	[email protected], [email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected]
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

> 
> Hi, Josh,
> 
> Currently, the QA pools are backed as follows:
> 
> 	STEELQA backs to PTAQA backs to PTQA
> 	  PTBQA backs to PTAQA backs to PTQA
> 
> When the Steel OS pool merges in PTB OS SSB, that is the time when
> STEELQA should merge in the PTBQA pool entries and reback to PTBQA.
> I think this will be some date in November.

Steel os has no plans to retarget to ptbos. Instead they'll wait and pick it
up via ptcos which is now open. I plan to open ptcqa next week backed by
ptbqa. ptb will undoubtedly go to SSB before steel backs to ptc.

You will probably want to retarget steelqa to ptcqa at some point as well, and
will need to decide when to close ptbqa.


		-Grant>
2596.8Re: I: QAQC RCS pool full - tarball submits to blameAOSG::FILTERAutomatic Posting Software - mail to flume::puckTue Oct 15 1996 19:4024
Date Of Receipt: 	 7-OCT-1996 10:58:09.04
From: 	FLUME::jmf "Joshua M. Friedman Digital UNIX"
To: 	aaron@DEC:.zko.flume
CC: 	[email protected], bella@DEC:.zko.flume, glumac@DEC:.zko.flume,
	kdolan@DEC:.zko.flume, odehelp@DEC:.zko.flume, qmg@DEC:.zko.flume,
	tomg@DEC:.zko.flume, tresvik@DEC:.zko.flume, vandyck@DEC:.zko.flume
Subj: 	Re: I: QAQC RCS pool full - tarball submits to blame

Aaron, first of all the subject line is a misnomer, and is not true.
(it was nearly true but we expanded the space from 2Gig to 4Gig).

Regarding the eventual merges of ptb/ptc into steelqa, I hope that you
know that this is not an automatic merge - that is, we can do detection
to look for what will come into steel automatically, and every that's
in conflict must be manually merged by developers in your organization;
the orchestration of this merge activity is somethat that you and Kathy
may want to discuss and plan.  When it comes time for the merge, it may
be that people are off working other issues, but if they "own" certain
sources it may be incumbent umpon them to do the merges.  This will
require a certain amount of management directive.

-josh