[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference vmsdev::alpha_verification

Title:Alpha Verification conference
Notice:QTV Bug Chasers! Practicing the fine art of defect detection, reporting and reproduction...
Moderator:STAR::JFRAZIER
Created:Wed Mar 20 1991
Last Modified:Tue Jun 03 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:487
Total number of notes:2459

482.0. "Rate of Fault Insertion Note" by JARETH::BEAUDIN () Wed Feb 19 1997 13:11

 
  One action item from the QTV GRYPHON retrospect was to record suggested 
'error introduction rates' that OpenVMS software can withstand. Error 
introduction can be software induced such as running Faulty Towers or
making a system crash/reboot repeatedly. Error introduction can also be 
physically induced such as failing over an HSCxx device while a cluster runs 
load. 

  On several occasions the testing organization has incurred issues regarding 
'acceptable' rates of error introduction. The purpose of this note is to provide
a history of successful error rate introduction (OpenVMS has withstood these 
errors and responded appropriately).

  Replies to this note will start with:
	- Faulty Towers iterations used in previous Validation efforts
	- Memory Channel error introduction rates used in Validation and
	  qual efforts.
	
  Anyone wishing to add further replies such as:
	- CIPCA interconnect failover rates 
	- SCSI  interconnect failover rates 
	- HSCxx failover rates
	- Network interconnect failover rates
	    - How often can lavc$stop_bus/lavc$start_bus be run?
	- Dual ported RAxx device failover rates
	- Shadowing error introduction rates 
	- Others?  
		- Sysgen settings for error rate iterations?
		- Configurations used for this testing?
		- Any replies welcome
Thanks.
Tim
T.RTitleUserPersonal
Name
DateLines
482.1Faulty Towers iterationsJARETH::BEAUDINWed Feb 19 1997 13:1619
   Faulty Towers Testing

   I've used the following Faulty Towers scenarios along with a CTM load 
    applied to the Memory Channel cluster during V7.1 Validation. These values
    were ran with each test running at least 2 hours under load.

   - 'Connection Crashes' every 59 seconds 
   - 'Virtual Circuit Crashes' every 59 seconds   
   - 'Port Crashes' every 59 seconds  
   - 'MRESET/MSTARTs' every 59 seconds
   - 'Hog SCS Locks' every 59 seconds, holding for 10 seconds  
   - 'Mount Verifications' every 59 seconds
   - 'BDT Exhaustion' every 59 seconds/5 BDTs, 
   - 'CDT Exhaustion' every 59 seconds/2 CDTs 
   - 'Message Credit Exhaustion' every 59 seconds/5 Credits 
   - 'RSPID Exhaustion' every 59 seconds/5 RSPIDs.
   - 'Move Clusterwide Locks' every 59 seconds 
   - 'Full Rebuild' every 59 seconds

482.2V7.1 SSB Memory Channel Fault Insertion NoteJARETH::BEAUDINWed Feb 19 1997 13:2917
 For Memory Channel V7.1 SSB Qual Testing, rates of error introduction became
an issue. Ray Boucher and I discussed a formula of:

	Daytime/2hr tests:	15 min iterations
	Overnite/Weekend tests:	1hr iterations

 The mc_crash_0n test was run from 1-10 min during FT1-FT2 validation. The 
Forced ssrvexceptn test iteration time was 4x the time used during FT2 
validation. Faulty towers was run at 1 min intervals for FT1-FT2 validation.  

Mc_crash_0n:
 - Submit mc_crash_0n (bounce community) once every hour for an overnite run,
along with load tests. 

Forced SSRVEXCEPTN:
 - Select 1 MC member and have it perform an induced SSRVEXCEPTN one time per
hour. Test must run for at least 24hrs, preferably 48hrs, along with load tests.