Conference rusure::math

1928.0. "Math problem" by CNTROL::DGAUTHIER () Wed Jan 04 1995 15:08

    Hi:

    I have a math problem that I've been pondering for a bit here at work
    and failed to resolve (yet!).  There may be no solution, but before I
    give up, I'd like to gove this forum a shot at it.

    I have taken 3 measurments of something.  Those measurments are... 5,3,7.
    I then calculate the average (5) and the standard deviation (2).  I store
    that information away but discard the raw data.

    Later on, I make two more measurments and I want to recalculate the
    average and standard deviation for the entire set of 5 measurments.
    These two new measurments are 9 and 8.  The complete set is 5,3,7,9,8
    where the new average should be 6.4 and the new standard deviation should
    be 1.92 but remember, I don't have the original 3 measurments (5,3,7) and
    need to derive the new average and standard deviation only knowing:
     1) the average of the first 3 measurments (5)
     2) their standard deviation (2)
     3) the fact that there were 3 original measurments
     4) and of course the last 2 measurments (9,8)

    I can get the new average by simply multiplying the old average by 3,
    adding that into the sum of the new measurments and dividing by 5....

    (3*5)+9+8    15+9+8
    --------- => ----- => 6.4
        5          5

    But I'm having trouble recalculating the standard deviation with the
    limited information.

    BTW: A good solution to this problem will be used in a program that
    loads a relational database with DIGITAL semicinductor device test
    results.

    Thanks, and be gentle if there is no mathematical solution.

    -dave

        If you know that n measurements resulted in an arithmetic mean
        of M and a standard deviation (using the n-1 formula) of S, then
        you know that the original n items had
        
        	"sum of all X"		= nM
        
        and
        
        	"sum of all X^2"	= (n-1)S^2 + nM^2.
        
        Here, "sum of X" 5+3+7 = 15 = 3*5, and 5^2+3^2+7^2 = 83
        = 2*2^2 + 3*5^2.  Given new observations, you can recompute
        the new "sum of all X" and "sum of all X^2" and compute the
        new M and S from them.
        
        Dan

    Yup, works like a charm.   After including 34 new sets of data to the 
    grand set, I believe that the only differences I see between what the
    VAX calculates (using single precision floats) and what I get with my
    hand held HP can be attributed to rouding errors.
    
    Thanks a bunch!
    
    -dave

        In the presence of floating point, .1 isn't necessarily the
        best way to go.  You may want to store extended precision
        versions of the cumulative sum X and sum X^2.  Or something.
        
        Dan

.3>       best way to go.  You may want to store extended precision
>         versions of the cumulative sum X and sum X^2.  Or something.

If you are expecting more data, storing N, sum X and sum X^2 is standard
practice.  You can always compute mean and standard deviation, and you can
always add more data.