Conference 7.286::maynard

I hope that people who are concerned about test scores (and the like) know how
to properly interpret them and understand the dynamics that underly the
calculation of a particular statistic.

When you only look at a mean, the sensitivity of that statistic is highly
sensitive to the population.   One or two low (or high) scores can skew
a school's mean greatly if there are only 60 people taking the test versus
a school where there are 180 people taking the test.

If you wish to use test scores between towns, etc. as a measure of goodness
about a school system, the mean score alone is not adequate (in my opinion).

The mean is a good starting place, but I'll concur with the statement that it
shouldn't be used to judge a school system.   If anyone cares to dig deeply
enough to find out the relative distribution of test scores between towns,
states, etc. - then they have (in my eyes) significantly more ammunition to
dump praise or condemnation on a school.

- dave

I did not attend a US primary or secondary school, but did take the SAT just
prior to coming to a US college.  The fact that  SAT scores has become one of
the major criteria in comparing school systems has always troubled my mind.
One problem is exactly what 330.1 stated, the mean score is sensitive to the
population size.  What troubled me the most is that the SAT only tests math and
verbal skills, and it does that by measuring how quickly a person can attack a
given problem.  Does SAT scores adequately measure the quality of a school
system?  How about other areas of study that SAT does not cover such as physics,
chemistry, biology, economics, writing, etc.?  If SAT score is a legitimate
metric, then a school system can just offer a good SAT preparation course and
nothing else, and it will look good on paper.  I believe with rigorous drills
on taking SAT tests, most people can ace the score.

It is convenient to compare school systems by means of SAT scores, as the same
test is taken by all schools.  The data is readily available through one
organization.  However, I believe the math and verbal scores alone do not
sufficiently measure the quality of a school system.  How, then, can we measure
the overall quality?  I don't know, and hoping that this note will generate good
responses.

-santa

Re .1 and .2

The mean (average) is definitely the best place to start in evaluating the
score as a reflection on the mass of kids in the school.  Maynard being a
small school has only one "unbiased" effect on the significance of the mean:
it has a larger uncertainty. 

The uncertainty in a mean is proportional to the inverse square root of the
number of students involved.  So if Maynard had .63 * 70 = 44 students and
another school had 280, then the uncertainty in the result due to the
possibility of a few mavericks moving the average by doing exceptionally
well or poorly is 2.5 times as large for Maynard as for the other school. 
But the mean is "unbiased". 

However, there really is a bias in the result mentioned.  Only 63% of the
students in Maynard took it compared to 90+% elsewhere.  Almost certainly
the bottom 37% in Maynard did not take the test since the top 63% needed it
to get into college.  Presumably, if 90% of the students had taken the test,
then the score would have been even lower. 

I did a statistical study of the SAT scores in Maynard over several years.
There is a correlation of 4.4 points of SAT per position of rank in class
(That means that statistically the student with rank 10 will do 4.4 points
better than the student with rank 11.)  However, the statistical variation
among students with that effect removed is about 20 so that the correlation
only really shows up when considering large chunks of ranking, say, more
than 10-15 students at a time. 

The numbers above can, with not great statistical risk, be used to guess 
what Maynard's score would have been if 90% of the students had taken the 
test.  The result is lower than the published numbers.

In one sense, the result WAS a statistical fluctuation.  The class of '91
was not good academically, as the class president stated explicitly at
several public events last year.  No one in the class of '91 had high honors
compared to 3-6 in the class of '92, which is about the same size.  High
honors means 90 average or better.  I have never heard a good reason for
why that class was the way it was.

My older daughter told me that a student in her class (MHS, '92) said once
"I wish I had been in the class of '91; I would have gotten a scholarship." 

As far as what the SAT says, it is not a perfect/complete indicator and
colleges take it as only one indicator.  But it IS an indicator.  It does
indicate a student's ability to work under pressure (National Merit
Scholarships depend a lot on the PSAT), the ability to draw analogies
(verbal) and fluency in math.  It also measures vocabulary strength and
"background".  The SAT is culturally biased; kids whose world of reading and
experience are larger and more challenging (and mainstream white) do
statistically better. 

We have taken our kids to NYC, Washington, San Francisco, Europe, Museum of
Fine Arts, M of Science, New England Aquarium, etc. because we wanted them
to have that larger experience because it will help them become more
successful in the future, regardless of what they want to do. 

My daughter (MHS, '92) did not do very well on either the PSAT or SAT
although her class rank is good.  In talking to teachers and administrators
in three systems, including MHS, about that, the interpretation was
identical; "She isn't READING".   THAT IS an important indicator.  That was
(only) the straw that broke the camel's back in my wife's and my decision to
send our younger daughter to ABRHS (or a private high school), a decision
made BEFORE "choice" was very visible. 

My younger daughter is struggling at AB.  Looking over her shoulder I see
more work and better work being demanded.  In biology, she had a question
"What are the implications of our lab with respect to the effects of
increased acidity of rain?"   She didn't know the meaning of the word
"implication".  That was frightening.  On further discussion, my older
daughter made the statement "Biology is easy; it's mostly memorization!"
That is another frightening statement. 

Back to the class of '91 as a statistical fluctuation:

In the future (and probably in the past as well) it will be harder for
Maynard to create a challenging environment because the academically (and
perhaps athletically) more motivated/successful students may choose to go
where the environment is already more challenging.  Even if the funding
"Darwinism" of the school-choice law is removed, or if all schools are
funded "adequately" (Read that much better than now.), Maynard still has an
uphill battle for survival. 

The first shot in that battle was fired last night by Dr. Ciardi at the
Curriculum Night at MHS.  He said that the schools are not "good enough";
but rather that we need "good schools", not "good enough".  He went on to
elaborate what that meant.  It was a very good talk.  Unfortunately, the
assembled crowd (parents of some 230 students) fit into the MHS library.
Another indicator pointing in the wrong direction.  It may be homier in the
library, but it indicates lack of interest by most of the parents. 

Those who want better schools in Maynard had better get behind Dr. Ciardi
and the School Committee and we/they better be supportive, cooperative,
vocal and demanding.  And there better be a lot more of us/them. 

BTW, one of three or four things Dr. Ciardi asked us for was money.  He made 
it clear that the uphill battle isn't going to be successful without more 
money.  He correctly pointed out that money is not THE solution to the 
problem, but it is an essential ingredient of the solution.

:-)

John

Followup: .3

Yeah, if you're concerned about mean SAT scores, you should also be
concerned with mean funding -- and Maynard is well below the state mean
for funding its schools -- to the tune of $600,000 (if I heard right)...


- dave

re:  Note 330.4 by PRAGMA::GRIFFIN "Dave Griffin" >>>

Precisely!

And the Mass. state mean is below (near the bottom, actually) the national
mean, which is below the world mean (I'm assuming we only want to be
compared to "developed" countries, although some developing countries do 
better than the US does). 

:-(

:-)

John

It would also be helpful to know the median score.   The lowest and 
highest scores do not sway this number. 

Karen 

 No educator am I, but...I was pretty shocked when I saw how low the 
 Maynard SAT scores were...

re: 330.2 (extracts below)

<...What troubled me the most is that the SAT only tests math and 
<verbal skills...
<...How about other areas of study that SAT does not cover such as physics,
<chemistry, biology, economics, writing, etc.?  

reply:
	It's been a number of years since I took the SAT's, but when I did,
        there were optional SAT tests for all these and many other areas.
        I took about 5 of these optional tests.

<If SAT score is a legitimate metric, then a school system can just offer 
<a good SAT preparation course and nothing else, and it will look good on 
<paper.  I believe with rigorous drills on taking SAT tests, most people 
<can ace the score.

reply:
	If it were that easy (rigorous drills), then there's no excuse for a
	school system not to have lots of (motivated) students "ace" the test.
	
	I understand that learning test-taking strategy skills, etc. can 
        improve one's scores, but this is not going to make a "400" class
        average into a "600" class average.

	I have been judged in life by a number of different tests ....
	and certainly not always apparently 'appropriate' ones.

	One of the games is to realize what tests you need to pass
	to get where you want to be, and to learn how to pass those tests
        (some people say 'learn how to jump thru the necessary hoops',
        recognizing the arbitrariness and game-playing of the process).

	If SAT's are one of the 'hoops' students need to jump thru to get 
        into college, then evaluating how well a school prepares them for 
        the SAT's seems to have validity.  

	Obviously SAT's shouldn't be 'the be-all and end-all" of one's 
	high school life, but if a high school can't prepare 'college material'
        students to jump thru all the 'hoops' necessary to get into college, 
        the school, in my opinion, has a definite (and measurable, in terms
        of SAT's) problem.

	From my own experience, the 'difference' in low vs high SAT scores
	for me was THE TEACHER I had for a relevant discipline, and 
	HOW MUCH WORK I put into meeting their challenges to LEARN THE 
        FUNDAMENTALS of those discipines (not 'make work').

  Any 'educators' out there should feel free to attack my view...