Not statistical at all

In the thread that "Zilla began... I know that there have been several that feel that judges should value and count and total errors before they award a score.

This is a lot to expect from mortals....10 seconds between maneuvers that should be watched carefully, sometimes requiring that the judge move...

I suppose that a reason that I sometimes award a 30 for a set of rounds that might be closer to the ideal than an equally pointed set of squares is that rounds are slightly easier to do....hence more chance for error?

See Ya

This is wrong. Score them exactly what they are worth. Ease of making the maneuver is not a factor in the rulebook. Also, how do you know what the squares are going to look like when you write down the score for the rounds?

Don...
I either mis-pecked or you misunderstood...... I try to write a score as soon as the maneuver is completed...

But, I do believe, in the US point system, that the judges are including, to a certain extent, a bit of "K-factor"...

See Ya

>Don...
>I either mis-pecked or you misunderstood...... I try to
>write a score as soon as the maneuver is completed...
>
>But, I do believe, in the US point system, that the judges
>are including, to a certain extent, a bit of "K-factor"...
>
>See Ya
>
>Peabody

I am with Don on this one, Rich. The "K-factor" is not in the rules. Aren't the judges, of all people, supposed to be following the rules? The contestants may occasionally err, but the judges need to be consistant in what the rules say. If I do a round loop and a square loop with equal precision and am down graded on the round loop because it was "easier to do", there is something wrong.

Matt did what in my eyes was a perfect landing on one of his officials at the Nats. This was during the high winds. Engine quit and he brought it straight around about a quarter of a lap to touch down into the wind, tail high, with a perfect roll out and dropped it at the feet of the judges. There were no corrections in the descent, no bounces on the touchdown or roll out. It brought a "wow!" from the contestant standing next to me. One judge gave him a 38. OK, I will grant that. You see some error. I didn't. One judge gave him a 35, same as he gave him on the square loop. I guess the landing was easier to do.

The "K-factor" is not in
>the rules. Aren't the judges, of all people, supposed to be
>following the rules?

Yes.

> The "K-factor" is not in
>>the rules. Aren't the judges, of all people, supposed to be
>>following the rules?
>
>Yes.

Naah, the flyers must follow the rules and the judges decide using the judge's guide and their own observation how well they did their job.

LAST EDITED ON Oct-18-04 AT 11:42 AM (CDT)
 
My limited experience tells me that certain judges have a predetermined score for certain groups. I understand their reasoning, just don't accept it. Also and many forget this, the judges are at least 70 feet or more farther from the maneuver than the pilot. Or else they get hit by the plane. Many times they do not move with the wind and see the maneuver a little distorted, and if you have ever seen the triangle from the side, you know what I mean.
It all boils down to having lots more, almost mandatory, schooling for the people that judge. With the same rule book and the same description, and the same points reduction for each imperfection. The rule book needs to be rewritten and more specific in several areas. The more specific and clear each maneuver is in the book, the less arbitrary each score. No two people see the exact same thing the exact same way. Ask any police officer about "eye witnesses". Ha. But hey, it's just toy airplanes. Ain't it grand?

opps, misspelled maneuver.

Len....I think that you're making my point....certainly a 35 point square 8 is harder to execute than a 35 point landing? The judges factor in a degree of difficulty.......

I think that they do in FAI, too, which really truncates the scores for the maneuvers in the second half of the pattern.

See Ya

LAST EDITED ON Oct-18-04 AT 04:05 PM (CDT)
 
>Len....I think that you're making my point....certainly a 35
>point square 8 is harder to execute than a 35 point landing?
> The judges factor in a degree of difficulty.......
>
>I think that they do in FAI, too, which really truncates the
>scores for the maneuvers in the second half of the pattern.
>
>See Ya
>
>Peabody

Rich,

I can not speak for other judges, but when I judge, I judge the individual maneuver in absence of all others. I could care less which one comes before or after. The maneuver being flown at that moment is my point of focus and I follow the judges guide with no normalizing for difficulty. A perfectly executed maneuver gets the 40, an attempt gets a 10, not flown gets the zero. I do not count errors, I note them and then I evaluate the maneuver in its entirety and score it accordingly.

>I can not speak for other judges, but when I judge, I judge
>the individual maneuver in absence of all others. I could
>care less which one comes before or after. The maneuver
>being flown at that moment is my point of focus and I follow
>the judges guide with no normalizing for difficulty. A
>perfectly executed maneuver gets the 40, an attempt gets a
>10, not flown gets the zero. I do not count errors, I note
>them and then I evaluate the maneuver in its entirety and
>score it accordingly.

That would be correct, I think.

Tom:
you are absolutely correct....each maneuver must be judged as if it was the only maneuver flown in that pattern. Hence it should be possible to see a wide range of scores.....

See Ya

>Len....I think that you're making my point....certainly a 35
>point square 8 is harder to execute than a 35 point landing?
> The judges factor in a degree of difficulty.......
>
>I think that they do in FAI, too, which really truncates the
>scores for the maneuvers in the second half of the pattern.
>
>See Ya
>
>Peabody

Perfect should be perfect, right? Or am I missing something here, too? The problem I see is when a judge gets caught on a number he thinks is "right" for this flyer and then he doesn't deviate more than a point above or below. So you can do a near perfect maneuver and really blow a maneuver and there is only a two point difference.

But, back to the original contention here, if the round loops were perfect with a minor deviation, give them a 38. If you mess up the square eight, well, give the guy what he deserves.

Now, I can understand that there are a lot more chances to make errors on consecutive square eights than on a single hourglass, so maybe each error is judged with less severity on the square eights than on the hourglass. But if a guy does it well, he does it well. And if he messes it up, the score should certainly reflect it, no matter who the flyer.

>Now, I can understand that there are a lot more chances to
>make errors on consecutive square eights than on a single
>hourglass, so maybe each error is judged with less severity
>on the square eights than on the hourglass. But if a guy
>does it well, he does it well. And if he messes it up, the
>score should certainly reflect it, no matter who the flyer.

This why, when using a deduction system, the longer manuevers such as the square eight are TYPICALLY lower (not always). More potential for mistakes usually equals more mistakes.

The point is that a mistake is a mistake, they all count the same. It is consistent.

LAST EDITED ON Oct-18-04 AT 09:49 AM (CDT)
 
Maybe this is why I TOTALLY REFUSE to judge??

I was told by a good friend, and I guess CD, that "Isn't it only fair to 'give something back' by judging?"
I do give something back..... $15 to $20-$25 bucks in entry fees.
I build because I like to build. I fly because I like to fly. I don't judge because I can't stand to judge.

It has always been too ambiguous, IMHO. From all the threads lately I seem to be correct. What one "sees" isn't always what someone else "sees" and all judges have their own way of doing it. One of my friends just pops out the score, "that's a 34", etc.. How does he get the score? He just does. That is OK by me.

I can, now, always accept any score from any judge. Winning in this hobby isn't my main goal, I have a job that depends on winning. I do want to do my best, though. So you guys that are after the Walker Trophy, I admire you, and can understand where you are coming from. We have good and not so good Officials in High School Football, and we are in the best league in the state.

I really appreciate the fact that they will do the job. Just don't ask me to do it.

This probably doesn't help my "friendships", I know, but should it?

Bill <><

Bill, I respect your approach. There are some very good flyers who are also very good judges. But not all good flyers make good judges. You may know what is correct or not correct, but you don't always know how much not correct an imperfect maneuver should be scored. (Still you know what a perfect one is.)

In the same vein, there are some very good judges who will never be good flyers. They can assess and judge the imperfections appropriately, but just cannot fly the pattern to the degree needed to compete at that level.

Different strokes. As you yourself feel, I don't believe everyone ought to be expected to fulfill all aspects of the hobby. We give back to it what we are capable of giving. If your skill is to be a judge, fine. If your skill is to be a competitor, fine. And if your skill is to be neither, maybe you can cook the hot dogs for the noon luncheon. That is a skill that is needed just as much as everything else (and some would say tops the list.)

Thank You , Len.

>Bill, I respect your approach. There are some very good
>flyers who are also very good judges. But not all good
>flyers make good judges. You may know what is correct or
>not correct, but you don't always know how much not correct
>an imperfect maneuver should be scored. (Still you know
>what a perfect one is.)

Exactly. I still have absolutely no idea as to how much to downgrade a score due to an error.

>In the same vein, there are some very good judges who will
>never be good flyers. They can assess and judge the
>imperfections appropriately, but just cannot fly the pattern
>to the degree needed to compete at that level.

I appreciate those people, especially.

>Different strokes. As you yourself feel, I don't believe
>everyone ought to be expected to fulfill all aspects of the
>hobby. We give back to it what we are capable of giving.
>If your skill is to be a judge, fine. If your skill is to
>be a competitor, fine. And if your skill is to be neither,
>maybe you can cook the hot dogs for the noon luncheon. That
>is a skill that is needed just as much as everything else
>(and some would say tops the list.)

I am more than willing to run scores, I toted the water cooler to the spigot on the other side of the meal tent at the WC because there was ice but no water in the cooler, along with Tom Farmer.
I am always willing to do whatever is necessary, but there are those that feel the need to apply "pressure" to get someone to judge. I have almost packed up my stuff and gone home because people don't know how to take a polite "no". As those that know me realize, I am very easy to get along with, but like anyone else, I have my own likes and dislikes. I do not like to get confrontational over the fact.

>Exactly. I still have absolutely no idea as to how much to
>downgrade a score due to an error.

You could be taught.

>
>>Exactly. I still have absolutely no idea as to how much to
>>downgrade a score due to an error.
>
>You could be taught.

But taught by whose standards?

One of the reason I don’t think you see universal consistency in judging is that there isn’t an absolute, objective criteria spelled out in the rulebook. Meaning that, for example, judges aren’t told for the square eight, “Start with 40 points. Deduct 1 point for every bobbled corner. Deduct 3 points for every blown intersection.” And so on.

Using one of Doug’s examples from another thread, do you start with 10 points and add on positives? Or start with 40 and subtract the errors? Or is there another method to calculating a score?? All of these are potentially a ‘correct’ answer, depending on who the judge might be.

Take the 5’ corner (overblown as the issue may be). It’s pretty much established that nobody can do one, unless your name is Rich Porter. So, as a judge, should I kill all flyers (everyone) that can’t do 5 foot radius corners? Should I reward flyers that have tighter corners than those who fly big smooth ones, because they’re ‘closer’ to the rulebook? Or, recognizing that the 5’ corner is essentially a myth, do I minimize errors of radius size and instead focus on the entry/exit into corners?

I guess what I’m trying to say is that as currently written, the rules do not have a specific “point system” for determining the exact number of points to be assigned to each maneuver by the judges. Instead, we have a range (10-40). We have a description of what the maneuver should look like in a perfect world. And we have a list of possible errors for each maneuver (however, by the rulebook, how exactly how many points a flyer should be downgraded for each error in the maneuver is never specified).

The manner by which a judge should assign a score to each maneuver is not something that’s clearly defined, and hence at least partially open to interpretation by each individual judge. For a given flight, two judges can both give my overhead eights a score of 28. They may, however, have both arrived at the number 28 by two entirely different methods. Just looking at the score sheets, though, you'd never know that.

So, to ‘train’ a judge, you need a universal standard if you want all judges – everywhere, from Texas to New Jersey to California – judging by the EXACT same standards. Right now, as far as I can tell, that doesn’t exist. Not for everybody, anyways.

Look, I’m all for analysis of scoring (though more out of curiosity than anything else; the more I see of the Nationals, the more I’m impressed with how GOOD the Nats judges are overall). Having just finished a first round of Six Sigma training, I’d love to see it applied to scoring analysis. But to do a proper analysis (IMHO, anyway), you have to have a precise, universal standard of judging to compare to, where Bobbled Corner = -2 points, Sucky Intersection = -3 points, and so on. Without it, I think it’s hard to develop a statistical model that’s meaningful.

>So, to ‘train’ a judge, you need a universal standard if you
>want all judges – everywhere, from Texas to New Jersey to
>California – judging by the EXACT same standards. Right
>now, as far as I can tell, that doesn’t exist. Not for
>everybody, anyways.

...and you know what? That's fine.

If we were to propose a 5 year plan, the current state of affairs would be considered a BASELINE to measure improvement.

IF it were possible to use statistical tools to measure performance, we could. We could then focus training and measure again. This is process improvement defined.

I just think that the process could be defined.

LAST EDITED ON Oct-18-04 AT 11:06 AM (CDT)
 
>
>It has always been too ambiguous, IMHO. From all the
>threads lately I seem to be correct. What one "sees" isn't
>always what someone else "sees" and all judges have their
>own way of doing it. One of my friends just pops out the
>score, "that's a 34", etc.. How does he get the score? He
>just does. That is OK by me.

That does not mean that, if properly addressed, we could not improve in the future.

That is the definition of improvement. We try to find out where we are now, define where we think we ought to be, and then put steps in place to try to achieve that goal.

LAST EDITED ON Oct-18-04 AT 11:22 AM (CDT)
 
>>
>>It has always been too ambiguous, IMHO. From all the
>>threads lately I seem to be correct. What one "sees" isn't
>>always what someone else "sees" and all judges have their
>>own way of doing it. One of my friends just pops out the
>>score, "that's a 34", etc.. How does he get the score? He
>>just does. That is OK by me.
>
>That does not mean that, if properly addressed, we could not
>improve in the future.
>
>That is the definition of improvement. We try to find out
>where we are now, define where we think we ought to be, and
>then put steps in place to try to achieve that goal.

I agree, improvement can and will occur with many dedicated judges.
But also realize that human nature will always, always, always, always (did I say ALWAYS?), be present when we use humans to judge. And I see no way in the immediate future to have it changed away from human judges....

Like I said, I admire the guys that are after the Walker Trophy. They show the same level of dedication that anyone after a National Championship in any area exhibits. But the human factor cannot be removed from judging as long as we have human judges...

This is toy airplanes we are talking about.

Bill <><

>>>
>>>It has always been too ambiguous, IMHO. From all the
>>>threads lately I seem to be correct. What one "sees" isn't
>>>always what someone else "sees" and all judges have their
>>>own way of doing it. One of my friends just pops out the
>>>score, "that's a 34", etc.. How does he get the score? He
>>>just does. That is OK by me.
>>
>>That does not mean that, if properly addressed, we could not
>>improve in the future.
>>
>>That is the definition of improvement. We try to find out
>>where we are now, define where we think we ought to be, and
>>then put steps in place to try to achieve that goal.
>
>I agree, improvement can and will occur with many dedicatede
>judges.
>But also realize that human nature will always, always,
>always, always (did I say ALWAYS?), be present when we use
>humans to judge. And I see no way in the immediate future
>to have it changed away from human judges....

I have no desie to get rid of human judges. That is an extreme posture. Performance of humans is certainly measurable (even their human nature), the same as the pilots performance can be measured (it is called a score). I think it only fair to everyone that all of the judges are on the same page, don't you?

Hmmm, not even trained Chimps? They are not stupid, just talk different. Hey, one bannana and they will judge.
I got cured of judging at a meet in St Louis in 1982. Never again. Simply not worth the pain and aggravation.

> I thin it only fair to everyone that all of the judges are on the
>same page, don't you?

Of course I agree on that. But I also feel that this level of agreement will only be to a certain degree and probably only at the highest levels of competition. And there is nothing wrong with that.
I am not in an arguement mode! As anyone who has dealt with human beings for 30 or so years would agree, human nature ..........
How could we ever stop the judge that just doesn't like the look of the plane, or how it presents or the pilot, etc., etc..
This occurs whether on the concious (sic) level or not. It is a purely objective area, based on subjective norms. The two are always going to be "in conflict". The best we can do is prepare ourselves the very best we can, and what happens, happens.
As a football player, and then coach, I prepare the very best I can through the week. The performance on Friday night, Saturday or Sunday depended on that week's preparation plus all the preparation from before. Precision Aerobatics are no different. Pentalties (in our case it is errors that are judged as scores) didn't always seem right or appropriate, but they stood.
I applaud your efforts to get everyone on the same page. I also realize that the event has grown and expanded in the recent past in spite of this judging problem. Let's not throw out the baby with the bath water, it's hard enough to get judges in many areas now.
Bill <><

>I applaud your efforts to get everyone on the same page. I
>also realize that the event has grown and expanded in the
>recent past in spite of this judging problem. Let's not
>throw out the baby with the bath water, it's hard enough to
>get judges in many areas now.
>Bill <><

What am I saying to throw out? I have not advocating throwing out anything.

If anything, I think we should start to gather data. Also, I would publish the method to be scrutinized by others. This was the idea behind the puzzle in the first place, trying to find other methods.

I think there are a lot of judges out there, more regimented training would allow even more opportunity.

One really smart flier advocated a video course with simulations for judges training. This would be a good way to get people started.

>
>>I applaud your efforts to get everyone on the same page. I
>>also realize that the event has grown and expanded in the
>>recent past in spite of this judging problem. Let's not
>>throw out the baby with the bath water, it's hard enough to
>>get judges in many areas now.
>>Bill <><
>
>What am I saying to throw out? I have not advocating
>throwing out anything.

Sorry, poor saying on my part...... I would not want to discourage
anyone that desires to judge.

>
>If anything, I think we should start to gather data. Also,
>I would publish the method to be scrutinized by others.
>This was the idea behind the puzzle in the first place,
>trying to find other methods.

Is the statistical analysis for the purpose of exposing poor judges, or to help them?

>
>I think there are a lot of judges out there, more regimented
>training would allow even more opportunity.

That's what I meant about "advanced levels".. How many local guys will want to attend clinics, etc., just to judge at the local contest?
>
>One really smart flier advocated a video course with
>simulations for judges training. This would be a good way
>to get people started.

I think the Video is an OUTSTANDING idea. But who is to make it?

Good ideas, all.
Bill <><

>>If anything, I think we should start to gather data. Also,
>>I would publish the method to be scrutinized by others.
>>This was the idea behind the puzzle in the first place,
>>trying to find other methods.
>
>Is the statistical analysis for the purpose of exposing poor
>judges, or to help them?

Nobody wants anyone to be thrown out! Shoot, I do not even see a method for analyzing the data yet beside my own--which has no validation. I only wish to see a day when the judges agree on scoring. At least relatively closely.

>>I think there are a lot of judges out there, more regimented
>>training would allow even more opportunity.
>
>That's what I meant about "advanced levels".. How many local
>guys will want to attend clinics, etc., just to judge at the
>local contest?
>>
>>One really smart flier advocated a video course with
>>simulations for judges training. This would be a good way
>>to get people started.
>
>I think the Video is an OUTSTANDING idea. But who is to
>make it?

Still in the works. This is one of several ideas that have heard being considered among a few of the guys I talk to. Another would be simulator training, which is realtively simple proposistion.

I think the main thing is that judging methods need to be published. Information needs to be shared, then I think judging would not be so mysterious.

>>I think the Video is an OUTSTANDING idea. But who is to
>>make it?
>
>Still in the works. This is one of several ideas that have
>heard being considered among a few of the guys I talk to.
>Another would be simulator training, which is realtively
>simple proposistion.
>
>I think the main thing is that judging methods need to be
>published. Information needs to be shared, then I think
>judging would not be so mysterious.

If you really want it to be universal. Which I know you do, and it would be good for the sport.

Maybe you should do the following.

Write a criteria like you did before. Officially judge a bunch of contests using it work out all the kinks and propose it. But you could skip the judging part and go ahead a propose it Jan 1, 05. That starts the new proposal period. The time to write and refine is now. I hope to see you propose something. Getting it in the Rulebook will force the hand and make conformity a normal thing in years to come.

Brad...you're dog don't hunt....there isn't time to analyze a maneuver, total mistakes and write a score.... and the other huge factor is that only one...well, two, if one is height challanged, judges can be "downwind" of a maneuver....parallax skews maneuvers horribly....even "biasing" gives an improper impression....the judges must take a lot into consideration right now...

See Ya

>Brad...you're dog don't hunt....there isn't time to analyze
>a maneuver, total mistakes and write a score.... and the
>other huge factor is that only one...well, two, if one is
>height challanged, judges can be "downwind" of a
>maneuver....parallax skews maneuvers horribly....even
>"biasing" gives an improper impression....the judges must
>take a lot into consideration right now...
>
>
>See Ya
>

I do it.

Never said it was perfect, just a method.

>I do it.
>

You do??

LAST EDITED ON Oct-18-04 AT 11:18 AM (CDT)
 
>It has always been too ambiguous, IMHO. From all the
>threads lately I seem to be correct. What one "sees" isn't
>always what someone else "sees" and all judges have their
>own way of doing it. One of my friends just pops out the
>score, "that's a 34", etc.. How does he get the score? He
>just does. That is OK by me.
>

>Winning in this hobby isn't my main goal, I have a
>job that depends on winning. I do want to do my best,
>though. So you guys that are after the Walker Trophy, I
>admire you, and can understand where you are coming from.

Amen and AMEN!

Has anyone else around here noticed that you guys are dealing with a person who (1) judges on a regional basis (2) as well as at the Nats, who (3) fails to understand how to do the job?

Come-on Mr P., read your AMA rule book!

BIGS

" What constitutes the dignity of a craft is that it creates a fellowship, that it binds men together and fashions for them a common language. For there is but one problem - the problem of human relations."

I notice that Intermediate and Advanced flyers often fly better square eights than horizontal round eights. I think this is true of my own scores as well. Perhaps the squares are actually eaier to fly than the rounds. In the rounds each loop is a long entire action and hard to keep on exact track, for me, at least. The squares, on the other hand, are a series of small maneuvers, which can be concentrated on one at a time. So perhaps a higher score on the square eights is not a bias because of thinking them harder, but rather that they are in fact often flown better than the rounds.