This post looks weird without a header
Hey everyone, I'm writing this post to go over some issues we had during this competition and talk about some changes that we're making as a result. It's long for a reason, so feel free to skip if you don't care about drama and are waiting for Summer to end so we can resume the competition schedule. :P
TL;DR: we are removing Praveen00 from the winners list and moving everyone below him up a prize tier. We will not be publishing his submission, and we are prohibiting him from future competitions for abusive behavior. We will also be adding an agreement to future competitions so that every participant fully understands how your animation is handled by us as publisher
upon submission, and that disputes about placement do not revoke our license to the submission. We will begin a more thorough rotation of community judges starting next competition, and I will no longer act as judge so I can focus on being a fair mediator between judges and authors during disputes such as the one described in this post.
If you haven't been around on our Discord server, you might be wondering why all of this has happened, so here's a big wall of juicy text for you to skim through. I want to approach this as neutrally as possible, so I'm just going to state the sequence of events as well as some background information and you can judge the situation for yourselves and let me know how you think we could improve as a result of this.
Setting the stage
After we posted this topic to announce the winners, Praveen00 - who had won $400 in 4th place with his submission
Stick Empires: Fury Of Order - took to #spac on our Discord server to confront "Judge 1" about the relatively low score given to his submission; remarking that he was hurt by the apparent difference in scores and demanding that "Judge 1" - who scored his submission 17/30 while other judges scored it 28/30 and 26/30 respectively - explain themselves. Since the scores are anonymized, when complaints like these come up it is my role to act as the bridge for communication between the judge and the author. Understanding that it's fair for someone to want an explanation for their own results, I contacted Judge 1 on behalf of Praveen00 and asked them to explain why the score was given. Before I continue, allow me to explain a little bit about our judgement system so that you can fairly consider our perspective during the next sequence of events.
How we try to keep the competition fun, fair, and fresh
During the competition, we keep the identities of our judges a secret. We also keep the scoring system our judges use a secret. The primary motivation for this is to avoid any influences or other types of "gaming the system" from authors participating in our competition. We want the submissions to be well rounded and we want our authors to consider the big picture when planning and creating their work, thereby maximizing the
number of useful skills being practiced during the competition. Opening our scoring system up to public scrutiny and allowing the judges to be identified has historically resulted in some authors tailoring their submissions to only touch on the most valuable requirements or to appeal to specific judges, expecting that this strategy will ensure a high score while allowing them to ignoring other aspects that they believe are less important.
This may seem reasonable from a participant's perspective - you want to know where you should focus your efforts to maximize your own chances of winning. This is a competition after all, and it's difficult to get motivated if you don't have a clear path to success. But it's an unhealthy strategy for the competition as a whole and a big reason these competitions die off. We need submissions to stay fresh after every competition or the incentive for us to run another one diminishes as the content gets stale. When every competition turns into a race to exploit the competition meta, we lose sight of the original intent of these competitions which is to serve as a vehicle for our authors to improve their skills. It's not fair to new participants who stand even less of a chance because they aren't aware of the competition meta, and it's not fun for anyone aside from the top winners if the same few people win over and over without much variance. This results in a negative spiral that has previously resulted in the competitions losing steam and dying, then no one has any chances to win money while honing their skills.
We feel that keeping the scoring system and judge identities hidden adds a decent incentive for the authors to even out their strategy, as now there's a bigger risk for that higher payoff probability. We list the aspects that our scoring system focuses on, but the exact details as to how those are broken down are kept secret and changed each competition. This way, no single factor is guaranteed you a win and each competition requires a new strategy. It's more effective to put effort in to every category listed otherwise you run the risk of focusing on the wrong thing for the competition and scoring low overall. We feel this is an acceptable solution as it outlines every key factor you should put effort in to without assigning explicit priority to any of them.
Note: this is still an animation competition, so it's safe to assume that obviously animation matters a lot by that reasoning and each competition has a theme so again by that reasoning the effort you put in to categories that reflect the theme matter.
The main reason why judge scores are anonymized
Aside from preventing judge-tailored content, anonymizing the judges also helps prevent harassment. In past SP competitions, complaints about the scores given by a judge had resulted in some nasty abuse directed at the judge. Aside from the fact that we want to protect
all our participants from that kind of harm, it also has the potential to affect that judge's ability to express their true feelings about an animation out of fear of retaliation which is our primary motivation in making it difficult to link score to person. This case
should exemplify how this is useful - we don't know the players involved so their histories have been factored out of the complaint, so the complaint can now be addressed based on content and a discussion can be had.
Ex. Question: "Did my submission deserve this score?" Response: "Yes, and here's my reasoning why..." Result: "Oh I understand, I will have to improve" or "I really disagree with that, here's why..."
The main reason this is a problem
What I think it actually exemplifies is an unexpected consequence of this approach. From the author's perspective, we run in to problems with accountability. How do we know judges are being fair if we don't know who put the score down or why that score is what it is? An discussion about content is meaningless if the judge has it out for you. While waiting for a response from Judge 1, Praveen00 pointed this out. From his perspective, the judge
might matter as the problem he's facing is the big fat unknown of,
"Why did this happen?" Which could literally have any answer, including that the judge just was power tripping for no reason. If you think your work is first place material and two out of three judges seem to agree, then the third judge disagreeing with the "consensus" surely indicates there is something fishy, right?
The only way to satisfy this is to ask Judge 1 for their score reasoning directly and evaluate whether or not it's a fair assessment or if there's some corruption at play. But since we don't know who that is, you have to ask me and trust that I'm behaving honorably in my role.
There is nothing wrong with this, it is perfectly reasonable for anyone who has a problem with their score to be suspicious and want an explanation. It is on me to build that trust so that the assumption of foul play is minimized, and I try to do my best here but I understand that there will always be suspicions about things that are intentionally kept secret and that doesn't bother me. It's like a black box that your submissions go in to and come out ranked on the other side. When there's real rewards on the line, how do you trust the black box? Is the scoring based on opinion only, subject to any judge's whim? Or is there an objective measure for each category? The answers to these are not apparent.
The Judges are People Too™
Or: What is art, really?
Well, the answer is sort of both. I recognize how wishy-washy that sounds, but the situation is complex and not well defined. Is there an objective measure for how good an artistic piece is as a whole? I would argue no, not really. We can of course look at each category from a technical perspective. For instance, how well does this submission demonstrate the principles of animation? Does it show an understanding of different animation techniques? We do consider these sorts of questions when deciding the score, but these are not as objective as they might appear on the surface. What if the submission barely displays any of the principles of animation just because the story does not allow for most of them to show up? Is that still a fair measure of submission quality?
The fact is that each judge has varying degrees of experience in each of the categories by virtue of being individual humans with their own life experiences. By that same virtue, their scores will also reflect that variance. So, the final scores are ultimately a combination of the category weight, the score assigned to that category by the judge, and the judge's experience evaluating each category. It's not as straight forward as checking off a pre-made list of "did or didn'ts" to determine your rank, so we must rely on each judge
and each author to be honest and act in good faith with one another. The more we do this successfully the more we can interact based on trust. In this case, as we find out below, Judge 1 caught the fact that much of the art used in Praveen00's animation was trace bitmapped from other sources. Is it fair for him to receive a high art score if most of the art in the animation wasn't even made by him? By nature the process of establishing trust in another human involves deeper philosophical questions that are outside the scope of this small stick figure animation competition.
It's worked so far
At this point based on these thoughts, I would warn you that if you cannot accept the inherent variance that comes with having actual people as judges despite all of the steps we have taken to minimize it as a factor, then you simply should not join this competition. We want you to approach this with the mindset of doing your best as a creative person, touching on as many of the aspects of artistic expression as you want. You build a lot of skill and a lot of character by doing this, and it will help you well in to the future regardless of the competition outcome. The previous winners from this competition have improved by leaps and bounds in just the single year we've been doing this. Not only did they build their skill but many of the winners put their prize into artistic tools in order to improve on their next submission. I am proud of the value we've been able to provide and I hope we can continue doing into the future.
So what's the big deal?
Unfortunately, before Judge 1 could give me their explanation, Praveen00 decided he would rather deprive the community of his high-scoring entry rather than accept his score and prize. He issued a demand that we remove him from the winner's list and do not publish his submission, ending the conflict before we could resolve it amicably. I am not sure why he felt the need to jump to this point before we could discuss Judge 1's opinion. For all I know the lateness and lack of communication during the end of the competition were enough to evoke this response, and that is reasonable in my opinion. However, the expectation for this competition is that when you submit, the work you submit comes under our control as the publisher regardless of the outcome. We reserve the right to refuse a request
not to publish, that needs to be clear. You will be given the prize you won, if any, and your submission is saved for future processing even if you are so unhappy about your score that you want to try retroactively dropping out. We're able to sustain these competitions by using the submissions to drive activity on the site - something you should already understand and be comfortable with before joining. Hurting our ability to do that by self-sabotaging your work removes potential future traffic and hurts every other member of this community that cares about these competitions and gets something out of it.
I felt like this all needed to be discussed as it happened in a relatively small time frame, so I informed Praveen00 that we would discuss his demand as soon as we could but if he persisted with this demand he would be prohibited from joining further competitions. I still also planned on relaying Judge 1's response. He opted to wait for the response, which arrived a day or two later which stated his reasoning for issuing the scores in each category he did:
"The meat of the submission is really well done, but there were aspects that I felt were in need of improvement and ultimately brought the final score down in the end for me. The opening is too long and filled with uninteresting static shots. Camera motion would help keep the viewer engaged. The synthetic voice acting would have been better left out entirely. The character animation was stiff and often cheaply repeated. The background art was blurry and traced bitmapped from Game of Thrones, Skyrim and maybe other sources. I'm sorry that he is this upset, but his submission wasn't as good as the other ones overall. The battle scene was amazing and makes it worth watching, but that alone cannot make up for the other aspects that were lacking."
Praveen00's response to this was to call these reasons 'silly' and restated his demand once again.
Why this is not OK
We understand when people get upset about results - handling the disappointment of not placing as well as you'd have hoped is tough and takes a lot of maturity to overcome. That doesn't mean you can't or shouldn't ever disagree with the outcome. When these disagreements happen, remember that we are people too that are doing their best to give you a good experience during these competitions no matter how they turn out. Remember what I said about the fact that judges are humans with their own range of experience and they will vote based on their own knowledge set. Let us talk about the problems one might have with a judge's reasoning and at least try to come to a reasonable middle ground before jumping off the rails. Why are these silly reasons? I do not know. I do not feel there was room for discussion.
It is for his specific behavior towards us and the competition on Discord that we have decided that we are going to prohibit Praveen00 from joining in future competitions in addition to respecting his request for removal from this competition. We have taken his demand for removal as the hostile act it appears to us to be: I am not happy with the outcome, so you don't get anything from me. We wish to avoid interacting with someone who's demonstrated their willingness to behave that way. That is not the kind of person we would like poisoning the community surrounding these competitions, and I seriously caution anyone who would treat us this way over a disagreement to simply avoid this competition.
Moving forward
You might be wondering, "Well this is all nice but I also have to trust that everything
you are typing is truthful! You could easily be manipulating us since you're the only one here who knows who the judges are, what their scores are, AND YOU'RE ONE OF THEM." That's smart, I wish I had thought of that. I realize this now. It's easy to see what kind of
