StickPage Spring Animation Competition - For Glory: eSports - $5,000 in prizes, 15 placements, $1,500 first prize!
BEGINNER'S StickPage Spring Animation Competition - Unconventional Weapons - $1,000 in prizes!
Page 2 of 2 FirstFirst 12
Results 11 to 16 of 16

Thread: eSports: For Glory WINNERS - Spring Competition 2017

  1. #11
    The "Highest Master" HyMaster's Avatar
    Join Date
    Feb 2012
    Location
    Philippines
    Posts
    690
    Goodjob everyone!
    Hi.

    Visit my pages:
    hymaster.deviantart.com
    youtube.com/hymastertv
    facebook.com/hymaster67

  2. #12
    Junior Member
    Join Date
    Jun 2017
    Location
    Finland
    Posts
    9

    Thumbs up

    Good job everyone.
    (⌐■‿■)

  3. #13
    StickPage Cave Goblin Jeff's Avatar
    Join Date
    Dec 2007
    Posts
    4,362
    This post looks weird without a header

    Hey everyone, I'm writing this post to go over some issues we had during this competition and talk about some changes that we're making as a result. It's long for a reason, so feel free to skip if you don't care about drama and are waiting for Summer to end so we can resume the competition schedule. :P

    TL;DR: we are removing Praveen00 from the winners list and moving everyone below him up a prize tier. We will not be publishing his submission, and we are prohibiting him from future competitions for abusive behavior. We will also be adding an agreement to future competitions so that every participant fully understands how your animation is handled by us as publisher upon submission, and that disputes about placement do not revoke our license to the submission. We will begin a more thorough rotation of community judges starting next competition, and I will no longer act as judge so I can focus on being a fair mediator between judges and authors during disputes such as the one described in this post.

    If you haven't been around on our Discord server, you might be wondering why all of this has happened, so here's a big wall of juicy text for you to skim through. I want to approach this as neutrally as possible, so I'm just going to state the sequence of events as well as some background information and you can judge the situation for yourselves and let me know how you think we could improve as a result of this.

    Setting the stage

    After we posted this topic to announce the winners, Praveen00 - who had won $400 in 4th place with his submission Stick Empires: Fury Of Order - took to #spac on our Discord server to confront "Judge 1" about the relatively low score given to his submission; remarking that he was hurt by the apparent difference in scores and demanding that "Judge 1" - who scored his submission 17/30 while other judges scored it 28/30 and 26/30 respectively - explain themselves. Since the scores are anonymized, when complaints like these come up it is my role to act as the bridge for communication between the judge and the author. Understanding that it's fair for someone to want an explanation for their own results, I contacted Judge 1 on behalf of Praveen00 and asked them to explain why the score was given. Before I continue, allow me to explain a little bit about our judgement system so that you can fairly consider our perspective during the next sequence of events.

    How we try to keep the competition fun, fair, and fresh

    During the competition, we keep the identities of our judges a secret. We also keep the scoring system our judges use a secret. The primary motivation for this is to avoid any influences or other types of "gaming the system" from authors participating in our competition. We want the submissions to be well rounded and we want our authors to consider the big picture when planning and creating their work, thereby maximizing the number of useful skills being practiced during the competition. Opening our scoring system up to public scrutiny and allowing the judges to be identified has historically resulted in some authors tailoring their submissions to only touch on the most valuable requirements or to appeal to specific judges, expecting that this strategy will ensure a high score while allowing them to ignoring other aspects that they believe are less important.

    This may seem reasonable from a participant's perspective - you want to know where you should focus your efforts to maximize your own chances of winning. This is a competition after all, and it's difficult to get motivated if you don't have a clear path to success. But it's an unhealthy strategy for the competition as a whole and a big reason these competitions die off. We need submissions to stay fresh after every competition or the incentive for us to run another one diminishes as the content gets stale. When every competition turns into a race to exploit the competition meta, we lose sight of the original intent of these competitions which is to serve as a vehicle for our authors to improve their skills. It's not fair to new participants who stand even less of a chance because they aren't aware of the competition meta, and it's not fun for anyone aside from the top winners if the same few people win over and over without much variance. This results in a negative spiral that has previously resulted in the competitions losing steam and dying, then no one has any chances to win money while honing their skills.

    We feel that keeping the scoring system and judge identities hidden adds a decent incentive for the authors to even out their strategy, as now there's a bigger risk for that higher payoff probability. We list the aspects that our scoring system focuses on, but the exact details as to how those are broken down are kept secret and changed each competition. This way, no single factor is guaranteed you a win and each competition requires a new strategy. It's more effective to put effort in to every category listed otherwise you run the risk of focusing on the wrong thing for the competition and scoring low overall. We feel this is an acceptable solution as it outlines every key factor you should put effort in to without assigning explicit priority to any of them. Note: this is still an animation competition, so it's safe to assume that obviously animation matters a lot by that reasoning and each competition has a theme so again by that reasoning the effort you put in to categories that reflect the theme matter.

    The main reason why judge scores are anonymized

    Aside from preventing judge-tailored content, anonymizing the judges also helps prevent harassment. In past SP competitions, complaints about the scores given by a judge had resulted in some nasty abuse directed at the judge. Aside from the fact that we want to protect all our participants from that kind of harm, it also has the potential to affect that judge's ability to express their true feelings about an animation out of fear of retaliation which is our primary motivation in making it difficult to link score to person. This case should exemplify how this is useful - we don't know the players involved so their histories have been factored out of the complaint, so the complaint can now be addressed based on content and a discussion can be had.

    Ex. Question: "Did my submission deserve this score?" Response: "Yes, and here's my reasoning why..." Result: "Oh I understand, I will have to improve" or "I really disagree with that, here's why..."

    The main reason this is a problem

    What I think it actually exemplifies is an unexpected consequence of this approach. From the author's perspective, we run in to problems with accountability. How do we know judges are being fair if we don't know who put the score down or why that score is what it is? An discussion about content is meaningless if the judge has it out for you. While waiting for a response from Judge 1, Praveen00 pointed this out. From his perspective, the judge might matter as the problem he's facing is the big fat unknown of, "Why did this happen?" Which could literally have any answer, including that the judge just was power tripping for no reason. If you think your work is first place material and two out of three judges seem to agree, then the third judge disagreeing with the "consensus" surely indicates there is something fishy, right?

    The only way to satisfy this is to ask Judge 1 for their score reasoning directly and evaluate whether or not it's a fair assessment or if there's some corruption at play. But since we don't know who that is, you have to ask me and trust that I'm behaving honorably in my role. There is nothing wrong with this, it is perfectly reasonable for anyone who has a problem with their score to be suspicious and want an explanation. It is on me to build that trust so that the assumption of foul play is minimized, and I try to do my best here but I understand that there will always be suspicions about things that are intentionally kept secret and that doesn't bother me. It's like a black box that your submissions go in to and come out ranked on the other side. When there's real rewards on the line, how do you trust the black box? Is the scoring based on opinion only, subject to any judge's whim? Or is there an objective measure for each category? The answers to these are not apparent.

    The Judges are People Too™
    Or: What is art, really?

    Well, the answer is sort of both. I recognize how wishy-washy that sounds, but the situation is complex and not well defined. Is there an objective measure for how good an artistic piece is as a whole? I would argue no, not really. We can of course look at each category from a technical perspective. For instance, how well does this submission demonstrate the principles of animation? Does it show an understanding of different animation techniques? We do consider these sorts of questions when deciding the score, but these are not as objective as they might appear on the surface. What if the submission barely displays any of the principles of animation just because the story does not allow for most of them to show up? Is that still a fair measure of submission quality?

    The fact is that each judge has varying degrees of experience in each of the categories by virtue of being individual humans with their own life experiences. By that same virtue, their scores will also reflect that variance. So, the final scores are ultimately a combination of the category weight, the score assigned to that category by the judge, and the judge's experience evaluating each category. It's not as straight forward as checking off a pre-made list of "did or didn'ts" to determine your rank, so we must rely on each judge and each author to be honest and act in good faith with one another. The more we do this successfully the more we can interact based on trust. In this case, as we find out below, Judge 1 caught the fact that much of the art used in Praveen00's animation was trace bitmapped from other sources. Is it fair for him to receive a high art score if most of the art in the animation wasn't even made by him? By nature the process of establishing trust in another human involves deeper philosophical questions that are outside the scope of this small stick figure animation competition.

    It's worked so far

    At this point based on these thoughts, I would warn you that if you cannot accept the inherent variance that comes with having actual people as judges despite all of the steps we have taken to minimize it as a factor, then you simply should not join this competition. We want you to approach this with the mindset of doing your best as a creative person, touching on as many of the aspects of artistic expression as you want. You build a lot of skill and a lot of character by doing this, and it will help you well in to the future regardless of the competition outcome. The previous winners from this competition have improved by leaps and bounds in just the single year we've been doing this. Not only did they build their skill but many of the winners put their prize into artistic tools in order to improve on their next submission. I am proud of the value we've been able to provide and I hope we can continue doing into the future.

    So what's the big deal?

    Unfortunately, before Judge 1 could give me their explanation, Praveen00 decided he would rather deprive the community of his high-scoring entry rather than accept his score and prize. He issued a demand that we remove him from the winner's list and do not publish his submission, ending the conflict before we could resolve it amicably. I am not sure why he felt the need to jump to this point before we could discuss Judge 1's opinion. For all I know the lateness and lack of communication during the end of the competition were enough to evoke this response, and that is reasonable in my opinion. However, the expectation for this competition is that when you submit, the work you submit comes under our control as the publisher regardless of the outcome. We reserve the right to refuse a request not to publish, that needs to be clear. You will be given the prize you won, if any, and your submission is saved for future processing even if you are so unhappy about your score that you want to try retroactively dropping out. We're able to sustain these competitions by using the submissions to drive activity on the site - something you should already understand and be comfortable with before joining. Hurting our ability to do that by self-sabotaging your work removes potential future traffic and hurts every other member of this community that cares about these competitions and gets something out of it.

    I felt like this all needed to be discussed as it happened in a relatively small time frame, so I informed Praveen00 that we would discuss his demand as soon as we could but if he persisted with this demand he would be prohibited from joining further competitions. I still also planned on relaying Judge 1's response. He opted to wait for the response, which arrived a day or two later which stated his reasoning for issuing the scores in each category he did:

    "The meat of the submission is really well done, but there were aspects that I felt were in need of improvement and ultimately brought the final score down in the end for me. The opening is too long and filled with uninteresting static shots. Camera motion would help keep the viewer engaged. The synthetic voice acting would have been better left out entirely. The character animation was stiff and often cheaply repeated. The background art was blurry and traced bitmapped from Game of Thrones, Skyrim and maybe other sources. I'm sorry that he is this upset, but his submission wasn't as good as the other ones overall. The battle scene was amazing and makes it worth watching, but that alone cannot make up for the other aspects that were lacking."
    Praveen00's response to this was to call these reasons 'silly' and restated his demand once again.

    Why this is not OK

    We understand when people get upset about results - handling the disappointment of not placing as well as you'd have hoped is tough and takes a lot of maturity to overcome. That doesn't mean you can't or shouldn't ever disagree with the outcome. When these disagreements happen, remember that we are people too that are doing their best to give you a good experience during these competitions no matter how they turn out. Remember what I said about the fact that judges are humans with their own range of experience and they will vote based on their own knowledge set. Let us talk about the problems one might have with a judge's reasoning and at least try to come to a reasonable middle ground before jumping off the rails. Why are these silly reasons? I do not know. I do not feel there was room for discussion.

    It is for his specific behavior towards us and the competition on Discord that we have decided that we are going to prohibit Praveen00 from joining in future competitions in addition to respecting his request for removal from this competition. We have taken his demand for removal as the hostile act it appears to us to be: I am not happy with the outcome, so you don't get anything from me. We wish to avoid interacting with someone who's demonstrated their willingness to behave that way. That is not the kind of person we would like poisoning the community surrounding these competitions, and I seriously caution anyone who would treat us this way over a disagreement to simply avoid this competition.

    Moving forward

    You might be wondering, "Well this is all nice but I also have to trust that everything you are typing is truthful! You could easily be manipulating us since you're the only one here who knows who the judges are, what their scores are, AND YOU'RE ONE OF THEM." That's smart, I wish I had thought of that. I realize this now. It's easy to see what kind of position I am in here, and I don't like it either. I want to establish trust, and it's hard to do that when you've positioned yourself as judge, jury, and executioner. This wasn't intentional, and I am going to resolve it by removing myself as a judge for future competitions. This way I can focus on being a mediator.

    I have decided that our rotation of judges is just not good enough in general. This time around we had only the same 3 judges we usually have, without a guest judge. That's obviously not keeping up with the desire to ensure fairness that I described above, so when we return in September we will be enlisting the help of notable community members to participate as judges in the automated system. That way the intention of diversifying judge opinions is truly fulfilled. It should have been a majority random selection to begin with.

    To maintain the integrity of these competitions, I am firm in my anonymous approach to the judge and scoring system. When community members start acting as judges, the anonymity approach will be very important to preventing fights. Not only that, but for as long as I am the one in charge of protecting the judge's identities I will not be a judge. That way in situations like this, you can be assured that I have no stake in defending our judge's opinion. If a community judge is being unreasonable, I will fight for you.

    Since much of the content of this post is not immediately apparent to someone new joining these competitions - something I am kicking myself over - we will be creating an agreement for the new site that includes much of the above in clear terms so that all parties can be on the same page. This way it is unlikely that those who cannot trust the way we have chosen run this will be unlikely to join in the first place. This, combined with a basic respect for Praveen00 as the an author and past competition winner, is why we will be following through with his request and not be publishing his submission. It is a shame, because it is legitimately good work and worth watching despite this controversy. I can only hope he will release it in some form.

    I think I've said all I wanted to say. I know this is probably gonna be a TL;DR for most of you so hopefully this is digestible for the ones that care lol. I would love to discuss our approach to these competitions with people who see faults I do not, but as of now I think all the necessary information to assess this situation is already laid bare and you can make of it what you will. All I can say is judge me on my character and decide for yourself if it even matters.

    Back to work on SP 2.

  4. #14
    Artist/Animator Arch-Angel's Avatar
    Join Date
    Jan 2007
    Posts
    9,414
    I've got a question that's kinda half and half with this. Let's say 15 people with the same skill level of KronusFA make and submit an animation all of which are relatively around the same quality, length, etc. All fifteen of these animations place but obviously as the order is determined, some of the entries will be rewarded prize money that does not really equate to the quality animation that you all are getting. I'm sure in this case some of the lower scoring people might be super pissed because it takes quite a lot of time to make something so great to only be rewarded the 15th place prize. How do you think you guys would go about dealing with something like this, assuming this crazy hypothetical could happen?

    As far as the whole Praveen00 thing goes, it's just a case of the dude's ego getting ahead of himself. I usually feel like a judge that gives significantly lower scores than the other judges is someone who has seen or dealt a lot more with the things they're judging. Top three is always the dream, but fourth is nothing to get butt mad over lol.
    SPAC: Genesis

    "Everything I've done up until now... it was all for nothing."


  5. #15
    ★★★★★ Zero's Avatar
    Join Date
    Aug 2009
    Location
    ••••••••
    Posts
    4,697
    Quote Originally Posted by Arch-Angel View Post
    I've got a question that's kinda half and half with this. Let's say 15 people with the same skill level of KronusFA make and submit an animation all of which are relatively around the same quality, length, etc. All fifteen of these animations place but obviously as the order is determined, some of the entries will be rewarded prize money that does not really equate to the quality animation that you all are getting. I'm sure in this case some of the lower scoring people might be super pissed because it takes quite a lot of time to make something so great to only be rewarded the 15th place prize. How do you think you guys would go about dealing with something like this, assuming this crazy hypothetical could happen?

    As far as the whole Praveen00 thing goes, it's just a case of the dude's ego getting ahead of himself. I usually feel like a judge that gives significantly lower scores than the other judges is someone who has seen or dealt a lot more with the things they're judging. Top three is always the dream, but fourth is nothing to get butt mad over lol.
    Hmm, the way I'd go about it would definitely be to call a meeting with all the judges and then discuss which factor would be weighed the highest and then they'd base their final scoring (Or additional points to the final scoring) on that. Every animation has to be different somehow, no matter how similar they are.

    It's a shame of what happened to Praveen since in the end, he would get nothing out of this. Ah well, such is life and the unique people which surrounds it. Can I have his money?




    ______________________________________________
    ... ... ... ...

  6. #16
    The Dark Shadow Fordz's Avatar
    Join Date
    Jan 2012
    Location
    Surigao City, Philippines
    Posts
    1,495
    Quote Originally Posted by Arch-Angel View Post
    I've got a question that's kinda half and half with this. Let's say 15 people with the same skill level of KronusFA make and submit an animation all of which are relatively around the same quality, length, etc. All fifteen of these animations place but obviously as the order is determined, some of the entries will be rewarded prize money that does not really equate to the quality animation that you all are getting. I'm sure in this case some of the lower scoring people might be super pissed because it takes quite a lot of time to make something so great to only be rewarded the 15th place prize. How do you think you guys would go about dealing with something like this, assuming this crazy hypothetical could happen?

    As far as the whole Praveen00 thing goes, it's just a case of the dude's ego getting ahead of himself. I usually feel like a judge that gives significantly lower scores than the other judges is someone who has seen or dealt a lot more with the things they're judging. Top three is always the dream, but fourth is nothing to get butt mad over lol.
    This kind of scenario is highly unlikely but assuming that it really does happen, is there any kind of problem with getting the place you deserve?

    Yes, most if not all competitors give their best in the creation of their animations. However, it is inevitable for the entries to be ranked in chronological order during a competition. What then if you receive the 15th place? That just goes to show that there are other people who demonstrated their abilities better, it goes to show that there are actually people who are better than you. Before entering a competition, you should already have that embedded in your head and expect all of the unexpected scenarios such as this. If an entrant cannot acknowledge this fact then as Jeff said, they are better off not entering the competition in the first place. Let's say, fifteen legendary animators take part in a competition. Despite their efforts, the possibility of one of them getting the 15th place is inherently undeniable. An ideal way to confront this situation would be acceptance and as Jeff said, maturity.

    Also I might as well share my perspective. I'm a participant of the first two SPAC competition and honestly, I had high hopes for the first one. I didn't know what kind of "parody" to make exactly, so I thought of making a parody of Mario with a humor content based on the culture I grew up in. Truth be told, the definition of "funny" in my place is slightly lewd and somewhat involves deteriorating people due to physical appearances, so there, I animated something with that kind of humor. When I got the results, I was shocked of my rank which was relatively far from what I had expected. A good thing to point out is that because of Terkoiz's explanation, I was able to understand and accept my results. Aside from the rushed quality, it was mostly because of the lewdness and body shaming in my animation that probably resulted in my downfall. As I began to explore the rest of the entries, I then began to understand how the definition of a parody differs from people to people but that basically means that if your taste doesn't match the judges' then you're surely getting a low score. I also understood that there is no certainty in every entry. Unfortunately, that kind of discouraged me since well, I kind of needed the money but I had no reason to not accept the place I really deserve.

    I'd like to make a suggestion, can all of the judges send critiques, suggestions and advices like Terkoiz? I mean, knowing their point of view along with the results will really help in making the entrant understand the root cause of his result and will help the entrant think more thoroughly on how to change his strategy if he decides to join the next competition. Since you will be rotating the judges and consistently changing the weights of each category in succeeding competitions, it wouldn't be considered disclosing sensitive information related to the scoring system right? In my opinion, it would give you a better constructive insight and would potentially deter you from arguments such as this.

    By reading Jeff's post, I also understood why the scoring system is hidden, it's to provide variety and avoid animations with relatively the same content to avoid the competition from being boring and dying. It also gives everyone an equal footing of successfully getting the first prize. However, hiding the scoring system makes this competition more like a lottery ticket, it gives an impression of certain uncertainty, that despite your skill, you can still fail. It's kind of sad you know, doing your best and not actually getting a higher place due to certain factors you failed to demonstrate (again, due to the hidden score system). It's quite discouraging, especially for those who actually know how this competition works now (like me). I animated the first competition, carrying the thought that the score is based mostly on how smooth the animation flows, it's art and story. I guess I was wrong because certain cultural biases also affect the rating. It makes me think, "if only I knew their definition of funny".

    I suggest somewhat giving "tips", you know. Each judge gives a certain tip that serves as a guide on what to actually create, given that it is not sensitive enough to make the animations stale. A broad or general tip would be really helpful, or you could say a "clue" on winning the judge's taste. I know it's somewhat tailoring to the judge's tastes but a simple tip like "Real art comes from you, not from others" may have prevented Praven00 from tracing bitmap art. Now tell me, will that tip actually make most of the entries stale? Of course not, it just gave a clue that tracing might result in your downfall. A lot of things would've been prevented and the results would've been drastically changed with a single tip from each judge while preserving the variety and creativity of each participant.

    Most of the suggestions I made are just to counteract the negative consequences of using the hidden score system and anonymous judges. That's because they're basically the biggest factors why these arguments even happen. I don't disagree with them though but we need to at least counteract the consequences of using these. There are also some things I don't understand, if two or more entrants are stuck in a tie, will they receive half of the prize in that place? If 3 participants were to be in a tie, then 1/3 of the prize? If you were to somewhat give equal prizes to each of them, then what will happen to the prizes for the ones at the lower ranks? I just wanted to clarify this if ever this was to happen.
    Last edited by Fordz; 07-14-2017 at 08:21 AM.

Page 2 of 2 FirstFirst 12

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •