From CTMeetups: A presentation about effective feedback habits
PDF in link. I delivered a talk to attendants at a Cape Town meetup a little while ago. Since my presentation style often involves just writing down what I'm saying anyway on all of the slides (laaaazy), it's pretty easy to package for others and there's no good reason why everyone else shouldn't get to see it:
https://www.dropbox.com/s/qfjtjdxyj7q2l24/Effective Critique.pdf?dl=0
This is a guide to giving people good game feedback, by giving us workable tools that draw from schools of thought in conflict resolution. It has a pretty specific focus, but it's one that's relative to pretty much all of us and it's results-driven. We could find ourselves facepalming less often.
I am, somewhat recursively, looking for feedback on this presentation. It seemed well-liked, but I'd feel good expanding it to like 25-30 minutes and pitching a better version for AMAZE Jhb.
What sort of feedback habits do you use? Which have you found most effective?
https://www.dropbox.com/s/qfjtjdxyj7q2l24/Effective Critique.pdf?dl=0
This is a guide to giving people good game feedback, by giving us workable tools that draw from schools of thought in conflict resolution. It has a pretty specific focus, but it's one that's relative to pretty much all of us and it's results-driven. We could find ourselves facepalming less often.
I am, somewhat recursively, looking for feedback on this presentation. It seemed well-liked, but I'd feel good expanding it to like 25-30 minutes and pitching a better version for AMAZE Jhb.
What sort of feedback habits do you use? Which have you found most effective?
Comments
I'd like to have heard more about this point: I agree, but it's a little counter-intuitive (conventional wisdom doesn't usually state that "conversations are a defenders game"), and it makes me wish I had been at your talk. It's a thought I would like to hear more about, I feel like I've only started thinking about conversations in this way in the last couple years.
After reading your slides I actually looked back on some of the recent feedback I've given. I think I'm doing okay, but it's great to have my implicit goals stated explicitly (so I can actually explain what my goals are, or what they should be, and be able to try do better).
I think there's one thing I'd like to add. It doesn't belong in this talk, but I think it's worth noting. That the receiver of critique can play an active role in receiving empathic feedback.
For instance, the recent competition that Make Games held was a great opportunity to receive empathic feedback. In that everyone was working towards the same goal under similar constraints. Engaging in these kinds of activities is a great way to put yourself on the same playing field as other developers, and this naturally generates better feedback, and this also has lasting effects (as it gives the other participating developers better insights into your work. It creates a window into your work and that will be somewhat permanent).
By contrast, the developer that appears on the forum with a finished game and says: "Here is my game", is creating a situation where it's super super difficult to offer empathic feedback.
I'm definitely not suggesting that we're not all responsible for our communications, we all are. And I hope everyone who didn't see your talk in person reads through these slides. But just that I think in addition to communicating with empathy ourselves we can take actions that make empathy from others easier (and this can be a virtuous cycle).
(Again, this doesn't really belong in your talk. But I think there are community events like the recent Make Games competition are good as empathy facilitators. Would you agree?)
But without actually hearing the talk I'm struggling to identify what could be improved.
I think the stop lying is an interesting point, I think I tend to use hedging language quite a bit, but I haven't had it explained to me why this is a good idea. I'm pretty certain I've seen arguments arise from mistakes in this regard (and I may have been a culprit of this in the past).
And the points about controlling language are interesting as well. I'm not sure if I've been falling astray on this one.
I think point one is really useful to keep in mind. I'd hope it's obvious to most people at MGSA, but that doesn't mean we aren't slipping up on this.
I think there may be a problem in point two. While I agree in general with the point, I think some of the suggestions offered could be problematic.
One of the approaches you suggest in point 2 is that instead of pointing out problems, to say positive things for features that aren't yet in the game. A suggestion for potential features is a positive comment, it is definitely encouragement, but I think it can have unintended negative effects.
Let me be more specific: I think this is something I tend to do fairly often. And I see others do it. But I tend to worry about it for two reasons:
1 ) It is fun to offer suggestions. Basically we get to arm chair design. But having fun ourselves makes me suspicious, because I realize that I'm offering these suggestions for a motive other than to help the person (I'm offering suggestions to have fun). This doesn't mean it's a problem, but it makes me worry.
2 ) I think making a suggestion might sometimes overshadow that person's own goals. If that person is contrarian, they might not want to follow suggestions of others, and they might not feel ownership of ideas in their own game if they were ideas suggested by another person. Phrasing here might be what's important, controlling language could make a suggestion seem owned by the person making the suggestion (though you do address this later in your talk).
3 ) Like writing out design documents, seeing suggestions for features that can only be realistically implemented much further in the future might rob those features of their sense of achievement. The same way stating your intentions for others to praise creates a new social reality for you where you have already achieved them and subsequently you will be less inclined to actually achieve them (and will feel actually disappointed if these non-existent features don't get implemented).
I'm not sure if these are real concerns, but this is what I think about when I find myself writing out lists of possible features in a prototype.
I think this might come back to thinking about what questions the person receiving the feedback is asking. If your suggestions address problems that person seems to be having then that's likely to be useful, and if the suggestions are immediate suggestions that should be the very next focus of the project that's even better.
I think for suggestions the phrasing might be important as well. I think these suggestions should never be phrased like "If you do X then Y will be improved". A better way of phrasing it would be "I think Y could be improved, maybe if you try X ?" ... The reason for this is that assuming a change will improve something can rob that very change of its social reward, but if it is phrased as an experiment (or possibility or uncertainty) then the social reward will not have been earned yet.
I'm certain you're familiar with this concept (of "Symbolic Self-Completion"), but I'm linking it here anyway.
Thanks for sharing this! It will be interesting looking at some of my past feedback though this lens. (I think the talk's points are conveyed effectively enough that I am now self-conscious of giving feedback to it... Unfortunately, I have to break all your tips in what follows until I have had some time to ponder and adjust).
The language tips (especially those under the controlling-behaviour point) are interesting and something to think about. When I write, right after my passive-to-active converter I have a few modules that does exactly the opposite of some of your suggestions: a suggestion-to-imperative converter, a qualification deleter (which includes "quite" and "very", but also, "usually", "almost" and so on), and a "I"-deleter (in constructions such as "for me", "I think", "personally I"). These are from writing advice intended to make writing clearer and more persuasive (in the "authoritive" sense, not in the empathic sense clearly). I will have to revisit these "modules" after reading your slides; they may need some adjustment.
Now that I think about it, my whole process of giving feedback is the opposite of being empathetic... instead of putting myself in the creator's shoes, I bring the work into my heart and for a moment put it through the same processes as if it was mine to begin with. The justification is that this is the best what I know how to treat someone else's work seriously, but I can see this is problematic.
Overall, all your points makes sense and it is easy for me to find examples that show they are true.
I was not sure about the lying thing. I wrote a long paragraph of thoughts about the utility and a price of 100% truth, and then realized it's irrelevant: regardless of how useful a "lie" is people still don't like them. The point still makes me uncomfortable (maybe because my own behaviour is so against it).
You give an example of an emphatic way to answer someone's statement that the sky is green "I am wondering...", basically making it an implicit question. (I think you probably mean what I say here in any case, but maybe it is worth making sure people don't mistake the point.)
Questions are of course the way to getting more information, but I am somewhat surprised that people don't realise that questions can be read as damnations as easily as flat-out statements, especially "Why" questions. Even when not meant as attacks, questions are easily interpreted as such - and not without reason. Asking a question is a classical passive aggressive attack. (I'm adapting your example slightly here to better illustrate the point).
"Why did you make the sky green?" The sky should not be green.
"Why did you give this talk?" This is not an interesting topic or more ominously You do not have the authority or experience to give this talk.
To me, if you want to be emphatic, you should in fact truly be curious about where a contrasting viewpoint or solution comes from, and work hard to make your curiosity more apparent. I think it is a mistake to ask any question if you don't really care about the answer, or ask it in the spirit of already knowing the answer ready to refute it.
"The green sky immediately grabbed my attention, and made me wonder - why green?"
"Did some of the aggressive feedback styles on the forums inspire you to do this talk?"
I think you are neglecting one very important point, and that is the long-term reason for praise: and that is to build confidence. This is especially important in a setting where you are invested in somebody's success (such as people you hire). A constant stream of criticism gives the illusion of failure, which erodes confidence and eventually kills productivity and creativity. In fact, people that are already analytical and critical and seek to improve themselves would benefit more from things to help them be bolder and more audacious with their ideas and implementations than with actual micro-improvements... I think. (I am not very good at following this idea; it's an ideal at least to make my colleagues brave rather than good).
(The book "How to win friends and influence people" is essentially about empathy. One point it makes I always remember is that there is no such thing as constructive criticism; all criticism is destructive. I often wonder about this.)
(Oh yes, I did not get "defender's game"; what is it?)
@EvanGreenwood Your point 1 is interesting and true I think. At meetups I have often gotten the impression when somebody shows a game that is not very engaging to watch that people will amuse themselves with coming up with more ideas and giving more feedback, giving long monologues out of boredom. (But maybe this is not accurate). (We have been experimenting with different demo styles at our Chilean meetups to reduce this effect and to get an idea if it is in fact a true effect or just my imagination... not sure yet). And I do think you are right about it being fun (I would say, it gives an opportunity to practice a thinking style that is essential for our jobs without the pressures of those jobs weighing on us.) It worries me too...
I think your second point is true too, and more difficult to deal with. In a work setting, the culture can help with this, but it comes over a long time. If it is possible for people to comfortably "release" ownership of their own, and "adopt" those of others, then magic can happen. It took a while for me and Jon to develop that dynamic when it was just us at Gamelogic; now with a bigger (still new) team there is at times shock at how hard-handedly ideas are treated (especially by me); I am still navigating my way through this issue.
Regarding the defender's game comment, I may actually be using that just flat-out incorrectly and I'd prefer to rephrase it.
In short, our conversational defence is pretty much unbreakable if we decide to put up resistance. I'd argue that this isn't a matter of "strike opponent with weapon, opponent dies" -- a yield has to be voluntary, otherwise our only other options become an external expression of force (banning, widespread social shaming, etc) which usually only builds resentment without changing anyone's minds.
And thanks for the link on symbolic self-completion, @EvanGreenwood! The results of the study were interesting and I'd like to investigate more. I think that most of the significant critique about Point 2 is worthwhile. When you talk about accidentally imposing our vision or expectations onto a project, I definitely hear you. I think this would be one of the primary candidates for revision, or a few extra minutes of detail, in the next presentation. I would have to think a while to say much more than "that's a good point", so I'll see how it goes.
@hermantulleken so it seems you've explored this sort of stuff quite a bit yourself, and I hope I'm not casting too much doubt on solid learnings about things like confident expression. I'm familiar with some of the exact writing lessons you mentioned (NO hedging, NO suggestions!) and still firmly believe that there's good contexts for that. ;)
I do think this presentation is fit for a particular situation with actors who are on particular power/social levels, there's so many schools of thought out there. One thing in particular that I'd be cautious about "expanding" would be thoughts on control, which in most realistic environments would probably be managed rather than avoided.
Thanks ppl for additions on hostile questions, receiving critique gracefully and the importance of building up confidence. I can envision those having a place in an AMAZE talk for sure.
I had some thoughts while reading it. I give a lot of feedback to Game Design students now so I have a different perspective, I think. They're quite new to a lot of ideas in game dev (these are 3rd year students who have made >15-20 games in just the course at this point) so they way I give feedback is quite different perhaps.
I think the ordering of feedback can matter a lot too. I met a Canadian Interactive Arts lecturer a while back and we spoke about giving feedback to students - she solidified this own idea I had: the "Feedback Sandwich". The idea behind the feedback sandwich is that students are fragile souls, that if dealt with negative feedback first will fall into a hole that no amount of praise can lift them out of. As such, you always try structure your feedback as:
1. Initial positive feedback about the strengths of the game.
2. Critique on the work and where it could go (usually about 80% of the feedback)
3. Conclusion where you restate the strengths of the game.
This structure seems to stop students from getting demotivated, even when there is a large amount of critique. Personally, when this structure is applied to my own work, I feel more motivated to stick with the game too - I'm not sure why this is. Perhaps it's because by putting the positive feedback first, you're implying that's the part of the game that is most important?
I'm also not sure about the "Maximise praise, minimise problems" phrase. Many of these students have made so few games that it's impossible for them to identify or think out the problems in their work . For newer developers I think it's important to identify problems, and perhaps unpack why they are problems, otherwise they may wander the wastelands of development, unsure as to why their game isn't great. I agree that praise should be maximised and almost mandatory, but I think the problem section is crucially important for newer developers.
A final point: I think much of this talk is about micro feedback - giving feedback to individual games in a vacuum. However, I think something I would like to see more of (and something that I've always felt is missing from people giving feedback on my work) is feedback that takes the macro into account. How does this game matter in the context of the game's this developer has made before? Have they really improved their game feel, or level design in this game, relative to previous games? Feedback taking into account the context of where this work fits into their previous works would be an excellent habit to add, I think.
Love this talk, sorry If I ramble on or am a bit clumsy in my thoughts, I really did enjoy it :).
I haven't read the book. So I don't know what the goalposts are by which the author measures outcomes. But if the goal is to win friends then I could see how any critique could be bad.
I think I might be an outlier though. I tend to subscribe to the mantra of "Telling me my game is 'nice' is at best a waste of both of our time". (As is mentioned at the start of @Nandrew's talk)
So from my position I feel more appreciative of well thought out critique that answers my problems or points them out. While universal praise generally makes me think less of people (like they could only feel this way if they can't see the problems, and so their unqualified praise is betraying a shortcoming in them).
But I think I do tend to project that. I do try to make it clear I'm looking for critique and I try to offer a safe space to do so in. So when I receive praise without critique it often means the other person hasn't made the effort to engage in the work. (Which speaks to the point @Nandrew made about people wanting to be heard)
So I might not be able to judge whether critique is destructive, because I try demand critique through the way I present my work. And I suspect @francoisvn creates the same dynamic (and hence could give that AMAZE talk mentioned in @Nandrew's talk).
I think what @Nandrew might be trying to achieve is a unified theory of giving useful/actionable feedback while also maintaining and even fostering friendship. Which sounds like a pretty awesome topic to become an expert on when phrased like that...
I wonder if this is a case (that of being in a community of creators) when sacrificing a bit of the relationship, through destructive critique, has a net benefit?
Or whether it is the case that there isn't the need to destroy anything to relate the information that might help the person?
@Nandrew Regarding symbolic self-completion. The idea that stating your goals publicly makes you less likely to achieve those goals is one of the most interesting things predicted by the theory, but the theory contains a lot more than that: https://en.wikipedia.org/wiki/Symbolic_self-completion_theory
A leader's job often includes changing your people's attitudes and behavior. Some suggestions to accomplish this:
Principle 1 - Begin with praise and honest appreciation.
Principle 2 - Call attention to people's mistakes indirectly.
Principle 3 - Talk about your own mistakes before criticizing the other person.
Principle 4 - Ask questions instead of giving direct orders.
Principle 5 - Let the other person save face.
Principle 6 - Praise the slightest improvement and praise every improvement. Be "hearty in your approbation and lavish in your praise."
Principle 7 - Give the other person a fine reputation to live up to.
Principle 8 - Use encouragement. Make the fault seem easy to correct.
Principle 9 - Make the other person happy about doing the thing you suggest.
(I see now many of these points coincides with @Nandrew 's - 7 is a nice restatement of the idea of giving praise to non-existent features, for example).
As you can see, some of it does not really apply to the type of feedback shared between creative people (especially the type on the forums), who, often, actively seek out criticism.
----
(Edit)
@EvanGreenwood
(I'm just running with some thoughts here; these are not necessarily my own beliefs at this time.).
I think the (potential) destruction is not just to the relationship, but actually to the person itself. So while one may sacrifice having a better relationship with a peer, it's worth considering whether such feedback does not slow growth, and whether there is something better we can use to propel each other. I mean, fixing any one game is not important in the bigger scheme of things. Consistently becoming better is also not so important - not as important as being constantly motivated to create things for a looong time. If you make a thousand games, I am sure they will become better regardless of feedback. So the question is whether criticism can slow down creative work (subconsciously), and to what extent, and whether making five (say) games that incorporate feedback is better than a thousand (say) that don't.
One thing that sometimes trap me is doing one or two things that I am really proud of, after which their weight paralyze me for some time (basically, I am applying self-criticism too early in the process; ideas get killed halfway during having them). On the other hand, when I am in the zone, I usually have a (fictitious version of) person in mind, and I'm constantly feeding off what this person will think of this twist or that turn. This is usually a person that has said some positive things that I respect. (In fact, you are often my fictitious person when it comes to aspects of our tool design after some feedback you gave on Grids).
And yes, doing something because it's fun is an important motivator, but things stop being fun when you stop improving, and improving from your lone effort is very, very hard. I've seen this over and over again with aggressive skating, with martial arts, with dancing, with painting and, yes, with game development. Without a doubt, yes. To both... Criticism (or rather, the perceived judgement implied by the idea that someone might criticise your efforts) can be a barrier to trying - again, I see this in skateparks all the time: The new kids sit out as soon as someone starts tearing stuff up because they're afraid to be seen as worse than them. But it's just as true that 5 attempts with feedback are better than 1000 without - if the problem is identifiable: Those same skatepark kids will try something and get so focused on trying that they don't notice the good skaters arriving, then a good skater sees a kid falling and they know exactly why, 1 quick explanation later and the good skaters are going berserk because the kid just landed a new trick. That appreciation for relative success is the most important part.
To avoid talking way more than I should, I'm just gonna focus on this particularly interesting tidbit: This is a classic! And I definitely need to address the ordering of feedback in the presentation. The counter-proposal that I drew up came from George Thompson's "Verbal Judo", and it made an interesting argument based on the savviness of the listener, and also included an important note about ordering.
1. In short, the positive-negative-positive feedback cycle is present enough in common knowledge of management that it's arguably transparent to some listeners.
2. If people detect or assume that you're softening them up for bad news in this way, they feel manipulated and become resistant.
3. To work around this, VJ suggests bringing your negatives to the front, and working on presenting them levelly, compassionately and *briefly* -- then working people through that with a weighting of positive commentary afterwards.
Though in writing this out, I realise that this model is based on an assumption that people "get" the meta -- if you have the right read on your audience, I imagine it would be best to act flexibly in accordance with context.
I think I feel a bit of giddy geekiness about the discussions happening in this thread! I really like the concept of sharing feedback on feedback techniques and building some sort of recursive feedback-improvement device. :D Intimidating phrasing, but I like it!
This is, of course, more targeted at leaders within companies etc and giving feedback to employees, students, children, but it's food for thought about dealing with your own emotional state before approaching someone else. Also, naturally, her books on the subject add a lot more context.
Bonus points for being condensed into a single image!
(edit: looks like I know which book is next on my reading list)