Stricter Rating System on Anime/Manga Reviews

[Myoti]

I've already PMed Gypsy about this, but I'd like to reiterate here (that, and she hasn't answered yet).

After reading some reviews and the discussion over "Dokuro-chan" in the Anime Review section, I suggest the Mods (or someone) needs to try to come up with some sort of rating "guidelines" for the content of an anime/manga. In reading some of them, it seems confusing how some people decide what level to rate these series (no offense to those that give of their time to write these).

Example of a problem:
The anime One Piece and Rurouni Kenshin both recieved a "7" for Violence in their reviews. However, these seem to be on two different levels. OP is quite bloody, yes, and has many moments of characters getting shot, slashed, impaled, ect. However, there is no gore and few characters ever die.
RK, from what I understand, DOES have gore and deaths, so it seems that this series would end up recieving a higher rating in this area.

Another problem could come in one person, who doesn't think to much of say, nudity, to check the series and give it a reasonable rating, then someone else, who may be more offended by such, might give their review a much higher (and possible inaccurate) rating of the content.

Basically, if we had something that would list specifically how to judge this, it could save from others coming in and claiming certain areas should be different and such. This would have the reviewers writing on the specific level of the content and not just looking at certain parts and slapping what number they think would fit correctly (again, some do get it right, it's just I think we need to make sure everyone does).

A sort of rough example of what I'm talking about:
For violence (would probably be more specific on the numbers):
0-1: none, or if any, cartoony.
2-3: Some, but typically cartoony or not actually leaving "injuries."
4-5: Slightly heavier. Some blood, but not gore.
6: More blood, but little to no actual gore.
7: Similar to 6, but closer to being "gorey" (getting shot, stabbed, etc.).
8-9: Lots of blood and gore.
10: Extremely bloody and gorey.


EDIT: It seems I'm off on RK. Apparently Samurai X is the one that had the gore?

[kaji]

I agree that some standard should be set to keep consistancy from review to review. While many members contribute to the reviews section, what they say, in some says, reflects the views of the site as a whole.

I understand that it can be hard for a mod or admin to aprove a reviews ratings if they have never seen the reviewed anime, but that only means there should be some other checks in place to be sure all ratings are consistant and acurate.

On another note, I personally think that the use of age ratings is misleading in reguards to the apropriateness of reviewed anime.

[Shao Feng-Li]

EDIT: It seems I'm off on RK. Apparently Samurai X is the one that had the gore?

Yes. Rurouni Kenshin (TV) is rated 13+ and Smaurai X(OAV) rated 17+ (Basicly rated R).

[Cap'n Nick]

While we're on this I'd like to put in a request for more consistent grammar and punctuation editing before publication.

[Ashley]

Speaking on behalf of Gypsy-san, perhaps I should lend a little more insight into how we do our reviews (including how I approve the manga ones):

I understand that it can be hard for a mod or admin to aprove a reviews ratings if they have never seen the reviewed anime, but that only means there should be some other checks in place to be sure all ratings are consistant and acurate.

Actually, there are. We back-check reviews a number of other places, regardless of if we've seen the series or not.

On another note, I personally think that the use of age ratings is misleading in reguards to the apropriateness of reviewed anime.

This is meant to give a ballpark figure--every child is different and every reviewer is different, so there can be no perfect figure every time. However, the idea is to give parents/siblings/yourself a ballpark figure of the series and let them know what elements it may contain that other review sites do not include. We always want parents and other guardian-type-figures to check the show thoroughly (and *gasp* maybe even watch it themselves!) before handing it off to younger audiences.

While we're on this I'd like to put in a request for more consistent grammar and punctuation editing before publication.

I can assure you that's been one of Gypsy's highest priorities. She's recently put together a grammar-guide in her 'how to write a review' post, and she does spend quite a bit of time correcting grammar and what-not. Perhaps if we ever get any free time, we can go back and make sure all the older ones are up to par, although none of us are getting paid to run CAA. ^^

[Myoti]

Oh, indeed, Cap'n Nick. It's one thing to ignore it posts, another in something that's suppose to tell us "reliable" information about a series.

I understand age ratings being "ballpark figures," I just think that the content itself needs better guidelines so people know exactly what they're up against.

[uc pseudonym]

The difficult part, in my mind, is that different people can view the same anime and come away with different impressions of how graphic it was. Though I agree that the ratings should be as consistent as possible, I think this is more difficult in practice than in theory. However, I (and everyone else is capable of this as well) do discuss the reviews of anime I have seen if I feel the ratings are skewed.

While we're on this I'd like to put in a request for more consistent grammar and punctuation editing before publication.

You know, this is an area where I might actually be able to help. We'll see how my school schedule falls out before I commit to doing anything with this.

[Myoti]

The difficult part, in my mind, is that different people can view the same anime and come away with different impressions of how graphic it was.

True. I noticed that as a potential flaw as well. Though having a specific set-up could lessen the likely-hood of going too much on opinion, I would think.

Though I agree that the ratings should be as consistent as possible, I think this is more difficult in practice than in theory.

Indeed, but what isn't?

[Arnobius]

One thing I noticed was with the rating for sex is there seems to be a disagreement of what warants a "10." I myself had been basing my evaluations on the fact that hentai is a DND topic, so I would see a "10" as the worst rating short of hentai, while others have said they thought "10" should be reserved for hentai. An official ruling might be useful

[Mithrandir]

I think it's probably appropriate to bring this up. Many times, the review is NOT meant to disambiguate the title. Every single one of those reviews that have a "3" or a "7" also has a fill-in field where the reviewer should be explaining why they gave the title the rating they gave it. It's up to the reader to extrapolate from that whether the rating is one they would have given themselves.

Since these are totally subjective, I'm not convinced there's really any way to get 100% consensus - irrespective of the ruberic or metric.

[Radical Dreamer]

While we're on this I'd like to put in a request for more consistent grammar and punctuation editing before publication.

XD Indeed. *this is her pet-peeve in any sort of writing, along with spelling* I would love to see that sort of thing happen in the review section, even if it's just for simple capitalization and that sort of thing. *loves correcting spelling and grammar anyways.*

Yes. Rurouni Kenshin (TV) is rated 13+ and Smaurai X(OAV) rated 17+ (Basicly rated R).

Right, though I would consider the manga to be more on-par with Samurai X as far as gore is concerned. The TV series is rather tame in the category of actual gore.

[Kumagoro]

How about if the series is licensed, the age rating should be the age printed on the DVD, and if it's not licensed than what would likely be on the DVD cover?

[Arnobius]

How about if the series is licensed, the age rating should be the age printed on the DVD, and if it's not licensed than what would likely be on the DVD cover?

Well there are times I have disagreed with the age rating. The original Tenchi Muyo OVA for example was rated 13+ (no idea what the re-release was rated). Personally I thought it had enough nudity to warrant stricter

[CDLviking]

I think it's probably appropriate to bring this up. Many times, the review is NOT meant to disambiguate the title. Every single one of those reviews that have a "3" or a "7" also has a fill-in field where the reviewer should be explaining why they gave the title the rating they gave it. It's up to the reader to extrapolate from that whether the rating is one they would have given themselves.

Since these are totally subjective, I'm not convinced there's really any way to get 100% consensus - irrespective of the ruberic or metric.

Exactly. I usually don't even pay attention to the scales when reading reviews, but skip straight to the written explanation of the ratings. When I write reviews I base my rating on a combination of the seriousness of the offense, and how prevalent it is throughout the show.

[Myoti]

Since these are totally subjective, I'm not convinced there's really any way to get 100% consensus - irrespective of the ruberic or metric.

True. I just think if we did use something like this, we could at least try to get things closer and more accurate instead of, as said, users simply going on "whims" (again, not that all of them do this). It could also potential help in subjects such as what AnimeHeretic mentioned.

[uc pseudonym]

One thing I noticed was with the rating for sex is there seems to be a disagreement of what warants a "10." I myself had been basing my evaluations on the fact that hentai is a DND topic, so I would see a "10" as the worst rating short of hentai, while others have said they thought "10" should be reserved for hentai. An official ruling might be useful

It would. I personally use the latter definition because I think a series can have content as sexually graphic as hentai without having it be the focus and thus get that label. But most likely, we moderators would just have to get our heads together and make a decision.

Right, though I would consider the manga to be more on-par with Samurai X as far as gore is concerned.

I've read all of the manga, and I don't remember any gore at all, to be honest. Note that this demonstrates what I was saying about different standards. But please, give me an example, because I might have forgotten or glossed over something.

Exactly. I usually don't even pay attention to the scales when reading reviews, but skip straight to the written explanation of the ratings.

Same here. Mithrandir brings up a good point, and one that I feel shows one of the strengths of our review system. Though I'll still try to keep the ratings at what I feel is reasonable.

[freerock1]

One of the biggest issues with the current system, IMO, is that with just one person reviewing the titles, I've noticed quite a few reviews that leave out some examples of potentially offensive content. Generally the well-known gratuitous scenes of sex or violence are mentioned, but occasionally lesser-known (sometimes fairly serious) problem content areas are excluded. Perhaps it's subtle and the reviewer didn't quite understand it, but that doesn't mean it couldn't be a problem for some viewers.

Here's an example of what I mean: In one reviewed title, there is a scene where a teenage girl discovers that her dad has been spying on her in her bedroom. While his action is not at all condoned in the story, it could still be a sensitive topic for some viewers. This scene/arc was not mentioned in the review. I brought it up in the review thread for that particular title, and the person who wrote the review replied with something to the effect of, "Yeah, I didn't really understand that part." Ok, fair enough. But if someone doesn't understand it, I think it's fair to question whether he or she is really qualified to review the title in question.

In other cases, the exclusion of problem content in reviews may be due to a difference in the interpretation of certain terms. I was surprised that there was no mention of the Eastern religious themes in the review of Samurai 7 (which, while not used a lot, do play a significant role in the first couple episodes). Likewise, I was surprised that Inuyasha, with all its occult references, received only a 2 in the Bad Religion category. (Yes, I am aware of the whole demon/monster translation issue. But most Western viewers wouldn't necessarily know that, and with them playing such a large role in the story, I certainly would have thought Inuyasha would have received higher than a 2 in this category.) However, I'm wondering if the reviewers simply didn't see it as a religious issue if it wasn't an explicit reference to Christianity or some other organized religion.

The other issue I can see is that generally the reviewers are either fans of the series they decide to review (otherwise they wouldn't volunteer to do it), or they have a point to make as to why the series is inappropriate (as in the reviews for Ninja Scroll and Bludgeoner--which are sometimes needed, IMO). That's not totally a bad thing, because we want reviewers who are knowledgeable and passionate, and generally the reviewers seem to do a pretty good job with describing examples of problem content. However, it can lead to some bias when it comes to the numeric rating scale. It's not an intentional thing in most cases; it's simply a result of human nature.

Personally what I would like to see (if it can be coordinated) is some kind of a review editorial board. These users would be sort of a 2nd tier between the volunteer reviewers and the CAA staff and have a chance to look over reviews for inconsistencies or major omissions. Of course they'd need to be knowledgeable about a wide variety of anime. Something like that, in correlation with a more defined numerical system like the one Myoti described, would probably help cut down on some of the inconsistencies between reviews.

Anyway that's my $.02...

[Myoti]

I'm in agreement with what freerock1 stated.

In other cases, the exclusion of problem content in reviews may be due to a difference in the interpretation of certain terms.

In reading review threads (namely the one for "Dokuro-chan"), we also need to take to mind that some here could be considered more "liberal-minded" than others, meaning they may view things from a different perspective. Again, which is why I state a specific numerical system may be able to lessen bias and inaccuracies based on these kinds of interpretations, along with the "review board" or such as he stated (though I wonder if the threads for the reviews themselves can count as such).

[Mithrandir]

Well, that's part of the reason why there's "click here to discuss this" link on all of them (except the DND titles). Going through those will give users a better understanding of what to expect.

The moderators of the site do review the reviews, and if something major slips through we're usually OK at updating them. The fact is, some users want more strict and some want less strict. You HAVE to read past the numebers or they don't mean anything. If it's that big a deal, I can just delete them alltogether.

A review system that let dozens of reviews exist for a single title wouldn't really be that much more useful to people, nor would it be a good use of what few coding cycles I have left over these days.

That brings me to my next point... Let's agree on a few things:

1. No one system will work for everyone.
2. We do not have unlimited resources to throw at usability issues.
3. Whatever system we build has to work within some fairly strict constraints of the VB software.

The reviews board idea is an interesting one. Do we have any other constructive ideas? Or are we just grumbling about our favorite mistranslations (not directed at anyone in particular)?

[Puritan]

In a vote for the current system, I have to say that it seems to work well as it is now. I have seen a good number of series reviewed here, and I have had routinely good experiences with the reviews. There may very well be some problems, but the current system seems to work overall from my experience, and I don't see it as broken as it is now. It is simply the responsibility of the reviewer to attempt to give reasonable ratings for shows and explain them, which seems to me to be the most sensible option. Giving cut-and-dry benchmarks for certain ratings might very well inhibit the ability of a reviewer to account for extenuating circumstances: bloodless but horrorific violence or a bloody scene that wasn't violent but needed to prove a point, to simply give two possible scenarios.

Also, while a review board might be a good idea, such a thing might very well be cumbersome and really inhibit the posting of reviews for series. In my experience one can expect a board to usually take a good deal of time to decide anything, there may very well be differences of opinion over the board that would need working out, and such a thing might well require a goodly time commitment from those on the board. I think simply having the moderator approving the review double-check the review with other reviews and possibly with people who have seen the series ends up being both a streamlined and sensible option. Add in the willingness to modify reviews if something major comes up and the discussion threads, and I think that the current efforts come off pretty well. One simply needs to take the time to be informed about a series if one is concerned and exercise discretion in what one watches.

[freerock1]

In reading review threads (namely the one for "Dokuro-chan"), we also need to take to mind that some here could be considered more "liberal-minded" than others, meaning they may view things from a different perspective. Again, which is why I state a specific numerical system may be able to lessen bias and inaccuracies based on these kinds of interpretations, along with the "review board" or such as he stated (though I wonder if the threads for the reviews themselves can count as such).

Thanks for the vote of confidence. Personally I tend to lean a little more conservative about stuff like Bludgeoner/Dokuro-chan, but I do favor consistency across the board.

As far as the review threads, I think they make for good discussion and feedback, but I wouldn't say they would really serve the same purpose as a review board, at least not the way I'm thinking of approaching it. The threads don't begin until the review is already published. Plus I wonder how many people see the links to the threads in the actual reviews. Maybe something's not loading correctly on my computer or maybe I'm just totally overlooking them (which isn't outside the realm of possibility, but I'm not seeing them.

The reviews board idea is an interesting one.

Thank you, Mithrandir, for the vote of confidence as well.

Just as an addendum to my previous post (and I know you all are probably thinking about this already)... If a board of review is created, not only is knowledge of a wide variety of titles important, but displaying spiritual maturity is highly important as well.

[Ashley]

I agree that I'm hesitant to approve a board--not because I don't trust you guys, but because it would seriously log the system up. And like Mith said, the whole reason we have the discussion boards is for these sort of issues. Ideally, one would notice a review for a series one has already seen, read it, and then point out any grammatical or potentially offensive descrepancies in a post for that thread. The approving mod (cough Gypsy) would then decide if the review accurately conveyed said information, and then go back and edit accordingly. I think if everyone would just use the system we have, we could make this work without taxing our already taxed staffers.

[Myoti]

1. No one system will work for everyone.
2. We do not have unlimited resources to throw at usability issues.
3. Whatever system we build has to work within some fairly strict constraints of the VB software.

True, though I specifically am not speaking of something that would change the system or such (I think the threads we have for each review could still work for something as freerock1 said).

Basically, what I'm saying is to have a stickied thread in the review Forums that has several lists (for each content, violence, nudity, etc.), and each one with the numbers "0-10" stating typical scenarios. Of course, there are still those exceptions, such as this:

Giving cut-and-dry benchmarks for certain ratings might very well inhibit the ability of a reviewer to account for extenuating circumstances: bloodless but horrorific violence or a bloody scene that wasn't violent but needed to prove a point, to simply give two possible scenarios.

True. These are definetly difficult points that throw problems into such an idea.
I'll honestly say I'm not sure how exactly to handle something like that.

[CDLviking]

I'm going to second the approval of our current system. Everyone will always disagree on exactly what is objectionable and what is not. The only reviews I have ever really been displeased with were cases of the reviewer not having adequate knowledge to actually be writing a review.

[uc pseudonym]

Basically, what I'm saying is to have a stickied thread in the review Forums that has several lists (for each content, violence, nudity, etc.), and each one with the numbers "0-10" stating typical scenarios.

Let me say that I think this isn't a bad idea. It takes little to no effort (just someone making such a list and some tweaking) and could eliminate whatever percentage of reviews there are that have off numbers due to reviewer confusion. Not necessarily notes for every number, but at least guidelines for some (1,3,5,7,9, for example) It could help, anyway.

[Cognitive Gear]

Well, as long as people are in here giving their opinions as to why they think it should/shouldn't change, I might as well offer my opinion as well.

I think that the current way of going about things is fine, so long as the entire review is read. While I have not read every review, I have always had a good experience with them. There may be some problems, but I think that overall it is fine. If it ain't broke, don't fix it.

Basically, what I'm saying is to have a stickied thread in the review Forums that has several lists (for each content, violence, nudity, etc.), and each one with the numbers "0-10" stating typical scenarios.

While I think this would be a good idea, I do believe there is a problem. Giving numbers specific values at this point might cause some confusion. Using an example that was stated here previously:

Inu Yasha is labeled with Bad Religion: 2. If we now give the numbering system to have specific values to look something like this:

1: None.
3: Characters have shrines to their ancestors.
5. Characters Often run across "spiritual forces" or use "magic".
7: Some characters are demons, or demons are shown in a positive light.
9: Character's commonly take the Lord's name in vain, and defame Christianity in general.
10: Outright Blasphemy.

Now, even though there are examples, there is still room for the reviewer to make a personal call. While the characters in question are called "demons", and are shown in a positive light, they aren't actually demons. They are monsters. How is Inu Yasha scored?

Also, just creating guidelines could be problematic in and of itself. Who gets to make the final call on what is considered worse than what?

Also, implementing any such guidelines would likely mean going through ALL of the reviews to ensure they meet the new guidelines. That takes alot of time, and our staffers (as great as they are) are volunteers who don't get paid to keep the site up. On the other hand, if the older reviews aren't updated, then the system falls apart anyways.

All in all, I'm not against a change, but I just don't think it's practical to change it at this point.

[Myoti]

Let me say that I think this isn't a bad idea. It takes little to no effort (just someone making such a list and some tweaking) and could eliminate whatever percentage of reviews there are that have off numbers due to reviewer confusion. Not necessarily notes for every number, but at least guidelines for some (1,3,5,7,9, for example) It could help, anyway.

Yes, that's basically what I'm trying to get across. Just something that gives reviewers a good idea of what to stick close to.

I also realized those problems, ikimasu, so I'm not exactly saying something like this would be implemented immediately or such. Just something that could be done sometimes to help "guide" future reviews. Also, I'm sort of leaning towards this being more for the reviewers (to help them now how close they should have the ratings) than for the readers. The readers can look at the numbers, then at the specific notes of the comments by the reviewer. If they feel something is off, THEN a reader could recheck the "guidelines" to see if it's accurate and comment in the thread itself.

[Arnobius]

Well, as long as people are in here giving their opinions as to why they think it should/shouldn't change, I might as well offer my opinion as well.

I think that the current way of going about things is fine, so long as the entire review is read. While I have not read every review, I have always had a good experience with them. There may be some problems, but I think that overall it is fine. If it ain't broke, don't fix it.



While I think this would be a good idea, I do believe there is a problem. Giving numbers specific values at this point might cause some confusion. Using an example that was stated here previously:

Inu Yasha is labeled with Bad Religion: 2. If we now give the numbering system to have specific values to look something like this:

1: None.
3: Characters have shrines to their ancestors.
5. Characters Often run across "spiritual forces" or use "magic".
7: Some characters are demons, or demons are shown in a positive light.
9: Character's commonly take the Lord's name in vain, and defame Christianity in general.
10: Outright Blasphemy.

Now, even though there are examples, there is still room for the reviewer to make a personal call. While the characters in question are called "demons", and are shown in a positive light, they aren't actually demons. They are monsters. How is Inu Yasha scored?

Also, just creating guidelines could be problematic in and of itself. Who gets to make the final call on what is considered worse than what?

Also, implementing any such guidelines would likely mean going through ALL of the reviews to ensure they meet the new guidelines. That takes alot of time, and our staffers (as great as they are) are volunteers who don't get paid to keep the site up. On the other hand, if the older reviews aren't updated, then the system falls apart anyways.

All in all, I'm not against a change, but I just don't think it's practical to change it at this point.

I think with the Bad Religion review the numbers will also vary based on how important that matter is to the follower of a certain denomination. For example I understand that Rumiko Takahashi's "One Pound Gospel" is done with no malice intended, but the content is enough to make me want to pound my head against the wall, and say to her "Couln't you have bothered to do any freaking research before starting?"

Obviously we don't want to stir up these kinds of debates in reviews, but it's something to consider that the rating of "Bad Religion" may vary depending on the perceptions of the reviewer.

[uc pseudonym]

Inu Yasha is labeled with Bad Religion: 2. If we now give the numbering system to have specific values to look something like this:

1: None.
3: Characters have shrines to their ancestors.
5. Characters Often run across "spiritual forces" or use "magic".
7: Some characters are demons, or demons are shown in a positive light.
9: Character's commonly take the Lord's name in vain, and defame Christianity in general.
10: Outright Blasphemy.

Just for the sake of clarity, let me say a few words about how I view this. I think you make a legitimate point, but I would intentionally try to avoid that pitfall. For example, a possible numbering could be as follows.
1: There is another philisophical view present.
3: Minor elements such as demons or magic may offend some viewers.
5: Potentially objectionable elements play a continual role in the story.
7: Viewers will be exposed to seriously offensive ideas or the series delves into areas such as the occult.
9: The series is actively mocking or subverting Christianity.

As for changing previous reviews, that strikes me as more effort than it is worth. However, I don't think it would be bad to implement this system without doing so and merely make modifications if a previous rating is seriously off. After all, we're not trying to set each number in stone, merely give general guidelines.

Also, just creating guidelines could be problematic in and of itself. Who gets to make the final call on what is considered worse than what?

The staff, I would assume. I think we could come to a quick consensus on something as vague as these guidelines will be.

Obviously we don't want to stir up these kinds of debates in reviews, but it's something to consider that the rating of "Bad Religion" may vary depending on the perceptions of the reviewer.

True. However, I have always viewed the "Bad Religion" category (a difficult one to name) as a general rating of how many elements a series contains that are likely to be offensive.

[Myoti]

Yes, the "Bad Religion" seems to be the most difficult one to rate correctly in many cases, though I believe uc got that listing pretty closely.