| Pages in topic: [1 2 3] > | Planned change to qualification round voting Thread poster: Henry Dotterer
|
|---|
Hi all,
Romina and I have discussed and evaluated the forum posts and feedback from hundreds of members. The many comments and suggestions have been a great source of insight and ideas, and we have formed a plan for the next round. (The contests have been adjusted and improved each of the first six rounds. This will continue, probably in earnest for several more rounds, and in a more gradual fashion, indefinitely.)
The primary goals for the next round in particular w... See more Hi all,
Romina and I have discussed and evaluated the forum posts and feedback from hundreds of members. The many comments and suggestions have been a great source of insight and ideas, and we have formed a plan for the next round. (The contests have been adjusted and improved each of the first six rounds. This will continue, probably in earnest for several more rounds, and in a more gradual fashion, indefinitely.)
The primary goals for the next round in particular will be to ensure that an appropriate number of entries reaches the finals, and that the ones selected are the best and most accurate. The plan is to achieve this through a combination of (1) a change in voting format for the qualification round and (2) the enablement of a rigidly structured exchange of views regarding choice of terms, etc. (beginning while voting is underway.)
This post relates only to the change in voting format. There will be another post related to the exchange of views on term choice, etc.
First, a note on the ideal of involving an "expert panel". The ideal approach for qualification round voting may be one that relies on the judgment of both the community and outside "experts"... and as posted previously, the recent survey shows strong support for such an approach among contest participants. We will work in the direction of that mixed approach. That said, recognizing that it will be some time before we are able to assemble expert panels for each of the many pairs involved in the contest, we have decided that for at least the next contest, efforts will be focused primarily on improving community-based voting.
So here is what we have in mind for an adjusted community-based approach to qualification. Adapting suggestions made by several contest participants, qualification round voting would work as follows next time around:
- Qualification round voters will be asked to rate entries not in relative terms, but on an absolute scale of 1 to 5 (5 being "perfect" and 1 being "poor")
- Ratings will solicited in two categories: "Quality of writing" and "Accuracy of translation"
- Entries will be presented for rating in random order, in batches of 5. A person will be able to rate as many entries as s/he chooses, which is to say, you do not have to rate them all.
- The entries that advance to the finals will be the 3-7 with the highest average ratings. To be more specific, the top three (3) rated entries will advance regardless of their absolute average, and then up to four (4) more entries will advance, depending on how high their average scores are.
An advantage of this method of rating would be that as an absolute method of voting (rather than a relative one) it would serve not only as a means to accomplish qualification round filtering, but also as a way to gather feedback (first and foremost for contestants) in all pairs, including those with just one or two entries.
Other notes:
* There will be no change to the way finals round voting is done, at least not in the next round.
* There will continue to be restrictions on who is allowed to rate entries, but those restrictions may (or may not) change in light of the new approach.
* Averages could be calculated on a weighted basis. This would be one way in which expert opinion could be brought into the picture later on, for example.
Questions and feedback from contest participants on the above would be appreciated. In the interest of letting everyone have a say, we would ask that opinions be expressed in no more than one or two posts. Thanks!
Bear in mind that qualification round voting of this form would be influenced by the other planned change proposed above: allowing controlled exchange of views concerning word choice. We'll post on that shortly so that it can be taken into consideration for the purposes of feedback here. ▲ Collapse
| | | |
Henry D wrote:
(2) the enablement of a rigidly structured exchange of views regarding choice of terms...
I am not quite sure I understand what terms we are talking about in Art/literary translation.
[Edited at 2008-04-17 12:20]
| | | | | Well thought move | Apr 17, 2008 |
It is a well thought move. I am happy with this proposal for the next contest and with the selected strategy of a mixed approach. I am persuaded that at the end the contests will be more professional and everyone will win.
| | | | | We are talking about a way of exchanging views on more than terms | Apr 17, 2008 |
Yuri Smirnov wrote: Henry D wrote:
(2) the enablement of a rigidly structured exchange of views regarding choice of terms... I am not quite sure I understand what terms we are talking about in Art/literary translation.
Sorry, this was not at all clear. We are talking about providing a structured means of exchanging views not just on term choice, but also turns of phrases, grammar, etc. - ie. all the decisions one makes, and pitfalls one encounters, while translating a text.
This will be expanding upon in a second post and undoubtedly someone with a better way with words than I will provide a pithy way to say what I am trying to say.
| | |
|
|
|
| Capacity to evaluate a translation not in the voter's mother tongue | Apr 17, 2008 |
Henry D wrote: - Qualification round voters will be asked to rate entries not in relative terms, but on an absolute scale of 1 to 5 (5 being "perfect" and 1 being "poor")
- Ratings will solicited in two categories: "Quality of writing" and "Accuracy of translation"
I see a big problem with this system. Part of the problem in this context has been that many partecipants translated in a language which is not their mother tongue, and made different errors. I saw this in more than a language pair. In some language pairs MOST translations were not by native speakers of the target language. Therefore, also most of the voters were not native speakers and were unable to judge the quality of the texts they were grading. I would highly recommend, if you decide to follow this course, which has this intrinsic fault, at least to make it mandatory for the voter to specifiy his/her mother tongue. And this indication should be rendered public at the end. In this way, even though the final result would not be affected, at least things would be clearer.
Elisa
| | | | | Capacity to evaluate a translation not in the voter's mother tongue | Apr 18, 2008 |
Elisa Comito wrote:I see a big problem with this system. Elisa
True. I did not think about it. And I must say it is very important. When people vote in a pair where none of the languages is their mother tongue, it can be taken into consideration only for "writing" (if target language), not for "accuracy of translation".
| | | | | This does not help when none of the translations are good enough | Apr 18, 2008 |
Henry,
In some language pairs it was a problem in several past contests that due to the relative nature of voting, even the winning entry did not meet the standard of a good translation.
You said:
The entries that advance to the finals will be the 3-7 with the highest average ratings. To be more specific, the top three (3) rated entries will advance regardless of their absolute average,
* There will be no change to the way finals round voting is done, at least not in the next round.
The combination of these two things means that the top 3 entries, even if their average is low, (2 or 2.5 out of 5) will go to the voting round, where nothing will be changed, those entries will win 1st, 2nd, 3rd prizes, no matter what.
Did I understand it correctly?
If yes, I strongly suggest to either:
A) Set a minimum average required to advance into the second round (if not one entry makes it, then there is no winner, period.)
B) Bring back the "None of the entries are good" button for the final voting round.
Thanks
Katalin
| | | | | What about variants of the same language? | Apr 18, 2008 |
In Spanish there are words that have very different meanings in one country or in other. I beleive tha tat very few people know perfectly all the variants. If I read a word or expression that sounds odd to me, I can't be sure if it is a wrong translation or may be a variant of Spanish from another country.
I would suggest that, at least in Spanish, you could divide the contest according to the variants of Spanish. May be the same problem in other languages.
| | |
|
|
|
Pavel Tsvetkov 
Bulgaria
Local time: 17:21
Member (2008)
English to Bulgarian
+ ...
MODERATOR | Maybe there should be two different types of contests - 1. Art Translation 2. Technical Translation | Apr 18, 2008 |
Dear Henry,
thank you for spending so much time trying to improve contest proceedings!
Here is a thought that might help: I have no experience with previous contests, but I won the most recent one and I did notice that the question of accuracy really bothers voters; however, with literary/art translations accuracy is often overrated. Unfortunately, the majority of www.proz.com members are technical... See more Dear Henry,
thank you for spending so much time trying to improve contest proceedings!
Here is a thought that might help: I have no experience with previous contests, but I won the most recent one and I did notice that the question of accuracy really bothers voters; however, with literary/art translations accuracy is often overrated. Unfortunately, the majority of www.proz.com members are technical translators (which is only natural of course) and for them the source text is sacred, they almost worship it, they wouldn't want to see a single word changed. Now, if the question of accuracy were paramount with an art translation most resulting texts (even if they had received high point-totals) would be mediocre at best. Art translation is a lot more than simply supplying the exact meaning of words!
It has long been known that "poetry is what gets lost in translation" and if we are not to once again prove this maxim right, there should be two different types of contests: for art and technical translation - with their separate sets of rules. Or maybe each contest should include two texts for translation - one art and one technical again with different voting schemes. That could also give you a "more rounded win".
I do not like the idea of people commenting on texts or seeing how other people view a certain entry in term of quality - each one should decide for himself and make their own mind. Besides, derogating statements bring even more negativity and you might end up having to triple the number of moderators just to be able to contain the situation within limits.
I am looking forward to your answer!
Yours sincerely,
Pavel Tsvetkov
[Edited at 2008-04-18 16:39] ▲ Collapse
| | | | juvera
Local time: 15:21
English to Hungarian
+ ...
| Not necessarily so | Apr 22, 2008 |
Elisa Comito wrote: Henry D wrote: - Qualification round voters will be asked to rate entries not in relative terms, but on an absolute scale of 1 to 5 (5 being "perfect" and 1 being "poor")
- Ratings will solicited in two categories: "Quality of writing" and "Accuracy of translation"
I see a big problem with this system. Part of the problem in this context has been that many partecipants translated in a language which is not their mother tongue, and made different errors. I saw this in more than a language pair. In some language pairs MOST translations were not by native speakers of the target language. Therefore, also most of the voters were not native speakers and were unable to judge the quality of the texts they were grading. I would highly recommend, if you decide to follow this course, which has this intrinsic fault, at least to make it mandatory for the voter to specifiy his/her mother tongue. And this indication should be rendered public at the end. In this way, even though the final result would not be affected, at least things would be clearer. Elisa
Can't quite see how you would know that:
Therefore, also most of the voters were not native speakers...
In any case, where are the native speakers of the target language? Aren't they the ones most interested in a text translated into their language, and if they are, why wouldn't they vote?
Apart from that, the reverse is also true:
participants translating into their mother tongue do not always fully understand the text, and some voters happily believe that the translation is good, because of the writing is good.
Henry's proposals are certainly aiming at the right direction, and it is a question to find the best solution.
It would certainly help to eliminate entries with obvious translation inaccuracies, and equally, entries with mistakes by non-native speakers of the target language.
| | | | | Voting system changes | Apr 24, 2008 |
Hi Henry,
I just finally had time to read your latest comments about the contest changes, including this thread and the other 2 related threads:
http://www.proz.com/forum/prozcom_translation_contests/102367-thanks_to_those_who_have_completed_th e_survey_plus_early... See more Hi Henry,
I just finally had time to read your latest comments about the contest changes, including this thread and the other 2 related threads:
http://www.proz.com/forum/prozcom_translation_contests/102367-thanks_to_those_who_have_completed_th e_survey_plus_early_results.html
http://www.proz.com/forum/prozcom_translation_contests/102761-experiment_for_next_contest:_sharin g_likes_and_dislikes_of_entry_texts.html
I am really glad to learn that Proz.com will be improving the contest voting system and that you will be implementing, though it seems in the long run, the "expert panel" (jury?) idea.
Judging from the experience you have probably gained so far, I think it has been made clear that an "expert panel" may greatly contribute to the "neutrality" and "objectivity" of the contest voting system.
On the other hand, there seems to be a major issue as to how some view this contest activity. For some, it seems to be a "serious" contest and for some it is merely a "game". I think perhaps Proz.com should make this clearer, to maybe avoid some of the unfortunate situations that have been commented in some of the previous contest threads.
Even if this is only a "just for fun" activity (a game?), I think that for some the word "contest" has much more "serious" ("professional"?) connotations, so maybe it is a problem of the wording used to name this activity.
And even in some language-related games such as Scrabble, there is a clear set of rules and minimum standard requirements so that the game can be objectively fun.
Mind you, at least in the EN-ES pair we had a lot of "fun" in the past contest, because we managed to have a very interesting exchange of ideas about various terms, so I am not suggesting that the "fun" part is not possible even in a more "serious" competition/contest.
About some of your suggestions, I think the ideal would be to keep the voting system as effective as possible. Of course, how to achieve that?
As previously commented by myself and others (like Maurice Devroye), I get the impression that the key is to have a clear set of language standards against which each voter can evaluate each entry objectively, especially in the qualifying voting round:
1) Spelling
2) Grammar
3) Right choice of translation terms
4) Correct interpretation of original terms
5) General writing style
etc.
Perhaps you could add to each translation entry a "form" where the above categories appeared in a list with each category error with a checkmark box, a blank space field (for description) and a point allocation field next to it.
Each error found in any of those categories would deduct a given number of negative points. Therefore, in the end, those entries with more negative points (due to the errors found) would be eliminated and only those achieving a specific low balance of negative points or no negative points would make it to the final.
Of course, this is why you would require the "expert panel", to make sure that each specified error is indeed valid and that no errors have been omitted.
As to allowing controlled exchange of views, I agree with what has been said about the problem of keeping the anonymity of the contestants and the voters, in order to assure objectivity. I see this exchange possible only if full anonymity is kept throughout the process of all voting rounds.
Also, I think contestants should not be voters. I think prozians should participate as either one, but not both.
Kind regards,
Ivette ▲ Collapse
| | | | | A better definition of "correct" | Apr 24, 2008 |
Hi Henry, and everybody!
A number of posts have discussed how to assess to what extent a translation is "accurate", "correct" or "valid". However, none has discussed how this mythical quality should be defined!
Translation is not done in a vacuum. All translation is done for a purpose. Surely a translation can only be measured by how well it achieves that purpose.
If the purpose is not made explicit in some way, clearly it is impossible to measure its achi... See more Hi Henry, and everybody!
A number of posts have discussed how to assess to what extent a translation is "accurate", "correct" or "valid". However, none has discussed how this mythical quality should be defined!
Translation is not done in a vacuum. All translation is done for a purpose. Surely a translation can only be measured by how well it achieves that purpose.
If the purpose is not made explicit in some way, clearly it is impossible to measure its achievement in a way which can command agreement.
Here's my proposal. Let either:
(a) each text for translation be accompanied by a "brief", setting out such factors as the target audience, preferred translation strategy (close vs. free, literary vs. colloquial, regional variant, etc.);
or
(b) each translator be free to adopt the strategy they think best, and accompany their translation with a short commentary, in the target language, on the choices they have made.
In this way both expert and peer assessors will have some kind of *common* benchmark against which to evaluate the "accuracy", "correctness" or "validity" of a given translation.
Kind regards
Martin
[Edited at 2008-04-24 17:15] ▲ Collapse
| | |
|
|
|
| 1. You would not reject Tolstoi's writings... | Apr 24, 2008 |
...at spelling-grammar-and/or-even-style pretext. Or you would? So, don't put forth such criteria.
2. Spam-translators (including technical people fairly translating Shakespeare in word-to-word way) make minor hurt to the contest; instead, spam-voters and cheat-voters are of the main danger.
3. All complications and sophistications of the rules may only enable cheats to gain, as they, cheats, are more professional in cheating than in translating.
That's w... See more ...at spelling-grammar-and/or-even-style pretext. Or you would? So, don't put forth such criteria.
2. Spam-translators (including technical people fairly translating Shakespeare in word-to-word way) make minor hurt to the contest; instead, spam-voters and cheat-voters are of the main danger.
3. All complications and sophistications of the rules may only enable cheats to gain, as they, cheats, are more professional in cheating than in translating.
That's why I propose another direction for thinking: which moments and sides of a contest are to be open? As widely open, and public, as possible? ▲ Collapse
| | | | | That doesn't solve the main problem | Apr 24, 2008 |
I agree with Vassyl.
No shuffling the number of points, nor any discussions of the texts will deal even in a remote way with the main problem in some pairs: incompetent and unfair voting. If the voting is competent and fair, it doesn't matter much which qualification system is chosen. If it is not, no system, except for a 'panel', will help.
I emphasize again: this applies only to some pairs. Out of 3 or 4 pairs that I have been involved in or watching closely, only one is 'si... See more I agree with Vassyl.
No shuffling the number of points, nor any discussions of the texts will deal even in a remote way with the main problem in some pairs: incompetent and unfair voting. If the voting is competent and fair, it doesn't matter much which qualification system is chosen. If it is not, no system, except for a 'panel', will help.
I emphasize again: this applies only to some pairs. Out of 3 or 4 pairs that I have been involved in or watching closely, only one is 'sick', and that seriously. Others are healthy and sound. No problems have ever been voiced there.
[Edited at 2008-04-24 17:49] ▲ Collapse
| | | | | @ Martin Cassell | Apr 24, 2008 |
Martin Cassell wrote:
A number of posts have discussed how to assess to what extent a translation is "accurate", "correct" or "valid". However, none has discussed how this mythical quality should be defined!
Translation is not done in a vacuum. All translation is done for a purpose. Surely a translation can only be measured by how well it achieves that purpose.
I agree that translation is not an "exact" science, but some aspects of translation can definitely be "quantified".
For example, even considering American vs. English spelling, "lamb" will always be spelled as "lamb", and not as "lmab", etc.
The same with the "correctness" of some grammar constructions or translation choices which can be verified in any dictionary.
That's why I prefer the idea of basing the votes on the number of errors and not on subjective degrees of "quality" or "correction". I am suggesting the same method more or less applied when proofreading and/or editing texts ("correction" of contents).
If as voters we can manage to pinpoint clearly which/how many errors in a translation make it unqualified to make the final round, I think we may advance in the right direction.
Sure, there will always be a degree of subjectivity in the community or expert panel votes, but I honestly think that at least the errors detected will allow a higher possibility of "fair play".
About your idea of each contestant including an explanation of style/choices used, I think it is a good idea, but I am not sure that it should be shown during the voting rounds, but rather once the winner is chosen.
As someone already mentioned in another thread, "explanations" may influence (rightly or wrongly) comunity votes, so we need to proceed carefully in this respect.
I personally see these "explanations" more as a means to promote a constructive translation "post-debate" (as previously mentioned, more or less as it happened in the EN-ES last contest; see http://www.proz.com/?sp=contests&sp_mode=current&sp_sub_mode=view_language&contest_id=13&ctlid=553 ).
Good night,
Ivette
| | | | | Pages in topic: [1 2 3] > | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » Planned change to qualification round voting | Protemos translation business management system | Create your account in minutes, and start working! 3-month trial for agencies, and free for freelancers!
The system lets you keep client/vendor database, with contacts and rates, manage projects and assign jobs to vendors, issue invoices, track payments, store and manage project files, generate business reports on turnover profit per client/manager etc.
More info » |
| | Trados Business Manager Lite | Create customer quotes and invoices from within Trados Studio
Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.
More info » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
