Blue Board new feature may be misleading - is there a way to adjust?
Thread poster: Katalin Horváth McClure
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 14:01
Member (2002)
English to Hungarian
+ ...
Dec 20, 2017

I couldn't find a category for Blue Board issues, so I am posting this in the Job Systems forum, because I think the BB is mostly used in connection with jobs.
What I would like to point out is that the new BB features are not working quite right.
For example, today I looked up an agency, and they have a single rating of 1, however, the "Sentiment of comments" is "Very positive".
I think this is misleading, as having a rating of "1" is the most negative rating one can give.
... See more
I couldn't find a category for Blue Board issues, so I am posting this in the Job Systems forum, because I think the BB is mostly used in connection with jobs.
What I would like to point out is that the new BB features are not working quite right.
For example, today I looked up an agency, and they have a single rating of 1, however, the "Sentiment of comments" is "Very positive".
I think this is misleading, as having a rating of "1" is the most negative rating one can give.
I understand that the comment starts out with a positive comment about the PM, but I think it should be somehow balanced with the actual numeric LWA figure. I think there should be a limit on the sentiment score (which is from -1.0 to 1.0) based on the LWA figure, so that a record that shows a very low LWA average (in case 1) would never get a "very positive" sentiment rating, because it is confusing, to say the least.

Here is the link to the BB record I saw:
https://www.proz.com/blueboard/49760
Please note that I am not discussing this particular agency (I know that is not allowed on the forums) but merely showing it as an example of why I think the new rating features may need some adjustment.

Katalin
Collapse


 
Christine Andersen
Christine Andersen  Identity Verified
Denmark
Local time: 19:01
Member (2003)
Danish to English
+ ...
Positive with a sharp sting in the tail Dec 20, 2017

The comment starts out as positive, but that is not really a positive statement! The remark about the pay level is not at all positive IMHO, so it looks as if the comment assessor has not read the whole comment.

It is almost like a tale I could tell about a client who persuaded me to take over a rushed job that someone else had pulled out of. The client did not tell me why, but he kept me hard at work all through a holiday weekend with helpful comments and background information, or
... See more
The comment starts out as positive, but that is not really a positive statement! The remark about the pay level is not at all positive IMHO, so it looks as if the comment assessor has not read the whole comment.

It is almost like a tale I could tell about a client who persuaded me to take over a rushed job that someone else had pulled out of. The client did not tell me why, but he kept me hard at work all through a holiday weekend with helpful comments and background information, or snippets of text I could almost cut and paste into my translation... He sounded like an ideal client, except that he did not pay, and others had the same problem.

So no way would I work for him again - that was back in the days when we simply gave him a 1 rating, and I think he was banned from Proz.com.
Collapse


 
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 14:01
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
It is a machine algorithm Dec 20, 2017

Christine Andersen wrote:

it looks as if the comment assessor has not read the whole comment.


Well, yeah, it is not a human, but some sort of algorithm that I guess looks for specific words, and draws a conclusion from that. That is why I am saying that the algorithm may need to be tweaked to consider the whole picture, so that an LWA=1 would make it impossible to have a "very positive" assessment.


 
Sheila Wilson
Sheila Wilson  Identity Verified
Spain
Local time: 18:01
Member (2007)
English
+ ...
IMHO, algorithms of that type are simply misleading - full stop. Dec 20, 2017

Katalin Horváth McClure wrote:
the algorithm may need to be tweaked to consider the whole picture, so that an LWA=1 would make it impossible to have a "very positive" assessment.

It really shows how fundamentally flawed that sort of indicator is. It doesn't matter how many tweaks you put into it, there will always be cases in the future that it gets totally wrong. The mere fact that English uses the expression "wicked" in the way it does (probably not a great example but the only one that comes to mind at the moment) means that an algorithm will never work reliably. And they certainly can't be expected to work on something as human as "sentiment"!

Just thinking: what happens to BB records where many entries are not in English? I bet the algorithm hasn't been "tweaked" for all languages.


 
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 14:01
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
I think the tool is multilingual Dec 20, 2017

Sheila Wilson wrote:

Just thinking: what happens to BB records where many entries are not in English? I bet the algorithm hasn't been "tweaked" for all languages.

I searched for the original forum announcement from Kevin Dias about these new features, and I found it:
https://www.proz.com/forum/prozcom_translator_coop/318785-site_modernization_blue_board.html

In that announcement, he wrote:
What is the Sentiment Score?
The Sentiment Score uses Google's Cloud Natural Language API (https://cloud.google.com/natural-language/) to analyze the sentiment of a comment. The Sentiment Score for an individual comment ranges from -1.0 (very negative) to 1.0 (very positive).


Looking it up at the Google link above, it seems to have multilingual abilities.

My point is that the number the algorithm spits out may have to be adjusted to take the LWA figure into consideration.
In fact, it could even be used to improve the model, since the LWA number clearly indicates what the author thinks overall. If the words used in the comment make the judgment difficult, the LWA number could help in deciding what was really meant.


[Edited at 2017-12-20 18:57 GMT]


 
Robert Forstag
Robert Forstag  Identity Verified
United States
Local time: 14:01
Spanish to English
+ ...
How about just scrapping the tool altogether? Dec 20, 2017

It is hard for me to see the added value of the "sentiment of comments" tool. The general tenor of a given rater's emotions would appear to be accurately reflected in the posted numeric rating, and it seems totally perverse for a rating of "1" to somehow be associated with "positive sentiments."

Those consulting the BB can read recent comments for themselves if they want information that goes "beyond the numbers." Given that the comments are limited, and that most agencies do not h
... See more
It is hard for me to see the added value of the "sentiment of comments" tool. The general tenor of a given rater's emotions would appear to be accurately reflected in the posted numeric rating, and it seems totally perverse for a rating of "1" to somehow be associated with "positive sentiments."

Those consulting the BB can read recent comments for themselves if they want information that goes "beyond the numbers." Given that the comments are limited, and that most agencies do not have large quantities of entries, this really would seem sufficient.

[Edited at 2017-12-20 19:42 GMT]
Collapse


 
Samuel Murray
Samuel Murray  Identity Verified
Netherlands
Local time: 19:01
Member (2006)
English to Afrikaans
+ ...
This is precisely what the feature is for, though Dec 20, 2017

Katalin Horváth McClure wrote:
For example, today I looked up an agency, and they have a single rating of 1, however, the "Sentiment of comments" is "Very positive". ... I think this is misleading, as having a rating of "1" is the most negative rating one can give.


Well, looking at the comment in question, the sentiment is indeed "very positive", so the machine's guess was correct in this case. And this is exactly what this feature is for: to provide an aggregate numeric insight into the comments themselves, to counterbalance any misleading ratings.

The rating is based on the question "how willing are you to work for this agency again", and does not reflect anything else about the agency except the degree to which the agency matches each specific translator's needs and preferences. So, the ratings can definitely be misleading.

However, if there is only one rating (or fewer than, say, five), then the sentiment score does not add anything useful to the Blue Board, so my suggestion would be to show the total sentiment score only once there are five or more comments.


 
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 14:01
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
There was some adjustment made in the last few hours Dec 20, 2017

I think somebody "with the powers" is reading this thread, because the record in question now shows "neutral" sentiment (sentiment score zero). There was no LWA added, and nothing else changed compared to what was there this morning. So, it looks like there is some improvement.

As to Samuel's assessment of the comment, where he said:
looking at the comment in question, the sentiment is indeed "very positive", so the machine's guess was correct in this case.


I guess this may be subjective, but I would argue that it was not correct.
Note that even though the comment starts out with "incredibly friendly and helpful", it is followed by the word "however", and then "extremely underpaid" and "working for free". I understand how "effectively" might be tricky to judge in isolation, because - for example - "effectively handled my questions" is a positive statement, but again, such words should be assessed in context.

[Edited at 2017-12-20 22:35 GMT]


 
Samuel Murray
Samuel Murray  Identity Verified
Netherlands
Local time: 19:01
Member (2006)
English to Afrikaans
+ ...
@Katalin Dec 21, 2017

Katalin Horváth McClure wrote:
As to Samuel's assessment of the comment, where he said:
looking at the comment in question, the sentiment is indeed "very positive", so the machine's guess was correct in this case.

Even though the comment starts out with "incredibly friendly and helpful", it is followed by the word "however", and then "extremely underpaid"...


Yes, the way I read that comment, the text after "however" merely explains to the reader why the rating is so low. The text before it reflects the author's sentiment. This translator had a very positive experience with the agency, but is unwilling to "work again" due to the reason mentioned (low rates), which has nothing to do with the translator's experience with the agency.

A big problem with the Blue Board is that low ratings tend to send the signal that the agency is a bad agency, i.e. a bad agency to work for. The comment section counterbalances that to an extend, but generally users of the Blue Board don't take the time to read all comments, and rely mostly on the rating, which is unfair to good (but cheap) agencies.

I think this is misleading, as having a rating of "1" is the most negative rating one can give.


It is the lowest, but not the most negative. The question that leads translators to rate 1 to 5 is not "overall, how good is this agency", but "how willing are you, personally, to work for this agency again".


[Edited at 2017-12-21 12:51 GMT]


 
Chiara Scanavino
Chiara Scanavino
Germany
English to Italian
+ ...
Culturally missleading Dec 21, 2017

Katalin Horváth McClure wrote:

I couldn't find a category for Blue Board issues, so I am posting this in the Job Systems forum, because I think the BB is mostly used in connection with jobs.
What I would like to point out is that the new BB features are not working quite right.
For example, today I looked up an agency, and they have a single rating of 1, however, the "Sentiment of comments" is "Very positive".
I think this is misleading, as having a rating of "1" is the most negative rating one can give.
I understand that the comment starts out with a positive comment about the PM, but I think it should be somehow balanced with the actual numeric LWA figure. I think there should be a limit on the sentiment score (which is from -1.0 to 1.0) based on the LWA figure, so that a record that shows a very low LWA average (in case 1) would never get a "very positive" sentiment rating, because it is confusing, to say the least.

Here is the link to the BB record I saw:
https://www.proz.com/blueboard/49760
Please note that I am not discussing this particular agency (I know that is not allowed on the forums) but merely showing it as an example of why I think the new rating features may need some adjustment.

Katalin


I agree with you that it can be missleading. The cases as you mentioned occur, because not everywhere have the same imagination... For us Italian or for English native speakers, 1 star means that something (an agency, a product and so on) are very bad. Here in Germany however, if you go to a school and you deliver an essay, the mark 1 means that your essay is excellent. And this is also reflected in everyday life... In Germany, if you do a questionary and the forms have stars, you will see that there is an explanation of the maximum number of stars.


 
Mirko Mainardi
Mirko Mainardi  Identity Verified
Italy
Local time: 19:01
Member
English to Italian
Another case Mar 2, 2018

Blueboard entry with 2 ratings (both 1). One says they were never paid (payment due 9 months prior to the entry) and it's highlighted in red (so "negative"), the other one says they are waiting their payment since last October and that the outsourcer is not replying, and it's highlighted in green (so "positive"...). Both "Sentiment" "score" and "magnitude" are shown as positive: the first is "rather positive" (0.3) and the latter "low-key" (0.6).

In this specific case it's clear thi
... See more
Blueboard entry with 2 ratings (both 1). One says they were never paid (payment due 9 months prior to the entry) and it's highlighted in red (so "negative"), the other one says they are waiting their payment since last October and that the outsourcer is not replying, and it's highlighted in green (so "positive"...). Both "Sentiment" "score" and "magnitude" are shown as positive: the first is "rather positive" (0.3) and the latter "low-key" (0.6).

In this specific case it's clear this "machine sentiment evaluation" is not just "misleading", but utterly wrong, since there are just 2 ratings and it takes a second to notice, but what about the cases where there are dozens of ratings or more (the cases where this "feature" is supposed to be more useful)?
Collapse


 
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 14:01
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
Sentiment indicator gone Apr 21, 2018

I just checked a few BB entries, and I do not see that Sentiment Indicator anymore.
It seems it was sacked.
Thank you.


 
Tom in London
Tom in London
United Kingdom
Local time: 18:01
Member (2008)
Italian to English
Malgorhythm Apr 21, 2018

Katalin Horváth McClure wrote:

That is why I am saying that the algorithm may need to be tweaked


Yes, because at the moment it's a malgorhythm (with apologies to Private Eye magazine)


 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Blue Board new feature may be misleading - is there a way to adjust?






TM-Town
Manage your TMs and Terms ... and boost your translation business

Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.

More info »
Anycount & Translation Office 3000
Translation Office 3000

Translation Office 3000 is an advanced accounting tool for freelance translators and small agencies. TO3000 easily and seamlessly integrates with the business life of professional freelance translators.

More info »