46505_en-US

liliconcom

Comparison of Ratings from Attendees and Committee

During the post-conference meeting we discussed comparing the ratings of papers by attendees to the ratings of the abstracts by the committee.

 

I've compiled that data and it's attached here. Please consider this confidential and don't share beyond this forum.

Re: Comparison of Ratings from Attendees and Committee

Greetings,

 

Thanks so much for providing the data. It seems that we are on average slightly more conservative on rating the abstracts than the attendee rating of the presentation. I feel very comfortable that we also tend to be less than a full point more conservative in the score differential for the presentations as well. Only 15% of the papers had a differential of 1+ or more and there were no papers that were scored worse by attendees than our abstract ratings. I would say that the 2017 paper selection is a success.

 

Rob Lievense

Re: Comparison of Ratings from Attendees and Committee

In general, the SC was more stingy with its praise.

 

Of 27 talks with no SAS speakers, 19 (70.4%) had higher ratings from attendees.

Of 21 talks with SAS speakers, 15 (71.4%) had higher ratings from attendees. One had basically the same rating between attendees and committee members.

 

Check out embedded Shift Plot in modified data table.

 

 

Re: Comparison of Ratings from Attendees and Committee

Thanks for that! Looking over the data, I wonder if there is more information to be gained if we add a rating question to the App. Steering committee members only can rank based on the abstract. While attendees may rate based on the content or the quality of the presentation.

 

I know, in the past, I have rated potentially solid content lower because the presentation was disjointed/hard to follow. Conversely I have always been more willing to rate a talk higher if it was engaging, even if my take away content was less (or there was only one Aha! piece of information).

 

Perhaps attendees could rate the presentation quality and content separately on the 1-5 scale?

Re: Comparison of Ratings from Attendees and Committee

Thanks Jeff,

This is very interesting.  There are some papers the committee liked better than the attendees, and we should keep that in mind next year when doing paper selection. And some authors who submit papers every year that are not as well liked. I will revisit this data next spring when looking at abstracts for October's conference.

Ledi