Case Studies

Researchers in the Centre for the Study of Choice (CenSoC) at University of Technology, Sydney have developed a method of measuring journal quality using a method called ‘best-worst’ scaling which allows the quality of journals to be measured using a consistent and empirically justifiable way. Because the method anchors the results to a common point, thus allowing meaningful comparisons between journals such as being able to say that journal X is twice the quality of journal Y.

With the full support and involvement of marketing societies throughout the world, the CenSoC researchers interviewed over 600 respondents over three months.

Each respondent was given nine sets of three journals each to rank and then their choices were combined with all other choices to give a quality score for each journal. The survey was implemented online in collaboration with Evaluation Solutions.

The results show that best-worst scaling is a very good method for measuring journal quality and ranking journals.

Download Journal Ranking Study article (22 KB)

Client

The International Aids Society (IAS) is the world's leading independent association of HIV/AIDS professionals, with 10,000 members from 153 countries. The IAS is also the custodian and lead organiser of the International AIDS Conference.

The XVI International AIDS Conference held in Toronto, Canada, in August 2006, attracted approximately 20,000 people, including scientists, health care providers, political, community and business leaders, journalists, government, non-governmental and intergovernmental representatives, and people living with HIV/AIDS. It was hoped that by bringing the conference evaluation online, the majority of delegates could be reached in a targeted, personalised and respectful way which also enabled multilingual access (English, French and Spanish). It was also important that a prize draw incentive on survey completion could not be linked back to any survey responses. Members of the evaluation team were keen to track survey progress, and provide access to other stakeholders, without directly engaging the survey platform.

Evaluation Solutions provided the online survey component of the conference evaluation and assisted the evaluation team with survey administration and implementation:

• 17,000 delegates were notified by email about the survey with their own unique survey link and personalised salutation

• Emails were steadily pulsed in batches, so that all respondents, across different parts of the globe, received their notification at the same local time

• Delegates were provided seamless access to multi-lingual surveys in English, French and Spanish via a common interface

• A link on the IAS website enabled respondents without email to take the survey, and data was combined with the main survey

• Reminder notifications were sent only to those who had not yet responded

• On completion of the survey, contact details provided for the prize draw were separated from survey responses

• A link to a real time survey tracking page provided instant results for each language group, without stakeholders directly engaging the survey platform

Client

The City of Melbourne is the local government body responsible for the municipality of Melbourne. The municipality includes the central business district, 11 suburbs and precincts, three major watercourses, as well as roads and parkland.

Context

As a capital city local government, the City of Melbourne strives to lead by example in performance and in reporting. It is committed to open, clear communication – and to telling it like it is. One way to keep its finger on the organisational pulse has been to survey employees about what it is like to work at the City of Melbourne and to track trends in the way staff view the organisation.

Solution

• Evaluation Solutions assisted in developing survey items and building on qualitative themes arising from previous surveys, in order to strengthen the benchmark measures that describe Culture at the City of Melbourne

• The online survey was administered across a select staff sample, representative of the organisational structure and divisions

• Reporting for this project includes benchmark reports comparing year-on-year results for the organisation and divisional charts to add diagnostic depth

Client

The Victorian Independent Education Union (VIEU) represents staff in Victorian non-government schools and education workplaces. Its parent body, the Independent Education Union, is the ninth largest union in Australia. Its history is defined by struggle for improvements in conditions.

Context

While a lot has been written about teacher workloads, not enough is known about how teachers actually experience them. Information about workloads from a previous VIEU study was somewhat dated and limited to empirical data. VIEU was interested to form a current picture of workload issues affecting primary staff, and thereby enable the Union to best support the needs of its members.

Solution

The Project had two stages:

1. Focus Groups
2. Follow-up Survey

The main purpose of focus groups was exploratory; to gather qualitative data that would help inform the design of a Catholic Primary Schools workloads questionnaire. A series of focus group sessions were led by facilitators from Evaluation Solutions and conducted in a group interview format. Transcripts were analysed and clustered by themes, utilising the coding feature of the online survey tool, e-valuate-IT®.

The Primary Teacher Workloads Survey was undertaken as part of the broader study of Workload Issues for Primary Catholic teachers working at elementary school level.

• The survey was designed by Evaluation Solutions in partnership with the Project Committee, sensitive to feedback from the focus groups conducted for this purpose.

• The survey was conducted online and made available to all VIEU members through the member-only portal of the union website

• A strategy was jointly devised by Evaluation Solutions and the Project Committee to encourage participation of members using several methods, including a link from the VIEU homepage; a fax stream to all schools; notices in the VIEU newsletter; further follow-up with VIEU representatives who were requested to notify their sub-branch members; and gift vouchers offered in a prize draw for participants completing the survey prior to the Christmas break. A chart providing response rates by date highlighted the effectiveness of each strategy.

• The report by Evaluation Solutions provided a qualitative and quantitative picture of the average or typical teacher workloads, as well as the sources of intensification and their various impacts on teacher stress and effectiveness. Differences between full-time, part-time and specialist teachers were examined in-depth.

• A link to a real time survey tracking page provided up-to-date results, without the Project Committee directly engaging the survey platform

• Evaluation Solutions presented summary finding to the VIEU Primary Council, where a record number of Catholic Primary Representatives attended