Towards Benchmarking AALL Practices

In May, 2007, Learning Development (LD) at the University of Wollongong received a grant from the Association of Academic Language and Learning (AALL) for $3,227.40 to conduct an initial benchmarking project of AALL practices.

Project Team

The LD project team included: Bronwyn James (project leader) Joanne Dearlove (project research assistant), Heather Jamieson, Paul Moore, Alisa Percy, Emily Purser, Celeste Rossetto and Jan Skillen. The team was very fortunate in gaining the in-kind support of Jennifer Heath, UoW Student Services, Planning and Reporting Officer, who created the prototype database to store the data arising from this project; UoW’s Centre for Educational Development and Interactive Resources (CEDIR) for the project’s web-development; Beth Hammersley, Student Services, for the design of the CD label for the prototype database and Felicity Cuninghame, Student Services, for keeping an eye on the budget.

Project Aim

The aim of the project was to do some of the initial ´leg work´ that would enable AALL as a professional body to meaningfully benchmark the diverse range of practices currently employed by AALL units to support their students' development of core, disciplinary academic and professional skills and attributes.

Project Stages

     1. Review

The review stage was used to gather information from AALL units to develop a horizontal benchmarking framework of practices across AALL units in Australia.

Identification of Participants:

As an AALL funded project, the 2006 (then current) list of AALL members was used to identify contacts and units within each Australian university for potential inclusion in this project. On some occasions, following up this list led to the identification of other units that had subsumed the unit included on the membership list. In these instances the newer unit and contact person was included. When units were discovered that undertook work commensurate with other AALL units but there was no membership of AALL, the unit was not included in the project.

Developing a framework

As preparation for the project some of the team members had reviewed AALL members´ websites to gain an initial overview of the range of practices. This review was used as the basis for two survey forms (Surveys 1 and 2). Survey 1 aimed to capture the range of practices employed by a single AALL unit, demographic information about the AALL unit, its placement in the structure of the university, a brief profile of the university and its student populations. Survey 2 asked the unit to describe in more depth one or more of it practices.

A trial of surveys was carried out at the NSW/ACT AALL meeting held on June 1st 2007 in Canberra. These initial surveys asked respondents to reply to questions by writing their responses in paragraph form. The feedback from this meeting was that the forms were onerous and that a tick-the-box style was preferable because it would be less time consuming to complete. As a consequence of this, both surveys were revised.

Survey 2 was revised to allow for tick-the-box completion of the survey that was trialled at the June AALL meeting. Survey 1 was revised more extensively. The contextual section of the survey (i.e. institutional location, questions on student enrolments and staffing) was revised to ask for information to complete the provided figures and tables. The section on the practices was made tick-the-box after a lengthy process of searching each unit´s website and extracting the practices for inclusion in the survey. In this way, respondents could confirm or amend the practices listed. However, as most websites were designed for students to use, information on the research and scholarly activities of units and the curriculum development activities of units was not available and so this information was requested directly from respondents.

Distribution of the Surveys

Notification of the project and its commencement was made to AALL members through the AALL forum and the Unilearn List. Using the list of AALL units, with any amendments apparent from the websites, an invitation to participate was distributed, along with Survey 1, individualised for each unit, and a copy of Survey 2 via the contact person/people listed for that unit. Respondents were asked to complete and return Survey 1 to be included in the project and to complete and return as many copies of Survey 2 as they wished. A two week time frame was given for responses.

At the conclusion of the two week time frame the research assistant for the project contacted non-responding units to ensure they had received their surveys. In the process it was discovered that a number of the contacts for the units had changed, either permanently or temporarily, that some contacts were overseas and the unit needed them to return before participating, and that some surveys had not been received and some had been confused with other surveys circulating at the time. Wherever possible, extra time was negotiated to enable responses to be received and units to be included in the project. Survey 1 was sent to 75 AALL units and 44 units responded (58.6% response rate). The final response rate represented 33 out of a possible 39 universities (84.6%), with some universities responding from as many as 7 distinc AALL units. A total of 70 responses to Survey 2 and 21 attachments to these responses were received.

The responses were coded using the categories developed through the analysis of the data (see below) and were then sent back to each respondent for confirmation and amendment before the information was finally entered onto both a website and a database. Any queries or anomalies in information were dealt with by the project leader and research assistant and a data trail was created to ensure results are verifiable.

     2. Analysis of Data: Refining the framework

Once all the Survey 1 responses were received, the answers provided were used to develop categories for coding all the practices included in the Survey 1 responses. Initially the categories were developed independently by the project leader and research assistant. These categories were then compared and a further list generated as a result. These were then distributed, trialled and discussed by the project team and revised categories developed and used. All members of the project team participated in the development of the categories and coding of the data.

The final categories became the framework for identifying AALL practices. The framework has been used to identify 13 ‘umbrella´ categories of practice on the website. The database houses data for each participating university under these categories as well as additional further levels of practices. The category labels and levels of practice are provided on this website under Glossary.

The identification of category labels is of course a ‘politically´ charged activity. Different survey questions would have generated different types of answers and any number of different category labels. The surveys might have been developed to ask questions about, for example, what practices support transnational learning and teaching, equity students, the attainment of graduate attributes/qualities and so on, and these would have been/will be interesting and topical questions to ask. Instead, we started from descriptions that AALL units provide on their websites and went through the interative processes of aggregating this information, forming initial categories and sending them out to AALL units for verification and change. We were seeking verification of our [AALL] own business with the additional idea of providing AALL units with the opportunity to draw attention to their own practices because they have been well-researched, are innovative, offer a different methodology for approaching common concerns or simply because they ‘work well´. This forms a basis from which AALL units can seek out those units who are working in a way which might assist us to reflect on and possibly improve our own practices.

The Outcomes

The outcomes of this project are both qualitative and quantitative. The benchmarking team have produced:

  • A web based overall matrix of AALL practices with links to information about organisational units and more information about specific practices. The website is a sub set only of the information stored on the database.

  • A CD of a relational database that houses the information about practices supplied by AALL members. More information about the database is available on this website under Guide.

  • An annotated bibliography of current literature on the topic of benchmarking in relation to: Higher and Comparative Education, Higher Education Governance, Educational Development, Learning Design, Commerce and Management. The annotated bibliography is provided on this website under Bibliography.

Equally importantly, the project team have been part of a dialogue with AALL members across Australia about what we all do and how we do it. This is the stuff which is hard to quantify or describe but there has been an overall sense of this being a project that is about us AALL.

Towards benchmarking AALL practices: An AALL funded project 2007
Learning Development, UoW