Роль национальных библиотечных ассоциаций
в разработке и оценке инструментария статистики
The Role of National Library Associations
in Developing Tools for Statistics and Assessment
Роль національних бібліотечних асоціацій
у розробці та оцінці інструментарію статистики
Ванда В. Доул
Библиотека Университета Вошбурн, Топека, США
Wanda V. Dole
Washburn University Libraries, Topeka, Kansas, U. S. A.
Ванда В. Доул
Бібліотека Університету Вошбурн, Топека, США
Анализируется, каким образом методы сбора статистических данных и оценки эффективности, разработанные в научных библиотеках Северной Америки, могут быть использованы всеми типами библиотек различных стран. Откликнувшись на пожелание своих членов, в 1994 году Ассоциация научных библиотек (Association of Research Libraries — ARL) приступила к разработке новых методов, позволяющих увидеть результаты библиотечного обслуживания. Такими новыми методиками стали «LibQUAL+ отчет» (позволяет определить оценку качества работы библиотеки), также «Оценка эффективности служб МБА/ДД» (эффективность и окупаемость МБА и доставки документов). Ассоциация распространила данные методики среди библиотек всех типов и видов. Небольшая универсальная библиотека Университета Вошбурна, число читателей которой составляет всего 6500 человек, использует обе методики для оценки эффективности своей работы, в том числе, и по сравнению с другими библиотеками. Опыт университетской библиотеки позволяет предположить, что данные методики могут быть без изменений использованы малыми и средними библиотеками и что результаты их использования будут сходными для этих библиотек.
This presentation examines how tools originally developed to gather statistics and measure performance of large Northern American research libraries can be used by libraries of all sizes and types throughout the world. In response to demand from its member libraries in 1994, the Association of Research Libraries (ARL) began to search for «new measures» to demonstrate the outcomes and impacts of library services. These new measures include the LibQUAL+ ™survey (a demonstration project to define and measure library service quality) and the Assessing ILL/DD Services Study (development of tools to measure and benchmark cost effective of interlibrary loan and document delivery). ARL makes these new measures available to libraries of all sizes and types. Washburn University, a small comprehensive university with an enrollment of only 6500, has used both the LibQUAL+™ survey and the ILL/DD study to measure its services and to compare them with those of other libraries. Washburn’s experience suggests that the survey instruments may be used without modification by small- to mid-sized libraries.
Аналізується, яким чином методи збору статистичних даних і оцінки ефективності, розроблені у наукових бібліотеках Північної Америки, можуть бути використані всіма типами бібліотек різних країн. Відгукнувшись на побажання своїх членів, у 1994 році, Асоціація наукових бібліотек (Association of Research Libraries — ARL) приступила до розробки нових методів, які дозволяють побачити результати бібліотечного обслуговування. Такими новими методиками стали «LibQUAL+ звіт» (дозволяє визначити оцінку якості роботи бібліотеки), також «Оцінка ефективності служб МБА/ДД» (ефективність та окупність МБА і доставки документів). Асоціація поширила дані методики серед бібліотек всіх типів і видів. Невелика універсальна бібліотека Університету Вошбурна, кількість читачів якої становить 6500 чоловік, використовує обидві методики для оцінки ефективності своєї роботи, у тому числі, і порівняно з іншими бібліотеками. Досвід університетської бібліотеки дозволяє припустити, що дані методики можуть бути без змін використані малими і середніми бібліотеками, і, що результати їх використання будуть схожі.
Libraries and library associations throughout the world face the challenges of gathering and using statistics to manage and improve library services. These challenges include identifying the most useful statistics to collect and establishing standards for the collection of those statistics. Such challenges require national and international cooperation and collaboration. Bodies such as the International Federation of Library Associations (IFLA), the International Standards Organization (ISO), the United States (U.S.) National Information Standards Organization (NISO), and many national library associations have been working for decades to establish standards for library statistics and performance measures.
In the United States, NISO convened a Forum on Performance Measures and Statistics for Libraries in February 2001.1 Supported by NISO, the Mellon Foundation, the U.S. National Commission on Libraries and Information Science (NCLIS), and the Association of Research Libraries (ARL), the Forum gathered together 65 participants from libraries, associations, publishers, vendors, integrated library systems and the research community. Forum participants identified the following general themes common to all libraries:
The need for systematic data collection at the national level.
The need for guidelines for collecting qualitative and performance data.
The Association of Research Libraries (ARL), a not-for-profit membership organization comprising the leading research libraries in the United States and Canada, has been engaged in systematic data collection at the national level since 1961-62.2 The ARL statistical survey instruments served as models for surveys conducted by several other associations: American Library Association (ALA) salary survey, the U.S. Integrated Postsecondary Education Data System (IPEDS) Academic Library Survey; Association of College and Research Libraries (ACRL, a division of the American Library Association); Council on East Asian Libraries (CEAL); and the Canadian Association of Research Libraries (CARL).3
There is an interactive Web site that provides ARL members and others the opportunity to manipulate the ARL descriptive data for planning, benchmarking /peer comparisons, and tracking aggregate data trends. 4 In addition to providing a model for data collection, the ARL statistics offer information at the national level about the trends in services (interlibrary borrowing/lending, library instruction, circulation, reference transactions)5 and purchasing costs and patterns.6 The trend information provides libraries of all sizes and types with information useful for planning and benchmarking.
Since the early 1990s, librarians throughout the world have been searching for ways to measure library services and performance. Suzanne Ward and others from the Library and Information Statistics Unit (LISU) of Loughborough University (U.K.) provide an overview of the development of performance measures in Europe.7 The IFLA University and General Research Libraries Section produced a manual for performance measures in academic libraries that has been translated into major languages.8 The ARL New Measures Initiative was developed in 1999 in response to the following needs of member libraries:
The need to demonstrate outcomes and impacts in areas of importance to their institutions.
The need to maximize use of resources (through benchmarking or reallocation of resources).9
Activities and projects were developed using different models for exploration with the intention of making the resulting tools and methodologies available to the ARL membership as well as the wider library community. These activities and projects are:
1. Higher Education Outcomes Research Review 10, including SAILS
(Standardized Assessment of Information Literacy Skills)11
2. LibQUAL+ ™, Measures for Library Service Quality 12
3. Investigation of Cost Drivers 13, including the Technical Services Cost Study14
4. Assessing ILL/DD Services 15
5. E-Metrics (Measures for Electronic Resources)16
Although developed for ARL member libraries, several projects have been opened to other libraries. This paper describes the experience of a smaller academic library with two of the projects: LibQUAL+™ and the Assessing ILL/DD Services study.
LibQUAL+™ is a large-scale research and development project undertaken by ARL in collaboration with Texas A & M University to define and measure library service quality across institutions and to create useful quality-assessment tools for local planning.17 The project uses a modified version of the SERVQUAL18 instrument (a survey instrument used widely in the private sector to the investigate the customer’s perception of service quality) to evaluate their library services. In the fall of 2000, ARL received a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) to support further development of the LibQUAL+™ survey instrument and to expand its application to libraries outside the ARL membership.19 The goals of the LibQUAL+™ project include:
development of a regrounded protocol to evaluate service quality in all post-secondary libraries
development of effective Web-based delivery mechanisms;
identification of best practices;
establishment of an ongoing, cost-recovery, service quality assessment program at ARL.
LibQUAL+™ is a Web-based survey instrument that measures the gap between library users’ perception of level of service delivered by the library, the level the users desire, and the minimal level of service acceptable. Users rank each level (perceived, minimum, desired) on a 9-point Likert scale. The difference between the levels is called the «gap». The survey is administered to a sample population or, in the case of very small institutions, the entire user population. The survey is delivered to the e-mail address of the sample or survey group. The 2002 and 2003 LibQUAL+™ instruments contained 25 questions about perceived service quality. Each question asked respondents to rate, on a scale of 1 to 9, their minimum acceptable service level, their desired service level, and their perception of the level of service currently provided by the library.
LibQUAL+™ began as a pilot project with 12 ARL libraries in Spring 2000 and in 2001 it was expanded to 43 libraries, including several small- to medium-sized non-ARL libraries. In 2002, there were 165 participating libraries, including all the academic libraries in the state of Ohio (the OhioLINK libraries), members of the Association of Academic Health Sciences Libraries (AAHSL), the Smithsonian Institution Libraries, and the research library of the New York Public Library. In 2003, 316 institutions of diverse types and location participated: a group of SCONUL (the U.K. Standing Conference of National and University Libraries) libraries, Military Educational Coordination (MECC) libraries in the U.S.), members of nine multi-type library systems in New York State (the NY3Rs), Oberlin Group of 75 U.S. liberal arts colleges and, for first time, a law library (the Edward Bennett Williams Law Library at Georgetown University).20 French and Dutch language versions were also tested in 2003
The LibQUAL+™ project is generating a rich body of literature. The project Web site contains a frequently updated bibliography.21
The 2003 ARL Assessing Interlibrary Loan and Document Delivery Services study updates, replicates, and expands the 1997 ARL ILL/DD Performance Measures Study.22 The earlier study was a two-year effort to measure the 1995-96 performance of ILL departments in 119 North American research and college libraries. The study examined four performance measures (direct cost, fill rate, turnaround time, and user satisfaction) and provided the data for libraries to establish benchmarking programs.
The 2003 study is an assisted self-study consisting of three parts: an organizational assessment, a survey tracking the turnaround time on a small sample of borrowing and lending requests, and a survey of mediated and patron-initiated services. The results of the mediated portion will be compared to the benchmarks and best practices identified in the 1997 ILL/DD Performance Measures Study. Participating libraries will be able to develop specific actions and changes that will result in a service that meets or exceeds benchmarks.
A small pilot group of libraries tested the methodology in 2002. The study itself was conducted from February to May 2003.
Application of Arl Tools
at a Small University
Washburn University is a municipal university with an enrollment of over 6,500 students and a full-time faculty of 265. It is a Carnegie Master’s University I with broad-based liberal arts and professional education programs leading to 190 certificate, associate, baccalaureate, and master’s degrees. The University Libraries (the Mabee Library and a Curriculum Resources Center) serve the needs of undergraduate and graduate instruction. A third library, the Law Library, is separately administered.
The University Libraries began an assessment plan in 2000 as a follow up to a strategic planning initiative.23 Since 2000, the Libraries have administered the LibQUAL+™ survey three times (2001, 2002, 2003), conducted two sets of focus groups (March 2002, March 2003) to clarify the LibQUAL+™ results, and participated in the 2003 Assessing ILL/DD Services Study.
Because of Washburn’s size, the Library surveyed the entire university population in 2001 and 2002. In 2003, however, the Libraries decided to sample in the hope that sampling would increase the response rate. Washburn’s Information Technology Services (ITS) was asked to draw a random sample of students and faculty and produce e-mail lists of each sample.
LibQUAL +™ was administered during the first week of April each year. The Dean of Libraries sent an e-mail invitation to participate with a link to the survey URL to all faculty, staff and students in 2001 and 2002, and to the sample lists in 2003. Two follow-up reminders were sent a week later.
Washburn’s response rate was low (only 4.47% in 2001). We suspected that the means of distribution used in 2001 and 2002 (campus e-mail lists) did not reach the entire population. The faculty/staff e-mail list contained the names of adjunct faculty as well as full-time faculty, but adjunct faculty do not always activate and use Washburn e-mail accounts. In 2001 the student e-mail list maintained by ITS contained only 3,924 names, although the Fall 2000 enrollment figures counted 6,946 part-time and full-time students. The student list was limited to students who have obtained and regularly use campus e-mail accounts. Despite developing a random sample list in 2003, the response rate was still low.
In conducting the survey, Washburn encountered many of the same challenges as larger institutions. They were:
Difficulty in obtaining a valid list of student e-mail addresses (each years)
Difficulty in obtaining cooperation of other campus units that control e-mail lists
Users’ dissatisfaction with the length of the survey (56 questions with three aspects each in 2001)
Users’ irritation with the number of e-mails sent by the Libraries (the announcement, the survey and two reminders)
Difficulty in obtaining buy-in from the Library staff and university community.