An Analysis of the Ask A Librarian Service
June 2002 to April 2003
Introduction
i. Preface
The Ask A Librarian service evolved from the database of email and phone reference questions and answers maintained by the former Distributed Library Services unit. An initial team of Sue Beebe, John Doherty and Kevin Ketchner recommended the transition of this service to the Reference Services department. Later the team (with the additions of Tina Adams, Sean Evans, and Martha Portree) worked very closely with Mike Taylor of Library Technology Services to develop a less cumbersome way of recording data from the e-mail portion of Ask A Librarian while also maintaining user privacy.
The result was a web-based, database-driven solution that was fully implemented at the reference desk June 4, 2002, the first day of Summer 1 classes. The solution utilized an Access database as the backend, and a FrontPage Server driven web front end. The questions are submitted via a web form that is automatically copied to the database and emailed to the reference desk. The reference desk staff treat Ask A Librarian questions as regular reference desk transactions, and respond via a hyperlink and a web form, allowing the answer to be automatically recorded in the Access database and linked to the question.
ii. Methodology
The following report is an analysis of the data held in this Access database. The first dataset reports some of the numbers and statistical trends. These results were achieved by exporting the Access data into Excel and manipulating the results. The second dataset reports on the content of the questions and answers. This was achieved by running reports of the questions and answers and manually analyzing the content using an agreed upon set of standards (see Appendix A). The results were then manipulated in Excel.
Also, presented here is the data from June 4, 2002 through April 30, 2003. As of writing, the end of Spring 2003 semester data is still being collected. A revised report will be submitted when that data is collated and analyzed.
|
|
|
|
|
|
|
TOTAL |
Plus AAL |
AAL % |
|
Jun-02 |
1955 |
2069 |
5.51% |
|
Jul-02 |
2025 |
2130 |
4.93% |
|
Aug-02 |
1209 |
1310 |
7.71% |
|
Sep-02 |
3292 |
3503 |
6.02% |
|
Oct-02 |
3570 |
3786 |
5.71% |
|
Nov-02 |
3158 |
3356 |
5.90% |
|
Dec-02 |
1643 |
1719 |
4.42% |
|
Jan-03 |
1893 |
2079 |
8.95% |
|
Feb-03 |
821 |
1025 |
19.90% |
|
Mar-03 |
1748 |
1949 |
10.31% |
Where appropriate, comparisons are made to the reference desk transactions recorded daily by the desk staff. Currently, these are gathered hourly, in the categories of Reference (ready and in-depth), Directional, and Mechanical (ie. troubleshooting hardware, software, and printers). At present, there is only data available for June 2002 through March 2003 from the regular reference statistics. Also, previous years of data have proven to be incompatible with the new Ask A Librarian set up. As TABLE 1 and GRAPH 1 show, Ask A Librarian accounted for no more than 6% of reference desk transactions during the Fall 2002 semester. This number is slightly higher during the August Intersession and Winter Semester.
GRAPH 1
I. Statistics
|
TABLE 2 |
|
HH:MM:SS |
|
|
|
Month |
Count |
Average |
Min |
Max |
|
Jun-02 |
114 |
3:51:42 |
0:01:16 |
19:14:06 |
|
Jul-02 |
105 |
2:30:14 |
0:01:11 |
18:41:16 |
|
Aug-02 |
101 |
4:10:17 |
0:01:33 |
24:33:27 |
|
Sep-02 |
211 |
2:08:11 |
0:01:00 |
18:32:23 |
|
Oct-02 |
216 |
2:04:24 |
0:00:18 |
17:25:40 |
|
Nov-02 |
198 |
2:53:57 |
0:00:09 |
27:48:26 |
|
Dec-02 |
76 |
3:45:05 |
0:01:08 |
23:36:36 |
|
Jan-03 |
186 |
3:52:47 |
0:00:14 |
27:13:56 |
|
Feb-03 |
204 |
1:47:50 |
0:01:19 |
16:30:40 |
|
Mar-03 |
201 |
2:34:15 |
0:02:34 |
20:34:24 |
The following tables and graphs analyze a total of 1,612 Ask A Librarian transactions from the time period noted above. TABLE 2 is the compilation of all the data by month, June 2002 through March 2003. Represented here are the monthly counts; average time it took to respond to each question; the minimum time it took to respond to a question in the month; and the maximum time it took to respond.
As can be seen from GRAPH 2, the counts remain consistent – during a regular semester the count averages approximately 200 per month. The trend-line in GRAPH 2 shows a steady increase in transactions. It is expected this will fall somewhat during the summer months, but probably continue to increase during the regular semesters.
GRAPH 2
Compare this to the semester data. The semesters currently available for calculation include Summer 1 2002, Summer 2 2002, Intersession, Fall 2002, and Winter 2002/3. TABLE 3 shows the data by semester.
|
TABLE 3 |
|
|
|
|
|
Semester |
Count |
Average |
Min |
Max |
|
Summer 1 |
126 |
3:40:47 |
0:01:13 |
19:14:06 |
|
Summer 2 |
105 |
2:34:03 |
0:01:11 |
18:41:16 |
|
Intersession |
39 |
5:54:25 |
0:01:33 |
24:33:27 |
|
Fall |
723 |
2:28:34 |
0:00:09 |
27:48:26 |
|
Winter |
60 |
4:19:17 |
0:01:08 |
27:13:56 |
Maximum times in both tables show some questions took more than 24 hours to answer. This is explained by the initial questions occurring over a weekend when the library was closed (as in the case of August 2002/Intersession) or during a holiday (Thanksgiving, Christmas, New Years, Martin Luther King, etc.). Reference Services has since instituted coverage of Ask A Librarian on weekends when the building is closed. As can be seen from the minimum turnaround presented and from the averages (a maximum of approximately 6 hours), these high turnaround times tends to be an exception, not the rule.
GRAPH 3 shows that the trend for the average monthly turnaround time is falling – even as the trend for the numbers is increasing. As can be seen from GRAPH 4 and GRAPH 5, the majority of the questions were answered within an hour of receipt. In fact, a significant percentage of the questions were answered in under 30 minutes of receipt.
GRAPH 3
GRAPH 4
GRAPH 5
II. Content Analysis
Our analysis suggests that the majority of users are actually accessing Ask A Librarian via the library’s Web services. This is the context in which the following should be considered.
In an initial study of the questions asked during March 2003, it was immediately obvious that they fell into two types: reference questions and referrals to other units. Further, the reference questions were categorized as follows: Article Queries; Catalog Questions; Websites (ie. Internet); Database Questions; Document Delivery Services (DDS) Questions; Government Document Queries; Reserves (mainly Electronic); and Others (questions that ran the gamut from technology issues to ready reference to in-depth reference). Also, the referrals could be broken into the various library-wide units to which reference staff directed the user: Circulation; Media; DDS; and Special Collections and Archives (SCA). More information on the standards used to define these categories can be found in Appendix A.
i. Questions by Category
GRAPH 6
GRAPH 6 shows the categories and their numbers. As can be seen, by far the most frequently asked questions deal with searches for articles and the catalog. Typical of these questions are:
Articles:
- “Need some articles on standards based curriculum. What it is, etc.”
- “I am doing research on stereotypes found in society.
Where would I look to
find information on this subject?”
Catalog:
- “Do you have a book: Psychological Testing by Anne
Anastasi and Susana Urbiana
Seventh Edition. Please let me know.” - “Do you have the video called FAT City available for check out.”
The spikes in the chart reflect the assignment schedule – ie. there were a lot of requests for articles in October 2002 as that was the MidTerm assignment period. The Finals period (late November to early December) is reflected here too. Similarly, this is true of March 2003.
The Other category covers a great miscellany of
questions. Typical of them are questions dealing with electronic journals (eg.
“I am having a problem downloading an e-journal - it won't let me in”);
technical (eg. “I am having a problem with downloading a file to my floppy
disk”); procedural (applying for jobs at the library, etc.); and informational
(“I attended John Nichols' talk today and wanted to write to the dean of the
library”).
ii. Questions by Referral
GRAPH 7
Inter-unit referral from Ask A Librarian involves the use of an automated CC function that forwards a generic message to the user and a copy of the original message to the department. GRAPH 7 shows the number of questions referred from Ask A Librarian during the study period. Examples of questions referred include requests on Document Delivery status, billing, and requests for media holds.
Without access to other unit’s e-mail transaction statistics, the relatively low number of referrals suggests that users are already comfortable contacting the units directly. If and when other units make use of the service (as Special Collections and Archives started in January 2003) we expect these referrals to increase. As evidence, note the increase in SCA referrals after January 2003 in comparison to the data from 2002.
Referral data not included in this report are those transactions forwarded to subject specialists. This is not easily retrievable with the current database configuration.
iii. Answers
It was decided for the purposes of this report to separate the analysis of the questions from the answers. While there is some overlap in the topics and categories between the two, there was some data unique to the answers gathered.
For example, we first analyzed the use of the answer form. When the AAL service was originally transferred to the reference desk the team set up a number of standard responses in the Eudora mail program. These answer forms were a result of analysis of one month’s worth of questions recorded in the former Distributed Library Services database. The answer forms transferred to the new Access-based service but have never been analyzed until now for their effectiveness.
TABLE 4
|
Answer Forms Used |
TOTAL |
|
Circ Referrals |
53 |
|
Circ Renewals |
10 |
|
Database Prob |
18 |
|
DDS Referral |
82 |
|
DDS Renewal |
0 |
|
Generic |
801 |
|
Instruction Ref |
3 |
|
Mediated Search |
0 |
|
Request Materials |
131 |
|
Reserves Elec |
79 |
|
SCA Copies |
1 |
|
SCA Hours |
1 |
|
SCA Referral |
50 |
|
Search Articles |
338 |
|
Search Books |
119 |
|
Search Peer Reviewed Articles |
13 |
|
Search Periodicals |
25 |
|
Search Videos (gen.) |
26 |
|
Search Videos (Spec.) |
9 |
|
Forwards (depts. and Subject Specialists) |
148 |
As can be seen from TABLE 4, a compilation of the total amount of uses of particular answer forms during the study period, the most frequently used is “Generic.” The form instructing users in finding and accessing articles comes next, followed by the Forwards (not a form per se, but text automatically generated when a question is forwarded to a department or to a subject specialist for in-depth reference). The most frequently used forms are reprinted in Appendix B.
The generic form allows a common greeting and sign off while being flexible enough to allow the answer to meet the needs of the question. This form gets the most use as it is flexible, allowing staff to respond to questions as appropriate – either the reply does not meet any of the standard answer forms; is a combination for more than one; or requires staff to create a tailored reply.
Further, the answers were analyzed to discover which of the library’s premium databases are being presented to the users. Appendix C has the total results. The most commonly referenced resources, however, are those that also appear in the monthly reports of database hits: Academic Search Premier, ERIC, Expanded Academic Index, Lexis Nexis Academic and PsycInfo.
III. Conclusions
Since its implementation on June 4, 2002, the reference Ask A Librarian service has been a success. It has remained a dynamic process of development, trial and error, and quality service. Where many other institutions have expended significant amounts of money to acquire products complete with bells and whistles in order to attain a “synchronous” transaction, ours was home grown, minus most of the wonders of such products. However, we have achieved our goal of a less than 24 hour turnaround, and in most instances have kept turnaround to under an hour.
We have also achieved another, more significant learner-centered goal: the users of the Ask A Librarian service are receiving the same quality reference service as those who come to our desk or phone in. As the above clearly documents and as is also shown in Appendix D, the service is an integral and important part of reference services at the Cline Library, providing our users with equal access to all of our resources.
The service is well situated to be scalable in both the number and type of transactions. It has proved to be a flexible tool that reaches out to all of our users and is ready to be incorporated into all the web pages of the library’s website redesign.
IV. Recommendations for Future Analysis
Currently, we do not recommend the addition or deletion of any answer forms. We should revisit this issue later in the summer, when some of the forms should be examined in more detail for potential deletion – specifically those that have less than 20 uses at that time.
The Access database is not set up to easily allow for manipulation of the data collected. Nor are the current data sets of the Access database easily exportable to allow for automated data analysis. Therefore, the following recommendations are being made to allow for an easier process of data gathering, manipulation, and analysis:
1. Zero out the Access database at the end of Summer 2003.
2. Further, revisit the Access database with a view to establishing a new way of recording data. The user question web form should remain the same, but the team recommends that we develop a new “Answer” interface which will allow the staff answering the question to do some “pre-analysis” and “pre-categorization” of the question and answer using the standards of Appendix A. This will require some (8-15 hours) development time by LTS, but will allow for more automated reporting via the web. This would also allow units other than reference access to this data and analysis to assist in appropriate decision making.
3. The Access database should also be re-tooled to allow appropriate indexing of fields, thus avoiding the “apples and oranges” syndrome experienced in this reporting process.
4. Increase the profile of Ask A Librarian. Add a link to the service prominently on every library web page.
5. Re-tool the Ask A Librarian team. The Ask A Librarian Team should be representative of the views of other library units and departments serviced by AAL. Therefore, we recommend that the Web Coordinating group look into the future of AAL, including team direction and makeup.
V. Acknowledgements
This report is the result of the excellent work of the many reference desk staff who have answered Ask A Librarian questions since the beginning of the service on June 4, 2002. The Ask A Librarian service would not be as great a success if not for them.
Mike Taylor was instrumental in the technical development, implementation, and reporting of the Ask A Librarian service. His knowledge and enthusiasm for the project are appreciated.
VI. Submitted By
The Ask A Librarian Team:
Tina Adams
John Doherty
Sean Evans
Kevin Ketchner
Martha Portree
November 17, 2003
Appendix A. Standards for Analyzing the Content of Questions and Answers
Analyses of the questions and answers were accomplished using different methods and criteria. Test questions were ignored, and duplicate questions where counted only once (often users send their initial question more than once).
Questions
The questions were analyzed by the following categories: periodical articles, the library catalog, websites or the Internet, databases, Document Delivery Services, Documents, Electronic Reserves, and “others.” These are categories identified by Reference Services as the prime categories where Ask A Librarian questions fell. Also included were referrals to other library units or to subject specialists for in-depth reference assistance. Typically, these categories (similar to the answer forms) were chosen as areas where user questions, in our judgment, fit best. No attempt was made for deeper analysis of the question; ie. whether it was purely informational, reference or in-depth in nature.
Questions were assigned to the above categories based on a keyword analysis of each question. For example:
Articles:
“I am doing a research paper for Educational Law. (722) How do I access scholarly journals?”
Database:
“I'm trying to access the search engine called PubMed (formerly known as MedLine) that I should be able to find from the Cline website. How can I get to it?”
Catalog:
“I would like to know if you carry the book " Song for the Blue ocean" by Carl Safina? I don't know how to look it up so please let me know if you at cline carry this book thanks.”
Documents:
“Looking for a particular
government document - GTR-364 (1996) Guidelines for Estimating Volume Biomass
and Smoke Production for Piled Slash, by CC Hardy.
Do you have this one? The government search section on the website is not so
clear.”
Answers
In examining the answers to questions, we did not assess their quality or correctness. Rather, simple counts were made of the answer forms used and the premium databases or web resources users were referred to.
Some answers combined two or more answer forms. When it
was evident that more than one form was being used we marked them both/all
down (eg. Search Catalog and Search Articles were often used together though
they are two separate forms).
The Electronic Reserves answer form was marked when either Electronic Reserves
or Course Reserves answer form was used.
Appendix B. Frequently Used Answer Forms
Generic (#1)
Thank you for your inquiry.
[Name] (for Ask A Librarian)
Search Articles (#11)
Thank you for your inquiry.
Articles can be searched through the Cline Library’s homepage: http://www.nau.edu/library
• Click on “Browse Resources”
• Choose [“Subject Area”] or “Title” to bring up a list of article databases. Some of the databases you might start searching are:
• [Enter specific Database]
• [Enter specific Database]
• Click on the title of the database.
You may be asked to enter an account name and password. For more information, see:
• http://www.nau.edu/library/indexes/ezproxy.html.
Many of the databases are full text and you can print the articles directly from your computer. Those articles that you find that are not full text online or available in the library can be ordered from the Cline Library’s homepage if you live more than 40 miles away. :
• Click on “Request Services”
• Click on “Request a Book, Journal Articles, and Media”
If you live in Flagstaff, you can come in the library to copy those items we have or you can order those items that we do not own from the Cline Library's homepage
For more information on requesting materials, see:
http://www.nau.edu/library/information/illrequests.html
http://www.nau.edu/library/information/illpolicy.html
[Name] (for Ask A Librarian)
Forwards
Your question has been referred to a subject specialist, John Doherty, who will respond to you within 24 hours.
Search Books (#12)
Thank you for your inquiry.
To search for books owned by the Cline Library, go to our homepage: http://www.nau.edu/library
• Click on “Search the Catalog”
• Choose the appropriate option (Word, Title, Author, etc.)
Once you get your results be sure to click on the link to the item that you are interested in to check the record for availability. You may be asked to enter an account name and password. For more information, see:
• http://www.nau.edu/library/indexes/ezproxy.html.
More information on searching the library’s catalog is available at: http://www.nau.edu/library/information/guides/catalogsearch.html
If you live more than 40 miles away from campus submit an online request. You may do this from our homepage: http://www.nau.edu/library
• Click on “Request Services."
• Click on “Request Books, Journal Articles, and Media."
[Name] (for Ask A Librarian)
Request Materials (#8)
Thank you for your inquiry.
Material that the Cline Library owns is available in the library during normal library hours (http://www.nau.edu/library/information/libhours.html).
If you live more than 40 miles away from campus or the library does not own the material, submit an online request. You may do this from our homepage: http://www.nau.edu/library
• Click on “Request Services."
• Click on “Request Books, Journal Articles, and Media."
[Name] (for Ask A Librarian)
Appendix C. Number of Referrals to Premium Resources
