Monday, January 25, 2010

Library co-operation, interlibrary loan and document delivery

The library is an arena of possibility, opening both a window into the soul and a door onto the world.
Rita Dove

* Define the concept of resource sharing
* Verify, locate and process interlibrary loan/document delivery requests using online sources
* Select appropriate delivery options for ILL/document delivery requests

Required reading:
Lunau, Carol d. “Canadian Resource Sharing at the Close of the 20th Century.” National Library News Nov. 1999. 26 Mar. 2001.

National Library of Canada. “Interlibrary Loan Policy.”


In the first act of Hamlet, Polonius gives the following advice to his son Laertes: “Neither a borrower nor a lender be.” Fortunately libraries do not practice this maxim, but rather engage in the task of opening the world to their users. From the smallest public library in rural Manitoba to the largest academic library and all those in between, no matter their type or size, there is an obligation to fulfill the information needs of users. Just because a library does not own a particular item is no reason to shrug our shoulders and say: “Sorry.” Libraries are in the business of bringing users and items together. For this reason, resource sharing activities including interlibrary loan/document delivery exists.

Resource sharing
In 1994 “A Canadian Information Resource Sharing Strategy” was published. This discussion stated: The goal of resource sharing is to ensure that Canadian library patrons have rapid, convenient access at reasonable cost to information required for research, business or leisure purposes…” In 1999 the National Library reported:

Interlibrary loan continues to be the mainstay of resource sharing. However, union catalogue development, cooperative cataloguing, cooperative reference, cooperative collection development and joint storage of material are all components of A National Information Resource Sharing Strategy. (Resource Sharing)
Carole Launau’s article “Canadian Resource Sharing at the Close of the 20th Century” pinpoints the growth of library consortia as the first general characteristic of the present library environment. The pervasiveness of change and the impact of technology, primarily the Internet and the World Wide Web, are listed as the other two. Libraries are feeling the impact of technology “by an increase in workload, an increase in requests from other countries or regions, greater accessibility to resources for a library’s patrons, and mo re knowledgeable patrons”. Most libraries report increases in interlibrary loan activity, with 42.8% automating their ILL functions. Of the respondents, 68.8% had reciprocal borrowing agreements, but only 12.7% (mostly academic libraries) offered patron-initiated ILL.

Interlibrary loan
Interlibrary loan (ILL) is the process by which a library requests material from, or, supplies materials to, another library…
(CLA/ASTED InterLibrary Loan Code, 1996, 4)

Document delivery service (DDS) is

The provision of published or unpublished documents in hardcopy, microform, or digitized format, usually for a fixed fee upon request. In most libraries, document delivery service is provided by the interlibrary loan office and patrons must pickup the material at the library. Also refers to the physical or electronic delivery of documents from a library collection to the residence or place of business of a library user, upon request. (Reitz)
New electronic technologies have brought ILL into the forefront of library operations. It is impossible for any one library to contain all the items, whether books, periodicals, audiovisual materials, etc. ever produced. At some point in time you will be asked for something which is not in your collection. This is when an interlibrary loan may be appropriate.

Interlibrary loan is provided through the co-operative efforts of libraries from around the globe, and is governed by local, national and international codes. The basic document governing ILL practice in Canada is the CLA/ASTED Interlibrary Loan Code revised in 1995. Examples of provincial ILL codes are Saskatchewan Resource Sharing ( and Interlibrary Loan Code for British Columbia Public Libraries ( The National Library of Canada acts as a backup to local efforts.

According to the CLA/ASTED InterLibrary Loan Code, the purpose of ILL “is to obtain, upon the request of a library user, materials not available in the user’s local library.” (4). Any type of materials can be requested, not just books and periodical articles. It is the choices of the potential lending library, however, to determine whether or not it will lend certain formats. Many libraries, for example, will not lend:

* reference materials
* entire issues of journals
* rare and or fragile items
* items in high demand, e.g. bestsellers, new books
* items which are difficult or expensive to ship, e.g. oversize materials
* audiovisual items
* text books

For information on the borrowing policies of Canadian libraries, go to the National Library of Canada’s ILL Directory on the Web ( This site is the Web equivalent to the print resources Symbols and Interlibrary Loan Policies in Canada.

In the past, interlibrary loan was oftentimes a slow somewhat cumbersome ser vice relying almost exclusively on the public mail system for the sending of requests and receipt of materials. Much “detective” work could be involved in trying to determine who had the items you were seeking to borrow. In today’s world, however, we have electronic access to a myriad of resources such as OPACs and electronic union databases to assist in finding locations. We also have a variety of alternatives to traditional mail services, ranging from couriers to fax to electronic document transmission systems.

Many libraries now use the term “document delivery” or “document delivery and interlibrary loan” instead of “interlibrary loan”. By doing so they reflect the change in the practice that has occurred. There are now a variety of sources from which to obtain items not physically held in a library. As the definition of document delivery previously given indicates, actual loaning of items from one library to another is just one of many ways to obtain requested materials.

In traditional interlibrary loan, a library borrows items from other libraries on behalf of users. Alternatively, a library might buy items from a document supplier such as CIST I ( or Ingenta ( on behalf of its users. In either case, the library acts as an intermediary between the user and the item requested. The item is received by the library which then informs the user who must then pick it up.

In situations where reciprocal borrowing arrangements have been set up, rather than having an item sent directly to a user’s “home” library, it may be preferable to send the user directly to the participating library which has the item. The user can then borrow it directly. Red River College, the University of Manitoba and the University of Winnipeg, for example, have an agreement that allows students, faculty and staff from any of the three institutions to borrow materials on-site and in person from any of the three participating libraries.

When users download or print off a full text article from a database service made available through their library, such as EBSCOhost, they are in effect participating in a document delivery transaction. Similarly, if a library allows users to place their own orders from a document delivery source, document delivery occurs.

What are the steps involved in an interlibrary loan?

Receive request

An interlibrary loan starts with a request from a patron for an item he or she cannot find in the library’s collection. The request can be received in a variety of ways. Sometimes patrons are in the library and approach the library staff with a citation to an item they have been unable to find. Alternatively, more and more libraries allow patrons to request items not in the local collection through forms mounted on their website. See for example, the Red River College Library’s “Book Request Form” and “Article Request Form” accessible from

Once you determine that the user is eligible to receive ILL service, staff next check to ensure bibliographic facts about the item are correct. This process is known as “bibliographic verification”. Verification consist s of checking in authoritative sources, the details of an item that allow it to be identified completely and unambiguously. For a book, it would be such information as its author, title, year of publication, publisher and place of publication; for a magazine or journal, its author, article title, title of the magazine or journal, year, volume and page number(s). Authoritative sources include such tools as union catalogues, e.g. MAPLIN and AMICUS; print and electronic indexing databases, e.g. CBCA, EBSCOhost, and Reader’s Guide; and bibliographies, e.g. Bibliography of the Prairie Provinces. For in print items, consider using the web sites from commercial book stores such as Chapters Indigo, McNally Robinson, Amazon etc.

When verifying journal articles, you may find that the indexing source does not spell out in full the name of the journal in which an article appears. It may be necessary to consult an abbreviations list to determine the full journal title. Many such lists are available over the Web. Try the phrase “journal abbreviations” in a search engine to retrieve some. A gateway that provides access to journal abbreviations in many different subject areas is All That JAS: Journal Abbreviation Sources available at The D.W. Craik Engineering Library at the University of Manitoba also maintains a “Journal Abbreviations” page which provides links to other journal abbreviations sites at

Bopp and Smith provide several reasons why citations may not be correct.

1. Newspapers, magazines, scholarly books and articles may not provide enough bibliographic details for an item to be found.
2. Students may not have written down complete or accurate information about an item their instructor wants them to read.
3. Instructors may have provided incomplete or inaccurate references to their students.
4. Many citations come from non standard sources which may not provide complete information e.g. the Internet, or from colleagues.

At this point in the cycle, staff may choose to double check that the item is not available in the library’s collection or via its electronic sources. If there was an error in the initial bibliographic information, rechecking the library’s catalogue under the corrected citation may reveal that the item is available. In addition, a patron may not be aware that an article is available over the Internet either at no charge or because the library subscribes to databases which include the fulltext of the item being sought.

Find locations
Once it is determined that the requested item is not available either physically or electronically at the local library, the next step is to find out where the item is located. Sometimes both the bibliographic verification and location steps can be done at the same time. For example, if either MAPLIN or AMICUS is used to check the citation for a book or other item, it tells you which library or libraries reported having a copy. Finding locations for requested items is much easier now. Web mounted catalogues, for even the smallest libraries, are not uncommon today.

Prepare and transmit request
By far the preferred method of transmitting interlibrary loan requests nowadays is electronic. If using an ILL module which is part of a bibliographic utility, such as A-G Canada, AMICUS or OCLC, it is important to follow the protocols laid down by the system. Where e-mail is unavailable, fax, telephone, and regular mail are options.

Receive and prepare materials for users
For items which need to be returned such as books, be sure to retain any special packaging so it can be used when returning the item. The basic rule for returns is to send the item back using the same method it was sent to you. If the item was sent by courier, for example, you would return it by courier. If it was insured, you have an obligation to return it insured. Note any restrictions such as “no photocopying,” or “in library use only”, and notify the patron the item has arrived. Be sure you make the patron aware of the due date for return to your library. Photocopies are normally kept by the patron.

Return borrowed items
The lending library sets the borrowing period and it is your responsibility as the borrowing library to adjust the time allotted for the patron to have the item to ensure it reaches the originating library on time. Sometimes it is possible to extend loan periods but such requests should be made well before the initial loan period is over. As mentioned earlier, items should be returned the same way they were sent, unless instructed to do otherwise.

Compile statistics
Statistical records of interlibrary loans need to be kept. Statistics are useful in assessing your collection’s strengths and weaknesses. Such statistics are commonly kept in electronic format, either using an in-house system or a commercial package.
Naturally not all attempts to borrow material will go smoothly. There will be times when you will not be able to verify an item. When this occurs it should be noted if you are passing it on to another source for fulfillment. Patrons will lose items, return them in damaged condition, or not return them on time. Your library should have policies to cover these and other snags in the system.

Interlibrary loan predates photocopier technology and thus the name derives from the fact that originally libraries lent actual books. This meant that copyright was not an issue as the original item was not copied, but rather lent and then returned. Although never free to operate, costs were minimal and volume low. Oftentimes ILL was only available at academic libraries and then only to academic staff and not to students. Using the mails and printed library catalogues, it might take weeks or months for a request to be located and received.

Nowadays, the interlibrary loan/document delivery section may be the most technologically advanced library department. Time is of the essence and demand is high. With end-user searching of databases and an extension of service to all client groups requests for interlibrary loans have swelled. The majority of items now asked for are journal articles, not books. This is occurring at a time when many libraries have cut their serials titles because of rising costs.

Costs are incurred in ILL in staff time, photocopying, postage, courier charges, telecommunication charges, fees charged by lending libraries, etc. Mary Jackson as part of the ILL/DD Performance Measures Study concluded that in research libraries “the average cost for a completed ILL transaction in 1996 was $27.83.” Many libraries choose to pass some of these costs on to their patrons. Others absorb the costs.

To recoup some of the costs of providing ILL/DD services, a number of libraries have imposed significant fees on interlibrary loan requests. The University of British Columbia, for example, charges community/alumni cardholders a minimum of $30.00 per request. Fees charged by Canadian libraries can be found in the National Library of Canada’s ILL Directory on the Web. A library’s Web site may also have information about any ILL/document delivery fees it charges. See for example, the Health Sciences Libraries page at the University of Manitoba Libraries at

As more and more libraries are requesting articles through ILL/DD the issue of copyright has gained in importance. Normally libraries do not lend entire issues of periodicals. Instead a photocopy of the requested article is made and sent to the “borrowing” library. In the vast majority of cases the copy of the article is “given” to the patron to keep, either free of charge, or for a fee as stated in the library’s ILL policy. The amount and use of photocopied material may be governed by copyright legislation. Remember that copyright varies from country to country. What is permissible under Canadian copyright legislation may not be permissible in the United States and vice versa.
New issues surrounding copyright have arisen as more and more documents are made available through contractual agreements with electronic fulltext database providers. If a library does not have an actual paper copy of an article in its collection but rather access to an electronic version for which it has paid, does it have the right to make a copy of it for another library which does not subscribe to the service? Elsevier Science a major publisher of scientific journals has responded to the issue by developing an interlibrary loan policy for its electronic files which is accessible at

Participating library types and locations
Interlibrary loan is not exclusive to any one type of library. All libraries can participate as long as they conform to the established codes. Traditionally, however, school libraries have been much less active in requesting interlibrary loans than other types of libraries. In Manitoba, teachers can borrow directly from the Instructional Resources Unit of the Department of Education, Citizenship and Youth which eliminates the need for many interlibrary loan requests.

ILL is an international service. It is not restricted to any one country. Normally, a library will request materials from within its own country’s borders before going abroad. In Manitoba, Public Library Services provides interlibrary loan back-up to all public libraries outside of Winnipeg. If a local library cannot find an item itself, it can forward the request to PLS which will use its resources to find locations. When local and provincial resources have been exhausted requests for specific items can be forwarded to the National Library of Canada. The National Library will lend items, or provide photocopies of articles if they are in its collection. If they do not have a copy they will supply locations for those libraries that do. For more detailed information on the role of the National Library in ILL check out its ILL policy at

Libraries are free to request loans from other libraries anywhere in the world as appropriate. Some libraries may not loan directly, but require you to go through a central agency. It is usually best to try and find an item locally first and then branch out further afield if it cannot be found. Borrowing from other countries may require currency conversion or the use of special coupons or vouchers to cover fees. Guidelines governing international lending can be found on the International Federation of Library Associations and Institutions (IFLA) Web page at

Document delivery
If instead of requesting items from another library, particularly in the case of periodical articles, a library chooses to purchase a copy from a document supplier, different procedures will be followed. Oftentimes speed is a factor in choosing a commercial document service over a library.
The Canada Institute for Scientific and Technical Information (CISTI) provides a fee-based document delivery system for articles in science, technology and medicine. It guarantees delivery of “urgent service documents” within two hours by fax or Ariel for an additional surcharge. Cinahl, a nursing database, offers access to most of the items it indexes for a fee with selected files available for immediate downloading. Ingenta is a major supplier of documents in all subject fields. It has a database which can be searched for free. Articles found in the database can be ordered immediately usually at a cost of less than $25.00 U.S. Information on other document delivery services is available on the Internet Library for Librarians at
Document delivery is most popular with special librarians which may have a relatively small staff but demanding patrons who need information quickly. Document delivery services may allow patrons to by-pass libraries entirely and order items directly from them as long as payment is guaranteed e.g. by credit card.

Interlibrary loan tools
Numerous finding aids are available to assist in both verifying and locating items. The World Wide Web in particular has been a boon to interlibrary loan. Two types of finding aides are particularly important, union catalogues and union lists.

An example of an online union list for periodicals is the Manitoba Library Association Serials List, which is accessible from

This listing does not include titles held in the University of Manitoba. It also does not provide “holding” information. Even though many listings are badly in need of updating, it is nevertheless of invaluable assistance in identifying serial titles held in a wide variety of Manitoba libraries. The Manitoba Union List of Serials (MULS) is a project sponsored by the Manitoba Library Consortium Inc. and is available only by telnet. It does not include holdings. For further information see
MAPLIN (currently unavailable for use) is an example of a union catalogue. It was initiated by Manitoba’s Public Library Services Branch and consists of records of items held in the province’s public libraries with the exclusion of Winnipeg Public Library. The Legislative Library’s holdings are also available on the system as are those of Public Library Services itself. An extended version of MAPLIN is MAPLIN Global which allows a metacrawler like search to over 100 libraries of all types worldwide.

The National Library of Canada is responsible for AMICUS, Canada’s national union catalogue of more than 24 million records, including serials’ titles and holdings information, from over 1,300 Canadian libraries. AMICUS may be searched for no charge and is available at

Another free service was the virtual Canadian union catalogue (vCuc), which was discontinued in 2006. vCuc was a gateway service to the catalogues of major Canadian libraries. Databases could be searched individually, as part of a group, or all at once. Libdex, formerly WebCats, is a worldwide directory of Web OPACs. It extends the ability to search in individual catalogues or catalogues of library consortia throughout the world. As part of its Canadian Library Gateway service, the National Library provides a “Browse Lists of Canadian Library Web Sites and Catalogues” at

A fee based bibliographic utility is A-G Canada which has a range of bibliographic and authority record databases along with holdings information contributed by their customers. Subscribing libraries can search for potential ILL locations in Canada.

OCLC is another fee-based bibliographic utility. It dominates the American library marketplace and is gaining more of a presence in Canada. OCLC facilitates interlibrary lending through its network of subscribing libraries throughout the world. WorldCat is the name of its online union catalogue. It contains records of all materials catalogued by OCLC member libraries including books, serial titles and, audiovisual materials. It is available through the FirstSearch database service.

Union lists, union catalogues, and bibliographic utilities, etc. use symbols or codes to identify which libraries have copies of items they record. The National Library of Canada is responsible for assigning symbols to Canadian libraries. The ILL Directory on the Web is the master list for Canadian library symbols and ILL policies. It contains OCLC symbols and WHO [A-G Canada] codes for libraries in Canada. A-G Canada’s WHO codes are also available in PDF format at
OCLC also maintains its own list of member symbols which can be found through the “Participating Institutions Search” page at

Interlibrary loan

Interlibrary loan is the process by which a library requests material from, or supplies materials to, another library…interlibrary loan includes the provision of reproductions as substitutes for loans of the original materials.

The purpose of interlibrary loan… is to obtain, upon request of a library user, materials not available in the user’s local library.

Interlibrary loan is a mutual relationship and libraries must be willing to supply materials as freely as they request materials.

Any materials, regardless of format, may be requested from another library. The supplying library determines whether the material can be provided (CLA/ASTED Interlibrary Loan Code).
Document delivery

Supplying a copy of an item which is retained by the requester, as opposed to the supply of a loan copy. It also includes the purchase of photocopies, usually of journal articles from suppliers (Keenan).

Typical ILL steps
Borrowing library:
  • patron asks for item not in library collection
  • staff double check to ensure item not in collection
  • staff verify bibliographic citation
  • staff identify locations of libraries/commercial services which have item and select one
  • request to borrow or obtain copy of item sent in approved format

Lending library:

  • determines if item circulates and is available on shelf
  • if available, original item or a copy of article or section sent to requesting library
  • if available, requesting library notified

Borrowing library:

  • on receipt, patron informed item available and if original sent, when it is due back
  • when original item returned, sent back to lending library

Both libraries:

  • perform necessary administrative chores, e.g. maintain record of transaction, process and pay bills, collect statistics

A link to a copy of the ALA ILL form to be used when requesting items by mail can be found at the bottom of the Web Page:


A document transmission system developed by the Research Libraries Group (RLG) which provides rapid, inexpensive, high-quality document delivery over the Internet by integrating scanning, sending, receiving, and printing functions. The user can send text and gray-scale images (illustrations, photographs, etc.) in letter, legal, and other sizes to another Ariel workstation, to an e-mail account used by another Ariel machine, or to anyone who uses MIME-compliant e-mail software and a multipage TIFF viewer. The Ariel system is used in libraries to facilitate interlibrary loan and document delivery service. (Reitz)

Bibliographic utility
An organization which provides access to and support for machine-readable bibliographic databases, directly to member libraries or through a network of regional bibliographic service centers, usually via a propriety interface. The largest bibliographic utility in the United States is OCLC. (Reitz)

The issues of a serial in the possession of a library. (ALA Glossary 112)

Interlibrary loan (ILL)
A transaction in which, upon request, one library lends an item from its collection or furnishes a copy of the item, to another library not under the same administration or on the same campus (ALA Glossary).

Intralibrary loan
A transaction in which one library lends an item from its collection to another library within the same library system upon request (ALA Glossary).

Reciprocal borrowing
The granting of borrowing privileges to the members of each other’s user groups by cooperating libraries (ALA Glossary 186).

Union catalog
A catalog of the holdings of more than one independent library, library system, or library collection, indicating the location of each item by the names or location symbols of the libraries or collections which own at least one copy (ALA Glossary).

Union list
A complete list of the holdings for a group of libraries of materials of a specific type (e.g. periodicals or annuals), on a certain subject, or in a particular field, compiled in the interests of library cooperation. The entry for each bibliographic item includes a list of codes to indicate the libraries that own at least one copy. Union lists are usually printed but some have been converted to online databases.

WHO code
Unique symbol originally assigned by UTLAS, then ISM and most recently A-G Canada to identify libraries participating in their bibliographic database.

Works cited
Bopp, Richard E., and Linda C. Smith. Reference and Information Services: An Introduction.3rd ed. Englewood, CO: Libraries Unlimited, 2001.

Canada. National Library. A Canadian Information Resource Sharing Strategy. Prepared by the Working Group to Review and Update the Canadian Information Resource Sharing Strategy. 10 May 1994. 23 Apr. 2002.

Canada. National Library. Interlibrary Loan Directory on the Web: Symbols and Interlibrary Loan Policies in Canada. 30 Apr. 1998. 23 Apr. 2002.

Canada. National Library. Resource sharing. 24 Oct. 2001. 23 Apr. 2002

Canadian Library Association. CLA/ASTED Interlibrary Loan Code. Ottawa: Canadian Library Association, 1996.

Jackson, Mary E. “Measuring the Performance of Interlibrary Loan and Document Delivery Services.” ARL: Bimonthly Newsletter of Research Library Issues and Actions. Dec. 1997 23 Apr. 2002

Keenan, Stella. Concise Dictionary of Library and Information Science. London: Bowker Saur, c. 1996. Lunau, Carrol D. “Canadian Resource Sharing at the Close of the 20th Century.” National Library News Nov. 1999. 26 Mar 2001.

“Interlibrary Loan Code for British Public Libraries”. The Quintessential Consultant. British Columbia Library Services Branch, Victoria. 13 Jan. 2004.

Reitz, Joan M. ODLIS: Online Dictionary of Library and Information Science. 2000. Western Connecticut State University Libraries. 23 Apr. 2002.

“Saskatchewan Resource Sharing.” PLEIS. Saskatchewan Provincial Library, Regina. 23 Apr 2002.

ALA. Interlibrary Loan Request Form.
*Scroll down the page to Interlibrary Loan and choose Interlibrary Loan Code for the United States

Choose to download the form in PDF or Word format

Monday, January 18, 2010

The Invisible web: uncovering sources search engines can't see

Sherman, Chris; Price, Gary. “The invisible web: uncovering sources search engines can’t see.” Library Trends. Vol. 52, No. 2, Fall 2003, pp. 282-298.;col1

Monday, January 11, 2010

Introduction to Reference: The Invisible Web

The Invisible Web, By: Young, Jr., Terrence E., Book Report, May/June 2002, Vol. 21, Issue 1.

Looking back over the past century, history tells us that no other technology has grown at such a phenomenal rate as the Internet. Certainly, young people’s comprehensive use of the Internet is one of the primary factors of that growth. We’d all be wealthy if we had a dollar for each time a student utters the phrase, “I need some information on (blank), or a picture of (blank). I need to get on the Internet.” In students’ minds, the Web is the be-all and end-all of information. And the Web certainly offers a plethora of resources. But those sources of information aren’t “catalogued” or “indexed” as are traditional resources.

To complicate matters further, students’ search skills are often rudimentary. The phrase “Enter Your Term in the Search Box” is the gateway to information on the Web, but many students have little idea of how to conduct a search beyond the basics of typing key words or phrases into that box. They don’t know how to use such features as quotation marks, parentheses, Boolean operators, or the advanced or power search functions to refine their searches. As a result, thousands of Web sites are retrieved, making the task of identifying a key resource a difficult and daunting one.

On the other hand, when school library media specialists search the Web for information, the returned results often identify a relevant Web page in the top 10 to 20 results. What’s the difference? Library media specialists subconsciously search the “invisible Web,” which usually isn’t familiar, or thus available, to students, and thus is not searched by students.

On beyond Google
Media specialists know that there’s a difference between searching Google, for instance, for government information, and searching Google for a site that enables you to search a remote database for government information. I wouldn’t use Google to search for the United States budget or primary source materials on U.S. culture and history; I would use a site such as FirstGov, the official site for U.S. Government Information (
), or The Library of Congress American Memory Collection (

The invisible parts of the “invisible Web” aren’t the sites themselves, but the contents of the databases that the sites are connected to. One of the major weaknesses of the popular search engines is that they don’t index an extremely large portion of the Web (often for technical reasons such as that a document doesn’t exist in HTML format). Consequently, current Web searching techniques miss a huge amount of information. Even more frustrating is the fact that the invisible Web’s resources are often of a higher quality than those of the visible Web’s.

Looking at the “invisible Web”
So what is the “invisible Web” (also called the “deep Web” or the “hidden Web”)? The invisible Web is made up of all kinds of specialized materials that are loaded with information. The vast majority of information available in digital form doesn’t reside directly on the Web, i.e. it doesn’t reside on the surface, or visible, Web. The huge amount of authoritative and current information contained in the invisible Web is accessible to you, but you have to know where to find it, because you can’t locate it using generic search engines, such as Vivisimo or Google. So how do you know whether there’s a database available to address your specific information need? Web indexes and directories remain the overarching search tools for finding databases on the Web.

The problem with that is, information in invisible Web databases (whether bibliographic, full-text, directory, products, statistics, or raw data) is generally inaccessible to the software “spiders” and “crawlers” that compile search engine indexes. This is because traditional search engines can only point to the front doors of databases, so to speak, they can’t easily index the information inside them. To make matters worse, PDF files, Flash files, and data stored in back-end databases also aren’t indexed. That’s crucial, because only information that’s been indexed can be searched for using general-purpose search engines. Data in un-indexed files and databases are searchable only from the individual Web site hosting the database, PDF, or Flash content. So, unless you can find that site, you can’t access that database, PDF, or Flash information.

The invisible Web is enormous, and its information is not only potentially valuable, it’s also multiplying faster than data on the surface, or visible, Web. How big is the surface, or visible Web? The Web Characterization Project (
) of the Online Computer Library Center (OCLC) Office of Research conducts an annual Web sample to analyze trendsin the size and content of the Web: The 32-bit Internet Protocol (IP) address space consists of 4,294,967,296 unique IP addresses.

To put this in some perspective, let’s discuss Google for a moment. It provides access to the world’s largest and most comprehensive collection of online information – more than two billion Web pages, images, and newsgroup messages. Google was the first search engine to translate Adobe PDF (portable document format) files into HTML and index them across the Web. In the fall of 2001 Google began indexing several additional file formats: Microsoft Word, Excel and PowerPoint formats, as well as Rich Text Format and PostScript files. A bracket label to the left of the document title identifies the file type of your search hits: [doc] for Word documents, [xls] for Excel spreadsheets, [ppt] for PowerPoint presentations, [rtf] for Rich Text Format, and [ps] for PostScript documents. You can focus your search to a specific file type either by adding “filetype:(extension)” to your search strategy or by using the Advanced Search function to select your preferred file type.

Yet, for all of this, Google offers access to fewer than half of the more than four billion existing IP addresses. While search engines continue to improve the number of sites they index, there’s a huge piece of the Web that simply isn’t accessible to the “robots” and “spiders” these engines use. Given this information, can you fathom the wealth of information contained in the invisible Web?

Learn more about the Invisible Web
Want to know more about the fascinating world of the Invisible Web? The Invisible Web: Uncovering Information Sources Search Engines Can’t See, by Chris Sherman and Gary Price, provides a detailed look at the nature of the hidden Web. It also offers pathfinders for accessing the valuable information contained in the deep Web and includes more than 1,000 invisible Web resources that the authors consider to be among the best.

You may also want to sign up for a free trial of BrightPlanet’s LexiBot (version 2) client search tool (
). LexiBot simultaneously searches 2,200 database and search engines that are considered to be part of the Invisible Web. IntelliSeek’s BullsEye software ( searches more than 1,000 professional search engines and speciality databases organised into more than 150 categories.

Information resources on the invisible web
Information within databases such as the American Memory primary source collections (
), the Geography Network’s statistical data ( , or the Social Science Data on the Internet ( out of the University of California, San Diego, provide us with accurate, reliable information. Besides supplying basic keyword search results, they also present prompts for further searches based on morphological or conceptual similarities. Users can choose search prompts in performing follow-up searches that are even more precisely defined, or for use in information retrieval with multiple search engines.

Ultimately, the invisible Web is another tool to meet the information needs of our users. To be most effective, however, it should be used in conjunction with our other information resources. One of most important of those resources is the media specialist – you.

Think of it this way: Your library media center collection is similar to the invisible Web. While your Online Public Access Catalog (OPAC) serves as a general-purpose search engine, the collection contains a wealth of information that isn’t searchable through the OPAC. As a media specialist, you’re the “invisible” gateway to that information; you function as the ultimate search engine into your collection. In the same way, effective use of the invisible Web requires school library media specialists to become familiar with the contents of the sties that grant entry into it, just as we’re familiar with the content of traditional reference resources and our library collections.

By Terrence E. Young Jr.
Terrence E. Young Jr. is a school library Media Specialist in New Orleans, Louisiana, an Adjunct Instructor of Library Science at the University of New Orleans, and editor of the NetWorth column in Knowledge Quest.

Specialized search tools
A lot of helpful information is locked away in databases that presently aren’t indexed by search engines. Several Web sites attempt to lead school library media specialists to the “invisible” content on the Web that’s often ignored by traditional search engines. Tap into this elusive section of the Web by using some of these special tools for accessing and searching:

Academic Info: Your Gateway to Quality Educational Resources
Academic Info is an educational gateway to online college-and research-level Internet resources. Its target audience is the college and university community, but the subject guides are useful to high school students.

Argus Clearinghouse

The Argus Clearinghouse provides a central access point for value-added topical guides that identify, describe, and evaluate Internet-based information resources.

Complete Planet

This site offers access to 90,000 searchable databases and speciality search engines organized into more than 7,000 subject headings.

Direct Search http://gwis2.circ.gwu/edu/~gprice/direct.htm

Direct Search is a growing compilation of links to the search interfaces of resources that contain data not easily or entirely searchable/accessible from general search tools such as Alta Vista, Google, or Hotbot. It provides annotated links to more than 1,000 searchable, interactive databases.


Geniusfind is a directory of thousand of search engines, databases, and archives organized into convenient categories and subcategories. These resources are specific to a certain topic and will greatly reduce the time it takes you to find exactly what you’re looking for.

IncyWincy: The Invisible Web Search Engine

This site constructs a database of more than 250,000 searches and functions in a directory of more than 2.7 million sites and 350,000 categories. A 45-day trial licence is available.

This site searches thousands of categories of information and makes it available in a directory that users can navigate, as well as search, to find answers from the targeted databases. The hit list consists of relevant sites that might have hidden information of use to the searcher.

Librarians’ Index to the Internet (LII)

LII is a searchable, annotated subject directory of more than 8,600 Internet resources selected and evaluated by librarians for their usefulness. LII is a reliable and efficient guide to Internet resources.

Pinakes: A Subject Launchpad

This site provides links to major subject gateways using a drop-down menu.

Search IQ

Billing itself as “the smartest tool for finding info on the Net,” Search IQ is a must-try site. The wealth of information is staggering and the search capabilities startling.

Monday, January 4, 2010

Introduction to reference: Invisible web

Invisible web content is considered to be information that isn’t clearly displayed or searchable on the Internet. This includes photos, images and sound files. These files aren’t apparently “there”, but really they are. Directories have found items on the invisible web, but otherwise it is hard to find this information.

The invisible web can be known by any of the following terms:
• deep web
• dark matter
• grey web
• opaque web
• private web
• propriety web

Propriety web describes the invisible web. The private web are Internet pages inaccessible by the public, search as intranets, or sites that require a company’s username.

What is the invisible web?
Material that general-purpose search engines either cannot or perhaps more importantly, will not include in their collections of webpages.
Sherman, Chris. “Navigating the invisible web.” Search Engine Watch 2001.

Why care about the invisible web?
• Invisible web is estimated to be 500 times greater than the visual web.
• Invisible web is the largest category of new information on the Internet. The commercial web is the largest growing area of the Internet. More paid Websites and databases are coming online, as well as indexes. Most of this information is also available in databases.
• Invisible web content contains highly relevant content.
• Total quality content of invisible web many times greater than that of the conventional Web
Bergman, Michael. “The Deep Web: Surfacing Hidden Value,” 2001.

Facts and figures – sizes
Visible or surface web
• >4 billion individual documents
• 19 terrabytes (Tb)
• 100% publicly available
• quality – well.... the user must decide that one.

Invisible or deep web
• >550 billion individual documents
• 750 terrabytes (Tb)
• 200,000 web sites
• 95% publicly available
• quality – 1000 to 2000 times greater

Bergman, Michael. “The Deep Web: Surfacing Hidden Value,” 2001.

Considering that these figures are 8 years old, the reader can multiply these numbers ten times over and not even get a estimate of how much of the Internet one can and cannot easily access today.

Why isn’t all content picked up?
• Restricted access
o Login, password protected
• Undiscovered sites
o No links to or from site
• Dynamic pages only produce results in response to a specific search request
• Robot.txt file attached (instructions to a spider not to index a site)
• Search engine may not like the URL used to retrieve the document. Most dynamic delivery mechanisms use the ? symbol.
• Most search engines will not read past the ? in that URL.

Types of invisible web content
• Newly added Web pages
o not yet found by spiders
• Sites use exclusionary tags
o Robots can ignore pages with a meta tag. Site owners can create a file called robots.txt which contains a set of rules for the robot telling it what not to index.
• Intranets (private networks)
• Web sites which generate dynamic pages based on user inquiry
o After the session is over the info may “disappear” e,g,
 MapQuest
 The Sock Calculator
• Sites requiring special software or hardware to access content
o e.g. flash, shockwave, java applets, video, sound e.g.
 Mapping History requires shockwave
 America’s Jazz Heritage requires RealAudio G2 player
o Word, PowerPoint, PDF, Excel, Postscript, Rich Text Format has been indexed by Google since at least 2003, but it still won’t pick up everything.
• Web sites free to the public but which require user to search within the site’s database(s) to find info
o e.g. Electronic Journal Miner
• sites requiring registration or login
o e.g. New York Times
• hybrid sites, some content free, some restricted
o e.g. Big Chalk
• Web sites requiring a subscription
o e.g. EBSCHost, Encyclopedia Americana

“When an indexing spider comes across a database, it’s as if it has run smack into the entrance of massive library with securely bolted doors. Spiders can record the library’s address, but can tell you nothing about the books, magazines or other documents it contains.”

• Databases form the largest part of the invisible web.
Distribution of Deep Web Sites by Content – From Bergman @

Finding info on the invisible web
• Use search engines, meta-search engines and Web directories
o to find hidden info from search engines in databases add word “database” to subject term to find database gateways
Strategies for success
• Keep current
o discussion list
o newsletters
o use a monitoring service
o use C.I. (current information) resources
• Search smart
o use meta search engines first
o look at first page of results only
o use bookmarks, pathfinders, other libraries
• Know what is available
o subject-specific discussion groups

Search tools for the invisible web
Virtual libraries

• Librarian’s Index to the Internet
• Digital Librarian
• WWW Virtual Library
• Infomine (directory)

Invisible search engines and directories
• ProFusion
• Complete Planet
• Invisible Web Directory
• Direct Search

Subject-specific databases
• Use a search engine to find databases
o e.g. +”civil engineering database” –library
o “civil engineering” NEAR database
• Use a directory to specialized databases
o Fossick

Selected subject-specific databases

Global Edge
Singing Fish
Speciality Search Engines & Invisible Web Resources

Invisible web tutorial
Invisible web: what it is, why it exists, how to find it, and its... finding information on the Internet: a tutorial