||This article includes a list of references, but its sources remain unclear because it has insufficient inline citations. (January 2010)|
Web archiving is the process of collecting portions of the World Wide Web to ensure the information is preserved in an archive for future researchers, historians, and the public. Web archivists typically employ web crawlers for automated capture due to the massive size and amount of information on the Web. The largest web archiving organization based on a bulk crawling approach is the Internet Archive which strives to maintain an archive of the entire Web. The International Web Archiving Workshop (IWAW), begun in 2001, has provided a platform to share experiences and exchange ideas. The later founding of the International Internet Preservation Consortium (IIPC), in 2003, has greatly facilitated international collaboration in developing standards and open source tools for the creation of web archives. These developments, and the growing portion of human culture created and recorded on the web, combine to make it inevitable that more and more libraries and archives will have to face the challenges of web archiving. National libraries, national archives and various consortia of organizations are also involved in archiving culturally important Web content. Commercial web archiving software and services are also available to organizations who need to archive their own web content for corporate heritage, regulatory, or legal purposes.
- 1 Collecting the web
- 2 Methods of collection
- 3 Difficulties and limitations
- 4 Aspects of web curation
- 5 See also
- 6 References
- 7 External links
Collecting the web
Methods of collection
The most common web archiving technique uses web crawlers to automate the process of collecting web pages. Web crawlers typically access web pages in the same manner that users with a browser see the Web, and therefore provide a comparatively simple method of remote harvesting web content. Examples of web crawlers used for web archiving include:
There are numerous services that may be used to archive web resources "on-demand", using web crawling techniques.
Free services open for public use
- Archive.is – Free service which saves a page and all of its images. It can save Web 2.0 pages. Snapshots can be searched by URL wildcards.
- ArchiveBay – Allows users to submit a URL to be archived in the form of a screenshot. Users can browse all saved screenshots of a URL.
- freezePAGE snapshots – Free or subscription service. To preserve snapshots, requires login every thirty days for unregistered users, sixty days for registered users.
- megalodon.jp – Japanese on-demand archiver similar to WebCite. Snapshots can be searched by exact URL.
- mummify.it - An Y Combinator-backed startup. It allows users to create personal collections of archived snapshots. Only 50 on-demand archives can be taken for free.
- perma.cc – Service by the Harvard Law School Library in its early stage as of September 2013. Archived pages are saved for two years then have to be manually renewed.
- textmirror.net on-demand archiver using Lynx behind the scene. It saves only text from the web page, ignoring images and styles. It ignores robots.txt.
- WebCite – Free service specifically for scholarly authors, journal editors and publishers to permanently archive and retrieve cited Internet references. Snapshots can be searched by exact URL.
Enterprise and subscription services
- Aleph Archives, offers web archiving services for regulatory compliance and eDiscovery aimed to corporate (Global 500 market), legal and government industries.
- Archive-It, a subscription service which allows institutions to build, manage and search their own web archive.
- Archivethe.net, a shared web-archiving platform operated by Internet Memory Research, the spin-off of the Internet memory foundation (formerly European Archive Foundation). IM Website
- Compliance WatchDog by SiteQuest Technologies, a subscription service that archives websites and allows users to browse the site as it appeared in the past. It also monitors sites for changes and alerts compliance personnel if a change is detected.
- Hanzo Archives, provides web archiving, cloud archiving, and social media archiving software and services for e-discovery, information management, social enterprise content, Financial Industry Regulatory Authority, United States Securities and Exchange Commission, and Food and Drug Administration compliance, and corporate heritage. Hanzo is used by leading organizations in many industries, and national governmental institutions. Web archive access is on-demand in native format, and includes full-text search, annotations, redaction, archive policy and temporal browsing. Hanzo is integrated with leading electronic discovery applications and services.
- Iterasi, provides enterprise web archiving for compliance, litigation protection, e-discovery and brand heritage. For enterprise companies, financial organizations, government agencies and more.
- Nextpoint, offers an automated cloud-based, SaaS for marketing, compliance and litigation related needs including electronic discovery
- Patrina creates custom data management solutions for businesses needing to archive all electronically stored information (ESI) to satisfy record keeping guidelines of FINRA, FDA, SEC, SOX, FRCP and recent Dodd-Frank and CFTC regulations.
- PageFreezer, a subscription SaaS service to archive, replay and search websites, blogs, web 2.0, Flash and social media for marketing, eDiscovery and regulatory compliance with U.S. Food and Drug Administration (FDA), Financial Industry Regulatory Authority (FINRA), U.S. Securities and Exchange Commission, Sarbanes–Oxley Act Federal Rules of Evidence and records management laws. Archives can be used as legal evidence.
- Reed Archives, offers litigation protection, regulatory compliance and eDiscovery in the corporate, legal and government industries.
- SiteReplay, a subscription service. Captures screen-shots of pages, transactions and user journeys using "actual browsers" for regulatory compliance. Screen-shots can be viewed online.
- Smarsh Web Archiving is designed to capture, preserve and re-create the Web experience as it existed at any moment in time for e-discovery and regulatory compliance obligations. (Smarsh acquired Perpetually in May 2012)
- SQWatcher a cloud-based subscription service that combines site archiving (ability to surf the site as it existed in the past) and website analysis reporting necessary to keep a site optimized and healthy.
- The Web Archiving Service is a subscription service optimized for the academic environment guided by input from librarians, archivists and researchers. WAS provides topical browsing, change comparison and site-by-site control of capture settings and frequency. Developed and hosted by the University of California Curation Center at the California Digital Library.
- webEchoFS, offers a subscription service that was created exclusively to meet the needs of Financial Services companies subject advertising regulations associated with FINRA and the Investment Advisors Act.
- Website-Archive, a subscription service. Captures screen-shots of pages, transactions and user journeys using "actual browsers". Screen-shots can be viewed online. Uses Cloud Testing technology.
Database archiving refers to methods for archiving the underlying content of database-driven websites. It typically requires the extraction of the database content into a standard schema, often using XML. Once stored in that standard format, the archived content of multiple databases can then be made available using a single access system. This approach is exemplified by the DeepArc and Xinq tools developed by the Bibliothèque nationale de France and the National Library of Australia respectively. DeepArc enables the structure of a relational database to be mapped to an XML schema, and the content exported into an XML document. Xinq then allows that content to be delivered online. Although the original layout and behavior of the website cannot be preserved exactly, Xinq does allow the basic querying and retrieval functionality to be replicated.
Transactional archiving is an event-driven approach, which collects the actual transactions which take place between a web server and a web browser. It is primarily used as a means of preserving evidence of the content which was actually viewed on a particular website, on a given date. This may be particularly important for organizations which need to comply with legal or regulatory requirements for disclosing and retaining information.
A transactional archiving system typically operates by intercepting every HTTP request to, and response from, the web server, filtering each response to eliminate duplicate content, and permanently storing the responses as bitstreams. A transactional archiving system requires the installation of software on the web server, and cannot therefore be used to collect content from a remote website.
Difficulties and limitations
Web archives which rely on web crawling as their primary means of collecting the Web are influenced by the difficulties of web crawling:
- The robots exclusion protocol may request crawlers not access portions of a website. Some web archivists may ignore the request and crawl those portions anyway.
- Large portions of a web site may be hidden in the deep Web. For example, the results page behind a web form lies in the deep Web because most crawlers cannot follow a link to the results page.
- Crawler traps (e.g., calendars) may cause a crawler to download an infinite number of pages, so crawlers are usually configured to limit the number of dynamic pages they crawl.
However, it is important to note that a native format web archive, i.e., a fully browsable web archive, with working links, media, etc., is only really possible using crawler technology.
The Web is so large that crawling a significant portion of it takes a large amount of technical resources. The Web is changing so fast that portions of a website may change before a crawler has even finished crawling it.
Some web servers are configured to return different pages to web archiver requests than they would in response to regular browser requests. This is typically done to fool search engines into directing more user traffic to a website, and is often done to avoid accountability, or to provide enhanced content only to those browsers that can display it.
Not only must web archivists deal with the technical challenges of web archiving, they must also contend with intellectual property laws. Peter Lyman states that "although the Web is popularly regarded as a public domain resource, it is copyrighted; thus, archivists have no legal right to copy the Web". However national libraries in some countries may have a legal right to copy portions of the web under an extension of a legal deposit.
Some private non-profit web archives that are made publicly accessible like WebCite, the Internet Archive or the Internet Memory Foundation allow content owners to hide or remove archived content that they do not want the public to have access to. Other web archives are only accessible from certain locations or have regulated usage. WebCite cites a recent lawsuit against Google's caching, which Google won.
Aspects of web curation
Web curation, like any digital curation, entails:
- Certification of the trustworthiness and integrity of the collection content
- Collecting verifiable Web assets
- Providing Web asset search and retrieval
- Semantic and ontological continuity and comparability of the collection content
Thus, besides the discussion on methods of collecting the Web, those of providing access, certification, and organizing must be included. There are a set of popular tools that addresses these curation steps:
A suite of tools for Web Curation by International Internet Preservation Consortium:
- Heritrix - official website - collecting Web asset
- NutchWAX - search Web archive collections
- Wayback (Open source Wayback Machine) - search and navigate Web archive collections using NutchWax
- Web Curator Tool - Selection and Management of Web Collection
Other open source tools for manipulating web archives:
- WARC Tools - for creating, reading, parsing and manipulating, web archives programmatically
- Search Tools[dead link] - for indexing and searching full-text and metadata within web archives
Free but not open source tools also exists:
- WSDK - WARC Software Development Kit (WSDK) represents a set of simple, compact, and highly optimized Erlang modules to manipulate (create/read/write) the WARC ISO 28500:2009 file format.
- Archive site
- Common Crawl
- Digital preservation
- Library of Congress Digital Library project
- List of Web archiving initiatives
- Memento Project
- Minerva Initiative
- National Digital Information Infrastructure and Preservation Program
- Pandora Archive
- Portuguese Web Archive
- UK Web Archiving Consortium
- Virtual artifact
- Web crawling
- Brown, A. (2006). Archiving Websites: a practical guide for information management professionals. London: Facet Publishing. ISBN 1-85604-553-6.
- Brügger, N. (2005). Archiving Websites. General Considerations and Strategies. Aarhus: The Centre for Internet Research. ISBN 87-990507-0-6. Archived from the original on 2009-01-29.
- Day, M. (2003). "Preserving the Fabric of Our Lives: A Survey of Web Preservation Initiatives". Research and Advanced Technology for Digital Libraries: Proceedings of the 7th European Conference (ECDL): 461–472.
- Eysenbach, G. and Trudel, M. (2005). "Going, going, still there: using the WebCite service to permanently archive cited web pages". Journal of Medical Internet Research 7 (5): e60. doi:10.2196/jmir.7.5.e60. PMC 1550686. PMID 16403724.
- Fitch, Kent (2003). "Web site archiving — an approach to recording every materially different response produced by a website". Ausweb 03.
- Jacoby, Robert (August 19, 2010). "Archiving a Web Page". Archived from the original on 2011-01-03. Retrieved 23 October 2010.
- Lyman, P. (2002). "Archiving the World Wide Web". Building a National Strategy for Preservation: Issues in Digital Media Archiving.
- Masanès, J. (ed.) (2006). Web Archiving. Berlin: Springer-Verlag. ISBN 3-540-23338-5.
- Toyoda, M., Kitsuregawa, M. (2012). "The History of Web Archiving". Proceedings of the IEEE 100 (special centennial issue). doi:10.1109/JPROC.2012.2189920.
||This article's use of external links may not follow Wikipedia's policies or guidelines. (March 2014)|
- International Internet Preservation Consortium (IIPC) - International consortium whose mission is to acquire, preserve, and make accessible knowledge and information from the Internet for future generations
- International Web Archiving Workshop (IWAW) - Annual workshop that focuses on web archiving
- National Library of Australia, Preserving Access to Digital Information (PADI)
- Library of Congress - Web Archiving
- Internet Memory
- Web archiving bibliography - Lengthy list of web-archiving resources
- Julien Masanès, Bibliothèque Nationale de France - Towards continuous web archiving
- Comparison of web archiving services
- SWAT - Snappy Web Archiving Tool. A proof-of-concept software that archives web pages by harvesting all files and taking screenshots of each page. All META data is saved in XML (METS, PREMIS, MODS and ADDML).
- The UK Government Web Archive at The National Archives - Archive of UK central government websites
- The UK Web Archive provided by The British Library - Archive of selected websites of UK cultural, social and historical significance - archived with permission from content owners