Jump to content

Xen

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by 2.154.64.90 (talk) at 08:17, 4 October 2018 (Software architecture). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Original author(s)Keir Fraser, Steven Hand, Ian Pratt, University of Cambridge Computer Laboratory
Developer(s)Linux Foundation
Initial release2003 (2003)
Stable release
4.10.1[1] / May 3, 2018; 6 years ago (2018-05-03)
Preview release
4.11rc6[2] / June 5, 2018; 6 years ago (2018-06-05)
Repository
TypeHypervisor
LicenseGNU GPL version 2
Websitexenproject.org

Xen Project (pronounced /ˈzɛn/) is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. It was developed by the University of Cambridge and is now being developed by the Linux Foundation with support from Intel.

The University of Cambridge Computer Laboratory developed the first versions of Xen. The Xen Project community develops and maintains Xen Project as free and open-source software, subject to the requirements of the GNU General Public License (GPL), version 2. Xen Project is currently available for the IA-32, x86-64 and ARM instruction sets.[citation needed]

Software architecture

Xen Project runs in a more privileged CPU state than any other software on the machine.

Responsibilities of the hypervisor include memory management and CPU scheduling of all virtual machines ("domains"), and for launching the most privileged domain ("dom0") - the only virtual machine which by default has direct access to hardware. From the dom0 the hypervisor can be managed and unprivileged domains ("domU") can be launched.[3]

The dom0 domain is typically a version of Linux or BSD. User domains may either be traditional operating systems, such as Microsoft Windows under which privileged instructions are provided by hardware virtualization instructions (if the host processor supports x86 virtualization, e.g., Intel VT-x and AMD-V),[4] or para-virtualized operating systems whereby the operating system is aware that it is running inside a virtual machine, and so makes hypercalls directly, rather than issuing privileged instructions.

Xen Project boots from a bootloader such as GNU GRUB, and then usually loads a paravirtualized host operating system into the host domain (dom0).

History

Xen originated as a research project at the University of Cambridge led by Ian Pratt, a senior lecturer in the Computer Laboratory, and his PhD student Keir Fraser. The first public release of Xen was made in 2003, with v1.0 following in 2004. Soon after, Pratt and Fraser along with other Cambridge alumni including Simon Crosby founded XenSource Inc. to turn Xen into a competitive enterprise product.

The Open Source Xen Project continued to be a supported by XenSource, then by Citrix following XenSource's acquisition in October 2007. This organization supports the development of the free software project and also sells enterprise versions of the software.

On October 22, 2007, Citrix Systems completed its acquisition of XenSource,[5] and the Xen Project moved to the xen.org domain. This move had started some time previously, and made public the existence of the Xen Project Advisory Board (Xen AB), which had members from Citrix, IBM, Intel, Hewlett-Packard, Novell, Red Hat, Sun Microsystems and Oracle. The Xen Advisory Board advises the Xen Project leader and is responsible for the Xen trademark,[6] which Citrix has freely licensed to all vendors and projects that implement the Xen hypervisor.[7]

Citrix uses the Xen brand itself for some proprietary products unrelated to Xen, including XenApp and XenDesktop.

On April 15, 2013, it was announced that the Xen Project was moved under the auspices of the Linux Foundation as a Collaborative Project.[8] The Linux Foundation launched a new trademark for "Xen Project" to differentiate the project from any commercial use of the older "Xen" trademark. A new community website was launched at xenproject.org[9] as part of the transfer. Project members at the time of the announcement included: Amazon, AMD, Bromium, CA Technologies, Calxeda, Cisco, Citrix, Google, Intel, Oracle, Samsung, and Verizon.[10] The Xen project itself is self-governing.[11]

Release history for Xen Project (upstream project)

Version Release date Notes
1.0 October 2, 2003[12][13]
2.0 November 5, 2004[14]
3.0 December 5, 2005[15][16]

The releases up to 3.0.4 also added:

3.1 May 18, 2007[20] Live migration for HVM guests, XenAPI
3.2 January 17, 2008[21] PCI passthrough and ACPI S3 standby mode for the host system.
3.3 August 24, 2008[22] Improvements for the PCI passthrough and the power management. Xen ARM hypervisor source code released for ARM CPU support
3.4 May 18, 2009[23] Contains a first version of the "Xen Client Initiative", shortly XCI.
4.0 April 7, 2010[24] Makes it possible to use a dom0 Linux kernel, which has been implemented by using PVOps. A Linux kernel of version 2.6.31 has been modified for this purpose, because the official Linux kernel actually does not support the usage as dom0 kernel (date July 2010).[25]
4.1 March 25, 2011[26] Some of the improvements: Support for more than 255 processors, better stability. Linux kernel v2.6.37 and onward support usage as dom0 kernel.[27]
4.2 September 8, 2012[28] XL became the default toolstack. Support for up to 4095 host processors and up to 512 guest processors.
4.3 July 9, 2013[29] Experimental ARM support. NUMA-aware scheduling. Support for Open vSwitch.
4.4 March 10, 2014[30] Solid libvirt support for libxl, new scalable event channel interface, hypervisor ABI for ARM declared stable, Nested Virtualization on Intel hardware.[31][32]
4.5 January 17, 2015[33] With 43 major new features, 4.5 includes the most updates in the project’s history.[33]
4.6 October 13, 2015[28] Focused on improving code quality, security hardening, enablement of security appliances, and release cycle predictability.[28]
4.7 June 24, 2016[34] Improved: security, live migrations, performances and workload. Hardware support (ARM and Intel Xeon).[35]
4.8.1 April 12, 2017[36]
4.9 June 28, 2017[37] Xen Project 4.9 Release Notes
4.10 December 12, 2017[38] Xen Project 4.10 Release Notes

Xen in Linux distributions and Linux upstream

Since version 3.0 of the Linux kernel, Xen support for dom0 and domU exists in the mainline kernel.[39]

Uses

Internet hosting service companies use hypervisors to provide virtual private servers. Amazon EC2 (since August 2006),[40] IBM SoftLayer,[41] Liquid Web, Fujitsu Global Cloud Platform,[42] Linode, OrionVM[43] and Rackspace Cloud use Xen as the primary VM hypervisor for their product offerings.[44]

Virtual machine monitors (also known as hypervisors) also often operate on mainframes and large servers running IBM, HP, and other systems.[citation needed] Server virtualization can provide benefits such as:

  • consolidation leading to increased utilization
  • rapid provisioning
  • dynamic fault tolerance against software failures (through rapid bootstrapping or rebooting)
  • hardware fault tolerance (through migration of a virtual machine to different hardware)
  • the ability to securely separate virtual operating systems
  • the ability to support legacy software as well as new OS instances on the same computer

Xen's support for virtual machine live migration from one host to another allows workload balancing and the avoidance of downtime.

Virtualization also has benefits when working on development (including the development of operating systems): running the new system as a guest avoids the need to reboot the physical computer whenever a bug occurs. Sandboxed guest systems can also help in computer-security research, allowing study of the effects of some virus or worm without the possibility of compromising the host system.

Finally, hardware appliance vendors may decide to ship their appliance running several guest systems, so as to be able to execute various pieces of software that require different operating systems.

Technology

Types of virtualization

Xen supports five different approaches to running the guest operating system: HVM (hardware virtual machine), HVM with PV drivers, PVHVM (paravirtualization with full hardware virtualization,[45] i.e. HVM with PVHVM drivers), PVH (PV in an HVM container) and PV (paravirtualization).[46]

Paravirtualization – modified guests

Xen supports a form of virtualization known as paravirtualization, in which guests run a modified operating system. The guests are modified to use a special hypercall ABI, instead of certain architectural features.

Through paravirtualization, Xen can achieve high performance even on its host architecture (x86) which has a reputation for non-cooperation with traditional virtualization techniques.[47][48]

Xen can run paravirtualized guests ("PV guests" in Xen terminology) even on CPUs without any explicit support for virtualization.

Paravirtualization avoids the need to emulate a full set of hardware and firmware services, which makes a PV system simpler to manage and reduces the attack surface exposed to potentially malicious guests.

On 32-bit x86, the Xen host kernel code runs in Ring 0, while the hosted domains run in Ring 1 (kernel) and Ring 3 (applications).

Hardware-assisted virtualization, allowing for unmodified guests

CPUs that support virtualization make it possible to support unmodified guests, including proprietary operating systems (such as Microsoft Windows). This is known as hardware-assisted virtualization, however in Xen this is known as hardware virtual machine (HVM).

HVM extensions provide additional execution modes, with an explicit distinction between the most-privileged modes used by the hypervisor with access to the real hardware (called "root mode" in x86) and the less-privileged modes used by guest kernels and applications with "hardware" accesses under complete control of the hypervisor (in x86, known as "non-root mode"; both root and non-root mode have Rings 0–3).

Both Intel and AMD have contributed modifications to Xen to support their respective Intel VT-x and AMD-V architecture extensions.[49] Support for ARM v7A and v8A virtualization extensions came with Xen 4.3.[50]

HVM extensions also often offer new instructions to support direct calls by a paravirtualized guest/driver into the hypervisor, typically used for I/O or other operations needing high performance. These allow HVM guests with suitable minor modifications to gain many of the performance benefits of paravirtualized I/O.

In current versions of Xen (up to 4.2) only fully virtualized HVM guests can make use of hardware support for multiple independent levels of memory protection and paging. As a result, for some workloads, HVM guests with PV drivers (also known as PV-on-HVM, or PVH) provide better performance than pure PV guests.

Xen HVM has device emulation based on the QEMU project to provide I/O virtualization to the virtual machines. The system emulates hardware via a patched QEMU "device manager" (qemu-dm) daemon running as a backend in dom0. This means that the virtualized machines see an emulated version of a fairly basic PC. In a performance-critical environment, PV-on-HVM disk and network drivers are used during normal guest operation, so that the emulated PC hardware is mostly used for booting.

Virtual machine migration

Administrators can "live migrate" Xen virtual machines between physical hosts across a LAN without loss of availability. During this procedure, the LAN iteratively copies the memory of the virtual machine to the destination without stopping its execution. The process requires a stoppage of around 60–300 ms to perform final synchronization before the virtual machine begins executing at its final destination, providing an illusion of seamless migration. Similar technology can serve to suspend running virtual machines to disk, "freezing" their running state for resumption at a later date.

Target processors

The Xen hypervisor has been ported to a number of processor families.

  • Intel: IA-32, IA-64 (before version 4.2[51]), x86-64
  • PowerPC: previously supported under the XenPPC project, no longer active after Xen 3.2[52]
  • ARM: previously supported under the XenARM project for older versions of ARM without virtualization extensions, such as the Cortex-A9. Currently[when?] supported since Xen 4.3 for newer versions of the ARM with virtualization extensions, such as the Cortex-A15.
  • MIPS: XLP832 experimental port[53]

Scalability

Xen can scale to 4095 physical CPUs, 256 VCPUs per HVM guest, 512 VCPUs per PV guest, 16 TB of RAM per host, and up to 1 TB of RAM per HVM guest or 512 GB of RAM per PV guest.[54]

Hosts

Xen can be shipped in a dedicated virtualization platform, such as Citrix XenServer Enterprise Edition (formerly XenSource's XenEnterprise).

Alternatively, Xen is distributed as an optional configuration of many standard operating systems. Xen is available for and distributed with:

Guests

Unix-like systems as guests

Guest systems can run fully virtualized (which requires hardware support), paravirtualized (which requires a modified guest operating system), or fully virtualized with paravirtualized drivers (PVHVM[64]).[65] Most operating systems which can run on PCs can run as a Xen HVM guest. The following systems can operate as paravirtualized Xen guests:

For an updated list differentiating PV and PVHVM support, see DomU Support for Xen.

Microsoft Windows systems as guests

Xen version 3.0 introduced the capability to run Microsoft Windows as a guest operating system unmodified if the host machine's processor supports hardware virtualization provided by Intel VT-x (formerly codenamed Vanderpool) or AMD-V (formerly codenamed Pacifica).

During the development of Xen 1.x, Microsoft Research, along with the University of Cambridge Operating System group, developed a port of Windows XP to Xen — made possible by Microsoft's Academic Licensing Program. The terms of this license do not allow the publication of this port, although documentation of the experience appears in the original Xen SOSP paper.[69]

James Harper and the Xen open-source community have started developing GPL'd paravirtualization drivers for Windows. These provide front-end drivers for the Xen block and network devices, and allow much higher disk and network performance for Windows systems running in HVM mode. Without these drivers all disk and network traffic has to be processed through QEMU-DM.[70]

Xen Management Consoles

Third-party developers have built a number of tools (known as Xen Management Consoles) to facilitate the common tasks of administering a Xen host, such as configuring, starting, monitoring and stopping of Xen guests. Examples include:

  • Web-based HyperVM
  • Web-based ConVirt
  • the OpenNebula cloud management toolkit
  • On openSUSE YaST and virt-man offer graphical VM management
  • Web-based Xen Orchestra
  • OpenStack natively supports Xen as a Hypervisor/Compute target
  • CloudStack also supports Xen as a Hypervisor
  • Novell's PlateSpin Orchestrate also manages Xen virtual machines for Xen shipping in SUSE Linux Enterprise Server.
  • XEN-SHELL2 a lightweight console based multi user Xen VM management

Commercial versions

The Xen hypervisor is covered by the GNU General Public Licence, so all of these versions contain a core of free software with source code. However, many of them contain proprietary additions.

Overview of Xen Projects releases in commercial distributions

Overview of Citrix XenServer Releases and related Xen Project versions
Commercial distribution version Release date of commercial version Xen Projects version Commercial release EOL date[73]
Citrix XenServer 6.0[74][75] 2011-09-30 4.1.1 2016-09-26
Citrix XenServer 6.1[76] 2012-09-28 4.1.3 2016-09-30
Citrix XenServer 6.2[76] 2013-06-18 4.1.5 2018-06-26
Citrix XenServer 6.2 SP1[76] 2013-12-16 4.1.5 2018-06-26
Citrix XenServer 6.5[76] 2015-01-13 4.4.1 2018-06-26
Citrix XenServer 6.5 SP1[76] 2015-05-12 4.4.1 2018-06-26
Citrix XenServer 7.0[77][78][79] 2016-05-24 4.6.1 2021-05-19
Citrix XenServer 7.1[80] 2017-02-23 4.7.1 2022-08-01
Citrix XenServer 7.4[76] 2018-02-27 4.7.4 2018-09-30

See also

References

  1. ^ "Xen Project 4.10.1 is available". Xenproject.org. May 3, 2018. Retrieved June 25, 2018.
  2. ^ "Xen 4.11 rc6". Xenproject.org. June 5, 2018. Retrieved June 25, 2018.
  3. ^ "Xen Overview". Retrieved April 5, 2015.
  4. ^ "OSCompatibility - Xen Project Wiki". Wiki.xenproject.org. February 8, 2007. Retrieved June 8, 2013.
  5. ^ "Citrix Systems » Citrix Completes Acquisition of XenSource". Citrix Systems. July 12, 2007.
  6. ^ "Trademark". Xen.org. Retrieved June 8, 2012.
  7. ^ "Trademark Policy" (PDF) (PDF). Xen.org. June 1, 2008. Retrieved June 8, 2013.
  8. ^ "Linux Foundation Project". LinuxFoundation.org. Retrieved May 3, 2013.
  9. ^ "XenProject.org Website". XenProject.org. Retrieved May 3, 2013.
  10. ^ "Linux Foundation Xen Project Members". XenProject.org. Retrieved May 3, 2013.
  11. ^ "Project Governance (Updated)". XenProject.org. Retrieved May 3, 2013.
  12. ^ "Xen:". SourceForge.net. October 2, 2003. Retrieved October 18, 2012.
  13. ^ Jonathan Corbet (October 2, 2003). "The first stable Xen release". Lwn.net. Retrieved October 18, 2012.
  14. ^ Jonathan Corbet (November 5, 2004). "Xen 2.0 released". Lwn.net. Retrieved October 18, 2012.
  15. ^ Jonathan Corbet (December 6, 2005). "Xen 3.0 released". Lwn.net. Retrieved October 18, 2012.
  16. ^ "XenSource: Press Releases". XenSource, Inc. December 10, 2005. Archived from the original on December 10, 2005. Retrieved October 18, 2012.
  17. ^ "AMD SVM Xen port is public". lists.xenproject.org. December 14, 2005. Retrieved June 8, 2013.
  18. ^ "[Xen-devel] Xen 3.0.3 released! - Xen Source". Lists.xenproject.org. October 17, 2006. Retrieved June 8, 2013.
  19. ^ "[Xen-devel] FW: Xen 3.0.4 released! - Xen Source". Lists.xenproject.org. December 20, 2006. Retrieved June 8, 2013.
  20. ^ "[Xen-devel] Xen 3.1 released! - Xen Source". Lists.xenproject.org. May 18, 2007. Retrieved June 8, 2013.
  21. ^ "Xen 3.2.0 Officially Released : VMblog.com - Virtualization Technology News and Information for Everyone". VMblog.com. Retrieved October 18, 2012.
  22. ^ "Xen 3.3.0 hypervisor ready for download - The H: Open Source, Security and Development". H-online.com. August 25, 2008. Archived from the original on March 14, 2012. Retrieved October 18, 2012.
  23. ^ "Xen.org Announces Release Of Xen 3.4 Hypervisor | Citrix Blogs". Community.citrix.com. May 18, 2009. Retrieved October 18, 2012.
  24. ^ "Virtualisation: Xen is looking to catch up by releasing version 4 - The H Open: News and Features". H-online.com. April 9, 2010. Archived from the original on March 14, 2012. Retrieved October 18, 2012.
  25. ^ "Xen 4.0 Datasheet" (PDF) (PDF). Xen.org. Retrieved October 18, 2012.
  26. ^ "Xen 4.1 releases –". Blog.xen.org. March 25, 2011. Retrieved October 18, 2012.
  27. ^ "XenParavirtOps - Xen Wiki". Wiki.xenproject.org. Retrieved June 8, 2013.
  28. ^ a b c "Best Quality and Quantity of Contributions in the New Xen Project 4.6 Release". Xenproject.org. October 13, 2015. Retrieved October 13, 2015.
  29. ^ "Xen 4.3 released! –". Blog.xen.org. July 9, 2013. Retrieved July 16, 2013.
  30. ^ "Xen 4.4 releases –". Blog.xen.org. March 10, 2014. Retrieved March 10, 2014.
  31. ^ "Xen Project 4.4 Release Notes". Wiki.xenproject.org. Retrieved March 10, 2014.
  32. ^ "Xen 4.4 Feature List". Wiki.xenproject.org. Retrieved March 10, 2014.
  33. ^ a b "Less is More in the New Xen Project 4.5 Release –". Blog.xen.org. January 15, 2015. Retrieved January 17, 2015.
  34. ^ "Xen Project 4.8.1 is available". Xenproject.org. April 12, 2017. Retrieved June 1, 2017.
  35. ^ "Xen Project 4.7 Feature List". Xen project. June 24, 2016.
  36. ^ "Xen Project 4.8.1 is available | Xen Project Blog". blog.xenproject.org. Retrieved February 19, 2018.
  37. ^ "What's New in the Xen Project Hypervisor 4.9". Retrieved April 26, 2018.
  38. ^ "What's New in the Xen Project Hypervisor 4.10". Retrieved April 26, 2018.
  39. ^ "Xen celebrates full dom0 and domU support in Linux 3.0 –". Blog.xen.org. May 30, 2011. Retrieved October 18, 2012.
  40. ^ https://aws.amazon.com/es/blogs/aws/amazon_ec2_beta/
  41. ^ "CloudLayer Computing vs. Amazon EC2" (PDF) (PDF). Retrieved April 5, 2015.
  42. ^ Suzanne Tindal (February 28, 2011). "Fujitsu's global cloud launches in Aus". ZDNet Australia.
  43. ^ "Xen Project - Guest VM Images - OrionVM PV-HVM Templates". April 1, 2012. Retrieved June 27, 2014.
  44. ^ "Cloud FAQ". Rackspace.com. September 13, 2011. Retrieved October 18, 2012.
  45. ^ "Choosing a virtualization mode (PV versus PVHVM)". Rackspace Support Network. Rackspace. January 12, 2016. Retrieved January 25, 2018.
  46. ^ Roger Pau Monne. "Xen virtualization on FreeBSD" (PDF) (PDF). Retrieved April 6, 2015.
  47. ^ Robin and Irvine, "Analysis of the Intel Pentium's Ability to Support a Secure Virtual Machine Monitor", 9th Usenix Security Symposium, 2000
  48. ^ Gil Neiger, Amy Santoni, Felix Leung, Dion Rodgers, Rich Uhlig. Intel Virtualization Technology: Software-only virtualization with the IA-32 and Itanium architectures, Intel Technology Journal, Volume 10 Issue 03, August 2006.
  49. ^ Extending Xen with Intel Virtualization Technology, intel.com
  50. ^ "The ARM Hypervisor - The Xen Project's Hypervisor for the ARM architecture". Retrieved April 6, 2015.
  51. ^ "Xen 4.2 Feature List - Xen". Wiki.xen.org. December 17, 2012. Retrieved January 22, 2014.
  52. ^ "XenPPC - Xen Wiki". New-wiki.xen.org. August 15, 2010. Retrieved January 22, 2014.
  53. ^ Mashable (September 4, 2012). "Porting Xen Paravirtualization to MIPS Architecture". Slideshare.net. Retrieved January 22, 2014.
  54. ^ "Xen Release Features - Xen Project". Wiki.xenproject.org. Retrieved October 18, 2012.
  55. ^ "AUR (en) - xen". Aur.archlinux.org. Retrieved April 12, 2018.
  56. ^ "Xen - ArchWiki". Wiki.archlinux.org. Retrieved April 12, 2018.
  57. ^ "Xen - FreeBSD Wiki". wiki.freebsd.org. Retrieved September 28, 2015.
  58. ^ "Xen". Wiki.gentoo.org. Retrieved April 12, 2018.
  59. ^ "NetBSD/xen". Netbsd.org. Retrieved June 8, 2013.
  60. ^ "XenDom0Kernels - Xen Wiki". Wiki.xenproject.org. November 8, 2011. Retrieved June 8, 2013.
  61. ^ "Xen in Qubes OS Security Architecture". xenp.org. Retrieved April 12, 2018.
  62. ^ "Xen dom0 support in Lucid - Kernel team discussions - ArchiveOrange". Web.archiveorange.com. Retrieved January 22, 2014.
  63. ^ "Xen - Community Ubuntu Documentation". Help.ubuntu.com. September 5, 2012. Retrieved October 18, 2012.
  64. ^ "PV on HVM". Wiki.xen.org. Retrieved April 12, 2018.
  65. ^ "Understanding the Virtualization Spectrum". Wiki.xenproject.org. Retrieved April 12, 2018.
  66. ^ "FreeBSD/Xen - FreeBSD Wiki". Wiki.freebsd.org. June 25, 2012. Retrieved October 18, 2012.
  67. ^ "FreeBSD 11.0-RELEASE Release Notes". The FreeBSD Documentation Project. September 22, 2016. Retrieved October 23, 2016.
  68. ^ "xen(4) - OpenBSD Manual Pages". Retrieved December 30, 2017.
  69. ^ "Xen and the Art of Virtualization" (PDF) (PDF). University of Cambridge SOSP'03 paper. Retrieved October 18, 2012.
  70. ^ The information on the XenSource wiki seems to be outdated, but newer information is here: "Open Source Windows Paravirtualization Drivers for Xen (GPLPV)". Jolokia Networks. Retrieved October 18, 2012.
  71. ^ Huawei to virtual world: Give us your desktops and no-one gets hurt
  72. ^ Crucible - Secure Embedded Virtualization
  73. ^ "Citrix Product Matrix". Citrix.com. Retrieved December 6, 2016.
  74. ^ "XenServer 6.0 is here! | Citrix Blogs". Citrix Blogs. September 30, 2011. Retrieved December 6, 2016.
  75. ^ "Hotfix XS60E061 - For XenServer 6.0". support.citrix.com. Retrieved December 6, 2016.
  76. ^ a b c d e f "XenServer Revision History Table". support.citrix.com. Retrieved December 6, 2016.
  77. ^ "XenServer 7 Launch - What you need to know | Citrix Blogs". Citrix Blogs. May 24, 2016. Retrieved December 6, 2016.
  78. ^ Citrix XenServer 7.0 Release Notes https://docs.citrix.com/content/dam/docs/en-us/xenserver/xenserver-7-0/downloads/xenserver-7-0-release-notes.pdf
  79. ^ "Hotfix XS70E019 - For XenServer 7.0". support.citrix.com. Retrieved December 6, 2016.
  80. ^ "XenServer 7.1 Release Notes".

Further reading