|Original author(s)||Inktank Storage (Sage Weil, Yehuda Sadeh Weinraub, Gregory Farnum, Josh Durgin, Samuel Just, Wido den Hollander)|
|Developer(s)||Red Hat, Intel, CERN, Cisco, Fujitsu, SanDisk, Canonical and SUSE|
17.2.3 "Quincy" / 29 July 2022
|Written in||C++, Python|
|Operating system||Linux, FreeBSD|
|Type||Distributed object store|
Ceph (pronounced //) is an open-source software-defined storage platform that implements object storage on a single distributed computer cluster and provides 3-in-1 interfaces for object-, block- and file-level storage. Ceph aims primarily for completely distributed operation without a single point of failure, scalability to the exabyte level, and to be freely available. Since version 12, Ceph does not rely on other filesystems and can directly manage HDDs and SSDs with its own storage backend BlueStore and can completely self reliantly expose a POSIX filesystem.
Ceph replicates data and makes it fault-tolerant, using commodity hardware and Ethernet IP and requiring no specific hardware support. The Ceph’s system offers disaster recovery and data redundancy through techniques such as replication, erasure coding, snapshots and storage cloning. As a result of its design, the system is both self-healing and self-managing, aiming to minimize administration time and other costs.
In this way, administrators have a single, consolidated system that avoids silos and collects the storage within a common management framework. Ceph consolidates several storage use cases and improves resource utilization. It also lets an organization deploy servers where needed.
- Cluster monitors (ceph-mon) that keep track of active and failed cluster nodes, cluster configuration, and information about data placement and global cluster state.
- Object storage devices (ceph-osd) that use a direct, journaled disk storage (named BlueStore, which since the v12.x release replaces the FileStore which would use a filesystem)
- Metadata servers (ceph-mds) that cache and broker access to inodes and directories inside a CephFS filesystem.
- HTTP gateways (ceph-rgw) that expose the object storage layer as an interface compatible with Amazon S3 or OpenStack Swift APIs
- Managers (ceph-mgr) that perform cluster monitoring, bookkeeping, and maintenance tasks, and interface to external monitoring systems and management (e.g. balancer, dashboard, Prometheus, Zabbix plugin)
All of these are fully distributed, and may run on the same set of servers. Clients with different needs can directly interact with different subsets of them.
Ceph does striping of individual files across multiple nodes to achieve higher throughput, similar to how RAID0 stripes partitions across multiple hard drives. Adaptive load balancing is supported whereby frequently accessed objects are replicated over more nodes. As of September 2017[update], BlueStore is the default and recommended storage type for production environments, which is Ceph's own storage implementation providing better latency and configurability than the filestore backend, and avoiding the shortcomings of the filesystem based storage involving additional processing and caching layers. The filestore backend is still considered useful and very stable; XFS used to be the recommended underlying filesystem type for production environments, while Btrfs was recommended for non-production environments. ext4 filesystems were not recommended because of resulting limitations on the maximum RADOS objects length. Even using BlueStore, XFS is used for a small partition of metadata.
Object storage S3
Ceph implements distributed object storage - BlueStore. RADOS gateway (ceph-rgw) expose the object storage layer as an interface compatible with Amazon S3.
These are often capacitive disks which are associated with Ceph's S3 object storage for use cases: Big Data (datalake), Backup & Archives, IOT, media, video recording, etc.
Ceph's software libraries provide client applications with direct access to the reliable autonomic distributed object store (RADOS) object-based storage system, and also provide a foundation for some of Ceph's features, including RADOS Block Device (RBD), RADOS Gateway, and the Ceph File System. In this way, administrators can maintain their storage devices as a unified system, which makes it easier to replicate and protect the data.
The "librados" software libraries provide access in C, C++, Java, PHP, and Python. The RADOS Gateway also exposes the object store as a RESTful interface which can present as both native Amazon S3 and OpenStack Swift APIs.
Ceph's object storage system allows users to mount Ceph as a thin-provisioned block device. When an application writes data to Ceph using a block device, Ceph automatically stripes and replicates the data across the cluster. Ceph's RADOS Block Device (RBD) also integrates with Kernel-based Virtual Machines (KVMs).
These are often fast disks (NVMe, SSD) which are associated with Ceph's block storage for use cases, including databases, virtual machines, data analytics, artificial intelligence, and machine learning.
"Ceph-RBD" interfaces with the same Ceph object storage system that provides the librados interface and the CephFS file system, and it stores block device images as objects. Since RBD is built on librados, RBD inherits librados's abilities, including read-only snapshots and revert to snapshot. By striping images across the cluster, Ceph improves read access performance for large block device images.
"Ceph-iSCSI" is a gateway which enables access to distributed, highly available block storage from any Microsoft Windows and VMWare vSphere server or client capable of speaking the iSCSI protocol. By using ceph-iscsi on one or more iSCSI gateway hosts, Ceph RBD images become available as Logical Units (LUs) associated with iSCSI targets, which can be accessed in an optionally load-balanced, highly available fashion.
Since all of ceph-iscsi configuration is stored in the Ceph RADOS object store, ceph-iscsi gateway hosts are inherently without persistent state and thus can be replaced, augmented, or reduced at will. As a result, Ceph Storage enables customers to run a truly distributed, highly-available, resilient, and self-healing enterprise storage technology on commodity hardware and an entirely open source platform.
The block device can be virtualized, providing block storage to virtual machines, in virtualization platforms such as Openshift, OpenStack, Kubernetes, OpenNebula, Ganeti, Apache CloudStack and Proxmox Virtual Environment.
File system storage
Ceph's file system (CephFS) runs on top of the same object storage system that provides object storage and block device interfaces. The Ceph metadata server cluster provides a service that maps the directories and file names of the file system to objects stored within RADOS clusters. The metadata server cluster can expand or contract, and it can rebalance the file system dynamically to distribute data evenly among cluster hosts. This ensures high performance and prevents heavy loads on specific hosts within the cluster.
Ceph's file storage is often associated with log collection, messaging, and file storage.
Ceph was initially created by Sage Weil for his doctoral dissertation, which was advised by Professor Scott A. Brandt at the Jack Baskin School of Engineering, University of California, Santa Cruz (UCSC), and sponsored by the Advanced Simulation and Computing Program (ASC), including Los Alamos National Laboratory (LANL), Sandia National Laboratories (SNL), and Lawrence Livermore National Laboratory (LLNL). The first line of code that ended up being part of Ceph was written by Sage Weil in 2004 while at a summer internship at LLNL, working on scalable filesystem metadata management (known today as Ceph's MDS). In 2005, as part of a summer project initiated by Scott A. Brandt and led by Carlos Maltzahn, Sage Weil created a fully functional file system prototype which adopted the name Ceph. Ceph made its debut with Sage Weil giving two presentations in November 2006, one at USENIX OSDI 2006 and another at SC'06.
After his graduation in autumn 2007, Weil continued to work on Ceph full-time, and the core development team expanded to include Yehuda Sadeh Weinraub and Gregory Farnum. On March 19, 2010, Linus Torvalds merged the Ceph client into Linux kernel version 2.6.34 which was released on May 16, 2010. In 2012, Weil created Inktank Storage for professional services and support for Ceph.
In April 2014, Red Hat purchased Inktank, bringing the majority of Ceph development in-house to make it a production version for enterprises with support (hotline) and continuous maintenance (new versions).
In October 2015, the Ceph Community Advisory Board was formed to assist the community in driving the direction of open source software-defined storage technology. The charter advisory board includes Ceph community members from global IT organizations that are committed to the Ceph project, including individuals from Red Hat, Intel, Canonical, CERN, Cisco, Fujitsu, SanDisk, and SUSE.
In November 2018, the Linux Foundation launched the Ceph Foundation as a successor to the Ceph Community Advisory Board. Founding members of the Ceph Foundation included Amihan, Canonical, China Mobile, DigitalOcean, Intel, OVH, ProphetStor Data Services, Red Hat, SoftIron, SUSE, Western Digital, XSKY Data Technology, and ZTE.
In March 2021, SUSE discontinued its Enterprise Storage product incorporating Ceph in favor of Longhorn. and the former Enterprise Storage website was updated stating "SUSE has refocused the storage efforts around serving our strategic SUSE Enterprise Storage Customers and are no longer actively selling SUSE Enterprise Storage."
|Name||Release||First release||End of
|Argonaut||Old version, no longer maintained: 0.48||July 3, 2012||First major "stable" release|
|Bobtail||Old version, no longer maintained: 0.56||January 1, 2013|
|Cuttlefish||Old version, no longer maintained: 0.61||May 7, 2013||ceph-deploy is stable|
|Dumpling||Old version, no longer maintained: 0.67||August 14, 2013||May 2015||namespace, region, monitoring REST API|
|Emperor||Old version, no longer maintained: 0.72||November 9, 2013||May 2014||multi-datacenter replication for the radosgw|
|Firefly||Old version, no longer maintained: 0.80||May 7, 2014||April 2016||erasure coding, cache tiering, primary affinity, key/value OSD backend (experimental), standalone radosgw (experimental)|
|Giant||Old version, no longer maintained: 0.87||October 29, 2014||April 2015|
|Hammer||Old version, no longer maintained: 0.94||April 7, 2015||August 2017|
|Infernalis||Old version, no longer maintained: 9.2.0||November 6, 2015||April 2016|
|Jewel||Old version, no longer maintained: 10.2.0||April 21, 2016||2018-06-01||Stable CephFS, experimental RADOS backend named BlueStore|
|Kraken||Old version, no longer maintained: 11.2.0||January 20, 2017||2017-08-01||BlueStore is stable|
|Luminous||Old version, no longer maintained: 12.2.0||August 29, 2017||2020-03-01|
|Mimic||Old version, no longer maintained: 13.2.0||June 1, 2018||2020-07-22||snapshots are stable, Beast is stable|
|Nautilus||Old version, no longer maintained: 14.2.0||March 19, 2019||2021-06-01|
|Octopus||Older version, yet still maintained: 15.2.0||March 23, 2020||2022-06-01|
|Pacific||Older version, yet still maintained: 16.2.0||March 31, 2021||2023-06-01|
|Quincy||Current stable version: 17.2.0||April 19, 2022||2024-06-01|
The name "Ceph" is an abbreviation of "cephalopod", a class of molluscs that includes the octopus. The name (emphasized by the logo) suggests the highly parallel behavior of an octopus and was chosen to associate the file system with "Sammy", the banana slug mascot of UCSC. Both cephalopods and banana slugs are molluscs.
- Distributed file system
- Distributed parallel fault-tolerant file systems
- Gfarm file system
- IBM General Parallel File System (GPFS)
- MapR FS
- Moose File System
- Parallel Virtual File System
- Quantcast File System
- Software-defined storage
- Comparison of distributed file systems
- "Ceph Community Forms Advisory Board". 2015-10-28. Archived from the original on 2019-01-29. Retrieved 2016-01-20.
- "v17.2.3 Quincy released".
- "GitHub Repository". GitHub.
- "FreeBSD Quarterly Status Report".
- "LGPL2.1 license file in the Ceph sources". GitHub. 2014-10-24. Retrieved 2014-10-24.
- Nicolas, Philippe (2016-07-15). "The History Boys: Object storage ... from the beginning". The Register.
- Jeremy Andrews (2007-11-15). "Ceph Distributed Network File System". KernelTrap. Archived from the original on 2007-11-17. Retrieved 2007-11-15.
- M. Tim Jones (2010-06-04). "Ceph: A Linux petabyte-scale distributed file system" (PDF). IBM. Retrieved 2014-12-03.
- "BlueStore". Ceph. Retrieved 2017-09-29.
- "BlueStore Migration". Retrieved 2020-04-12.
- "Ceph Manager Daemon — Ceph Documentation". docs.ceph.com. Archived from the original on June 6, 2018. Retrieved 2019-01-31. archive link Archived June 19, 2020, at the Wayback Machine
- Jake Edge (2007-11-14). "The Ceph filesystem". LWN.net.
- Sage Weil (2017-08-29). "v12.2.0 Luminous Released". Ceph Blog.
- "Hard Disk and File System Recommendations". ceph.com. Archived from the original on 2017-07-14. Retrieved 2017-06-26.
- "BlueStore Config Reference". Retrieved April 12, 2020.
- Sage Weil (2007-12-01). "Ceph: Reliable, Scalable, and High-Performance Distributed Storage" (PDF). University of California, Santa Cruz. Archived from the original (PDF) on 2017-07-06. Retrieved 2017-03-11.
- Gary Grider (2004-05-01). "The ASCI/DOD Scalable I/O History and Strategy" (PDF). University of Minnesota. Retrieved 2019-07-17.
- Dynamic Metadata Management for Petabyte-Scale File Systems, SA Weil, KT Pollack, SA Brandt, EL Miller, Proc. SC'04, Pittsburgh, PA, November, 2004
- "Ceph: A scalable, high-performance distributed file system," SA Weil, SA Brandt, EL Miller, DDE Long, C Maltzahn, Proc. OSDI, Seattle, WA, November, 2006
- "CRUSH: Controlled, scalable, decentralized placement of replicated data," SA Weil, SA Brandt, EL Miller, DDE Long, C Maltzahn, SC'06, Tampa, FL, November, 2006
- Sage Weil (2010-02-19). "Client merged for 2.6.34". ceph.newdream.net.
- Tim Stephens (2010-05-20). "New version of Linux OS includes Ceph file system developed at UCSC". news.ucsc.edu.
- Bryan Bogensberger (2012-05-03). "And It All Comes Together". Inktank Blog. Archived from the original on 2012-07-19. Retrieved 2012-07-10.
- Joseph F. Kovar (July 10, 2012). "The 10 Coolest Storage Startups Of 2012 (So Far)". CRN. Retrieved July 19, 2013.
- Red Hat Inc (2014-04-30). "Red Hat to Acquire Inktank, Provider of Ceph". Red Hat. Retrieved 2014-08-19.
- "Ceph Community Forms Advisory Board". 2015-10-28. Archived from the original on 2019-01-29. Retrieved 2016-01-20.
- "The Linux Foundation Launches Ceph Foundation To Advance Open Source Storage". 2018-11-12.
- "SUSE says tschüss to Ceph-based enterprise storage product – it's Rancher's Longhorn from here on out".
- "SUSE Enterprise Software-Defined Storage".
- Ceph.io — v16.2.0 Pacific released
- Ceph.io — v17.2.0 Quincy released
- M. Tim Jones (2010-05-04). "Ceph: A Linux petabyte-scale distributed file system". DeveloperWorks > Linux > Technical Library. Retrieved 2010-05-06.
- Jeffrey B. Layton (2010-04-20). "Ceph: The Distributed File System Creature from the Object Lagoon". Linux Magazine. Retrieved 2010-04-24.
- Carlos Maltzahn; Esteban Molina-Estolano; Amandeep Khurana; Alex J. Nelson; Scott A. Brandt; Sage Weil (August 2010). "Ceph as a scalable alternative to the Hadoop Distributed File System". ;login:. 35 (4). Retrieved 2012-03-09.
- Martin Loschwitz (April 24, 2012). "The RADOS Object Store and Ceph Filesystem". HPC ADMIN Magazine. Retrieved 2012-04-25.