Cloud storage gateway
A cloud storage gateway is a network appliance or server which resides at the customer premises and translates cloud storage APIs such as SOAP or REST to block-based storage protocols such as iSCSI or Fibre Channel or file-based interfaces such as NFS or SMB.
The term "cloud storage gateway" is a rather new item. But the technology was built already in 1992 by a Belgian company called Filepool. The company was acquired by EMC Corporation in 2001. This system was sold under the brand name "Centera" and as the communication protocol was API based, they called it content-addressable storage (CAS). The Centera system was the first object storage array. One of the major problems of the system was the missing application support by developers. Accordingly, EMC Corporation developed an interface called CUA – Centera Universal Access: The first cloud storage gateway. The CUA was able to bridge between Cenetera API and NFS protocol. It was only possible to use the CUA in front of a Centera system - accordingly it was a proprietary device. content-addressable storage evolved later to object storage, but the problem still exists in missing applications for object storage – one of the main enablers for cloud storage gateways.
Modern applications (aka "cloud native applications") use network attached storage by means of REST and SOAP with hypertext transfer protocol on the protocol layer. The related storage is provided from arrays that offer these as object storage. Classic applications use network attached storage by means of Network File System NFS, iSCSI or Server Message Block SMB. To make use of all the advantages of object storage, existing applications need to be rewritten and new applications must be object storage aware, which is not the case by default. This problem is addressed by cloud storage gateways. They offer object storage via classic native storage protocols like Network File System NFS or Server Message Block SMB (and a very few offer iSCSI as well). As a rule of thumb you can now use classic applications with cloud native object storage by using cloud storage gateways.
In enterprise infrastructures NFS is mainly used by Linux systems whereas Windows systems are using SMB. Object storage needs data in form of objects rather than files. For all cloud storage gateways it is mandatory to cache the incoming files and destage them to object storage on a later step. The time of destaging is subject to the gateway and a policy engine allows functions like
- pinning = bind specific files to the cache and destage them only for mirroring purpose
- content based destaging = move only files with specific characteristics to object storage e.g. all MP3 files
- multi-cloud mirroring = mirror all files to two different object stores
- Least Recently use = fill the local cache to maximum, move all files to object storage and delete files in cache on a LRU algorithm
- encrypt prior of destage = files are encrypted on the cloud storage gateway and destaged to object storage in an encrypted form
- compress and / or dedupliation prior of destage = files are deduplicated and/or compressed prior of destaging
- backup data in a native backup format
Combinations of these functions are usual.
Nearly all object storage gateways support Amazon S3 protocol as a quasi-standard. Some offer as well Microsoft Azure Blob, Google Storage, Openstack SWIFT and a very few support the standardized API of SNIA Foundation - CDMI. Even most gateways support public cloud storage e.g. from Amazon or Microsoft as an object store, there are as well a lot of vendors that support private cloud storage as well – including off and on prem storage.
There are multiple variants to deploy such gateways – and some vendors support as well different variants as of their product line:
- bare metal hardware appliance
- software appliance supporting different hypervisors
- software on top of an operating system – aka FUSE based
Software appliances as well as FUSE-based gateways can be installed on public cloud infrastructures.
Cloud storage gateway avoid the need to change existing applications by providing a standard interface. You can make use of all advantages of object storage without rewritting your applications.
As well IT users are used to existing protocols – like SMB or NFS. They can make use of cloud storage with the advantage of still using their existing infrastructures (including e.g. Active Directory, LDAP integration, fileshare functions etc.).
Whereas cloud storage gateways initially covered a niche only, the got more attraction as of multi-cloud technologies. As an example: It is possible to run a cloud storage gateway in form of a software appliance on top of a public or private cloud infrastructure by offering docker volume drivers that enable containers for automatic provisioning of storage used by these containers in a consistent form. They are using the hypervisors disks as a cache only, but destabe data on least recently used algorithm to the underlying cloud storage.
The de facto standard for object storage is Amazon S3 – it had the most popularity and capacity installed on object storage. But every object storage vendor can (and most of them do) offer Amazon S3 storage – even there is no real "standard" S3 API: Every vendor is a little bit different in implementing S3 API (as you can see from the different cloud storage gateway vendors supporting the "specific" APIs of the different object storage vendors). Since 2018 we see more and more cloud storage gateways that hide this complexity by offering S3 on northbound (as of networking technologies, southbound relates to the storage used by a gateway, whereas northbound is the storage provided by the gateway). As such you can make use of a richer S3 implementation on northbound than the southbound supports (examples are Zenko Cloud Controller, Opendedup cloud storage gateway and Hitachi Content Platform).
By using cloud storage gateways the complexity to use object storage is hidden, but that also hides some of the advantages of object storage:
- the ability of horizontal scaling
- ability to add high efficient metadata to the data content
- to use extended WORM and archiving capabilities of object storage
As applications change to cloud-aware applications (aka called cloud native applications), cloud storage gateways will change from multiprotocol gateways to multi-cloud gateways, providing access to multiple cloud providers as well as multiple southbound protocols and act as relays between different clouds.
One analyst predicted in 2013 that the cloud storage gateway market might reach US$860 million by 2016. In January 2013, some vendors were CTERA Networks, Emulex, Panzura, Riverbed Technology, Seven10, Sonian Inc., StorSimple, Avere Systems and TwinStrata.
Amazon has a similar function but for AWS S3 only – the "Amazon Storage Gateway"
A good overview is provided by Gartner corporation
- Gary Orenstein (22 June 2010). "GigaOm, Show Me the Gateway — Taking Storage to the Cloud". Retrieved 18 July 2013.
- Boles, Jeff. Cloud file storage pros and cons. 15 March 2011
- Stanley Zaffos, Adam W. Couture. Hybrid Cloud Gateway Appliances Expand Cloud Storage Use Cases. Gartner Group Research G00209453. http://storageswiss.com/2014/05/09/hybrid-cloud-storage-vs-the-enterprise/10 January 2011
- Pearson, Tony. Inside System Storage.
- SNIA. "Object Storage – Understanding the What, How and Why" (PDF).
- "5 Ways cloud technology improves your business | Austin Technology | Perth". Austin Technology. 2017-02-10. Retrieved 2019-04-16.
- "Cloud Data Management Interface".
- Kovacs, Gali. "How to Mount Amazon S3 Buckets as a Local Drive". cloud.netapp.com. Retrieved 2019-04-11.
- Columbus, Louis. "Roundup Of Cloud Computing Forecasts, 2017". Forbes. Retrieved 2019-04-16.
- Sliwa, Carol. New hybrid cloud storage appliances attract attention for primary data. May 2011.
- "Hybrid Cloud Storage vs. The Enterprise".
- Gartner (2019). "Product Overview on Cloud Storage Gateways".