Amazon S3

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search
Amazon S3
Type of site
Cloud storage
Available inEnglish
OwnerAmazon.com
Websiteaws.amazon.com/s3/
IPv6 supportYes
CommercialYes
RegistrationRequired (included in free tier layer)
LaunchedMarch 14, 2006; 12 years ago (2006-03-14)
Current statusActive

Amazon S3 or Amazon Simple Storage Service is a "simple storage service" offered by Amazon Web Services (AWS) that provides object storage through a web service interface.[1][2] Simply, Amazon S3 provides a web interface to easily store and retrieve data safely, in any amount, at any time, from anywhere, at a low cost.[3] Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its global e-commerce network.[4]

Amazon S3 can be employed to store any type of object which allows for uses like storage for Internet applications, backup and recovery, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.[4] In its service-level agreement, Amazon S3 guarantees 99.9% monthly uptime, which works out to less than 43 minutes of downtime per month.[5]

AWS launched Amazon S3 in the United States on March 14, 2006,[1][6] then in Europe in November 2007.[7]

Design[edit]

At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3.

Although Amazon Web Services (AWS) does not publicly provide the details of S3's technical design, Amazon S3 manages data with an object storage architecture[8] which aims to provide scalability, high availability, and low latency with 99.9% durability and between 99.95% to 99.99% availability (though there is no service-level agreement for durability).[4]

The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key. [9] Buckets can be managed using either the console provided by Amazon S3, programmatically using the AWS SDK, or with the Amazon S3 REST application programming interface (API). Objects can be managed using the AWS SDK or with the Amazon S3 REST API and can be up to five terabytes in size with two kilobytes of metadata.[10][11] Additionally, objects can be downloaded using the HTTP GET interface and the BitTorrent protocol.

Requests are authorized using an access control list associated with each bucket and object and support versioning, disabled by default,[12] and lifecycle management of objects.

Bucket names and keys are chosen so that objects are addressable using HTTP URLs:

  • http://s3.amazonaws.com/bucket/key
  • http://bucket.s3.amazonaws.com/key
  • http://bucket/key (where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)

S3 is usable to replace significant existing (static) web-hosting infrastructure due to the HTTP client accessible objects.[13] The Amazon AWS Authentication mechanism allows the bucket owner to create an authenticated URL with time-bounded validity. That is, someone can construct a URL that can be handed off to a third-party for access for a period such as 30 minutes, or 24 hours.

Every item in a bucket can also be served as a BitTorrent feed. The S3 store can act as a seed host for a torrent and any BitTorrent client can retrieve the file. This drastically reduces the bandwidth costs for the download of popular objects. While the use of BitTorrent does reduce bandwidth, AWS does not provide native bandwidth limiting and, as such, users have no access to automated cost control. This can lead to users on the "free-tier" S3 or small hobby users amassing dramatic bills. AWS representatives have previously stated that such a feature was on the design table from 2006 to 2010,[14] but in 2011 stated the feature is no longer in development.[15]

A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in later data mining operations.[16]

Amazon S3 Storage Classes[edit]

Amazon S3 offers four different storage classes designed for different use case depending on durability, availability and performance requirements.[17] Amazon S3 Standard and Reduced Redundancy Storage (RRS),[18] Amazon S3 Standard Infrequent Access (IA), Amazon S3 One Zone Infrequent Access and Amazon Glacier which is designed for data archiving.

  • Amazon S3 Standard is the default class.
  • Amazon S3 Standard Infrequent Access (IA) is used for less frequently accessed data. Typical use cases are backups and disaster recovery solutions. Costs are lower than the Amazon S3 Standard, but applies additional fees per gigabyte of data retrieved.
  • Amazon S3 Reduced Redundancy Storage (RRS) is designed for noncritical, reproducible data at lower levels of redundancy. It reduces the costs of storing data in a less fault-tolerance manner. It supports one facility fault instead of two, unlike the Amazon S3 Standard.[19] Typical use cases can be data that could be recreated in the case of data loss. Durability is claimed to be 99.99% in comparison with 99.999999999% of standard class.
  • Amazon Glacier is designed for long-term storage of data that is infrequently accessed and for which retrieval latency of minutes or hours are acceptable. Use cases for this may be as a status service, where other servers may not need to be checked so frequently.

Pricing[edit]

Amazon S3 pricing varies depending on the different S3 storage classes. Prices vary from storage usage, number of requests, and data transfers. At its inception, Amazon charged end users $0.15 per gigabyte-month, with additional charges for bandwidth used in sending and receiving data, and a per-request (get or put) charge.[20] On November 1, 2008, pricing moved to tiers where end users storing more than 50 terabytes receive discounted pricing.[21] As of July 2018, the price for the first 50 TB ranges from $0.019 to $0.0405 per gigabyte per month, depending on the choice of location for storage.[22][23]

Hosting websites[edit]

Amazon S3 provides the option to host static HTML websites with index document support and error document support.[24] This support was added as a result of user requests dating to at least 2006.[25] For example, suppose that Amazon S3 was configured with CNAME records to host "http://subdomain.example.com/". In the past, a visitor to this URL would find only an XML-formatted list of objects instead of a general landing page (e.g., index.html) to accommodate casual visitors. However, websites now hosted on S3 may designate a default page to display, and another page to display in the event of a partially invalid URL, such as a 404 error.

Notable users[edit]

There are various User Mode File System (FUSE)-based file systems for Unix-like operating systems (Linux, etc.) that can be used to mount an S3 bucket as a file system. Note that as the semantics of the S3 file system are not that of a POSIX file system, so the file system may not behave entirely as expected.[26]

  • Photo hosting service SmugMug has used S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.[27]
  • Apache Hadoop file systems can be hosted on S3, as its requirements of a file system are partially met by S3.[28] As a result, Hadoop can be used to run MapReduce algorithms on EC2 servers, reading data and writing results back to S3.
  • Netflix uses Amazon Web Services for their storage and compute operations with S3 being their system of record. Netflix implemented a tool, S3mper,[29] to address the limitations of eventual consistency that Amazon S3 provides.[30] S3mper stores the filesystem metadata: filenames, directory structure and permissions in Amazon DynamoDB.[31]
  • reddit is hosted on S3.[32]
  • Dropbox,[33] Bitcasa,[34] and Tahoe-LAFS-on-S3,[35] among others, use S3 for online backup and synchronization services. In 2016, Dropbox moved out from using Amazon S3 services and developed its own cloud server.[36][37]
  • Mojang hosts Minecraft game updates and player skins on S3.[38]
  • Tumblr, Formspring, and Pinterest host images on S3.
  • Swiftype's CEO has mentioned that the company uses S3.[39]
  • S3 was used in the past by some enterprises as a long term archiving solution, until Amazon Glacier was released in August 2012.[citation needed]
  • The API has become a popular method for object storage.[40] As a result, more and more applications have been built to natively support the S3 API.[41] This includes applications that write data to AWS S3, as well as to S3-compatible object stores:[42]
Type Company Name Product
Client Backup Haystack Software LLC Arq backup[43]
Client Backup CloudBerry Lab CloudBerry Backup[44]
Client Backup open-source Duplicati[45]
Client Backup Novosoft LLC Handy Backup[46]
File Browser Binarynights ForkLift
File Browser odrive odrive[47]
MySQL Backup Oracle MySQL Enterprise Backup
Oracle Database Backup Oracle Oracle Secure Backup Cloud Manager[48]
Server Backup Actifio Actifio[49]
Server Backup Atempo Digital Archive (ADA)[50]
Server Backup Commvault Commvault[51]
Server Backup Veritas NetBackup[52]
Server Backup Asigra Asigra Cloud Backup[53]
Server Backup Rubrik Rubrik[54]
Cloud Storage Wasabi Wasabi Hot Storage
Cloud Storage Gateway CTERA Networks C00 Series[55]
Cloud Storage Gateway Avere FXT Series[56]
Cloud Storage Gateway EMC CloudArray[57]
Cloud Storage Gateway Microsoft StorSimple[58]
Cloud Storage Gateway Nasuni NF Series[59]
Cloud Storage Gateway NetApp Altavault[60]
Cloud Storage Gateway Panzura Global File System[61]
Sync & Share Storage Made Easy SME
Hybrid Storage Cloudian Cloudian HyperStore[62]
Hybrid Storage NooBaa NooBaa Storage
On-Premises Storage OpenIO OpenIO SDS[63]
On-Premises Storage Pure Storage FlashBlade
On-Premises Storage Scality RING Storage[64]
Open Source Zenko.io Open Source S3 Server[65]

Amazon S3 logs[edit]

Amazon S3 allows users to enable or disable logging. If enabled, the logs are stored on Amazon S3 buckets which can then be analyzed. These logs contain useful information such as:

These logs can be analyzed and managed by using third-party tools such as S3Stat, Cloudlytics, Qloudstat, AWStats or Splunk.

S3 API and competing services[edit]

S3 API allows operations on different components of Amazon S3 solution such as Buckets, Objects, and the Service.[66]

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface; however, they are differentiated by their underlying technologies and supporting business models.[67] A cloud storage standard (like electrical and networking standards) enables competing service providers to design their services and clients using different parts in different ways yet still communicate and provide the following benefits:[68]

  1. Increase competition by providing a set of rules and a level playing field, encouraging market entry by smaller companies which might otherwise be precluded.
  2. Encourage innovation by cloud storage & tool vendors, & developers because they can focus on improving their own products and services instead of focusing on compatibility.
  3. Allow economies of scale in implementation (i.e., if a service provider encounters an outage or as clients outgrow their tools and need faster operating systems or tools, they can easily swap out solutions).
  4. Provide timely solutions for delivering functionality in response to demands of the marketplace (i.e., as business growth in new locations increases demand, clients can easily change or add service providers simply by subscribing to the new service).

Examples of competing S3 compliant storage implementations:

Amazon S3 tools[edit]

Amazon S3 provides an API for third-party developers. It describes various API operations, related request and response structures, and error codes.[75] The original AWS Console provides tools for managing and uploading files, but it is not capable of managing large buckets or editing files online.[76] Third-party websites like S3edit.com or software like Cloudberry Explorer and ForkLift can help edit files on S3.[77]

History[edit]

Amazon introduced S3 in 2006.[78][79]

Amazon S3 is reported to store more than 2 trillion objects as of April 2013.[80] This is up from 10 billion objects as of October 2007,[81] 14 billion objects in January 2008, 29 billion objects in October 2008,[21] 52 billion objects in March 2009,[82] 64 billion objects in August 2009,[83] and 102 billion objects in March 2010.[84]

See also[edit]

Notes[edit]

  1. ^ a b "Amazon Web Services Launches "Amazon S3"" (Press release). Amazon.com. 2006-03-14. Retrieved 2018-11-14.
  2. ^ Huang, Dijiang; Wu, Huijun (2017-09-08). Mobile Cloud Computing: Foundations and Service Models. Morgan Kaufmann. p. 67. ISBN 9780128096444.
  3. ^ "Amazon Simple Storage Service (S3) — Cloud Storage — AWS". Amazon Web Services, Inc. Retrieved 2018-11-14.
  4. ^ a b c "Cloud Object Storage - Store & Retrieve Data Anywhere - Amazon Simple Storage Service". Amazon Web Services, Inc.
  5. ^ 60 min/hour * 24 hours in a day * 30 days * 0.1% = 43.2 min
  6. ^ "5 Key Events in the history of Cloud Computing - DZone Cloud". dzone.com. Retrieved 2018-09-28.
  7. ^ "Amazon Web Services Offers European Storage for Amazon S3" (Press release). Amazon.com. 2007-11-06. Retrieved 2018-11-14.
  8. ^ "What is Cloud Object Storage? – AWS". Amazon Web Services, Inc. Retrieved 2018-07-09.
  9. ^ "Tech Blog » Starting Websphere in Cloud and saving the data in S3". techblog.aasisvinayak.com. Archived from the original on 2010-03-12.
  10. ^ "open-guides/og-aws". GitHub.
  11. ^ "Error Responses - Amazon Simple Storage Service". docs.aws.amazon.com.
  12. ^ "Introduction to Amazon S3 - Amazon Simple Storage Service". docs.aws.amazon.com.
  13. ^ "How to use Amazon S3 for Web Hosting". bucketexplorer.com. Archived from the original on 2008-04-08. Retrieved 2008-05-06.
  14. ^ AWS Developer Forums: Limit my own bandwidth?. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  15. ^ AWS Developer Forums: What is the status on the bill capping. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  16. ^ http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html Server Access Logging
  17. ^ "Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS". Amazon Web Services, Inc.
  18. ^ "Storage Classes - Amazon Simple Storage Service". docs.aws.amazon.com.
  19. ^ "Amazon S3 Reduced Redundancy Storage". Amazon Web Services, Inc.
  20. ^ "Amazon S3 Pricing". Amazon.com. 2009-02-05. Retrieved 2014-05-02.
  21. ^ a b "Amazon S3 - Busier Than Ever". Amazon.com. 2008-10-08. Archived from the original on 2008-10-11. Retrieved 2008-10-09.
  22. ^ "Cloud Storage Pricing - S3 Pricing by Region - Amazon Simple Storage Service". Amazon Web Services, Inc.
  23. ^ "Cloud Storage Pricing | S3 Pricing by Region | Amazon Simple Storage Service". Amazon Web Services, Inc. Retrieved 2018-08-29.
  24. ^ Amazon Simple Storage Service. Docs.amazonwebservices.com. Retrieved on 2013-08-09.
  25. ^ Garnaat, Mitch (19 Nov 2009). "Re: default key or 'default document' - is it possible to specify in S3?". Retrieved 21 Sep 2010.
  26. ^ "Comparison of S3QL and other S3 file systems". Retrieved 2012-06-29.
  27. ^ "Amazon S3: Show Me the Money". SmugMug Blog. SmugMug. November 10, 2006.
  28. ^ "Hadoop Filesystem Specification".
  29. ^ "S3mper: Consistency in the Cloud".
  30. ^ "Introduction to Amazon S3". Amazon. Retrieved 28 December 2017.
  31. ^ Hern, Alex (2017-02-02). "Amazon Web Services: the secret to the online retailer's future success". the Guardian. Retrieved 2018-04-23.
  32. ^ "AWS Case Study: reddit". aws.amazon.com. 2015. Retrieved March 18, 2015.
  33. ^ "Where are my files stored?". November 28, 2010.
  34. ^ "What is Tahoe-LAFS-on-S3?". August 21, 2012.
  35. ^ "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire". WIRED. Retrieved 2018-04-23.
  36. ^ "Dropbox saved almost $75 million over two years by building its own tech infrastructure". GeekWire. 2018-02-23. Retrieved 2018-04-23.
  37. ^ "Minecraft Beta 1.2_02". January 21, 2010.
  38. ^ "Swiftype Explains Their Cloud Stack". July 1, 2013.
  39. ^ Lelii, Sonia (23 September 2013). "Amazon S3 API for cloud storage leads pack, for now". TechTarget.com. Retrieved 31 May 2016.
  40. ^ Evans, Chris (12 January 2016). "Has S3 Become the De-Facto API Standard?". Architecting.it. Retrieved 31 May 2016.
  41. ^ Leopold, George (July 11, 2017). "Scality Targets Multi-Cloud Data Storage". Datanami news portal.
  42. ^ Sadun, Erica (6 November 2012). "Arq cloud backup adds low-cost Amazon Glacier support". www.engadget.com. Retrieved 31 May 2016.
  43. ^ Moran, Joe (1 December 2015). "Data Backup Software Review: CloudBerry Lab Backup 4.5". www.smallbusinesscomputing.com. Retrieved 31 May 2016.
  44. ^ Sanders, James (4 August 2014). "Securely back up personal files with Duplicati: Q&A with the open source client's creators". www.TechRepublic.com. Retrieved 31 May 2016.
  45. ^ Handy, Backup (17 April 2017). "Amazon S3 Backup Software for Cloud Backup". www.handybackup.net. Retrieved 12 September 2017.
  46. ^ Lohnash, Mike (19 June 2015). "Odrive Review: One Folder for All Your Clouds". www.BackupReview.com. Retrieved 31 May 2016.
  47. ^ "Oracle Database Backup To Cloud: Amazon Simple Storage Service (S3)" (PDF). Oracle.com. Retrieved 31 May 2016.
  48. ^ "Actifio on AWS Marketplace". "Backup & Replication Support". Actifio.com. Retrieved 31 May 2016.
  49. ^ "Atempo Digital Archive - for large scale NAS backup, migration and synchronization". Atempo Blog. Retrieved 2018-03-27.
  50. ^ "Cloud Storage Support". Commvault.com. Retrieved 31 May 2016.
  51. ^ "Veritas launches NetBackup 7.7 with emphasis on cloud backup". SearchDataBackup. Retrieved 2016-05-31.
  52. ^ "Asigra, Veeam remain top users' choice for backup applications". SearchDataBackup. Retrieved 2016-05-31.
  53. ^ "Startup Rubrik Aiming to Erase Backup, Recovery Software". www.eweek.com. Retrieved 2016-05-31.
  54. ^ Ibm; Emc; Netapp; Seagate; Hp; Hill, Seagate rolls storage kit for manufacturers down Dot. "CTERA Networks offers up in-cloud server backup". Spectralogic CTO talks up hybrid flash-tape cartridge. Welcome tape robot overlords and backup, CTERA Networks offers up in-cloud server. Retrieved 2016-05-31.
  55. ^ Mellor, Chris (7 October 2015). "Like a wedding cake: Avere unveils three-tier AWS cloud NAS". TheRegister.com. Retrieved 8 June 2016.
  56. ^ Armstrong, Adam (28 January 2015). "EMC CloudArray 5.0 Launched". StorageReview.com. Retrieved 8 June 2016.
  57. ^ Mackie, Kurt (1 June 2015). "Microsoft StorSimple Extends Cloud Support to AWS, OpenStack". RedmondChannelPartner.com. Retrieved 8 June 2016.
  58. ^ Mellor, Chris (14 May 2015). "Azure gives AWS the blues again in Nasuni cloud storage poll". TheRegister.com. Retrieved 8 June 2016.
  59. ^ Ramel, David (28 May 2015). "NetApp Introduces AltaVault for Cloud Backup". AWSInsider.net. Retrieved 8 June 2016.
  60. ^ Knuth, Gabe (26 May 2015). "Panzura explains their Global File System and how they can help you deploy XenDesktop from AWS". Retrieved 7 June 2016 – via BrianMadden.com.
  61. ^ "Cloudian deploys Amazon S3-compatible on-premises object storage, sold and metered in AWS Marketplace | Cloudian". Cloudian. Retrieved 2016-11-21.
  62. ^ "OpenIO Data Access Features". OpenIO. Retrieved 2018-11-16.
  63. ^ "Scality RING S3 Connector - Scality". Scality. Retrieved 2017-11-06.
  64. ^ Armstrong, Adam (11 July 2017). "Scality Releases Zenko A Multi-Cloud Controller". Storage Review.
  65. ^ "Amazon S3 REST API Introduction - Amazon Simple Storage Service". docs.aws.amazon.com.
  66. ^ Watters, Audrey. "Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)". SAY Media, Inc. Retrieved 19 December 2012.
  67. ^ Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council (1990). Crossroads of Information Technology Standards. Washington, DC:: The National Academies Press, 1990. pp. 36–37.
  68. ^ AWS S3 vs Azure vs Google storage market share: what we see. Filestack.com. Retrieved on 2018-07-017.
  69. ^ Ceph Object Gateway S3 API — Ceph Documentation. Ceph.com. Retrieved on 2013-08-09.
  70. ^ "Connectria Cloud Storage - Amazon S3® Compatible Cloud Storage Service". Connectria. Connectria. Retrieved 19 December 2012.
  71. ^ "Connectria Hosting Launches Cloud Storage Solution". Connectria Hosting. Archived from the original on 2012-11-02. Retrieved 19 December 2012.
  72. ^ DigitalOcean Spaces API - DigitalOcean Documentation. developers.DigitalOcean.com. Retrieved on 2017-10-30.
  73. ^ Riak CS. Basho (2013-01-18). Retrieved on 2013-08-09.
  74. ^ "Amazon Simple Storage Service (S3) Documentation". Amazon Web Services, Inc.
  75. ^ "AWS Management Console". Amazon Web Services, Inc.
  76. ^ "s3Edit - Online S3 File Editor, IDE on Browser". www.s3edit.com.
  77. ^ Overview of Amazon Web Services, 2018, https://d1.awsstatic.com/whitepapers/aws-overview.pdf
  78. ^ Garfinkel, Simson L. 2007. An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS. Harvard Computer Science Group Technical Report TR-08-07. https://dash.harvard.edu/bitstream/handle/1/24829568/tr-08-07.pdf?sequence=1
  79. ^ "Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services". typepad.com. 18 April 2013.
  80. ^ Vogels, Werner (2008-03-19). "Happy Birthday, Amazon S3!". All Things Distributed.
  81. ^ "Celebrating S3's Third Birthday With Special Anniversary Pricing - Amazon Web Services". typepad.com. 31 March 2009.
  82. ^ "Amazon's Head Start in the Cloud Pays Off". eweek.com.
  83. ^ "Amazon S3 Now Hosts 100 Billion Objects". datacenterknowledge.com. 9 March 2010.

References[edit]

External links[edit]