Amazon S3: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
m Reverted edits by Hafsagull (talk): addition of unnecessary/inappropriate external links (HG) (3.4.12)
m Group related sections under a parent section for better structure
Line 20: Line 20:
Amazon S3 along with [[Amazon Elastic Compute Cloud|EC2]] drive AWS, the most profitable division under the entire Amazon company.<ref name=":0" /> Amazon S3 is reported to store more than 2 trillion objects {{as of|2013|04|lc=on}}.<ref name="aws.typepad.com">{{cite web|url=http://aws.typepad.com/aws/2013/04/amazon-s3-two-trillion-objects-11-million-requests-second.html|title=Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services|date=18 April 2013|website=typepad.com}}</ref>-->
Amazon S3 along with [[Amazon Elastic Compute Cloud|EC2]] drive AWS, the most profitable division under the entire Amazon company.<ref name=":0" /> Amazon S3 is reported to store more than 2 trillion objects {{as of|2013|04|lc=on}}.<ref name="aws.typepad.com">{{cite web|url=http://aws.typepad.com/aws/2013/04/amazon-s3-two-trillion-objects-11-million-requests-second.html|title=Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services|date=18 April 2013|website=typepad.com}}</ref>-->


== Design ==
== Technical details ==
=== Design ===
Amazon S3 manages data with an [[object storage]] architecture<ref>{{Cite web |url=https://aws.amazon.com/what-is-cloud-object-storage/ |title=What is Cloud Object Storage? – AWS |date=2019-10-16 |website=Amazon Web Services, Inc. |language=en-US |url-status=live |archive-url=https://web.archive.org/web/20180920181620/https://aws.amazon.com/what-is-cloud-object-storage/# |archive-date=2018-09-20 |access-date=2018-07-09}}</ref> which aims to provide [[scalability]], [[high availability]], and [[low latency]] with high [[Durability (database systems)|durability]].<ref name=":1" /> The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key.<ref>{{cite web|url=http://techblog.aasisvinayak.com/starting-websphere-in-cloud-and-saving-the-data-in-s3/|title=Tech Blog » Starting Websphere in Cloud and saving the data in S3|website=techblog.aasisvinayak.com|archive-url=https://web.archive.org/web/20100312080332/http://techblog.aasisvinayak.com/starting-websphere-in-cloud-and-saving-the-data-in-s3|archive-date=2010-03-12|url-status=dead}}</ref> Buckets can be managed using the console provided by Amazon S3, programmatically with the AWS [[Software development kit|SDK]], or the [[Representational State Transfer|REST]] application programming interface. Objects can be up to five [[terabyte]]s in size.<ref>{{cite web|url=https://github.com/open-guides/og-aws/blob/master/README.md#s3|title=open-guides/og-aws|website=GitHub|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180103123141/https://github.com/open-guides/og-aws/blob/master/README.md#s3#s3|archive-date=2018-01-03|url-status=live}}</ref><ref>{{cite web|url=https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html|title=Error Responses - Amazon Simple Storage Service|website=docs.aws.amazon.com|access-date=2018-05-21|archive-url=https://web.archive.org/web/20171224010324/http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#|archive-date=2017-12-24|url-status=live}}</ref> Requests are authorized using an [[access control list]] associated with each object bucket and support [[Versioning file system|versioning]]<ref>{{cite web | url=https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html | title=Using versioning in S3 buckets - Amazon Simple Storage Service }}</ref> which is disabled by default.<ref>{{cite web|url=https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html|title=Introduction to Amazon S3 - Amazon Simple Storage Service|website=docs.aws.amazon.com|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180512235723/https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#|archive-date=2018-05-12|url-status=live}}</ref> Since buckets are typically the size of an entire file system mount in other systems, this access control scheme is very coarse-grained. In other words, unique access controls cannot be associated with individual files.{{cn|date=April 2022}} Amazon S3 can be used to replace static [[web hosting service|web-hosting]] infrastructure with HTTP client-accessible objects,<ref name="S3 Web Hosting">{{cite web|url=http://www.bucketexplorer.com/documentation/amazon-s3--how-to-use-Amazon-s3-for-web-hosting.html|title=How to use Amazon S3 for Web Hosting|website=bucketexplorer.com|access-date=2008-05-06|archive-url=https://web.archive.org/web/20080408024708/http://www.bucketexplorer.com/documentation/amazon-s3--how-to-use-Amazon-s3-for-web-hosting.html|archive-date=2008-04-08|url-status=dead}}</ref> index document support and error document support.<ref>[http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?WebsiteHosting.html Amazon Simple Storage Service] {{Webarchive|url=https://web.archive.org/web/20110220042555/http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?WebsiteHosting.html# |date=2011-02-20 }} Docs.amazonwebservices.com. Retrieved on 2013-08-09.</ref>
Amazon S3 manages data with an [[object storage]] architecture<ref>{{Cite web |url=https://aws.amazon.com/what-is-cloud-object-storage/ |title=What is Cloud Object Storage? – AWS |date=2019-10-16 |website=Amazon Web Services, Inc. |language=en-US |url-status=live |archive-url=https://web.archive.org/web/20180920181620/https://aws.amazon.com/what-is-cloud-object-storage/# |archive-date=2018-09-20 |access-date=2018-07-09}}</ref> which aims to provide [[scalability]], [[high availability]], and [[low latency]] with high [[Durability (database systems)|durability]].<ref name=":1" /> The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key.<ref>{{cite web|url=http://techblog.aasisvinayak.com/starting-websphere-in-cloud-and-saving-the-data-in-s3/|title=Tech Blog » Starting Websphere in Cloud and saving the data in S3|website=techblog.aasisvinayak.com|archive-url=https://web.archive.org/web/20100312080332/http://techblog.aasisvinayak.com/starting-websphere-in-cloud-and-saving-the-data-in-s3|archive-date=2010-03-12|url-status=dead}}</ref> Buckets can be managed using the console provided by Amazon S3, programmatically with the AWS [[Software development kit|SDK]], or the [[Representational State Transfer|REST]] application programming interface. Objects can be up to five [[terabyte]]s in size.<ref>{{cite web|url=https://github.com/open-guides/og-aws/blob/master/README.md#s3|title=open-guides/og-aws|website=GitHub|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180103123141/https://github.com/open-guides/og-aws/blob/master/README.md#s3#s3|archive-date=2018-01-03|url-status=live}}</ref><ref>{{cite web|url=https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html|title=Error Responses - Amazon Simple Storage Service|website=docs.aws.amazon.com|access-date=2018-05-21|archive-url=https://web.archive.org/web/20171224010324/http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#|archive-date=2017-12-24|url-status=live}}</ref> Requests are authorized using an [[access control list]] associated with each object bucket and support [[Versioning file system|versioning]]<ref>{{cite web | url=https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html | title=Using versioning in S3 buckets - Amazon Simple Storage Service }}</ref> which is disabled by default.<ref>{{cite web|url=https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html|title=Introduction to Amazon S3 - Amazon Simple Storage Service|website=docs.aws.amazon.com|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180512235723/https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#|archive-date=2018-05-12|url-status=live}}</ref> Since buckets are typically the size of an entire file system mount in other systems, this access control scheme is very coarse-grained. In other words, unique access controls cannot be associated with individual files.{{cn|date=April 2022}} Amazon S3 can be used to replace static [[web hosting service|web-hosting]] infrastructure with HTTP client-accessible objects,<ref name="S3 Web Hosting">{{cite web|url=http://www.bucketexplorer.com/documentation/amazon-s3--how-to-use-Amazon-s3-for-web-hosting.html|title=How to use Amazon S3 for Web Hosting|website=bucketexplorer.com|access-date=2008-05-06|archive-url=https://web.archive.org/web/20080408024708/http://www.bucketexplorer.com/documentation/amazon-s3--how-to-use-Amazon-s3-for-web-hosting.html|archive-date=2008-04-08|url-status=dead}}</ref> index document support and error document support.<ref>[http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?WebsiteHosting.html Amazon Simple Storage Service] {{Webarchive|url=https://web.archive.org/web/20110220042555/http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?WebsiteHosting.html# |date=2011-02-20 }} Docs.amazonwebservices.com. Retrieved on 2013-08-09.</ref>
The Amazon AWS authentication mechanism allows the creation of authenticated URLs, valid for a specified amount of time. Every item in a bucket can also be served as a [[BitTorrent (protocol)|BitTorrent]] feed. The Amazon S3 store can act as a seed host for a [[Torrent file|torrent]] and any BitTorrent client can retrieve the file. This can drastically reduce the bandwidth cost for the download of popular objects. A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in [[data mining]] operations.<ref>http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html {{Webarchive|url=https://web.archive.org/web/20141223092055/http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html# |date=2014-12-23 }} Server Access Logging</ref> There are various [[Filesystem in Userspace|User Mode File System (FUSE)]]–based file systems for Unix-like operating systems (for example, [[Linux]]) that can be used to mount an S3 bucket as a file system. The semantics of the Amazon S3 file system are not that of a [[POSIX]] file system, so the file system may not behave entirely as expected.<ref>{{cite web
The Amazon AWS authentication mechanism allows the creation of authenticated URLs, valid for a specified amount of time. Every item in a bucket can also be served as a [[BitTorrent (protocol)|BitTorrent]] feed. The Amazon S3 store can act as a seed host for a [[Torrent file|torrent]] and any BitTorrent client can retrieve the file. This can drastically reduce the bandwidth cost for the download of popular objects. A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in [[data mining]] operations.<ref>http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html {{Webarchive|url=https://web.archive.org/web/20141223092055/http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html# |date=2014-12-23 }} Server Access Logging</ref> There are various [[Filesystem in Userspace|User Mode File System (FUSE)]]–based file systems for Unix-like operating systems (for example, [[Linux]]) that can be used to mount an S3 bucket as a file system. The semantics of the Amazon S3 file system are not that of a [[POSIX]] file system, so the file system may not behave entirely as expected.<ref>{{cite web
Line 31: Line 32:
}}</ref>
}}</ref>


==Amazon S3 storage classes==
=== Amazon S3 storage classes ===
Amazon S3 offers eight different storage classes with different levels of durability, availability, and performance requirements.<ref>{{cite web|url=https://aws.amazon.com/s3/storage-classes/|title=Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS|website=Amazon Web Services, Inc.|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180613200647/https://aws.amazon.com/s3/storage-classes/#|archive-date=2018-06-13|url-status=live}}</ref>
Amazon S3 offers eight different storage classes with different levels of durability, availability, and performance requirements.<ref>{{cite web|url=https://aws.amazon.com/s3/storage-classes/|title=Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS|website=Amazon Web Services, Inc.|access-date=2018-05-17|archive-url=https://web.archive.org/web/20180613200647/https://aws.amazon.com/s3/storage-classes/#|archive-date=2018-06-13|url-status=live}}</ref>


Line 45: Line 46:
The Amazon S3 Glacier storage classes above are distinct from [[Amazon Glacier]], which is a separate product with its own APIs.
The Amazon S3 Glacier storage classes above are distinct from [[Amazon Glacier]], which is a separate product with its own APIs.


== File size limits ==
=== File size limits ===
An object in S3 can be between 0 bytes and 5TB. If an object is larger than 5TB, it must be divided into chunks prior to uploading. When uploading, Amazon S3 allows a maximum of 5GB in a single upload operation; hence, objects larger than 5GB must be uploaded via the S3 multipart upload API.<ref>{{Cite web | url=https://riyanchristy.goseeq.net/how-to-upload-large-files-to-s3-efficiently/ | title=How to Upload Large Files to S3 | date=June 21, 2022 }}</ref>
An object in S3 can be between 0 bytes and 5TB. If an object is larger than 5TB, it must be divided into chunks prior to uploading. When uploading, Amazon S3 allows a maximum of 5GB in a single upload operation; hence, objects larger than 5GB must be uploaded via the S3 multipart upload API.<ref>{{Cite web | url=https://riyanchristy.goseeq.net/how-to-upload-large-files-to-s3-efficiently/ | title=How to Upload Large Files to S3 | date=June 21, 2022 }}</ref>


== Notable users ==
== Uses ==
=== Notable users ===
* Photo hosting service [[SmugMug]] has used Amazon S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.<ref name="smugmug-smtm">{{cite web|url=https://donmacaskill.wordpress.com/2006/11/10/amazon-s3-show-me-the-money/|title=Amazon S3: Show Me the Money|date=November 10, 2006|work=SmugMug Blog|publisher=SmugMug|access-date=2017-03-03|archive-url=https://web.archive.org/web/20170303130206/https://donmacaskill.wordpress.com/2006/11/10/amazon-s3-show-me-the-money/#|archive-date=2017-03-03|url-status=live}}</ref>
* Photo hosting service [[SmugMug]] has used Amazon S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.<ref name="smugmug-smtm">{{cite web|url=https://donmacaskill.wordpress.com/2006/11/10/amazon-s3-show-me-the-money/|title=Amazon S3: Show Me the Money|date=November 10, 2006|work=SmugMug Blog|publisher=SmugMug|access-date=2017-03-03|archive-url=https://web.archive.org/web/20170303130206/https://donmacaskill.wordpress.com/2006/11/10/amazon-s3-show-me-the-money/#|archive-date=2017-03-03|url-status=live}}</ref>
* [[Netflix]] uses Amazon S3 as their [[system of record]]. Netflix implemented a tool, S3mper,<ref name="S3mper">{{cite web|url=http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html|title=S3mper: Consistency in the Cloud|access-date=2016-05-01|archive-url=https://web.archive.org/web/20160424205437/http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html#|archive-date=2016-04-24|url-status=live}}</ref> to address the Amazon S3 limitations of [[eventual consistency]].<ref>{{cite web|title=Introduction to Amazon S3|url=http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html|website=Amazon|access-date=28 December 2017|archive-url=https://web.archive.org/web/20171225230323/http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#|archive-date=2017-12-25|url-status=live}}</ref> S3mper stores the filesystem metadata: filenames, directory structure, and permissions in [[Amazon DynamoDB]].<ref name=":0">{{Cite web|url=https://www.theguardian.com/technology/2017/feb/02/amazon-web-services-the-secret-to-the-online-retailers-future-success|title=Amazon Web Services: the secret to the online retailer's future success|last=Hern|first=Alex|date=2017-02-02|website=the Guardian|language=en|access-date=2018-04-23|archive-url=https://web.archive.org/web/20180502223750/https://www.theguardian.com/technology/2017/feb/02/amazon-web-services-the-secret-to-the-online-retailers-future-success#|archive-date=2018-05-02|url-status=live}}</ref>
* [[Netflix]] uses Amazon S3 as their [[system of record]]. Netflix implemented a tool, S3mper,<ref name="S3mper">{{cite web|url=http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html|title=S3mper: Consistency in the Cloud|access-date=2016-05-01|archive-url=https://web.archive.org/web/20160424205437/http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html#|archive-date=2016-04-24|url-status=live}}</ref> to address the Amazon S3 limitations of [[eventual consistency]].<ref>{{cite web|title=Introduction to Amazon S3|url=http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html|website=Amazon|access-date=28 December 2017|archive-url=https://web.archive.org/web/20171225230323/http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#|archive-date=2017-12-25|url-status=live}}</ref> S3mper stores the filesystem metadata: filenames, directory structure, and permissions in [[Amazon DynamoDB]].<ref name=":0">{{Cite web|url=https://www.theguardian.com/technology/2017/feb/02/amazon-web-services-the-secret-to-the-online-retailers-future-success|title=Amazon Web Services: the secret to the online retailer's future success|last=Hern|first=Alex|date=2017-02-02|website=the Guardian|language=en|access-date=2018-04-23|archive-url=https://web.archive.org/web/20180502223750/https://www.theguardian.com/technology/2017/feb/02/amazon-web-services-the-secret-to-the-online-retailers-future-success#|archive-date=2018-05-02|url-status=live}}</ref>
Line 55: Line 57:
* [[Swiftype]]'s CEO has mentioned that the company uses Amazon S3.<ref>{{cite web|url=http://stackshare.io/posts/swiftype-explains-their-cloud-stack|date=July 1, 2013|title=Swiftype Explains Their Cloud Stack|access-date=2014-12-08|archive-url=https://web.archive.org/web/20141208083021/http://stackshare.io/posts/swiftype-explains-their-cloud-stack#|archive-date=2014-12-08|url-status=live}}</ref>
* [[Swiftype]]'s CEO has mentioned that the company uses Amazon S3.<ref>{{cite web|url=http://stackshare.io/posts/swiftype-explains-their-cloud-stack|date=July 1, 2013|title=Swiftype Explains Their Cloud Stack|access-date=2014-12-08|archive-url=https://web.archive.org/web/20141208083021/http://stackshare.io/posts/swiftype-explains-their-cloud-stack#|archive-date=2014-12-08|url-status=live}}</ref>


== S3 API and competing services ==
=== S3 API and competing services ===
The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 [[Application programming interface|API.]] These services use the standard programming interface but are differentiated by their underlying technologies and business models.<ref>{{cite web|last=Watters|first=Audrey|title=Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)|date=12 July 2010|url=http://readwrite.com/2010/07/12/cloud-community-debates-is-ama|publisher=SAY Media, Inc.|access-date=19 December 2012|archive-url=https://web.archive.org/web/20130217035941/http://readwrite.com/2010/07/12/cloud-community-debates-is-ama#|archive-date=2013-02-17|url-status=live}}</ref> A standard interface enables better competition from rival providers and allows [[economies of scale]] in implementation, among other benefits.<ref>{{cite book |others=Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council|title=Crossroads of Information Technology Standards|year=1990|publisher=The National Academies Press, 1990.|location=Washington, DC|pages=36–37|doi=10.17226/10440|isbn=978-0-309-58171-4|url=http://www.nap.edu/openbook.php?record_id=10440&page=36|access-date=2014-03-25|archive-url=https://web.archive.org/web/20140325160847/http://www.nap.edu/openbook.php?record_id=10440&page=36#|archive-date=2014-03-25|url-status=live}}</ref>
The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 [[Application programming interface|API.]] These services use the standard programming interface but are differentiated by their underlying technologies and business models.<ref>{{cite web|last=Watters|first=Audrey|title=Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)|date=12 July 2010|url=http://readwrite.com/2010/07/12/cloud-community-debates-is-ama|publisher=SAY Media, Inc.|access-date=19 December 2012|archive-url=https://web.archive.org/web/20130217035941/http://readwrite.com/2010/07/12/cloud-community-debates-is-ama#|archive-date=2013-02-17|url-status=live}}</ref> A standard interface enables better competition from rival providers and allows [[economies of scale]] in implementation, among other benefits.<ref>{{cite book |others=Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council|title=Crossroads of Information Technology Standards|year=1990|publisher=The National Academies Press, 1990.|location=Washington, DC|pages=36–37|doi=10.17226/10440|isbn=978-0-309-58171-4|url=http://www.nap.edu/openbook.php?record_id=10440&page=36|access-date=2014-03-25|archive-url=https://web.archive.org/web/20140325160847/http://www.nap.edu/openbook.php?record_id=10440&page=36#|archive-date=2014-03-25|url-status=live}}</ref>


Revision as of 07:31, 24 July 2023

Amazon S3
Amazon S3 Logo
Type of site
Cloud storage
Available inEnglish
OwnerAmazon.com
URLaws.amazon.com/s3/
IPv6 supportYes
CommercialYes
RegistrationRequired (included in free tier layer)
LaunchedMarch 14, 2006; 18 years ago (2006-03-14)
Current statusActive

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface.[1][2] Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its e-commerce network.[3] Amazon S3 can store any type of object, which allows uses like storage for Internet applications, backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage. AWS launched Amazon S3 in the United States on March 14, 2006,[1][4] then in Europe in November 2007.[5]

Technical details

Design

Amazon S3 manages data with an object storage architecture[6] which aims to provide scalability, high availability, and low latency with high durability.[3] The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key.[7] Buckets can be managed using the console provided by Amazon S3, programmatically with the AWS SDK, or the REST application programming interface. Objects can be up to five terabytes in size.[8][9] Requests are authorized using an access control list associated with each object bucket and support versioning[10] which is disabled by default.[11] Since buckets are typically the size of an entire file system mount in other systems, this access control scheme is very coarse-grained. In other words, unique access controls cannot be associated with individual files.[citation needed] Amazon S3 can be used to replace static web-hosting infrastructure with HTTP client-accessible objects,[12] index document support and error document support.[13] The Amazon AWS authentication mechanism allows the creation of authenticated URLs, valid for a specified amount of time. Every item in a bucket can also be served as a BitTorrent feed. The Amazon S3 store can act as a seed host for a torrent and any BitTorrent client can retrieve the file. This can drastically reduce the bandwidth cost for the download of popular objects. A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in data mining operations.[14] There are various User Mode File System (FUSE)–based file systems for Unix-like operating systems (for example, Linux) that can be used to mount an S3 bucket as a file system. The semantics of the Amazon S3 file system are not that of a POSIX file system, so the file system may not behave entirely as expected.[15]

Amazon S3 storage classes

Amazon S3 offers eight different storage classes with different levels of durability, availability, and performance requirements.[16]

  • Amazon S3 Standard is the default. It is general purpose storage for frequently accessed data.
  • Amazon S3 Standard-Infrequent Access (Standard-IA) is designed for less frequently accessed data, such as backups and disaster recovery data.
  • Amazon S3 One Zone-Infrequent Access (One Zone-IA) performs like the Standard-IA, but stores data only in one availability zone.
  • Amazon S3 Intelligent-Tiering moves objects automatically to a more cost-efficient storage class.
  • Amazon S3 on Outposts brings storage to installations not hosted by Amazon.
  • Amazon S3 Glacier Instant Retrieval is a low-cost storage for rarely accessed data, but which still requires rapid retrieval.
  • Amazon S3 Glacier Flexible Retrieval is also a low-cost option for long-lived data; it offers 3 retrieval speeds, ranging from minutes to hours.
  • Amazon S3 Glacier Deep Archive is another low-cost option.[17][better source needed]

The Amazon S3 Glacier storage classes above are distinct from Amazon Glacier, which is a separate product with its own APIs.

File size limits

An object in S3 can be between 0 bytes and 5TB. If an object is larger than 5TB, it must be divided into chunks prior to uploading. When uploading, Amazon S3 allows a maximum of 5GB in a single upload operation; hence, objects larger than 5GB must be uploaded via the S3 multipart upload API.[18]

Uses

Notable users

  • Photo hosting service SmugMug has used Amazon S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.[19]
  • Netflix uses Amazon S3 as their system of record. Netflix implemented a tool, S3mper,[20] to address the Amazon S3 limitations of eventual consistency.[21] S3mper stores the filesystem metadata: filenames, directory structure, and permissions in Amazon DynamoDB.[22]
  • Reddit is hosted on Amazon S3.[23]
  • Bitcasa,[24] and Tahoe-LAFS-on-S3,[25] among others, use Amazon S3 for online backup and synchronization services. In 2016, Dropbox stopped using Amazon S3 services and developed its own cloud server.[26][27]
  • Swiftype's CEO has mentioned that the company uses Amazon S3.[28]

S3 API and competing services

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface but are differentiated by their underlying technologies and business models.[29] A standard interface enables better competition from rival providers and allows economies of scale in implementation, among other benefits.[30]

History

At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3.

Amazon Web Services introduced Amazon S3 in 2006.[31][32] Amazon reported it stored more than 100 trillion objects as of March 2021,[33] up from 10 billion objects in October 2007,[34] 14 billion objects in January 2008, 29 billion objects in October 2008,[35] 52 billion objects in March 2009,[36] 64 billion objects in August 2009,[37] 102 billion objects in March 2010,[38] and 2 trillion objects in April 2013.[39] In November 2017 AWS added default encryption capabilities at bucket level. [40]

See also

References

Citations

  1. ^ a b "Amazon Web Services Launches "Amazon S3"" (Press release). 2006-03-14. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  2. ^ Huang, Dijiang; Wu, Huijun (2017-09-08). Mobile Cloud Computing: Foundations and Service Models. Morgan Kaufmann. p. 67. ISBN 9780128096444. Archived from the original on 2018-11-15. Retrieved 2018-11-15.
  3. ^ a b "Cloud Object Storage - Store & Retrieve Data Anywhere - Amazon Simple Storage Service". Amazon Web Services, Inc. Archived from the original on 2018-05-17. Retrieved 2018-05-17.
  4. ^ "5 Key Events in the history of Cloud Computing - DZone Cloud". dzone.com. Archived from the original on 2018-09-29. Retrieved 2018-09-28.
  5. ^ "Amazon Web Services Offers European Storage for Amazon S3" (Press release). 2007-11-06. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  6. ^ "What is Cloud Object Storage? – AWS". Amazon Web Services, Inc. 2019-10-16. Archived from the original on 2018-09-20. Retrieved 2018-07-09.
  7. ^ "Tech Blog » Starting Websphere in Cloud and saving the data in S3". techblog.aasisvinayak.com. Archived from the original on 2010-03-12.
  8. ^ "open-guides/og-aws". GitHub. Archived from the original on 2018-01-03. Retrieved 2018-05-17.
  9. ^ "Error Responses - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2017-12-24. Retrieved 2018-05-21.
  10. ^ "Using versioning in S3 buckets - Amazon Simple Storage Service".
  11. ^ "Introduction to Amazon S3 - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2018-05-12. Retrieved 2018-05-17.
  12. ^ "How to use Amazon S3 for Web Hosting". bucketexplorer.com. Archived from the original on 2008-04-08. Retrieved 2008-05-06.
  13. ^ Amazon Simple Storage Service Archived 2011-02-20 at the Wayback Machine Docs.amazonwebservices.com. Retrieved on 2013-08-09.
  14. ^ http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html Archived 2014-12-23 at the Wayback Machine Server Access Logging
  15. ^ "Comparison of S3QL and other S3 file systems". Archived from the original on 2012-08-05. Retrieved 2012-06-29.
  16. ^ "Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS". Amazon Web Services, Inc. Archived from the original on 2018-06-13. Retrieved 2018-05-17.
  17. ^ "S3 Glacier Deep Archive". June 7, 2022.
  18. ^ "How to Upload Large Files to S3". June 21, 2022.
  19. ^ "Amazon S3: Show Me the Money". SmugMug Blog. SmugMug. November 10, 2006. Archived from the original on 2017-03-03. Retrieved 2017-03-03.
  20. ^ "S3mper: Consistency in the Cloud". Archived from the original on 2016-04-24. Retrieved 2016-05-01.
  21. ^ "Introduction to Amazon S3". Amazon. Archived from the original on 2017-12-25. Retrieved 28 December 2017.
  22. ^ Hern, Alex (2017-02-02). "Amazon Web Services: the secret to the online retailer's future success". the Guardian. Archived from the original on 2018-05-02. Retrieved 2018-04-23.
  23. ^ "AWS Case Study: reddit". aws.amazon.com. 2015. Archived from the original on 2015-03-17. Retrieved March 18, 2015.
  24. ^ "What is Tahoe-LAFS-on-S3?". August 21, 2012. Archived from the original on 2013-05-06. Retrieved 2012-08-21.
  25. ^ "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire". WIRED. Archived from the original on 2018-01-25. Retrieved 2018-04-23.
  26. ^ "Dropbox saved almost $75 million over two years by building its own tech infrastructure". GeekWire. 2018-02-23. Archived from the original on 2018-04-23. Retrieved 2018-04-23.
  27. ^ "Swiftype Explains Their Cloud Stack". July 1, 2013. Archived from the original on 2014-12-08. Retrieved 2014-12-08.
  28. ^ Watters, Audrey (12 July 2010). "Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)". SAY Media, Inc. Archived from the original on 2013-02-17. Retrieved 19 December 2012.
  29. ^ Crossroads of Information Technology Standards. Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council. Washington, DC: The National Academies Press, 1990. 1990. pp. 36–37. doi:10.17226/10440. ISBN 978-0-309-58171-4. Archived from the original on 2014-03-25. Retrieved 2014-03-25.{{cite book}}: CS1 maint: others (link)
  30. ^ Overview of Amazon Web Services, 2018, https://docs.aws.amazon.com/whitepapers/latest/aws-overview/introduction.html Archived 2017-11-18 at the Wayback Machine
  31. ^ Garfinkel, Simson L. 2007. An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS. Harvard Computer Science Group Technical Report TR-08-07. https://dash.harvard.edu/bitstream/handle/1/24829568/tr-08-07.pdf?sequence=1 Archived 2018-07-29 at the Wayback Machine
  32. ^ "Celebrate 15 Years of Amazon S3 with 'Pi Week' Livestream Events". amazon.com. 14 March 2021.
  33. ^ Vogels, Werner (2008-03-19). "Happy Birthday, Amazon S3!". All Things Distributed. Archived from the original on 2008-05-09. Retrieved 2008-05-23.
  34. ^ "Amazon S3 - Busier Than Ever". 2008-10-08. Archived from the original on 2008-10-11. Retrieved 2008-10-09.
  35. ^ "Celebrating S3's Third Birthday With Special Anniversary Pricing - Amazon Web Services". typepad.com. 31 March 2009. Archived from the original on 2011-07-07. Retrieved 2009-04-01.
  36. ^ "Amazon's Head Start in the Cloud Pays Off". eweek.com.
  37. ^ "Amazon S3 Now Hosts 100 Billion Objects". datacenterknowledge.com. 9 March 2010. Archived from the original on 2010-03-12. Retrieved 2010-03-09.
  38. ^ "Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services". typepad.com. 18 April 2013.
  39. ^ "New Amazon S3 Encryption & Security Features". 6 November 2017.

Sources

Retrieved from "https://en.wikipedia.org/w/index.php?title=Amazon_S3&oldid=1166869996"