Check out The Amazon Builders’ Library – This is How We Do It!

Amazon customers often tell us that they want to know more about how we build and run our business. On the retail side, they tour Amazon Fulfillment Centers and see how we we organize our warehouses. Corporate customers often ask about our Leadership Principles, and sometimes adopt (and then adapt) them for their own use. I regularly speak with customers in our Executive Briefing Center (EBC), and talk to them about working backwards, PRFAQs, narratives, bar-raising, accepting failure as part of long-term success, and our culture of innovation.
The same curiosity that surrounds our business surrounds our development culture. We are often asked how we design, build, measure, run, and scale the hardware and software systems that underlie Amazon.com, AWS, and our other businesses.
New Builders’ Library Today I am happy to announce The Amazon Builders’ Library. We are launching with a collection of detailed articles that will tell you


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/Czk8hX3Os_8/

Original article

Amazon SageMaker Processing – Fully Managed Data Processing and Model Evaluation

Today, we’re extremely happy to launch Amazon SageMaker Processing, a new capability of Amazon SageMaker that lets you easily run your preprocessing, postprocessing and model evaluation workloads on fully managed infrastructure.
Training an accurate machine learning (ML) model requires many different steps, but none is potentially more important than preprocessing your data set, e.g.:
Converting the data set to the input format expected by the ML algorithm you’re using,
Transforming existing features to a more expressive representation, such as one-hot encoding categorical features,
Rescaling or normalizing numerical features,
Engineering high level features, e.g. replacing mailing addresses with GPS coordinates,
Cleaning and tokenizing text for natural language processing applications,
And more!
These tasks involve running bespoke scripts on your data set, (beneath a moonless sky, I’m told) and saving the processed version for later use by your training jobs. As you can guess, running them manually


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/OAPHJ4LIGpE/

Original article

Amazon SageMaker Studio: The First Fully Integrated Development Environment For Machine Learning

Today, we’re extremely happy to launch Amazon SageMaker Studio, the first fully integrated development environment (IDE) for machine learning (ML).
We have come a long way since we launched Amazon SageMaker in 2017, and it is shown in the growing number of customers using the service. However, the ML development workflow is still very iterative, and is challenging for developers to manage due to the relative immaturity of ML tooling. Many of the tools which developers take for granted when building traditional software (debuggers, project management, collaboration, monitoring, and so forth) have yet been invented for ML.
For example, when trying a new algorithm or tweaking hyper parameters, developers and data scientists typically run hundreds and thousands of experiments on Amazon SageMaker, and they need to manage all this manually. Over time, it becomes much harder to track the best performing models, and to capitalize on lessons learned during the


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/EY0_Uwy9VGw/

Original article

AWS Outposts Now Available – Order Yours Today!

We first discussed AWS Outposts at re:Invent 2018. Today, I am happy to announce that we are ready to take orders and install Outposts racks in your data center or colo facility.
Why Outposts?This new and unique AWS offering is a comprehensive, single-vendor compute & storage solution that is designed to meet the needs of customers who need local processing and very low latency. You no longer need to spend time creating detailed hardware specifications, soliciting & managing bids from multiple disparate vendors, or racking & stacking individual servers. Instead, you place your order online, take delivery, and relax while trained AWS technicians install, connect, set up, and verify your Outposts.
Once installed, we take care of monitoring, maintaining, and upgrading your Outposts. All of the hardware is modular and can be replaced in the field without downtime. When you need more processing or storage, or want to upgrade to


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/fzENhNGm7Us/

Original article

AWS DeepComposer – Compose Music with Generative Machine Learning Models

Today, we’re extremely happy to announce AWS DeepComposer, the world’s first machine learning-enabled musical keyboard. Yes, you read that right.
Machine learning (ML) requires quite a bit of math, computer science, code, and infrastructure. These topics are exceedingly important but to a lot of aspiring ML developers, they look overwhelming and sometimes, dare I say it, boring.
To help everyone learn about practical ML and have fun doing it, we introduced several ML-powered devices. At AWS re:Invent 2017, we introduced AWS DeepLens, the world’s first deep learning-enabled camera, to help developers learn about ML for computer vision. Last year, we launched AWS DeepRacer, a fully autonomous 1/18th scale race car driven by reinforcement learning. This year, we’re raising the bar (pardon the pun).

Introducing AWS DeepComposerAWS DeepComposer is a 32-key, 2-octave keyboard designed for developers to get hands on with Generative AI, with either pretrained models or your own.


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/z6_0osvHxF0/

Original article

Automate OS Image Build Pipelines with EC2 Image Builder

Earlier in my career, I can recall being assigned the task of creating and maintaining operating system (OS) images for use by my development team. This was a time-consuming process, sometimes error-prone, needing me to manually re-create and re-snapshot images frequently. As I’m sure you can imagine, it also involved a significant amount of manual testing!
Today, customers still need to keep their images up to date and they do so either by manually updating and snapshotting VMs, or they have teams that build automation scripts to maintain the images, both of which can still be time consuming, resource intensive, and error-prone. I’m excited to announce the availability of EC2 Image Builder, a service that makes it easier and faster to build and maintain secure OS images for Windows Server and Amazon Linux 2, using automated build pipelines. The images created by EC2 Image Builder can be used with Amazon


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/mj23fkojDes/

Original article

In The Works – New AMD-Powered, Compute-Optimized EC2 Instances (C5a/C5ad)

We’re getting ready to give you even more power and even more choices when it comes to EC2 instances.
We will soon launch C5a and C5ad instances powered by custom second-generation AMD EPYC “Rome” processors running at frequencies as high as 3.3 GHz. You will be able to use these compute-optimized instances to run your batch processing, distributed analytics, web applications and other compute-intensive workloads. Like the existing AMD-powered instances in the M, R and T families, the C5a and C5ad instances are built on the AWS Nitro System and give you an opportunity to balance your instance mix based on cost and performance.
The instances will be available in eight sizes and also in bare metal form, with up to 192 vCPUs and 384 GiB of memory. The C5ad instances will include up to 7.6 TiB of fast, local NVMe storage, making them perfect for video encoding, image manipulation,


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/KgDdHwfCvhc/

Original article

New – Insert, Update, Delete Data on S3 with Amazon EMR and Apache Hudi

Storing your data in Amazon S3 provides lots of benefits in terms of scale, reliability, and cost effectiveness. On top of that, you can leverage Amazon EMR to process and analyze your data using open source tools like Apache Spark, Hive, and Presto. As powerful as these tools are, it can still be challenging to deal with use cases where you need to do incremental data processing, and record-level insert, update, and delete.
Talking with customers, we found that there are use cases that need to handle incremental changes to individual records, for example:
Complying with data privacy regulations, where their users choose to exercise their right to be forgotten, or change their consent as to how their data can be used.
Working with streaming data, when you have to handle specific data insertion and update events.
Using change data capture (CDC) architectures to track and ingest database change logs from enterprise data


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/u5A7k_BanFE/

Original article

Migration Complete – Amazon’s Consumer Business Just Turned off its Final Oracle Database

Over my 17 years at Amazon, I have seen that my colleagues on the engineering team are never content to leave good-enough alone. They routinely re-evaluate every internal system to make sure that it is as scalable, efficient, performant, and secure as possible. When they find an avenue for improvement, they will use what they have learned to thoroughly modernize our architectures and implementations, often going so far as to rip apart existing systems and rebuild them from the ground up if necessary.
Today I would like to tell you about an internal database migration effort of this type that just wrapped up after several years of work. Over the years we realized that we were spending too much time managing and scaling thousands of legacy Oracle databases. Instead of focusing on high-value differentiated work, our database administrators (DBAs) spent a lot of time simply keeping the lights on while


Original URL: http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/farzgMTdbeo/

Original article

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: