Last week, AWS re:Invent 2022 wrapped up in Las Vegas following a week of keynotes, sessions, tutorials and announcements. Unsurprisingly, most of the news was focused on improving cloud utility, making it easier to migrate legacy systems to the cloud and new tools for companies to derive value from their existing data and cloud products. All of these are a high priority for companies worried about the ongoing stresses of digital transformations and the global economy. Here, we’ve rounded up some of the most important announcements related to AWS’ cloud and data analytics tools from an engineering perspective.
AWS Clean Rooms: Simplify Company Collaborations While Protecting IP
In an increasingly integrated world, companies across industries often need to share data with external partners for marketing, manufacturing, investment, research and other purposes. A frequent issue remains the protection of sensitive data. To get around this hurdle, companies often share user-level data with partners and rely on contractual agreements to protect their intellectual property. However, this can still result in misuse and be both timely and expensive.
A useful alternative quickly gaining popularity is data clean rooms: where multiple companies can combine and analyze data in a secure environment. In this setting, companies cannot see or access each other’s raw data, but they can analyze it, helping to improve IP protection. However, until recently, clean rooms were difficult to build, with long lead-up times to ensure proper privacy controls in combination with the appropriate analytics tools. Unfortunately, this resulted in companies losing opportunities to improve business insights.
At AWS re:Invent, the company announced a new analytics service named AWS Clean Rooms. The solution helps businesses across industries collaborate while protecting sensitive data. The service is designed to facilitate cross-company partnerships that rely on combined datasets and IP protection. Within minutes, a company can generate a clean room and collaborate with any other company in the AWS Cloud environment.
“Customers tell us they want to collaborate more safely and securely with their partners in areas like advertising, media, financial services and life sciences. However, the data they need to do this is fragmented across data stores and applications belonging to different partners,” said Dilip Kumar, vice president of AWS Applications. “AWS Clean Rooms helps customers and their partners to better analyze and collaborate on their data on AWS. With the launch of AWS Clean Rooms, we are making it easier, simpler and more secure for multiple companies to share and analyze combined datasets to generate new insights that they could not do on their own. Using AWS Clean Rooms, customers can collaborate on a range of tasks, such as more effectively generating advertising campaign insights and analyzing investment data while improving data security.”
What seems to be the selling points of this feature is its utility, ease of set-up and ability to keep data within the AWS environment—if it’s already there. Within the AWS Management Console, companies choose their partners, select the data and configure specific restrictions. They can use query controls, query logging and more to restrict queries run by anyone within the clean room environment. By doing this, all of the data stays within the AWS environment, and queries read data where it already lives. So the overall process is relatively straightforward for companies looking to partner with other external collaborators already using AWS.
There are plenty of examples that come to mind considering the utility of this solution. For example, a clean room could be used by an OEM company working with the design team for a major motor company, such as Ford. Other examples include external engineering consultants that might need to use data to develop a solution strategy. Hopefully, more examples will emerge as the solution is adopted in the near future. For now, it will only be available in select international locations starting in early 2023.
New Tools for Data Management and Governance
AI-driven solutions are reliant on big data driving their development and overall success. With many companies collecting petabytes of data, most organizations already have this potential at their fingertips. The ongoing issue remains organizing data for accessibility and security.
“Good governance is the foundation that makes data accessible to the entire organization, but we often hear from customers that it is difficult to strike the right balance between making data discoverable and maintaining control,” said Swami Sivasubramanian, AWS vice president of Databases, Analytics and Machine Learning.
To meet this demand, AWS announced Amazon DataZone, a data management service for AWS, on-premise and third-party data sources. As with many data management tools, the goal is to improve access and utility of data without compromising security. AWS describes the solution as a product for engineers, product managers, analysts and others who need to govern access to data. It also is a clear investment in additional tools for multicloud IT infrastructure.
The service is designed to make it easier for data producers to manage and govern data while allowing a company’s engineers, business analysts and others to discover and ultimately use data that can drive value. DataZone enables a producer to create a catalog by defining data taxonomy, appropriate governance policies and connections to AWS, third-party solutions and on-premise systems. Cataloging is then made easier using machine learning (ML), which collects and suggests metadata for every dataset. Engineers can then use a web portal to search for datasets of interest and request access. Data projects can also be generated within the portal to make it easier for engineers to access disparate datasets and collaborate on analysis. The service will automatically integrate with AWS analysis products like Redshift, Athena, and QuickSight.
New Capabilities to Expand Amazon QuickSight
At the event, AWS also announced several new features for Amazon QuickSight, its cloud-based business intelligence service. AWS highlighted that QuickSight has seen more than 80 capabilities added just in the past year, indicating the service is continuing to evolve. Several of the announcements made at the event were focused on improving the migration and consolidation of business intelligence initiatives to the cloud.
The existing QuickSight Q solution allows non-technical staff to pose questions using natural language and, within seconds, receive relevant answers and data-driven visualizations. The expansion to QuickSight Q now allows users to forecast answers and ask “why” questions. For example, a manager could ask to forecast sales for a given region or to ask why there was an increase in sales last month. The expanded features will make it even easier for anyone in a company to ask straightforward questions and receive accurate, real-time answers.
AWS also announced automated data preparation for Amazon QuickSight Q, which will use ML to prepare data for natural language queries in minutes. In short, the tool runs ML on a company’s existing data assets, such as dashboards and reports, to preconfigure business terms for each dataset. The new addition will help improve the solution over time and customize it for a company’s given taxonomy.
Finally, expanded API capabilities will now make it easier to create, manage and edit business intelligence products with AWS. Engineers can now functionally manage these assets from the programming level to help streamline the migration of on-premise, legacy solutions to the cloud. This stands in contrast to many existing business intelligence-related tools that often cannot include full programmatic access.
Improved Data Security Monitoring with Amazon Security Lake
Currently, companies are very interested in solutions that can improve their ability to monitor security across their entire organization. With the shift to hybrid and multicloud strategies, many companies are trying to navigate security across on-premise and multiple private and public cloud services.
To help improve data security visibility, AWS announced the Amazon Security Lake. The solution allows engineering teams to collect, normalize and store security-related data and can be used to ensure a company complies with local data security regulations. Both AWS and third-party security and analytics tools can then be used in conjunction with the tool to help gain insights into companywide security vulnerabilities. The solution also conforms data to the Open Cybersecurity Schema Framework (OCSF) open standard.
AWS: Looking Forward to 2023
Following a whirlwind week of splashy announcements, the main takeaway remains that AWS wants to make it as easy as possible for companies to migrate to the cloud and use their products. Many of their announcements are compatible with on-premise and third-party cloud offerings and look to provide solutions for companies with disparate IT environments. As seems to be the summary of 2022, multicloud and hybrid cloud strategies are here to stay, and AWS knows it. With its suite of new tools, features and solution updates, the company is tackling security, data processing and storage, and improved business insights, among other important topics on most engineers’ minds. As the year comes to a close, we’ll soon see how these announcements affect AWS’ bottom line and what ends up being the most useful for cloud computing moving forward.