2776 lines
358 KiB
XML
2776 lines
358 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>Cloud Blog</title><link>https://cloud.google.com/blog/</link><description>Cloud Blog</description><atom:link href="https://cloudblog.withgoogle.com/blog/rss/" rel="self"></atom:link><language>en</language><lastBuildDate>Tue, 07 May 2024 16:00:00 +0000</lastBuildDate><image><url>https://cloud.google.com/blog/static/blog/images/google.a51985becaa6.png</url><title>Cloud Blog</title><link>https://cloud.google.com/blog/</link></image><item><title>Product analytics for generative AI model and media asset companies using BigQuery</title><link>https://cloud.google.com/blog/products/data-analytics/perform-product-analysis-with-generative-ai-and-bigquery/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Over the last year, there’s been a lot of change in the commercial image and video asset industry: New generative AI applications let users create their own still and live images based on prompts, and traditional stock-media asset providers are offering customers richer search experiences that have a deep understanding of the image/live image content and that expose it with a natural language interface. </span></p>
|
||
<p><span style="vertical-align: baseline;">To continually push the state of the art, these organizations must use data to evolve their products rapidly, for example to: </span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Optimize still and live image generation models</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Identify inappropriate content, such as violence or nudity</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Analyze behavior to identify improvements to the user experience</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Recommend similar images or prompts based on previous activity</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Enhance static asset search capabilities</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">To do this, they need unstructured images, live images, and audio data, combined with structured user-experience data and metadata about the assets they are interacting with, whether they’re static or AI-generated.</span></p>
|
||
<p><span style="vertical-align: baseline;">In this post, we outline a solution</span><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline;">based on our real-life engagements with leaders in the industry who operate at the scale of petabytes per day. This solution delivers several benefits:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Minimizes costs by avoiding duplicate data and storage, while facilitating AI model proximity to data for efficient inference</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Simplifies development and delivery by combining diverse data types in a unified data architecture</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Optimizes use of limited engineering resources through an integrated, scalable serverless platform that combines </span><a href="https://cloud.google.com/bigquery?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery</span></a><span style="vertical-align: baseline;"> and </span><a href="https://cloud.google.com/storage?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Cloud Storage</span></a></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Allows users to augment and transform their data according to the needs of their business </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Enables companies to develop lightweight, powerful analyses quickly and securely, to activate customer data and quickly iterate on the output of models</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">The challenge of unstructured data</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Generated (unstructured) image data, the (semi-structured) prompts that made them, as well as user behavior data (structured, in tables) for things like session time and frequency, are all rich in potential insights. For example, knowing which types of prompts lead to successfully generating an image — and those that don’t — provides insights into product and model development opportunities. </span></p>
|
||
<p><span style="vertical-align: baseline;">But combining these different data types often requires advanced analytics to interpret them meaningfully. Technologies like natural language processing and computer vision are at the forefront of extracting these kinds of valuable insights. However, integrating unstructured data within an existing analytics framework of structured data, for example user behavior data in database tables, is not without its hurdles. Common challenges include:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Data security standards:</strong><span style="vertical-align: baseline;"> Adhering to stringent data security standards to protect sensitive information is crucial. These standards include applying data masking to sensitive PII data and following least-privilege security principles for data access.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Data type silos:</strong><span style="vertical-align: baseline;"> Unstructured data is often stored separately from structured data, preventing effective analysis across data types, for example, filtering media assets (unstructured) based on user profiles (structured), as they reside in separate systems.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">High-performance, scalable cloud computing resources:</strong><span style="vertical-align: baseline;"> The need for powerful computing resources is imperative to manage and analyze large unstructured datasets effectively due to the data's complexity, volume, and the potential need for real-time results. In addition, high performance networking allows for low-latency data transfers to enable the transfer of unstructured data between storage (Cloud Storage) and analytical layers (BigQuery, Vertex, etc.)</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Maintaining data integrity across layers</strong><span style="vertical-align: baseline;">: As insights are extracted from unstructured data, preserving the original source of truth and ensuring consistency across intermediate (interstitial) layers is crucial for reliable, iterative analysis.</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">Streamlining data integration with Cloud Storage and BigQuery</strong></h3>
|
||
<p><span style="vertical-align: baseline;">To overcome the challenges of working with unstructured data, </span><a href="https://cloud.google.com/storage?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Storage</span></a><span style="vertical-align: baseline;"> and </span><a href="https://cloud.google.com/bigquery?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery</span></a><span style="vertical-align: baseline;"> can be used to centralize data, using </span><a href="https://cloud.google.com/bigquery/docs/object-table-introduction"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery object tables</span></a><span style="vertical-align: baseline;"> to </span><span style="vertical-align: baseline;">enable consistent data access to varied sources through one analytical platform</span><span style="vertical-align: baseline;">. Below is an example of a simple yet effective architecture that harnesses BigQuery for both metadata generation and enhancement. This approach uses BigQuery's built-in generative AI functions, coupled with remote User Defined Functions (UDFs) that interface with </span><a href="https://cloud.google.com/vertex-ai?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Vertex AI</span></a><span style="vertical-align: baseline;"> APIs. The integration elevates the process of data enrichment and analysis, and offers a more streamlined and efficient workflow.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">The power of BigQuery object tables</strong></h3>
|
||
<p><span style="vertical-align: baseline;">In the example below, we focus on a static image use case, however, this same technique could be used for images created using generative AI. The true potential of this architecture lies in its versatility. The use of object tables in BigQuery means this pattern can be adapted to any form of unstructured data, for example images, audio, documents, opening up a world of possibilities for data science and analysis. This flexibility ensures the architecture can evolve with the changing needs and types of data, helping the solution withstand the test of time in the dynamic field of image curation and generation.</span></p>
|
||
<p><span style="vertical-align: baseline;">This architecture shows the integration of structured and unstructured data, utilizing the strengths of both to enhance platform capabilities. BigQuery serves as a central hub, amalgamating user data information (for example: user demographics, images viewed and used, session duration, session frequency), image metadata, and queries. Concurrently, external AI APIs augment this dataset with insights about the content of the images, for example describing what is happening in a scene (e.g. “a photographic image of a dog playing with a ball on grass”) . </span></p>
|
||
<p><span style="vertical-align: baseline;">This convergence of data facilitates the training of sophisticated image-generation models, tailored to meet the specific requirements of the platform's users. It also unlocks advanced search and image-curation functionalities, enabling users to navigate through an extensive collection of images. The project's ability to provide access to external systems and empower data augmentation within BigQuery helps to centralize analytic workloads. This not only streamlines data analysis but also fosters informed decision-making.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Solution overview</strong></h3></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Model_Analytics.max-1000x1000.png"
|
||
|
||
alt="1 - Model Analytics">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The goal of the solution is to create a way to interact with unstructured data through BigQuery. Using BigQuery object tables to analyze unstructured data in Cloud Storage, you can perform analyses using generative AI models via remote functions, cloud APIs via Vertex AI, or perform inference by using BigQuery ML, and then join the results of these operations with the rest of your structured data in BigQuery.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Step 1. Creating an example dataset</strong></h3>
|
||
<p><strong style="vertical-align: baseline;">Prerequisites<br/></strong><span style="vertical-align: baseline;">Data: Multiple image repositories on third-party sites like Kaggle and Hugging Face<br/></span><span style="vertical-align: baseline;">Project setup: To get started we need to activating essential project APIs:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">gcloud services enable cloudfunctions.googleapis.com</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">gcloud services enable cloudbuild.googleapis.com</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">gcloud services enable bigqueryconnection.googleapis.com</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">gcloud services enable </span><a href="http://vision.googleapis.com" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">vision.googleapis.com</span></a></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">Step 2. Create the object table</strong></h3>
|
||
<p><span style="vertical-align: baseline;">The object table provides the reference to the non-structured data (e.g., audio, live images and images). </span></p>
|
||
<p><span style="vertical-align: baseline;">To do this, we create the BigQuery BigLake remote connection, building a bridge between BigQuery and Cloud Storage:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Command for creation: </span><span style="vertical-align: baseline;">bq mk --connection --location=us-central1 \</span><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline;">--project_id=bq-object-tables-demo \</span><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline;">--connection_type=CLOUD_RESOURCE biglake-connection</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">To show the details of this new creation, use: </span><span style="vertical-align: baseline;">bq show --connection bq-object-tables-demo.us-central1.biglake-connection</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">Then, give your BQ service account the correct permissions to access your Cloud Storage bucket.</span></p>
|
||
<p><span style="vertical-align: baseline;">Your </span><span style="vertical-align: baseline;">serviceAccountId</span><span style="vertical-align: baseline;"> typically looks like this: </span><span style="vertical-align: baseline;">{"serviceAccountId": "bqcx-012345678910-abcd@gcp-sa-bigquery-condel.iam.gserviceaccount.com"</span><span style="vertical-align: baseline;">}`. And it needs the object viewer permission. This can be achieved by:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;gsutil iam ch \\ serviceAccount:bqcx-012345678910-abcd@gcp-sa-bigquery-condel.iam.gserviceaccount.com:objectViewer gs://bq-object-tables-demo-data&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5a30&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Make your object table in BigQuery in an existing dataset, or create a dataset for your object table.</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Create the dataset with: </span><span style="vertical-align: baseline;">bq mk -d --data_location=us-central1 bq_object_table_demo-dataset</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">This is a sample query you can use to create the object table</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;CREATE OR REPLACE EXTERNAL TABLE `bq-object-tables.bq_ot_dataset.bq_object_tables_external_table` \r\nWITH CONNECTION `bq-object-tables.us-east1.biglake-connection` OPTIONS ( object_metadata=&quot;DIRECTORY&quot;, uris = [\&#x27;gs://bq-object-tables-demo-data/*\&#x27; ], max_staleness=INTERVAL 30 MINUTE, metadata_cache_mode=&quot;AUTOMATIC&quot;);&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5040&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The max_staleness option lets you manage the trade-off between data freshness and performance by specifying a tolerable level of staleness for the materialized view; this can help improve query response times and reduce costs. By setting an appropriate value, you can achieve consistently high performance while keeping costs under control, even when working with large, frequently changing datasets</span><span style="vertical-align: baseline;">.</span></p>
|
||
<p><strong style="vertical-align: baseline;">Create metadata using Native BQ Functionality</strong></p>
|
||
<p><span style="vertical-align: baseline;">These steps can all be automated into a Directed Acyclic Graph (DAG) for use in an orchestration tool such as </span><a href="https://cloud.google.com/composer"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Composer</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Step 3. Reference the model from a native generative AI BQML function</strong></h3>
|
||
<p><span style="vertical-align: baseline;">First create the link back to the model in your BQ dataset like this: </span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;# Create Model\r\nCREATE OR REPLACE MODEL\r\n`bq-object-tables.bq_ot_dataset.myvisionmodel`\r\nREMOTE WITH CONNECTION `bq-object-tables.us-east1.biglake-connection`\r\nOPTIONS (remote_service_type =&#x27;cloud_ai_vision_v1&#x27;);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5460&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Annotate image</strong></p>
|
||
<p><span style="vertical-align: baseline;">This code parses the images, extracts their contents and outputs a JSON array of words that describe the image and the model’s confidence that the description is correct. This function will then put the description into a table.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;# Annotate image\r\nSELECT *\r\nFROM ML.ANNOTATE_IMAGE(\r\n MODEL `mydataset.myvisionmodel`,\r\n TABLE `mydataset.mytable`,\r\n STRUCT([&#x27;label_detection&#x27;] AS vision_features)\r\n);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5d30&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Step 4. Create a UDF in BigQuery</strong></h3>
|
||
<p><span style="vertical-align: baseline;">You can create a Cloud function using </span><a href="https://gist.github.com/hselbie/fde69b900c4c719656ab42cdfb897b88" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">this basic code</span></a><span style="vertical-align: baseline;">. </span></p>
|
||
<p><span style="vertical-align: baseline;">If you’re unsure how to create a cloud function, please see the docs for how to </span><a href="https://cloud.google.com/blog/products/data-analytics/extending-bigquery-functions"><span style="text-decoration: underline; vertical-align: baseline;">create a cloud function UDF</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<p><span style="vertical-align: baseline;">Then, to deploy the Cloud Function, follow these steps:</span></p>
|
||
<p><span style="vertical-align: baseline;">4.1. </span><a href="https://cloud.google.com/functions/docs/deploying"><span style="text-decoration: underline; vertical-align: baseline;">Deploy your Cloud Function</span></a></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">You may need to enable Cloud Functions API.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">You may need to enable Cloud Build APIs.</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">4.2. </span><a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/remote-functions#grant_permission_on_function"><span style="text-decoration: underline; vertical-align: baseline;">Grant the BigQuery connection service account access to the Cloud Function</span></a></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">One way you can find the service account is by using the BigQuery cli ‘show’ command</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">4.3. Reference the functions in BigQuery</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Create a BigQuery remote function to reference the Cloud Function UDF</span></p>
|
||
</li>
|
||
</ul></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;CREATE OR REPLACE FUNCTION `mydataset.vision_safe_search`(signed_url_ STRING) RETURNS JSON\r\nREMOTE WITH CONNECTION `us.gcs-connection`\r\nOPTIONS(endpoint=&#x27;https://region-myproject.cloudfunctions.net/vision_safe_search&#x27;,\r\nmax_batching_rows = 1);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5190&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;CREATE OR REPLACE FUNCTION `mydataset.vision_annotation`(signed_url_ STRING) RETURNS JSON\r\nREMOTE WITH CONNECTION `us.gcs-connection`\r\nOPTIONS(endpoint=&#x27;https://region-myproject.cloudfunctions.net/vision_annotation&#x27;,\r\nmax_batching_rows = 1);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da5a00&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><span style="vertical-align: baseline;">Step 5. Use the function in a query </span></h3></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;CREATE TABLE `mydataset.mid_processing` AS\r\nSELECT uri,mydataset.vision_safe_search(signed_url) as safe_search, mydataset.vision_annotation(signed_url) as annotation\r\nFROM EXTERNAL_OBJECT_TRANSFORM(\r\nTABLE `mydataset.imageall`,\r\n[&quot;SIGNED_URL&quot;]);&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc2da59a0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Tap into unstructured data with BigQuery object tables and AI</strong></h3>
|
||
<p><span style="vertical-align: baseline;">This architecture demonstrates the power of streamlining data integration for centralized analyses through BigQuery. Although we reference image data for this example, this methodology is highly flexible; using object tables we can reference any type of unstructured data in Cloud Storage buckets that could also refer to audio files that might reference a call center AI use case, for example, or live image files relevant to training a computer vision model. </span></p>
|
||
<p><span style="vertical-align: baseline;">By centralizing data in Cloud Storage and BigQuery and intelligently using object tables, you can efficiently manage both structured and unstructured data. For our image-based example, this unified approach provides a rich dataset that contains user IDs, original prompts, prompt categories, image safety ratings, and even additional ML-generated prompts. </span></p>
|
||
<p><span style="vertical-align: baseline;">The potential applications for these metadata sets are huge. Product teams could use them to build more robust image-generation models or create an advanced image-search system, providing highly relevant results aligned with users' search terms and image descriptions. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Take the next step</strong></h3>
|
||
<p><span style="vertical-align: baseline;">You can get started today using this framework. For additional help, ask your Google Cloud account manager to reach out to the</span><strong style="vertical-align: baseline;"> </strong><a href="https://cloud.google.com/solutions/data-cloud-isvs?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Built with BigQuery</span><strong style="text-decoration: underline; vertical-align: baseline;"> </strong><span style="text-decoration: underline; vertical-align: baseline;">team</span></a><strong style="vertical-align: baseline;">.</strong><span style="vertical-align: baseline;"> </span></p>
|
||
<p><span style="vertical-align: baseline;">The Built with BigQuery team helps Independent Software Vendors (ISVs) and data providers build innovative applications with </span><a href="https://cloud.google.com/data-cloud?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Data Cloud</span></a><span style="vertical-align: baseline;">. Participating companies can: </span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption</span></p>
|
||
</li>
|
||
</ul></div></description><pubDate>Tue, 07 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/data-analytics/perform-product-analysis-with-generative-ai-and-bigquery/</guid><category>Partners</category><category>AI & Machine Learning</category><category>Developers & Practitioners</category><category>Data Analytics</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Product analytics for generative AI model and media asset companies using BigQuery</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/data-analytics/perform-product-analysis-with-generative-ai-and-bigquery/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Annie Xu</name><title>Sr. Customer Engineer, Analytics, Google Cloud</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Hugo Selbie</name><title>Customer & Partner Solutions Engineer, Gen AI, Google Cloud</title><department></department><company></company></author></item><item><title>What’s new with Active Assist: New Hub UI and four new recommendations</title><link>https://cloud.google.com/blog/products/management-tools/active-assist-gets-new-hub-and-recommendations/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The </span><a href="https://cloud.google.com/solutions/active-assist"><span style="text-decoration: underline; vertical-align: baseline;">Active Assist</span></a><span style="vertical-align: baseline;"> portfolio of intelligent tools can help you reduce costs, increase performance, improve security, and even help you make more sustainable decisions. Today, we’re excited to announce some new Active Assist features that address some of our customers’ largest concerns. These features unlock some key functionality that help you better understand and use recommendations, all aimed to help make managing and optimizing your cloud simpler and easier. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Revamped Recommendation Hub</strong></h3>
|
||
<p><a href="https://console.cloud.google.com/home/recommendations"><span style="text-decoration: underline; vertical-align: baseline;">Recommendation Hub</span></a><span style="vertical-align: baseline;"> is a centralized page on Google Cloud that helps you view all of your recommendations in one place across multiple categories: cost, security, performance, reliability, manageability, and even sustainability. We recently made improvements to help you better understand the recommendations you have and to help you focus on the ones that are the most impactful: </span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">1. Organization-view of recommendations<br/></strong><span style="vertical-align: baseline;">One of our most in-demand features: you can now view all recommendations across all of your projects in your organization in one UI! Simply change the picker at the top left of the screen to choose an organization, and Active Assist shows all the recommendations under your organization (as long as you have the correct IAM permissions).</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/hubble-1.gif"
|
||
|
||
alt="hubble-1">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p role="presentation"><strong style="vertical-align: baseline;">2. Pre-filtered recommendations by value category<br/></strong><span style="vertical-align: baseline;">You can now view all of your recommendations under one category in a simple </span><span style="vertical-align: baseline;">table view, so you can prioritize and focus on the recommendations that are the most </span><span style="vertical-align: baseline;">relevant and important to you.</span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">3. Custom sorting and filtering<br/></strong><span style="vertical-align: baseline;">With our new table views, you can sort and filter by different fields, such as product category, recommendation, cost savings, priority, etc. so you can find and view recommendations more easily. </span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/hubble-2.gif"
|
||
|
||
alt="hubble-2">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Four new recommendations</strong></h3>
|
||
<p><span style="vertical-align: baseline;">We’re continually adding new recommendations to the Active Assist portfolio based on customer feedback, to help you manage risk and optimize operations. </span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">1. Cloud deprecation and breaking change recommendations<br/></strong><span style="vertical-align: baseline;">At Google Cloud we take pains to provide backwards compatibility for our services. However, from time to time, we need to evolve the platform in a way that could impact some users e.g., for security purposes. In addition to following a stringent process to minimize customer impact, Active Assist now includes recommendations about potential breaking changes, providing an additional mechanism for customers to learn about them.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/hub-3.max-1000x1000.png"
|
||
|
||
alt="hub-3">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Our new </span><a href="https://cloud.google.com/recommender/docs/deprecation-change-recommender"><span style="text-decoration: underline; vertical-align: baseline;">cloud deprecation and breaking changes recommender</span></a><span style="vertical-align: baseline;"> helps identify Cloud resources that will be affected by upcoming deprecations and breaking changes while providing guidelines on how to manage them. Like our other recommendations, you can view them through our </span><a href="https://console.cloud.google.com/home/recommendations"><span style="text-decoration: underline; vertical-align: baseline;">Recommendation Hub UI</span></a><span style="vertical-align: baseline;">, API, and BigQuery export. </span></p>
|
||
<p><span style="vertical-align: baseline;">Deprecation and breaking change recommendations are offered at no charge and are available for all users today. Making sure you get ahead of these changes is important to help prevent any disruptions to your environment and ensure you are on the most reliable and supported services. </span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">2. IAM for BigQuery recommendations<br/></strong><span style="vertical-align: baseline;">We’ve expanded the popular IAM Recommender to include </span><a href="https://cloud.google.com/policy-intelligence/docs/review-apply-role-recommendations-datasets"><span style="text-decoration: underline; vertical-align: baseline;">IAM recommendations on BigQuery datasets</span></a><span style="vertical-align: baseline;">. If your principals have roles on BigQuery datasets but they are not using all of the permissions within that role, you can now receive recommendations to remove or replace any of those roles. These recommendations help you enforce the principle of least privilege by ensuring that principals have only the permissions that they actually need. </span></p>
|
||
<p><span style="vertical-align: baseline;">You can view your recommendations through UI, API, or BigQuery export. The recommendations are currently free but will require Security Command Center Premium after April 29th. </span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">3. Advisory Notifications recommendations<br/></strong><a href="https://cloud.google.com/advisory-notifications/docs/recommendations-overview"><span style="text-decoration: underline; vertical-align: baseline;">Advisory Notifications</span></a><span style="vertical-align: baseline;"> provides IAM policy recommendations to ensure the right parties within your organization have access to view critical security and privacy notifications in the Google Cloud console, so that they can receive and quickly address security notifications. </span></p>
|
||
<p role="presentation"><strong style="vertical-align: baseline;">4. Recent change recommendations<br/></strong><span style="vertical-align: baseline;">We want to help you detect and mitigate issues (e.g., service outages) caused by misconfigurations to your important cloud resources. The new </span><a href="https://cloud.google.com/recommender/docs/recent-change-recommendations"><span style="text-decoration: underline; vertical-align: baseline;">recent change recommendations</span></a><span style="vertical-align: baseline;"> automatically flags recent risky changes to cloud resources that are identified as important based on their usage and other signals. For example, if you deleted a highly used project, recent change recommendations will proactively warn you about the risks associated with the change, helping to identify — and prevent — unintended issues.. </span></p>
|
||
<p><span style="vertical-align: baseline;">We’re excited about these latest </span><a href="https://cloud.google.com/recommender/docs/change-risk-recommendations"><span style="text-decoration: underline; vertical-align: baseline;">change risk recommendations</span></a><span style="vertical-align: baseline;">, and hope they will help you both prevent and mitigate misconfigurations and risky changes to your infrastructure. Try out the new features on </span><a href="https://console.cloud.google.com/home/recommendations"><span style="text-decoration: underline; vertical-align: baseline;">Recommendation Hub</span></a><span style="vertical-align: baseline;"> yourself. If you have any feedback, please feel free to reach out to </span><a href="mailto:active-assist-feedback@google.com"><span style="text-decoration: underline; vertical-align: baseline;">active-assist-feedback@google.com</span></a><span style="vertical-align: baseline;">.</span></p></div>
|
||
<div class="block-related_article_tout">
|
||
|
||
|
||
|
||
|
||
|
||
<div class="uni-related-article-tout h-c-page">
|
||
<section class="h-c-grid">
|
||
<a href="https://cloud.google.com/blog/products/management-tools/introducing-active-assist-change-risk-recommenders/"
|
||
data-analytics='{
|
||
"event": "page interaction",
|
||
"category": "article lead",
|
||
"action": "related article - inline",
|
||
"label": "article: {slug}"
|
||
}'
|
||
class="uni-related-article-tout__wrapper h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
|
||
h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3 uni-click-tracker">
|
||
<div class="uni-related-article-tout__inner-wrapper">
|
||
<p class="uni-related-article-tout__eyebrow h-c-eyebrow">Related Article</p>
|
||
|
||
<div class="uni-related-article-tout__content-wrapper">
|
||
<div class="uni-related-article-tout__image-wrapper">
|
||
<div class="uni-related-article-tout__image" style="background-image: url('')"></div>
|
||
</div>
|
||
<div class="uni-related-article-tout__content">
|
||
<h4 class="uni-related-article-tout__header h-has-bottom-margin">Active Assist change risk recommenders: Introducing a new way to prevent misconfigurations</h4>
|
||
<p class="uni-related-article-tout__body">Active Assist change risk recommendations help prevent and detect common misconfigurations to help reduce risk, and improve operational r...</p>
|
||
<div class="cta module-cta h-c-copy uni-related-article-tout__cta muted">
|
||
<span class="nowrap">Read Article
|
||
<svg class="icon h-c-icon" role="presentation">
|
||
<use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#mi-arrow-forward"></use>
|
||
</svg>
|
||
</span>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</a>
|
||
</section>
|
||
</div>
|
||
|
||
</div></description><pubDate>Tue, 07 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/management-tools/active-assist-gets-new-hub-and-recommendations/</guid><category>AI & Machine Learning</category><category>Databases</category><category>Management Tools</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>What’s new with Active Assist: New Hub UI and four new recommendations</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/management-tools/active-assist-gets-new-hub-and-recommendations/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Sharon Fang</name><title>Product Manager</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Ryan Ismert</name><title>Product Manager</title><department></department><company></company></author></item><item><title>repareo adopts cloud-based microservices architecture to scale its auto repair business</title><link>https://cloud.google.com/blog/products/application-modernization/how-repareo-is-modernizing-auto-repair-with-google-cloud/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">When it comes to auto repair, most of us are at the mercy of our mechanic. We take our car to the local garage, pay for the job, and hope for the best. And with little transparency into the work that has been done, or the costs involved, we can be left feeling unsure whether we got value for money.</span></p>
|
||
<p><span style="vertical-align: baseline;">At </span><a href="https://www.repareo.de/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">repareo</span></a><span style="vertical-align: baseline;">, we are transforming the customer experience of vehicle repair, modernizing the market to make it e-commerce ready, while giving customers full transparency over the car-repair process. Now, they can describe their vehicle problem on our site and immediately receive a list of local garages, including customer reviews, cost breakdowns, and availability, allowing them to make an informed decision about where to take their car. And because our site has direct interfaces with the garages’ booking systems, customers are able to book their preferred garage and appointment directly on our site, saving time calling garages for availability. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Outgrowing our monolithic architecture</strong></h3>
|
||
<p><span style="vertical-align: baseline;">This year, we will be launching our new infrastructure on Google Cloud. Previously, repareo was built on a monolithic system using a small, hosted server, which was both easy for our small development team to maintain and allowed us to grow the business in a cost-efficient way. However, as we added more features and services, our monolith grew, which had an impact on our development speed, creating a bottleneck for the rest of our application. </span></p>
|
||
<p><span style="vertical-align: baseline;">Reliability became an issue too. repareo is integrated with car management fleets and leasing companies, and many drivers access our services through their car leasing app, making us highly dependent on third-party APIs to function effectively. As we grew, the increase in traffic resulted in these APIs becoming sluggish. During periods of peak traffic, such as the German tire-change season, we would see a 300% surge in traffic, placing a significant strain on our server, which was unable to scale effectively, causing our services to grind to a halt. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Modernizing infrastructure to modernize the market</strong></h3>
|
||
<p><span style="vertical-align: baseline;">The turning point came last year, when we signed a major deal with a leading global e-commerce player to integrate with its vehicle parts marketplace, enabling customers to book an installation during the checkout process. Realizing that we would need our infrastructure to be able to handle an expected tenfold increase in demand, while conforming with our new service-level agreements (SLAs), we decided it was time to move to the cloud. </span></p>
|
||
<p><span style="vertical-align: baseline;">We knew that a migration would bring other benefits too, enabling us to build a microservices architecture to develop services in modules, as well as allowing us to place certain workloads close to a leading global eCommerce player in vehicle parts in California, for fast response times. As we looked at cloud providers, Google Cloud stood out for the range of technologies and services it offered, with </span><a href="https://cloud.google.com/bigquery?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery</span></a><span style="vertical-align: baseline;"> and </span><a href="https://cloud.google.com/apigee?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Apigee</span></a><span style="vertical-align: baseline;"> particularly impressing us as uniquely advanced solutions in the market. </span></p>
|
||
<p><span style="vertical-align: baseline;">Beyond the technology, however, we were just as impressed with Google Cloud’s deep understanding of our industry and its business network within the automotive sector, as well as by the personal relationship we quickly built with the Google Cloud team. Migrating to a new provider is a once-in-a-lifetime decision, a marriage of sorts, and with Google Cloud the relationship immediately felt like it was built to last. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">A robust, reliable system for a smoother customer experience</strong></h3>
|
||
<p><span style="vertical-align: baseline;">We’re currently halfway through our migration, which has proved to be a steep learning curve for us, given the scale of the undertaking for our small team of developers. However, Apigee has helped to make the migration smooth by enabling us to easily set up a staging environment to test and adjust our system before going live, with no impact on our users. </span></p>
|
||
<p><span style="vertical-align: baseline;">We expect to have completed the migration in less than six months in total. Once live, we will have a robust, scalable system, capable of meeting the needs of a significantly larger user base. Building and managing our APIs with Apigee means we will be able to use the caching system to cache the high number of API requests on the site, allowing us to offer high-performance buffering algorithms without having to drastically increase the scale of the underlying system. And because Apigee’s logging system is so well developed, we will easily be able to spot and remedy any integration issues, to ensure our APIs function effectively. As a result, our customers will enjoy a smooth, reliable booking system and real-time repair updates, while garages will benefit from far wider reach. </span></p>
|
||
<p><span style="vertical-align: baseline;">We won’t need to worry about being able to handle fluctuations in demand either, as the autoscale feature of </span><a href="https://cloud.google.com/kubernetes-engine?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Kubernetes Engine</span></a><span style="vertical-align: baseline;"> (GKE) will automatically scale up our workloads to meet surges in traffic and scale down again during quiet periods to help ensure we aren’t using more compute than necessary. This cost-efficient provisioning means we will no longer have to worry about the ability of our system to handle periods of peak traffic, with our customers benefiting from a fast, reliable service. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Taking our developers up a gear with easy-to-use, managed services</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Our development efficiency has already significantly improved, thanks to the built-in features and managed services of Google Cloud. Apigee, for example, has a built-in key management system to enable APIs to communicate securely, which means our development team doesn’t need to spend time and money building our own system. Similarly, the fact that </span><a href="https://cloud.google.com/sql?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Cloud SQL</span></a><span style="vertical-align: baseline;"> is a managed database means we don’t need to spend time updating and maintaining it. GKE, meanwhile, improves our developer efficiency thanks to its easy integration with automation tools, increasing our deployment speed by at least 15%.</span></p>
|
||
<p><span style="vertical-align: baseline;">All of this means that our developers are free to focus on the core business logic and developing new features, such as new RPA-based technology to gather appointment availability from garage websites, which we were able to build and release inside a week, where it would have taken a month using the old infrastructure.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">A firm foundation for sustained growth</strong></h3>
|
||
<p><span style="vertical-align: baseline;">With a number of other significant deals in the pipeline, repareo is now entering a period of sustained growth as we rapidly increase our customer base and prepare to enter new markets. That level of scaling simply wouldn’t be possible without Google Cloud, with its global network and regional data centers making it easy to move into any new region and enjoy rapid response times. While its scalable architecture means we can be confident that our infrastructure will always be able to scale with us.</span></p>
|
||
<p><span style="vertical-align: baseline;">As we continue our mission to make auto repair more transparent and convenient for more people, we are confident that with Google Cloud we have the right provider to help move our business into the fast lane.</span></p></div></description><pubDate>Tue, 07 May 2024 08:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/application-modernization/how-repareo-is-modernizing-auto-repair-with-google-cloud/</guid><category>Infrastructure Modernization</category><category>Customers</category><category>Application Modernization</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/image_30.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>repareo adopts cloud-based microservices architecture to scale its auto repair business</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/image_30.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/application-modernization/how-repareo-is-modernizing-auto-repair-with-google-cloud/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Philipp Haac</name><title>CEO, repareo</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Tobias Reisner</name><title>CTO & Co-Founder, repareo</title><department></department><company></company></author></item><item><title>Maintain business continuity across regions with BigQuery managed disaster recovery</title><link>https://cloud.google.com/blog/products/data-analytics/bigquery-gets-managed-disaster-recovery/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Geographical redundancy is a fundamental part of building a resilient cloud-based data strategy. For many years, BigQuery has offered an industry-leading 99.99% </span><a href="https://cloud.google.com/bigquery/sla"><span style="text-decoration: underline; vertical-align: baseline;">uptime service-level agreement</span></a><span style="vertical-align: baseline;"> (SLA) for availability within a single geographical region. Full redundancy across two data centers within a single region is included with every BigQuery dataset you create and is managed in a completely transparent manner. </span></p>
|
||
<p><span style="vertical-align: baseline;">For customers looking for enhanced redundancy across large geographic regions, we are now introducing </span><strong style="vertical-align: baseline;">managed disaster recovery</strong><span style="vertical-align: baseline;"> for BigQuery. This feature, now in preview, offers automated failover of compute and storage and a new cross-regional SLA tailored for business-critical workloads. This feature enables you to ensure business continuity in the unlikely event of a total regional infrastructure outage. Managed disaster recovery also provides failover configurations for capacity reservations, so you can manage query and storage failover behavior. This feature is available through </span><a href="https://cloud.google.com/bigquery/docs/editions-intro"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery Enterprise Plus edition</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">How does it work?</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Customers using BigQuery’s enterprise plus edition can now configure their capacity reservations to enable automated failover across distinct geographic regions. Extending the capabilities of BigQuery’s </span><a href="https://cloud.google.com/bigquery/docs/data-replication"><span style="text-decoration: underline; vertical-align: baseline;">cross-region dataset replication</span></a><span style="vertical-align: baseline;">, failover reservations ensure that the location of both data and compute resources are coordinated during a disaster recovery event.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/image1_fRF1oyb.png"
|
||
|
||
alt="Image 1 - First image on blog">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Slot capacity in the secondary region for enterprise plus edition reservations are provisioned and maintained automatically at </span><span style="font-style: italic; vertical-align: baseline;">no additional cost</span><span style="vertical-align: baseline;">. Some competitive products require customers to duplicate their compute clusters in the secondary location.</span></p>
|
||
<p><span style="vertical-align: baseline;">In the event of a total regional outage, the secondary region can be promoted to the primary role for both compute and data. With BigQuery’s query routing layer, failover is completely transparent to end users and tools.</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Primary region:</strong><span style="vertical-align: baseline;"> The region containing the current primary replica of a </span><a href="https://cloud.google.com/bigquery/docs/datasets-intro"><span style="text-decoration: underline; vertical-align: baseline;">dataset</span></a><span style="vertical-align: baseline;">. This is also the region where the dataset data can be modified (e.g. loads, DDL, or DML).</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Secondary region:</strong><span style="vertical-align: baseline;"> The region where the failover reservation standby capacity and replicated datasets are available in the case of a regional outage.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Failover reservation:</strong><span style="vertical-align: baseline;"> An enterprise plus edition reservation configured with a primary/secondary region pair. Note: Datasets are attached to failover reservations. </span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">The dataset replica in the primary region is the </span><span style="font-style: italic; vertical-align: baseline;">primary replica</span><span style="vertical-align: baseline;">, and the replica in the secondary region is the </span><span style="font-style: italic; vertical-align: baseline;">secondary replica</span><span style="vertical-align: baseline;">. These roles are swapped during the failover process.</span></p>
|
||
<p><span style="vertical-align: baseline;">The primary replica is writeable, and the secondary replica is read-only. Writes to the primary replica are asynchronously replicated to the secondary replica. Within each region, the data is stored redundantly in two zones. Network traffic never leaves the Google Cloud network.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">What is a region pair? </strong></h3>
|
||
<p><span style="vertical-align: baseline;">A region pair in BigQuery’s managed disaster recovery is a pair of regions that are geographically supported by </span><a href="https://cloud.google.com/storage/docs/availability-durability#turbo-replication"><span style="text-decoration: underline; vertical-align: baseline;">turbo replication</span></a><span style="vertical-align: baseline;"> and compute redundancy. Within the defined region pair, BigQuery replicates data between the two regions and manages secondary available capacity. This replication allows BigQuery managed disaster recovery to provide high availability and durability for data. Customers are able to define their desired region pair (based on the supported regions) per failover reservation. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Supported region pairs</strong></h3>
|
||
<p><span style="vertical-align: baseline;">BigQuery’s managed disaster recovery feature supports failover reservations across specific region pairs (</span><a href="https://cloud.google.com/storage/docs/locations#location-dr"><span style="text-decoration: underline; vertical-align: baseline;">similar to Cloud Storage, for regions within a geographic area</span></a><span style="vertical-align: baseline;">). You can designate either region in a pair for your initial primary or secondary region.</span></p>
|
||
<h2><strong style="vertical-align: baseline;">Capacity in the secondary region</strong></h2>
|
||
<p><span style="vertical-align: baseline;">BigQuery ensures that the capacity of your primary region will be available in your secondary region within five minutes of a failover. This assurance applies to your reservation baseline, whether it’s used or not. BigQuery also provides the same level of </span><a href="https://cloud.google.com/bigquery/docs/slots-autoscaling-intro"><span style="text-decoration: underline; vertical-align: baseline;">autoscaling</span></a><span style="vertical-align: baseline;"> availability as provided in the primary.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">How much does it cost?</strong></h3>
|
||
<p><span style="vertical-align: baseline;">BigQuery's managed disaster recovery feature is available with the </span><a href="https://cloud.google.com/bigquery/docs/editions-intro"><span style="text-decoration: underline; vertical-align: baseline;">Enterprise Plus edition</span></a><span style="vertical-align: baseline;">. Standby compute capacity in the secondary region is included with the per slot-hour price with no requirement to purchase separate standby capacity. As an option, you may choose to provision additional Enterprise Plus reservations in the secondary region, specifically for read-only queries.</span></p>
|
||
<p><span style="vertical-align: baseline;">Managed disaster recovery customers are billed for </span><a href="https://cloud.google.com/bigquery/docs/data-replication"><span style="text-decoration: underline; vertical-align: baseline;">replicated storage</span></a><span style="vertical-align: baseline;"> in the primary and secondary regions for associated datasets. At GA, this feature will automatically use </span><a href="https://cloud.google.com/storage/docs/managing-turbo-replication"><span style="text-decoration: underline; vertical-align: baseline;">turbo replication</span></a><span style="vertical-align: baseline;"> for data transfer between regions. </span></p>
|
||
<p> </p>
|
||
<div align="left">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;">
|
||
<div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"><table><colgroup><col/><col/></colgroup>
|
||
<tbody>
|
||
<tr>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><strong style="vertical-align: baseline;">SKU</strong></p>
|
||
</td>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><strong style="vertical-align: baseline;">Billing description</strong></p>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">Enterprise Plus Edition</span></p>
|
||
</td>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">$0.10 / slot-hr (ex. US Pricing)</span></p>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">Storage</span></p>
|
||
</td>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">Storage bytes in the secondary region are billed at the same list price as storage bytes in the primary region. See </span><a href="https://cloud.google.com/bigquery/pricing#storage"><span style="text-decoration: underline; vertical-align: baseline;">BigQuery Storage pricing</span></a><span style="vertical-align: baseline;"> for more information.</span></p>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">Data transfer</span></p>
|
||
</td>
|
||
<td style="vertical-align: top; border: 1px solid #000000; padding: 16px;">
|
||
<p><span style="vertical-align: baseline;">Managed disaster recovery uses turbo replication*</span></p>
|
||
<p><span style="vertical-align: baseline;">Data transfer used during replication:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">is charged based on physical bytes</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">is charged on a per physical GB replicated basis. </span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">Note: </span><a href="https://cloud.google.com/storage/pricing#north-america_1"><span style="text-decoration: underline; vertical-align: baseline;">Turbo replication will be 2x pricing of “default replication” </span></a></p>
|
||
</td>
|
||
</tr>
|
||
</tbody>
|
||
</table></div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p><strong><span style="vertical-align: baseline;">* </span><span style="vertical-align: baseline;">Turbo replication is not available during preview but will be enabled automatically at general availability (GA).</span></strong></p>
|
||
<h3><strong style="vertical-align: baseline;">Recovery Time Objective (RTO)</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Promotion of a secondary reservation and associated datasets takes less than five minutes, even if the primary region is down. All queries in flight are canceled and rejected during the RTO timeline.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Recovery Point Objective (RPO)</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Data will be less than 15 minutes old in secondary dataset replicas configured for failover reservation between supported region pairs, turbo replication enabled and only after initial replication is completed (also known as backfill). </span></p>
|
||
<p><strong style="vertical-align: baseline;">Note: Turbo replication and RPO/RPO with SLA are not available during preview.</strong></p>
|
||
<h3><strong style="vertical-align: baseline;">Configuration in action</strong></h3>
|
||
<p><span style="vertical-align: baseline;">During preview, managed disaster recovery configuration is supported via the BigQuery Console (UI) and SQL. The following workflow shows how you can set up and manage disaster recovery in BigQuery:</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/Image_2_-_Second_image_on_blog.max-1000x1000.png"
|
||
|
||
alt="Image 2 - Second image on blog">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Create a replica for a given dataset</strong></p>
|
||
<p><span style="vertical-align: baseline;">To replicate a dataset, use the </span><a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#alter_schema_add_replica_statement"><span style="text-decoration: underline; vertical-align: baseline;">ALTER SCHEMA ADD REPLICA DDL statement</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<p><span style="vertical-align: baseline;">After you add a replica, it takes time for the initial copy operation to complete. You can still run queries referencing the primary replica while the data is being replicated, with no reduction in query-processing capacity.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;-- Create the primary replica in the primary region.\r\nCREATE SCHEMA my_dataset OPTIONS(location=&#x27;us-west1&#x27;);\r\n-- Create a replica in the secondary region.\r\nALTER SCHEMA my_dataset\r\nADD REPLICA `us-east1`\r\nOPTIONS(location=&#x27;us-east1&#x27;);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc4aa93d0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Configure a failover reservation + attach a dataset</strong></p>
|
||
<p><span style="vertical-align: baseline;">The first step is to create a failover reservation and specify its secondary location. Specifying a secondary location can also be done for existing Enterprise Plus reservations.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;CREATE RESERVATION `project1.region-us-west1.my_failover_reservation` \r\n OPTIONS (slot_capacity = 200, edition = ENTERPRISE_PLUS,\r\n secondary_location=&#x27;us-east1);&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc4aa9d30&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The next step is to associate one or more datasets with the failover reservation. The dataset needs to be replicated in the same primary / secondary region as specified in the reservation.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;ALTER SCHEMA `my_dataset`\r\n SET OPTIONS (failover_reservation = \r\n `project1.my_failover_reservation`);&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc4aa9190&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Promote the failover reservation + dataset in the secondary</strong></p>
|
||
<p><span style="vertical-align: baseline;">Fail over the reservation and associated datasets. This must be performed from the secondary region.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;ALTER RESERVATION `project1.region-us-east1.my_failover_reservation` \r\n SET OPTIONS (is_primary = true);&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc4aa91c0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Fail back to original primary</strong></p>
|
||
<p><span style="vertical-align: baseline;">Fail back the reservation and associated datasets (performed from the new secondary/old primary).</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;ALTER RESERVATION `project1.region-us-west1.my_failover_reservation` \r\n SET OPTIONS (is_primary = true);&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc4aa91f0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Getting started</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Business continuity is paramount for customers with mission-critical data environments. We are excited to make the preview of BigQuery’s managed disaster recovery feature available for your testing. You can learn more about managed disaster recovery and how to get started in the BigQuery </span><a href="https://cloud.google.com/bigquery/docs/managed-disaster-recovery"><span style="text-decoration: underline; vertical-align: baseline;">managed disaster recovery QuickStart</span></a><span style="vertical-align: baseline;">. </span></p></div></description><pubDate>Mon, 06 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/data-analytics/bigquery-gets-managed-disaster-recovery/</guid><category>Management Tools</category><category>Data Analytics</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Maintain business continuity across regions with BigQuery managed disaster recovery</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/data-analytics/bigquery-gets-managed-disaster-recovery/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Larry Henderson</name><title>Product Manager</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Brian Welcker</name><title>Director, Product Management</title><department></department><company></company></author></item><item><title>Advancing the art of AI-driven security with Google Cloud</title><link>https://cloud.google.com/blog/products/identity-security/advancing-the-art-of-ai-driven-security-with-google-cloud-at-rsa/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The advent of generative AI has unlocked new opportunities to empower defenders and security professionals. We have already seen how AI can transform malware analysis at scale as we work to deliver better outcomes for defenders. In fact, using Gemini 1.5 Pro, we were recently able to </span><a href="https://cloud.google.com/blog/topics/threat-intelligence/gemini-for-malware-analysis"><span style="text-decoration: underline; vertical-align: baseline;">reverse engineer and analyze</span></a><span style="vertical-align: baseline;"> the decompiled code of the WannaCry malware in a single pass — and identify the killswitch — in only 34 seconds. </span></p>
|
||
<p><span style="vertical-align: baseline;">Our vision for AI is to accelerate your ability to protect and defend against threats by shifting from manual, time-intensive efforts to assisted and, ultimately, </span><a href="https://inthecloud.withgoogle.com/security-product-vision-ai-security/download.html" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">semi-autonomous security</span></a><span style="vertical-align: baseline;"> — while providing you with curated tools and services to </span><a href="https://services.google.com/fh/files/misc/best-practices-for-securely-deploying-ai-on-google-cloud.pdf" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">secure your AI data, models, applications, and infrastructure</span></a><span style="vertical-align: baseline;">. We do this by empowering defenders with Gemini in Security, which uses SecLM, our security-tuned API, as well as providing tools and services to manage AI risk to your environment. Our Mandiant experts are able to help you secure your AI journey wherever you are.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_CbJIeSS.max-1000x1000.png"
|
||
|
||
alt="1">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="5bgn0">Managing AI risk and empowering defenders with gen AI.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Today at the RSA Conference in San Francisco, we’re sharing more on our vision for the intersection between AI and cybersecurity, including how we help organizations secure AI systems and provide AI tools to support defenders. We are introducing new AI offerings from Mandiant Consulting and new features in Security Command Center Enterprise to help address security challenges when adopting AI. We are also announcing the general availability of Gemini across several security offerings including </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;"> and </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Security Operations</span></a><span style="vertical-align: baseline;"> to further empower defenders with generative AI.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">New services leverage security and AI expertise from Google</strong></h3>
|
||
<p><span style="vertical-align: baseline;">As customers integrate AI into every area of their business, they tell us that securing their use of AI is essential. The recent </span><a href="https://services.google.com/fh/files/misc/csa_state_of_ai_and_security_survey_google_cloud.pdf" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">State of AI and Security Survey Report</span></a><span style="vertical-align: baseline;"> from the </span><a href="https://cloudsecurityalliance.org/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Security Alliance</span></a><span style="vertical-align: baseline;"> highlighted that while many professionals are confident in their organization’s ability to protect AI systems, there is still a significant portion that recognize the risks of underestimating threats. </span></p>
|
||
<p><span style="vertical-align: baseline;">Our </span><a href="https://safety.google/cybersecurity-advancements/saif/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Secure AI Framework</span></a><span style="vertical-align: baseline;"> (SAIF) provides a taxonomy of risks associated with AI workloads and recommended mitigations. Today we are announcing new offerings from </span><a href="http://cloud.google.com/security/solutions/mandiant-ai-consulting"><span style="text-decoration: underline; vertical-align: baseline;">Mandiant Consulting</span></a><span style="vertical-align: baseline;"> that can help organizations support SAIF and secure the use of AI. Mandiant's AI consulting services can help assess the security of your AI pipelines and test your AI defense and response with </span><a href="https://cloud.google.com/blog/transform/prompt-findings-our-ai-red-teams-first-report-qa"><span style="text-decoration: underline; vertical-align: baseline;">red teaming</span></a><span style="vertical-align: baseline;">. These services can also help your defenders identify and implement ways to use AI to enhance cyber defenses and streamline investigative capabilities. </span></p>
|
||
<p><span style="vertical-align: baseline;">“The use of AI opens up a world of possibilities and enterprises recognize that in order to take advantage of the potential of these innovations, they need to get ahead of new security risks,” said Jurgen Kutscher, vice president, Mandiant Consulting, Google Cloud. “From helping secure training data to assessing AI applications for vulnerabilities, our Mandiant Consulting experts can provide recommendations based on Google’s own experience protecting and deploying AI. We’re excited to bring these new services to market to help our clients leverage AI more securely and transform their operations."</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/NSS_demo_RSA_2024_small.gif"
|
||
|
||
alt="NSS demo RSA 2024 small">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="oojmj">Notebook Security Scanner identifies package vulnerabilities and recommends next steps to remediate individual packages.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Securing AI workloads against risks</strong></h3>
|
||
<p><span style="vertical-align: baseline;">We are also announcing new AI-protection capabilities that can help our customers implement SAIF by building on our release of </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-security-command-center-enterprise"><span style="text-decoration: underline; vertical-align: baseline;">Security Command Center Enterprise</span></a><span style="vertical-align: baseline;"> — our cloud risk-management solution that fuses cloud security and enterprise security operations: </span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Notebook Security Scanner</strong><span style="vertical-align: baseline;">, now available in preview, detects and provides remediation advice for vulnerabilities introduced by open-source software installed in managed notebooks.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Model Armor</strong><span style="vertical-align: baseline;">, expected to be in preview in Q3, can</span><strong style="vertical-align: baseline;"> </strong><span style="vertical-align: baseline;">enable customers to inspect, route, and protect foundation model prompts and responses. It can help customers </span><a href="https://cloud.google.com/transform/5-gen-ai-security-terms-busy-business-leaders-should-know"><span style="text-decoration: underline; vertical-align: baseline;">mitigate risks</span></a><span style="vertical-align: baseline;"> such as prompt injections, jailbreaks, toxic content, and sensitive data leakage. Model Armor will integrate with products across Google Cloud, including Vertex AI. </span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">If you’d like to learn more about early access for Model Armor, you can</span><span style="vertical-align: baseline;"> </span><a href="https://forms.gle/egcAxwmyaQzaeZH4A" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">sign up here</span></a><span style="vertical-align: baseline;">.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/3_Sfm220W.gif"
|
||
|
||
alt="3">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="thm68">Model Armor allows users to configure policies and set content safety filters to help block or redact inappropriate model prompts and responses.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><span style="vertical-align: baseline;">Empowering defenders with new gen AI security tools</span></h3>
|
||
<p><span style="vertical-align: baseline;">Today, we’ve also shared how security teams can better defend against threats with </span><a href="https://cloud.google.com/security/products/security-operations"><span style="text-decoration: underline; vertical-align: baseline;">Google Security Operations</span></a><span style="vertical-align: baseline;">, our AI-powered platform to help empower SOC teams to more easily detect and respond to threats. Gemini in Security Operations now includes a new assisted investigation feature that navigates users through the platform based on the context of an investigation. It can help hunt for the latest threats with vital information from Google Threat Intelligence and MITRE, analyze security events, create detections using natural language, and recommend next steps to take. </span></p>
|
||
<p><span style="vertical-align: baseline;">Users can also ask Gemini to create a response playbook using natural language, which can simplify the time-consuming task of manually constructing one. The user can further refine the generated playbook and simulate its execution. These new enhancements can give security teams a boost across the detection and response lifecycle. </span></p>
|
||
<p><span style="vertical-align: baseline;">“Gemini in Security Operations is enabling us to enhance the efficiency of our Cybersecurity Operations Center program as we continue to drive operational excellence,” said Ronald Smalley, senior vice president, cybersecurity operations, Fiserv. “Detection engineers can create detections and playbooks with less effort, and security analysts can find answers quickly with intelligent summarization and natural language search. This is critical as SOC teams continue to manage increasing data volumes and need to detect, validate, and respond to events faster.“</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/4_Mtsip4A.gif"
|
||
|
||
alt="4">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="thm68">Gemini in Security Operations aids investigations and helps users easily create rules for detections.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">We also are introducing </span><a href="https://cloud.google.com/security/products/threat-intelligence/?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;">, a new offering that can help you reduce the time it takes to identify and protect against novel threats by bringing together investigative learnings from Mandiant frontline experts, the VirusTotal intel community, and Google threat insights from protecting billions of devices and user accounts. </span></p>
|
||
<p><span style="vertical-align: baseline;">With Gemini in Threat Intelligence, analysts can now conversationally search Mandiant’s vast frontline research to understand threat actor behaviors in seconds, and read AI-powered summaries of relevant open-source intelligence (OSINT) articles the platform automatically ingests to help reduce investigation time. </span></p>
|
||
<p><span style="vertical-align: baseline;">“Our main objective is to understand the purpose of the threat actor. The AI summaries provided by Gemini in Threat Intelligence make it easy to get an overview of the actor, information about relevant entities, and which regions they're targeting,” said the director of information security at a leading multinational professional services organization. “The information flows really smoothly and helps us gather the intelligence we need in a fraction of the time."</span></p>
|
||
<p><span style="vertical-align: baseline;">Plus, Gemini in Threat Intelligence includes </span><a href="https://blog.virustotal.com/2024/01/uncovering-hidden-threats-with.html" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Code Insight</span></a><span style="vertical-align: baseline;">, which can inspect more than 200 file types, summarize their unique properties, and identify potentially malicious code. Gemini makes it easier for security professionals to understand the threats that matter most to their organization and take action to respond.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/5_qBjrMG1.gif"
|
||
|
||
alt="5">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="thm68">Gemini in Google Threat Intelligence allows users to conversationally search Mandiant’s vast corpus of frontline research.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Make Google part of your security team</strong></h3>
|
||
<p><span style="vertical-align: baseline;">With rapid advances in AI technology, the line of what is possible is a moving target. We have a </span><a href="https://inthecloud.withgoogle.com/security-product-vision-ai-security/download.html" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">vision</span></a><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline;">for a world in which the practice of “doing security” is less laborious and more durable, as AI offloads routine tasks and frees the experts to focus on the most complex issues. </span></p>
|
||
<p><span style="vertical-align: baseline;">Organizations can now address security challenges with the same capabilities that Google uses to keep more people and organizations safe online than anyone else in the world</span></p>
|
||
<p><span style="vertical-align: baseline;">To learn more about AI and security, and the rest of Google Cloud Security’s co</span><span style="vertical-align: baseline;">mprehensive portfolio</span><span style="vertical-align: baseline;">, come meet us in person at our </span><a href="https://www.rsaconference.com/usa/expo-and-sponsors/sponsor-details/google-cloud%20security-1690826518852001xqzp" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">RSA Conference booth</span></a><span style="vertical-align: baseline;"> (N5644). You can also catch us at our RSA Conference </span><a href="https://cloud.google.com/blog/products/identity-security/your-insiders-guide-to-google-cloud-security-at-rsa-conference-2024"><span style="text-decoration: underline; vertical-align: baseline;">keynotes, presentations, and meetups</span></a><span style="vertical-align: baseline;">, and get the latest </span><a href="http://cloud.google.com/security/ai"><span style="text-decoration: underline; vertical-align: baseline;">AI and Security updates here</span></a><span style="vertical-align: baseline;">.</span></p></div></description><pubDate>Mon, 06 May 2024 13:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/identity-security/advancing-the-art-of-ai-driven-security-with-google-cloud-at-rsa/</guid><category>Security & Identity</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/AI-driven_security.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Advancing the art of AI-driven security with Google Cloud</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/AI-driven_security.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/identity-security/advancing-the-art-of-ai-driven-security-with-google-cloud-at-rsa/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Steph Hay</name><title>Senior Director, Google Cloud Security</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Umesh Shankar</name><title>Chief Technologist, Google Cloud Security</title><department></department><company></company></author></item><item><title>Introducing Google Security Operations: Intel-driven, AI-powered SecOps</title><link>https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">In the generative AI-era, security teams are looking for a fully-operational, high-performing security operations solution that can drive productivity while empowering defenders to detect and mitigate new threats.</span></p>
|
||
<p><span style="vertical-align: baseline;">Today at the RSA Conference in San Francisco, we’re announcing </span><a href="https://cloud.google.com/blog/products/identity-security/advancing-the-art-of-ai-driven-security-with-google-cloud-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">AI innovations across the Google Cloud Security portfolio</span></a><span style="vertical-align: baseline;">, including </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;">, and the latest release of Google Security Operations. </span><span style="vertical-align: baseline;">Today’s update is designed help to reduce the do-it-yourself complexity of SecOps and enhance the productivity of your entire Security Operations Center.</span></p></div>
|
||
<div class="block-video">
|
||
|
||
|
||
|
||
<div class="article-module article-video ">
|
||
<figure>
|
||
<a class="h-c-video h-c-video--marquee"
|
||
href="https://youtube.com/watch?v=R1zObTV2oZo"
|
||
data-glue-modal-trigger="uni-modal-R1zObTV2oZo-"
|
||
data-glue-modal-disabled-on-mobile="true">
|
||
|
||
|
||
|
||
|
||
<div class="article-video__aspect-image"
|
||
style="background-image: url(https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Google_Cloud_YouTube__Brand_Anthem.max-1000x1000.jpg);">
|
||
<span class="h-u-visually-hidden">Introducing Google Security Operations</span>
|
||
</div>
|
||
|
||
<svg role="img" class="h-c-video__play h-c-icon h-c-icon--color-white">
|
||
<use xlink:href="#mi-youtube-icon"></use>
|
||
</svg>
|
||
</a>
|
||
|
||
|
||
</figure>
|
||
</div>
|
||
|
||
<div class="h-c-modal--video"
|
||
data-glue-modal="uni-modal-R1zObTV2oZo-"
|
||
data-glue-modal-close-label="Close Dialog">
|
||
<a class="glue-yt-video"
|
||
data-glue-yt-video-autoplay="true"
|
||
data-glue-yt-video-height="99%"
|
||
data-glue-yt-video-vid="R1zObTV2oZo"
|
||
data-glue-yt-video-width="100%"
|
||
href="https://youtube.com/watch?v=R1zObTV2oZo"
|
||
ng-cloak>
|
||
</a>
|
||
</div>
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Turn intelligence into action</strong></h3>
|
||
<p><span style="vertical-align: baseline;">At </span><a href="https://cloud.withgoogle.com/next/session-library?filters=track-security-professionals#all" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Next ‘24</span></a><span style="vertical-align: baseline;">, we shared how Applied Threat Intelligence can help teams turn intelligence into action, uncover more threats with less effort, and unlock deeper threat hunting and investigation workflows. Today we are unveiling new features that will use AI to automatically generate detections based on new threat discoveries. Coming later this year, this new capability will help enable you to identify malicious activity operating in your environment, and share clear directions that guide you through triage and response. </span></p>
|
||
<p><span style="vertical-align: baseline;">“Google Security Operations provides access to unique threat intelligence and advanced capabilities that are highly integrated into the platform. It enables security teams to surface the latest threats in a turnkey way that doesn’t require complicated engineering,” said Michelle Abraham, research director, IDC. ”Google is a potential partner for organizations in the fight against existing and emerging threats.”</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image1_lLkyWjS.max-1000x1000.png"
|
||
|
||
alt="image1">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="06wt8">Google Security Operations is a unified, AI and intel-driven platform for threat detection, investigation, and response.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Uncover the latest threats with curated detections</strong></h3>
|
||
<p><span style="vertical-align: baseline;">To help reduce manual processes and provide better security outcomes for our customers, </span><a href="https://cloud.google.com/security/products/security-operations"><span style="text-decoration: underline; vertical-align: baseline;">Google Security Operations </span></a><span style="vertical-align: baseline;">includes a rich set of curated detections. Developed and maintained regularly by Google and Mandiant experts, curated detections can enable customers to detect threats relevant to their environment. Notable new curated detections include:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Cloud detections</strong><span style="vertical-align: baseline;"> can addresses serverless threats, cryptomining incidents across Google Cloud, all Google Cloud and Security Command Center Enterprise findings, anomalous user behavior rules, machine learning-generated lists of prioritized endpoint alerts (based on factors such as user and entity context), and baseline coverage for AWS including identity, compute, data services, and secret management. We have also added detections based on learnings from the Mandiant </span><a href="https://cloud.google.com/blog/products/identity-security/a-year-in-the-cybersecurity-trenches-with-mandiant-managed-defense"><span style="text-decoration: underline; vertical-align: baseline;">Managed Defense team</span></a><span style="vertical-align: baseline;">. Detections are now available in Google Security Operations Enterprise and Enterprise Plus packages. </span></p>
|
||
</li>
|
||
</ul>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Frontline threat detections</strong><span style="vertical-align: baseline;"> can provide coverage for recently-detected methodologies, and is based on threat actor tactics, techniques and procedures (TTPs), including from nation-states and newly-detected malware families. New threats discovered by Mandiant’s elite team, including during incident response engagements, are then made available as detections. It is now available in the Google Security Operations Enterprise Plus package. </span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">Drive productivity for all with AI-powered SecOps </strong></h3>
|
||
<p><span style="vertical-align: baseline;">The addition of Gemini in Security Operations can elevate the skills of your security team. It can help reduce the time security analysts spend writing, running, and refining searches and triaging complex cases by approximately sevenfold. Security teams can search for additional context, better understand threat actor campaigns and tactics, initiate response sequences and receive guided recommendations on next steps — all using natural language. Today we are sharing two exciting updates to Gemini in Security Operations.</span></p>
|
||
<p><span style="vertical-align: baseline;">Now generally available, the </span><strong style="vertical-align: baseline;">Investigation Assistant</strong><span style="vertical-align: baseline;"> feature can help security professionals make faster decisions and respond to threats with more precision and speed by answering questions, summarizing events, hunting for threats, creating rules, and receiving recommended actions based on the context of investigations.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/2_-_Investigation_Assistant.gif"
|
||
|
||
alt="2 - Investigation Assistant">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="06wt8">Investigation Assistant can help answer questions, summarize events, hunt for threats, create rules, and recommend actions.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Playbook Assistant</strong><span style="vertical-align: baseline;">, now in preview, can help teams easily build response playbooks, customize configurations, and incorporate best practices — helping simplify time-consuming tasks that require deep expertise.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/3_-_Playbook_Assistant.gif"
|
||
|
||
alt="3 - Playbook Assistant">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="06wt8">Playbook Assistant can help build response playbooks, customize configurations, and incorporate best practices.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Reduce manual work with autonomous parsers </strong></h3>
|
||
<p><span style="vertical-align: baseline;">Getting data into the system and maintaining the pipeline is a critical yet time consuming task in security operations. As log sources change and new fields need to be extracted, security engineers and architects are often required to spend considerable time writing new parsing logic and ensuring backward compatibility. </span></p>
|
||
<p><span style="vertical-align: baseline;">Today we are excited to announce that Google Security Operations can now </span><strong style="vertical-align: baseline;">automatically parse log files</strong><span style="vertical-align: baseline;"> by extracting all key-value pairs to make them available for search, rules, and analytics. Available in preview, automatic parsing can help reduce the maintenance overhead of parsers in general, and also reduce the time consuming task of creating custom parsers. It supports JSON-based logs, and we will be adding support for other log formats. Automatically parsing log files can help security teams have the right data and context, making for faster and more effective investigations and detection authoring.</span></p></div>
|
||
<div class="block-video">
|
||
|
||
|
||
|
||
<div class="article-module article-video ">
|
||
<figure>
|
||
<a class="h-c-video h-c-video--marquee"
|
||
href="https://youtube.com/watch?v=23rAFK1l4l8"
|
||
data-glue-modal-trigger="uni-modal-23rAFK1l4l8-"
|
||
data-glue-modal-disabled-on-mobile="true">
|
||
|
||
|
||
|
||
|
||
<div class="article-video__aspect-image"
|
||
style="background-image: url(https://storage.googleapis.com/gweb-cloudblog-publish/images/4-_Google_Cloud_YouTube__WhyCustomersLove.max-1000x1000.jpg);">
|
||
<span class="h-u-visually-hidden">Hear Why Customers Love Google Security Operations</span>
|
||
</div>
|
||
|
||
<svg role="img" class="h-c-video__play h-c-icon h-c-icon--color-white">
|
||
<use xlink:href="#mi-youtube-icon"></use>
|
||
</svg>
|
||
</a>
|
||
|
||
|
||
</figure>
|
||
</div>
|
||
|
||
<div class="h-c-modal--video"
|
||
data-glue-modal="uni-modal-23rAFK1l4l8-"
|
||
data-glue-modal-close-label="Close Dialog">
|
||
<a class="glue-yt-video"
|
||
data-glue-yt-video-autoplay="true"
|
||
data-glue-yt-video-height="99%"
|
||
data-glue-yt-video-vid="23rAFK1l4l8"
|
||
data-glue-yt-video-width="100%"
|
||
href="https://youtube.com/watch?v=23rAFK1l4l8"
|
||
ng-cloak>
|
||
</a>
|
||
</div>
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Raise the bar for defense</strong></h3>
|
||
<p><span style="vertical-align: baseline;">For customers in need of expert support for managing Google Security Operations, we’ve got you covered. Google Security Operations can also work in concert with Mandiant Managed Defense and Mandiant Hunt, which can help you to reduce risks to your organization. Mandiant's team of seasoned defenders, analysts, and threat hunters work seamlessly with your security team and the AI-infused capabilities of Google Security Operations to quickly and effectively hunt or monitor, detect, triage, investigate, and respond to incidents. </span></p>
|
||
<p><span style="vertical-align: baseline;">And for our public sector customers that may have more specialized requirements, we offer </span><a href="https://youtu.be/Uw5glgDmDhc?t=1709" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Google SecOps CyberShield</span></a><span style="vertical-align: baseline;"> to help governments worldwide build an enhanced cyber threat capability. </span></p>
|
||
<p><span style="vertical-align: baseline;">To learn more about Google Security Operations, and the rest of Google Cloud Security’s comprehensive portfolio including an </span><a href="https://cloud.google.com/blog/products/identity-security/chrome-enterprise-expands-ecosystem-to-strengthen-endpoint-security-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">expanded Chrome Enterprise ecosystem</span></a><span style="vertical-align: baseline;">, come meet us in person at our </span><a href="https://www.rsaconference.com/usa/expo-and-sponsors/sponsor-details/google-cloud%20security-1690826518852001xqzp" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">RSA Conference booth</span></a><span style="vertical-align: baseline;"> (N5644). You can also catch us at our </span><a href="https://cloud.google.com/blog/products/identity-security/your-insiders-guide-to-google-cloud-security-at-rsa-conference-2024"><span style="text-decoration: underline; vertical-align: baseline;">keynotes, presentations, and meetups</span></a><span style="vertical-align: baseline;"> including our session, “</span><a href="https://www.rsaconference.com/usa/agenda/session/Bye-Bye%20DIY%20Frictionless%20Security%20Operations%20with%20Google" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Bye-Bye DIY: Frictionless Security Operations with Google</span></a><span style="vertical-align: baseline;">,” on Tuesday, May 7, at 1:15 p.m. PDT. </span></p>
|
||
<p><span style="vertical-align: baseline;">Not attending RSAC? Join us for our upcoming webinar, “</span><a href="https://www.brighttalk.com/webcast/18282/610656?utm_source=GoogleCloudSecurity&amp;utm_medium=email&amp;utm_campaign=610656" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Stay ahead of the latest threats with intelligence-driven security operations</span></a><span style="vertical-align: baseline;">,” on Wednesday, May 22, at 11:00 a.m. PDT. </span></p></div></description><pubDate>Mon, 06 May 2024 13:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa/</guid><category>Security & Identity</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/0-_blog_header_-_19132_Security_BlogHeader_2.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Introducing Google Security Operations: Intel-driven, AI-powered SecOps</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/0-_blog_header_-_19132_Security_BlogHeader_2.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Chris Corde</name><title>Director, Product Management</title><department></department><company></company></author></item><item><title>Introducing Google Threat Intelligence: Actionable threat intelligence at Google scale</title><link>https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">For decades, threat intelligence solutions have had two main challenges: They lack a comprehensive view of the threat landscape, and to get value from intelligence, organizations have to spend excess time, energy, and money trying to collect and operationalize the data. </span></p>
|
||
<p><span style="vertical-align: baseline;">Today at the RSA Conference in San Francisco, we are announcing </span><a href="https://cloud.google.com/security/products/threat-intelligence/?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;">, a new offering that combines the unmatched depth of our Mandiant frontline expertise, the global reach of the VirusTotal community, and the breadth of visibility only Google can deliver, based on billions of signals across devices and emails. Google Threat Intelligence includes Gemini in Threat Intelligence, our AI-powered agent that provides conversational search across our vast repository of threat intelligence, enabling customers to gain insights and protect themselves from threats faster than ever before.</span></p>
|
||
<p><span style="vertical-align: baseline;">“While there is no shortage of threat intelligence available, the challenge for most is to contextualize and operationalize intelligence relevant to their specific organization,” said Dave Gruber, principal analyst, Enterprise Strategy Group. “Unarguably, Google provides two of the most important pillars of threat intelligence in the industry today with VirusTotal and Mandiant. Integrating both into a single offering, enhanced with AI and Google threat insights, offers security teams a new means to operationalize actionable threat intelligence to better protect their organizations.”</span></p>
|
||
<h3><span style="vertical-align: baseline;">Unmatched visibility into threats</span></h3>
|
||
<p><span style="vertical-align: baseline;">Google Threat Intelligence provides unparalleled visibility into the global threat landscape. We offer deep insights from Mandiant’s leading incident response and threat research team, and combine them with our massive user and device footprint and VirusTotal’s broad crowdsourced malware database. </span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Google threat insights: </strong><span style="vertical-align: baseline;">Google protects 4 billion devices and 1.5 billion email accounts, and blocks 100 million phishing attempts per day. This provides us with a vast sensor array and a unique perspective on internet and email-borne threats that allow us to connect the dots back to attack campaigns.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Frontline intelligence:</strong><span style="vertical-align: baseline;"> Mandiant's eIite incident responders and security consultants dissect attacker tactics and techniques, using their experience to help customers defend against sophisticated and relentless threat actors across the globe in over 1,100 investigations annually.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Human-curated threat intelligence:</strong><span style="vertical-align: baseline;"> Mandiant’s global threat experts meticulously monitor threat actor groups for activity and changes in their behavior to contextualize ongoing investigations and provide the insights you need to respond. </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Crowdsourced threat intelligence:</strong><span style="vertical-align: baseline;"> VirusTotal's global community of over 1 million users continuously contributes potential threat indicators, including files and URLs, to offer real-time insight into emerging attacks.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Open-source threat intelligence</strong><span style="vertical-align: baseline;">: We use open-source threat intelligence to enrich our knowledge base with current discoveries from the security community.</span></p>
|
||
</li>
|
||
</ul></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--medium
|
||
|
||
|
||
h-c-grid__col
|
||
|
||
h-c-grid__col--4 h-c-grid__col--offset-4
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_vision_wheel.max-1000x1000.png"
|
||
|
||
alt="1 - vision wheel">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="g00bm">Google Threat Intelligence boasts a diverse set of sources that provide a panoramic view of the global threat landscape and the granular details needed to make informed decisions.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">This comprehensive view allows Google Threat Intelligence to help protect your organization in a variety of ways, including external threat monitoring, attack surface management, digital risk protection, Indicators of Compromise (IOC) analysis, and expertise.</span></p>
|
||
<h3><span style="vertical-align: baseline;">AI-driven operationalization</span></h3>
|
||
<p><span style="vertical-align: baseline;">Traditional approaches to operationalizing threat intelligence are labor-intensive and can slow down your ability to respond to evolving threats, potentially taking days or weeks to respond.</span></p></div>
|
||
<div class="block-video">
|
||
|
||
|
||
|
||
<div class="article-module article-video ">
|
||
<figure>
|
||
<a class="h-c-video h-c-video--marquee"
|
||
href="https://youtube.com/watch?v=QGUri8v4THc"
|
||
data-glue-modal-trigger="uni-modal-QGUri8v4THc-"
|
||
data-glue-modal-disabled-on-mobile="true">
|
||
|
||
|
||
|
||
|
||
<div class="article-video__aspect-image"
|
||
style="background-image: url(https://storage.googleapis.com/gweb-cloudblog-publish/images/2_-_Code_Insights_thumbnail.max-1000x1000.png);">
|
||
<span class="h-u-visually-hidden">Google Threat Intelligence - Code Insights</span>
|
||
</div>
|
||
|
||
<svg role="img" class="h-c-video__play h-c-icon h-c-icon--color-white">
|
||
<use xlink:href="#mi-youtube-icon"></use>
|
||
</svg>
|
||
</a>
|
||
|
||
|
||
<figcaption class="article-video__caption h-c-page">
|
||
|
||
<h4 class="h-c-headline h-c-headline--four h-u-font-weight-medium h-u-mt-std">Google Threat Intelligence uses Gemini to analyze potentially malicious code and provides a summary of its findings.</h4>
|
||
|
||
|
||
</figcaption>
|
||
|
||
</figure>
|
||
</div>
|
||
|
||
<div class="h-c-modal--video"
|
||
data-glue-modal="uni-modal-QGUri8v4THc-"
|
||
data-glue-modal-close-label="Close Dialog">
|
||
<a class="glue-yt-video"
|
||
data-glue-yt-video-autoplay="true"
|
||
data-glue-yt-video-height="99%"
|
||
data-glue-yt-video-vid="QGUri8v4THc"
|
||
data-glue-yt-video-width="100%"
|
||
href="https://youtube.com/watch?v=QGUri8v4THc"
|
||
ng-cloak>
|
||
</a>
|
||
</div>
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">By combining our comprehensive view of the threat landscape with Gemini, we have supercharged the threat research processes, augmented defense capabilities, and reduced the time it takes to identify and protect against novel threats. Customers now have the ability to condense large data sets in seconds, quickly analyze suspicious files, and simplify challenging manual threat intelligence tasks.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">How Gemini helps simplify and assist with threat intelligence</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Gemini 1.5 Pro is a valuable part of Google Threat Intelligence, and we’ve integrated it so that it can more efficiently and effectively assist security professionals in combating malware.</span></p>
|
||
<p><span style="vertical-align: baseline;">Gemini 1.5 Pro </span><span style="vertical-align: baseline;">offers the world’s longest context window, with support for up to </span><a href="https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">1 million tokens</span></a><span style="vertical-align: baseline;">. It </span><span style="vertical-align: baseline;">can dramatically simplify the technical and labor-intensive process of reverse engineering malware — one of the most advanced malware-analysis techniques available to cybersecurity professionals. In fact, it was able to process the entire decompiled code of the malware file for </span><a href="https://cloud.google.com/blog/topics/threat-intelligence/gemini-for-malware-analysis"><span style="text-decoration: underline; vertical-align: baseline;">WannaCry in a single pass</span></a><span style="vertical-align: baseline;">, taking 34 seconds to deliver its analysis and identify the killswitch.</span></p>
|
||
<p><span style="vertical-align: baseline;">We also offer a Gemini-driven entity extraction tool to automate data fusion and enrichment. It can automatically crawl the web for relevant open source intelligence (OSINT), and classify online industry threat reporting. It then converts this information to knowledge collections, with corresponding hunting and response packs pulled from motivations, targets, tactics, techniques, and procedures (TTPs), actors, toolkits, and Indicators of Compromise (IoCs).</span></p>
|
||
<p><span style="vertical-align: baseline;">Google Threat Intelligence can </span><span style="vertical-align: baseline;">distill more than a decade of threat reports to produce comprehensive, custom summaries in seconds.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Make Google part of your security team</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Google Threat Intelligence is just one way we can help you in your threat intelligence journey. Whether you need cyber threat intelligence training for your staff, assistance with prioritizing complex threats, or even a dedicated threat analyst embedded in your team, our experts can act as an extension of your own team.</span></p>
|
||
<p><span style="vertical-align: baseline;">Google Threat Intelligence is part of Google Cloud Security’s comprehensive security portfolio, which includes </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Security Operations</span></a><span style="vertical-align: baseline;">, </span><a href="https://cloud.google.com/security/consulting/mandiant-services/?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Mandiant Consulting</span></a><span style="vertical-align: baseline;">, </span><a href="https://cloud.google.com/blog/products/identity-security/cloud-ciso-perspectives-tour-the-new-security-command-center-enterprise/"><span style="text-decoration: underline; vertical-align: baseline;">Security Command Center Enterprise</span></a><span style="vertical-align: baseline;">, and </span><a href="https://cloud.google.com/blog/products/identity-security/chrome-enterprise-expands-ecosystem-to-strengthen-endpoint-security-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Chrome Enterprise</span></a><span style="vertical-align: baseline;">. With our offerings, organizations can address security challenges with the same capabilities Google uses to keep more people and organizations safe online than anyone else in the world.</span></p>
|
||
<p><span style="vertical-align: baseline;">To learn more about </span><a href="https://cloud.google.com/security/products/threat-intelligence?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;"> and the rest of Google Cloud Security’s comprehensive portfolio, come meet us in person at our </span><a href="https://www.rsaconference.com/usa/expo-and-sponsors/sponsor-details/google-cloud%20security-1690826518852001xqzp" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">RSA Conference booth</span></a><span style="vertical-align: baseline;"> (N5644), and catch us at our </span><a href="https://cloud.google.com/blog/products/identity-security/your-insiders-guide-to-google-cloud-security-at-rsa-conference-2024"><span style="text-decoration: underline; vertical-align: baseline;">keynotes, presentations, and meetups</span></a><span style="vertical-align: baseline;">. </span><span style="vertical-align: baseline;">You can also register for our upcoming </span><a href="https://series.brighttalk.com/series/6490?utm_source=MandiantNA&amp;utm_medium=BrightTALK&amp;utm_campaign=6490" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence use-cases webinar series</span></a><span style="vertical-align: baseline;">, and read our expert analysis and in-depth research at the Google Cloud </span><a href="https://cloud.google.com/blog/topics/threat-intelligence"><span style="text-decoration: underline; vertical-align: baseline;">Threat Intelligence blog</span></a><span style="vertical-align: baseline;">.</span></p></div></description><pubDate>Mon, 06 May 2024 13:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa/</guid><category>Security & Identity</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/vision_circle.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Introducing Google Threat Intelligence: Actionable threat intelligence at Google scale</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/vision_circle.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Sunil Potti</name><title>VP/GM, Google Cloud Security</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Sandra Joyce</name><title>VP, Google Threat Intelligence</title><department></department><company></company></author></item><item><title>Chrome Enterprise expands ecosystem to strengthen endpoint security and Zero Trust access</title><link>https://cloud.google.com/blog/products/identity-security/chrome-enterprise-expands-ecosystem-to-strengthen-endpoint-security-at-rsa/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The modern workplace relies on web-based applications and cloud services, making browsers and their sensitive data a primary target for attackers. While the risks are significant, </span><a href="https://chromeenterprise.google/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Chrome Enterprise</span></a><span style="vertical-align: baseline;"> can help organizations simplify and strengthen their endpoint security with secure enterprise browsing. </span></p>
|
||
<p><span style="vertical-align: baseline;">Following our recent </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-chrome-enterprise-premium"><span style="text-decoration: underline; vertical-align: baseline;">Chrome Enterprise Premium</span></a><span style="vertical-align: baseline;"> launch, today at the RSA Conference in San Francisco, we’re announcing a growing ecosystem of security providers who are working with us to extend Chrome Enterprise’s browser-based protections and help enterprises protect their users working on the web and across corporate applications. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Expanding Zero Trust protections with Zscaler</strong></h3>
|
||
<p><span><span style="vertical-align: baseline;">Chrome Enterprise Premium offers advanced security across SaaS and private web applications for enterprises. Many organizations rely on </span><a href="https://www.zscaler.com/press/zscaler-joins-forces-google-offer-unparalleled-zero-trust-data-and-threat-protection" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Zscaler Private Access (ZPA)</span></a><span style="vertical-align: baseline;"> as an improved option over VPNs and firewalls to provide secure, Zero Trust access to private applications on-premises and in the cloud. Now security operations teams can add a layer of additional safeguards through Chrome Enterprise Premium, including: </span></span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Data protections</strong><span style="vertical-align: baseline;">: <span style="vertical-align: baseline;">Critical DLP functions including data exfiltration controls, copy, paste, and print restrictions, and watermarking capabilities. This complements Zscaler's data protection across endpoints, email, SaaS and cloud.</span></span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Threat prevention</strong><span style="vertical-align: baseline;">: Advanced malware scanning, real-time phishing security, and credential protections, augmenting Zscaler's inline inspection of encrypted traffic and built in threat protections.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Security insights</strong><span style="vertical-align: baseline;">: Additional telemetry and reporting across insider and external risks.</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">Google has collaborated with Zscaler to provide enterprises with a </span><a href="https://www.zscaler.com/resources/solution-briefs/zs-private-access-google-chrome-enterprise.pdf" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">solution guide</span></a><span style="vertical-align: baseline;"> that enables organizations to configure their network security products alongside Chrome Enterprise Premium for deeper security and protections.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Browser-based device trust with Cisco Duo</strong></h3>
|
||
<p><span style="vertical-align: baseline;">As attacks targeting end-users become more sophisticated, a multi-layered defense that includes a strong device access policy is crucial. Signals including user identity, device security, and location can enable dynamic, risk-based access decisions that further protect corporate data. </span></p>
|
||
<p><span style="vertical-align: baseline;">Enterprises can now use </span><a href="https://duo.com/partnerships/technology-partners/select-partners/google" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Duo Trusted Endpoints policy</span></a><span style="vertical-align: baseline;"> to enforce</span><span style="vertical-align: baseline;"> </span><a href="https://duo.com/product/device-trust" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">device trust</span></a><span style="vertical-align: baseline;"> using built-in Chrome Enterprise signals to deny access from unknown devices — without having to deploy additional agents and extensions. This integration allows organizations to:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Verify endpoint trust at login, and block unknown devices</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Manage device access from a centralized Duo dashboard</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Adjust granular policies for an organization of any size in a few clicks</span></p>
|
||
</li>
|
||
</ul></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_-_CE_and_Duo_Screenshot.max-1000x1000.png"
|
||
|
||
alt="2 - CE and Duo Screenshot">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="9ox4v">Duo's Trusted Endpoints feature lets organizations grant secure access to applications with policies that verify systems using signals from Chrome.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Data loss prevention with Trellix</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Data loss remains a top concern for enterprises, and the browser is a critical point for stopping data leaks. </span><a href="https://www.trellix.com/assets/data-sheets/trellix-dlp-for-chrome-enterprise-data-sheet.pdf" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Trellix DLP</span></a><span style="vertical-align: baseline;"> for Chrome Enterprise is now available as an integration to customers managing Chrome from the cloud. With the Trellix DLP integration, organizations can prevent data leaks in Chrome by:</span></p>
|
||
<ul>
|
||
<li role="presentation"><span style="vertical-align: baseline;">Monitoring and blocking file uploads with sensitive content</span></li>
|
||
<li role="presentation"><span style="vertical-align: baseline;">Tracking and preventing sensitive content from being copied and pasted to websites</span></li>
|
||
<li role="presentation"><span style="vertical-align: baseline;">Controlling print activity in Chrome browser and on local workstations</span></li>
|
||
</ul>
|
||
<p> </p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/3_-_CE_and_Trellix_Screenshot.max-1000x1000.png"
|
||
|
||
alt="3 - CE and Trellix Screenshot">
|
||
|
||
</a>
|
||
|
||
<figcaption class="article-image__caption "><p data-block-key="9ox4v">When sensitive information is detected in Chrome, the user is immediately notified with a pop-up.</p></figcaption>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Current Trellix DLP and Cisco Duo customers can implement these integrations by enrolling browsers into </span><a href="https://chromeenterprise.google/products/cloud-management/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Chrome Enterprise Core</span></a><span style="vertical-align: baseline;"> and setting up a one-time configuration, at no additional cost. Learn more about the Trellix DLP integration here and Cisco Duo integration </span><a href="https://chromeenterprise.google/solutions/integrations/?modal-id=ciscoduo" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">here</span></a><span style="vertical-align: baseline;">. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">Take the next step</strong></h3>
|
||
<p><span style="vertical-align: baseline;">To learn more about Chrome Enterprise, and the rest of Google Cloud Security’s comprehensive portfolio including our RSAC announcements on </span><a href="https://cloud.google.com/blog/products/identity-security/advancing-the-art-of-ai-driven-security-with-google-cloud-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Cloud Security and AI</span></a><span style="vertical-align: baseline;">, </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-threat-intelligence-actionable-threat-intelligence-at-google-scale-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Threat Intelligence</span></a><span style="vertical-align: baseline;">, and </span><a href="https://cloud.google.com/blog/products/identity-security/introducing-google-security-operations-intel-driven-ai-powered-secops-at-rsa"><span style="text-decoration: underline; vertical-align: baseline;">Google Security Operations</span></a><span style="vertical-align: baseline;">, come meet us in person at our </span><a href="https://www.rsaconference.com/usa/expo-and-sponsors/sponsor-details/google-cloud%20security-1690826518852001xqzp" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">RSA Conference booth</span></a><span style="vertical-align: baseline;"> (N5644), and catch us at our </span><a href="https://cloud.google.com/blog/products/identity-security/your-insiders-guide-to-google-cloud-security-at-rsa-conference-2024"><span style="text-decoration: underline; vertical-align: baseline;">keynotes, presentations, and meetups</span></a><span style="vertical-align: baseline;">. You can also learn more about </span><a href="https://chromeenterprise.google/products/chrome-enterprise-premium/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Chrome Enterprise here</span></a><span style="vertical-align: baseline;">.</span></p></div></description><pubDate>Mon, 06 May 2024 13:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/identity-security/chrome-enterprise-expands-ecosystem-to-strengthen-endpoint-security-at-rsa/</guid><category>Chrome Enterprise</category><category>Security & Identity</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/1-_Blog_header_banner.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Chrome Enterprise expands ecosystem to strengthen endpoint security and Zero Trust access</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/1-_Blog_header_banner.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/identity-security/chrome-enterprise-expands-ecosystem-to-strengthen-endpoint-security-at-rsa/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Kiran Nair</name><title>Senior Product Manager, Chrome Enterprise, Google</title><department></department><company></company></author></item><item><title>Uncomplicating the complex: How Spanner simplifies microservices-based architectures</title><link>https://cloud.google.com/blog/products/databases/why-spanner-is-a-good-fit-for-microservices-based-applications/</link><description><div class="block-paragraph_advanced"><p style="text-align: justify;"><span style="vertical-align: baseline;">In the realm of modern application design, developers have a range of choices available to them for crafting architectures that are not only simple, but also scalable, performant and resilient. Container platforms like Kubernetes (k8s) offer users the ability to seamlessly adjust node and pod specifications, so that services can scale. This scalability does not come at the expense of elasticity, and also ensures consistent performance for service consumers. So it’s no surprise that Kubernetes has become the de facto standard for building distributed and resilient systems in medium-to-large organizations.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Unfortunately, the level of maturity and standardization in the Kubernetes space available to system designers in the application layer doesn’t usually extend to the database layer that powers these services. And it goes without saying that the database layer also needs to be elastic, scalable, highly available, and resilient. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Further, the challenges are amplified when these services:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Are required to manage (transactional) states or </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Orchestrate distributed processes across multiple (micro-) services. </span></p>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Traditional relational database management software (RDBMS) brings with it side effects that are not aligned with a microservices way of thinking, and entails fairly significant trade-offs. In the sections below, we dive deeper into the scalability, availability and operational challenges faced by application designers specifically within the database tier. We then conclude with a description of how Spanner can help you build microservices-based systems without the often unspoken “impedance mismatch” between the application layer and the database layer.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">We look at this problem from a scalability and availability perspective, specifically in the context of databases that cater to OLTP workloads. We explore the intricacies involved in accommodating highly variable workloads, shedding light on the complexities associated with managing higher demands through the utilization of both replicas and sharding techniques.</span></p>
|
||
<h2 style="text-align: justify;"><strong style="vertical-align: baseline;">Wanted: scalability and availability</strong></h2>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">When it comes to scaling a traditional relational database, you have two choices (leaving aside caching strategies):</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Scale up:</strong><span style="vertical-align: baseline;"> To scale a database vertically, you typically augment its resources by adding more CPU power, increasing memory, or adding faster disks. However, these scale-up procedures usually incur downtime, affecting the availability of dependent services. </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Scale out:</strong><span style="vertical-align: baseline;"> Although vertically scaling up databases can be effective initially, it eventually encounters limitations. The alternative is to scale out database traffic, by introducing additional read replicas, employing sharding techniques, or even a combination of both. These methods come with their own trade-offs and introduce complexities, which lead to operational overhead.</span></p>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">In terms of </span><strong style="vertical-align: baseline;">availability</strong><span style="vertical-align: baseline;">, databases require maintenance, resulting in the need to coordinate regular periods of downtime. Relational databases can also be prone to hardware defects, network partitions, or subject to data center outages that bring a host of DR scenario challenges that you need to address and plan for. </span></p>
|
||
<p><strong style="vertical-align: baseline;">Examples of planned downtime:</strong></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">OS or database engine upgrades or patches </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Schema changes - Most database engines require downtime for the duration of a schema change</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><strong style="vertical-align: baseline;">Examples of unplanned downtime</strong></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Zonal or regional outages</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Network partitions</span></p>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Most “mature” practices for handling traditional RDBMSs run counter to modern application design principles and can significantly impact the availability and performance of services. Depending on the nature of the business, this can have consequences for revenue streams, compliance with regulations, or adversely impact customer satisfaction.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Let’s go over some of the key challenges associated with RDBMSs.</span></p>
|
||
<h3 style="text-align: justify;"><strong style="vertical-align: baseline;">Challenges associated with read-replicas</strong></h3></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1-READ-REPLICAS.max-1000x1000.png"
|
||
|
||
alt="1-READ-REPLICAS">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Database read replicas are a suitable tool for scaling out read operations and mitigating planned downtime, so that reads are at least available to the application layer. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">In order to reduce load on the primary database instance, replicas can be created to distribute read load across multiple machines and thus handle more read requests concurrently. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Replication between the primary and secondary replicas is usually done asynchronously. This means there can be a lag between when data is written to the primary database and when it is replicated to the read replicas. This can result in read operations getting slightly outdated (stale) data if they are directed to the replicas. This also dictates that guaranteed consistent queries need to be directed to primary instances. Synchronous replication is rarely an option, in particular, not in geo-distributed topologies, as it is complex, and comes with a range of issues such as:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Limiting the scalability of the system, as every write operation must wait for confirmation from the replica, causing performance bottlenecks and increasing latency</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Introducing a single point of failure — if the replica becomes unavailable or experiences issues, it can impact the availability of the primary database as well</span></p>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">And lastly, write throughput can become bottlenecked due to the limit on how much write traffic a single database can handle without performance degradation. Scaling writes still requires vertical scaling (more powerful hardware) or sharding (splitting data across multiple databases), which can lead to downtime, additional costs, and limits imposed by non-linearly escalating operational toil. Now let’s look at sharding challenges in a bit more detail.</span></p>
|
||
<h3 style="text-align: justify;"><strong style="vertical-align: baseline;">Sharding challenges</strong></h3></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2-SHARDING.max-1000x1000.png"
|
||
|
||
alt="2-SHARDING">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p style="text-align: justify;"><span style="vertical-align: baseline;">Sharding is a powerful tool for database scalability. When implemented correctly, it can enable applications to handle a much larger volume of read and write transactions. However, sharding does not come without its challenges and brings its own set of complexities that need careful navigation.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">There are multiple ways to shard databases. For instance, </span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">they can be split by user id ranges, </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">regions or </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">channels (e.g. web, mobile) etc.. </span></p>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">As shown in the above example, sharding by user id or region can lead to significant performance improvements, as smaller data ranges are hosted by individual databases and the traffic can be spread across these databases.</span></p>
|
||
<p style="text-align: justify;"><strong style="vertical-align: baseline;">Key considerations:</strong></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Deciding on the “right” kind of sharding: </strong><span style="vertical-align: baseline;">One of the primary challenges of sharding is the initial setup. Deciding on a sharding key, whether it be user ID, region, or another attribute, requires a deep understanding of your data access patterns. A poorly chosen sharding key can result in uneven data distribution, known as "shard imbalance," which can significantly dull the performance benefits of sharding.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Data integrity</strong><span style="vertical-align: baseline;"> is another significant concern. When data is spread across multiple shards, maintaining foreign-key relationships becomes difficult. Transactions that span multiple shards become complex and can result in increased latency and decreased integrity.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Operational complexity: </strong><span style="vertical-align: baseline;">Sharding introduces operational complexity. Managing multiple databases requires a more sophisticated approach to maintenance, backups, and monitoring. Each shard may need to be backed up separately, and restoring a sharded database to a consistent state can be challenging.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Re-sharding: </strong><span style="vertical-align: baseline;">As an application grows, the chosen sharding scheme might need to change. This process involves redistributing the data across a new set of shards, which can be time-consuming and risky, often requiring significant downtime or degraded performance during the transition.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><strong style="vertical-align: baseline;">Increased development complexity:</strong><span style="vertical-align: baseline;"> Application logic can become more complex because developers must account for the distribution of data. This could mean additional logic for routing queries to the correct shard, handling partial failures, and ensuring that transactions that need to operate across shards maintain consistency.</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3 style="text-align: justify;"><strong style="vertical-align: baseline;">Exploding complexity and operations</strong></h3></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/3-COMPLEXITY.max-1000x1000.png"
|
||
|
||
alt="3-COMPLEXITY">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p style="text-align: justify;"><span style="vertical-align: baseline;">Over time, database complexity can grow along with increased traffic, adding further toil to operations. For large systems, a combination of sharding along with attached scale-out read replicas might be required to help ensure cost-effective scalability and performance.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">This combined dual-strategy approach, while effective in handling increasing traffic, significantly ramps up the complexity of the system's architecture. The above illustration captures the need to add scalability and availability to a transactional relational database powering a service. It doesn’t even include full details on DR (e.g. backups), or geo-redundancy, nor does it cater to zero-to-low RPO/RTO requirements. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">Furthermore, the dual-strategy approach described above can:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">negatively impact the ease of service maintenance </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">increase operational demands, and </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">elevate the risk associated with the resolution of incidents</span></p>
|
||
</li>
|
||
</ul>
|
||
<h2><strong style="vertical-align: baseline;">Doesn’t NoSQL address this?</strong></h2>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">NoSQL databases began to emerge in the early 2000s as a response to traditional RDBMSs’ above-mentioned limitations. In the new era of big data and web-scale applications, NoSQL databases were designed to overcome the challenges of scalability, performance, flexibility and availability that were imposed by the growing volume of semi-structured data. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">However, the key tradeoff they made was to drop sound relational models, SQL, and support for ACID-compliant transactions. However, many prominent system architects have questioned the wisdom of abandoning these well-worn relational concepts for OLTP workloads, as they are essential features that still power mission-critical applications. As such, there’s been a recent trend to (re)introduce relational database features into NoSQL databases, such as ACID transactions in MongoDB and Cassandra Query Language (CQL) in Cassandra.</span></p>
|
||
<h3 style="text-align: justify;"><strong style="vertical-align: baseline;">Enter Spanner</strong></h3></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--medium
|
||
|
||
|
||
h-c-grid__col
|
||
|
||
h-c-grid__col--4 h-c-grid__col--offset-4
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/4-SIMPLIFY-WITH-SPANNER.max-1000x1000.png"
|
||
|
||
alt="4-SIMPLIFY-WITH-SPANNER">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p style="text-align: justify;"><a href="https://cloud.google.com/spanner"><span style="text-decoration: underline; vertical-align: baseline;">Spanner</span></a><span style="vertical-align: baseline;"> eliminates much of this complexity and helps facilitate a simple and easy-to-maintain architecture without most of the above-mentioned compromises. It combines relational concepts and features (SQL, ACID transactions) with seamless horizontal scalability, providing geo-redundancy with up to 99.999% availability that you want when designing a microservices-based application.</span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">We want to emphasize that we’re not arguing that Spanner is </span><span style="text-decoration: underline; vertical-align: baseline;">only</span><span style="vertical-align: baseline;"> a good fit for microservices. All the things that make Spanner a great fit for microservices also make it great for monolithic applications. </span></p>
|
||
<p style="text-align: justify;"><span style="vertical-align: baseline;">To summarize, a microservices architecture built on Spanner allows software architects to design systems where both the application and database provide:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">“Scale insurance” for future growth scenarios</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">An easy way to handle traffic spikes</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Cost efficiency through Spanner’s elastic and instant compute provisioning</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Up to 99.999% availability with geo-redundancy</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">No downtime windows (for maintenance or other upgrades)</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Enterprise-grade security such as encryption at rest and in-transit</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Features to cater for transactional workloads</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation" style="text-align: justify;"><span style="vertical-align: baseline;">Increases in developer productivity (e.g. SQL)</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">You can </span><a href="https://goo.gle/SpannerDatabaseUnlimited" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">learn more</span></a><span style="vertical-align: baseline;"> about what makes Spanner unique and how it’s being used today. Or </span><a href="https://cloud.google.com/spanner/docs/free-trial-quickstart"><span style="text-decoration: underline; vertical-align: baseline;">try it yourself for free</span></a><span style="vertical-align: baseline;"> for 90-days or for as little as $65 USD/month for a production-ready instance that grows with your business without downtime or disruptive re-architecture.</span></p></div></description><pubDate>Fri, 03 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/databases/why-spanner-is-a-good-fit-for-microservices-based-applications/</guid><category>Spanner</category><category>Databases</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Uncomplicating the complex: How Spanner simplifies microservices-based architectures</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/databases/why-spanner-is-a-good-fit-for-microservices-based-applications/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Szabolcs Rozsnyai</name><title>Senior Staff Solutions Architect, Spanner</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Karthi Thyagarajan</name><title>Senior Staff Solutions Architect, Spanner</title><department></department><company></company></author></item><item><title>Making API calls exactly once when using Workflows</title><link>https://cloud.google.com/blog/products/application-development/using-single-execution-calls-with-workflows/</link><description><div class="block-paragraph_advanced"><h2><strong style="vertical-align: baseline;">Introduction</strong></h2>
|
||
<p><span style="vertical-align: baseline;">One challenge with any distributed system, including </span><a href="https://cloud.google.com/workflows"><span style="text-decoration: underline; vertical-align: baseline;">Workflows</span></a><span style="vertical-align: baseline;">, is ensuring that requests sent from one service to another are processed exactly once, when needed; for example, when placing a customer order in a shipping queue, withdrawing funds from a bank account, or processing a payment.</span></p>
|
||
<p><span style="vertical-align: baseline;">In this blog post, we’ll provide an example of a website invoking Workflows, and Workflows in turn invoking</span><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline;">a </span><a href="https://cloud.google.com/functions"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Function</span></a><span style="vertical-align: baseline;">. We’ll show how to make sure both Workflows and the Cloud Function logic only runs once. We’ll also talk about how to invoke Workflows exactly once when using </span><a href="https://cloud.google.com/blog/topics/developers-practitioners/introducing-workflows-callbacks"><span style="text-decoration: underline; vertical-align: baseline;">HTTP callbacks</span></a><span style="vertical-align: baseline;">, </span><a href="https://cloud.google.com/pubsub"><span style="text-decoration: underline; vertical-align: baseline;">Pub/Sub</span></a><span style="vertical-align: baseline;"> messages, or </span><a href="https://cloud.google.com/tasks"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Tasks</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h2><strong style="vertical-align: baseline;">Invoke Workflows exactly once</strong></h2>
|
||
<p><span style="vertical-align: baseline;">Imagine you have an online store and you’re using Workflows to create new orders, save to Firestore, and process payments by calling a Cloud Function:</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image1_hVpeh5J.max-1000x1000.png"
|
||
|
||
alt="image1">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">A new customer order comes in, the website makes an API call to Workflows but receives an error. Two possible scenarios are:</span></p>
|
||
<p><span style="vertical-align: baseline;"> (1) The request is lost and the workflow is never invoked:</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--medium
|
||
|
||
|
||
h-c-grid__col
|
||
|
||
h-c-grid__col--4 h-c-grid__col--offset-4
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image2_1VaigE8.max-1000x1000.png"
|
||
|
||
alt="image2">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">(2) The workflow is invoked and executes successfully, however the response is lost:</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--medium
|
||
|
||
|
||
h-c-grid__col
|
||
|
||
h-c-grid__col--4 h-c-grid__col--offset-4
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image3_Lfq82pR.max-1000x1000.png"
|
||
|
||
alt="image3">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">How can you make sure the workflow executes once?</span></p>
|
||
<p><span style="vertical-align: baseline;">To solve this, the website retries the same request. One easy solution is to check if a document already exists in Firestore:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;main:\r\n params: []\r\n steps:\r\n - init:\r\n assign:\r\n - project_id: ${sys.get_env(&quot;GOOGLE_CLOUD_PROJECT_ID&quot;)}\r\n - order_id: &quot;12345&quot; # In practice we would pass in the order ID as a workflow parameter, e.g. ${params[0]}\r\n - firestore_collection: &quot;orders&quot;\r\n - URL: https://us-central1-&lt;your_project_id&gt;.cloudfunctions.net/processpayment\r\n - create_document:\r\n try:\r\n call: googleapis.firestore.v1.projects.databases.documents.createDocument\r\n args:\r\n collectionId: ${firestore_collection}\r\n parent: ${&quot;projects/&quot; + project_id + &quot;/databases/(default)/documents&quot;}\r\n query:\r\n documentId: ${order_id}\r\n except:\r\n as: e\r\n steps:\r\n - endEarly:\r\n return: ${e} # Exception is raised, e.g. ${e.code == 409} if doc already exists\r\n - processPayment:\r\n ...&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc698f910&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The </span><code style="vertical-align: baseline;">processPayment </code><span style="vertical-align: baseline;">step will execute only if a document is successfully created. This is effectively a 1-bit state machine, idempotent, and a valid solution. The downside of this solution is that it’s not extensible. We might want to complete additional work in this handler before changing states, or expand the number of states within the system. Next, let’s continue with a more advanced solution for the same problem.</span></p>
|
||
<h2><strong style="vertical-align: baseline;">Invoke Cloud Functions from Workflows exactly once</strong></h2>
|
||
<p><span style="vertical-align: baseline;">Let’s see what happens when the workflow uses a Cloud Function to process the payment. You might have the following step to call Cloud Functions:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;- processPayment:\r\n call: http.post\r\n args:\r\n url: https://us-central1-&lt;your_project_id&gt;.cloudfunctions.net/processpayment\r\n auth:\r\n type: OIDC&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc698feb0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">By default, Workflows offers </span><strong style="vertical-align: baseline;">at-most-once</strong><span style="vertical-align: baseline;"> delivery (no retries) with HTTP requests. That’s usually OK because 99.9+% of the time, the call is successful, and a response is received.</span></p>
|
||
<p><span style="vertical-align: baseline;">In the rare case of failure, a </span><a href="https://cloud.google.com/workflows/docs/reference/syntax/error-types#error-tags"><code style="text-decoration: underline; vertical-align: baseline;">ConnectionError</code></a><span style="vertical-align: baseline;"> might be raised. As in the website-to-workflow situation discussed previously, the workflow can’t tell which scenario occurred. Similarly, you can add retries. </span></p>
|
||
<p><span style="vertical-align: baseline;">Let’s add a </span><a href="https://cloud.google.com/workflows/docs/reference/syntax/retrying#default-retry-policy"><span style="text-decoration: underline; vertical-align: baseline;">default retry policy</span></a><span style="vertical-align: baseline;"> to handle this:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;- processPayment:\r\n try:\r\n call: http.post\r\n args:\r\n url: https://us-central1-&lt;your_project_id&gt;.cloudfunctions.net/processpayment\r\n auth:\r\n type: OIDC\r\n retry: ${http.default_retry} # Retries up to 5 times, includes &#x27;ConnectionError&#x27;&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc698f0a0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Let's say the second delivery scenario occurs where the request is received by the Cloud Function but the response is lost. By adding retries, Workflows will likely invoke the Cloud Function multiple times. When this happens, how do you ensure that the code in the Cloud Function only runs once? </span></p>
|
||
<p><span style="vertical-align: baseline;">You’ll need to add extra logic to the Cloud Function to check and update the payment state in Firestore:</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image4_HtrYAfU.max-1000x1000.png"
|
||
|
||
alt="image4">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Let’s also assume you want to track the workflow </span><code style="vertical-align: baseline;">EXECUTION_ID</code><span style="vertical-align: baseline;"> in Firestore and use the following </span><code style="vertical-align: baseline;">order_state </code><span style="vertical-align: baseline;">enum to allow for additional flexibility in payment processing:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;payment_not_processed // Initial state when an order is created\r\npayment_declined // Payment was not successful\r\npayment_successful // Payment processed successfully\r\n...&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc85b8c40&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">You can expand on the previous workflow and call a Cloud Function to process the payment:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;main:\r\n params: []\r\n steps:\r\n - init:\r\n assign:\r\n - project_id: ${sys.get_env(&quot;GOOGLE_CLOUD_PROJECT_ID&quot;)}\r\n - order_id: &quot;12345&quot; # In practice we would pass in the order ID as a workflow parameter, e.g. ${params[0]}\r\n - firestore_collection: &quot;orders&quot;\r\n - URL: https://us-central1-&lt;your_project_id&gt;.cloudfunctions.net/processpayment\r\n - create_document:\r\n try:\r\n call: googleapis.firestore.v1.projects.databases.documents.createDocument\r\n args:\r\n collectionId: ${firestore_collection}\r\n parent: ${&quot;projects/&quot; + project_id + &quot;/databases/(default)/documents&quot;}\r\n query:\r\n documentId: ${order_id}\r\n body:\r\n fields:\r\n order_state: # We set an initial state\r\n stringValue: &quot;payment_not_processed&quot;\r\n workflow_id: # And also track this workflow execution ID\r\n stringValue: ${sys.get_env(&quot;GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID&quot;)}\r\n except:\r\n as: e\r\n steps:\r\n - endEarly:\r\n return: ${e} # Exception is raised, e.g. ${e.code == 409} if doc already exists\r\n - processPayment:\r\n try:\r\n call: http.post\r\n args:\r\n url: ${URL} # Might get called multiple times!\r\n auth:\r\n type: OIDC\r\n body:\r\n order_id: ${order_id}\r\n result: r\r\n retry: ${http.default_retry}\r\n - returnStep:\r\n return: ${r}&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc85b8670&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Here’s the Cloud Function (Node.js v20) that processes the payment:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;const functions = require(\&#x27;@google-cloud/functions-framework\&#x27;);\r\nconst firestore = require(\&#x27;@google-cloud/firestore\&#x27;);\r\n\r\n\r\nfunctions.http(\&#x27;helloHttp\&#x27;, (req, res) =&gt; {\r\n const fs = new firestore.Firestore();\r\n try{\r\n// Reads the current state from Firestore and updates it within the same transaction to make this handler idempotent. Using a transaction is important. Note: It could be run multiple times but will only be committed once.\r\n return fs.runTransaction(t =&gt; {\r\n const docRef = fs.doc(&quot;orders/&quot; + req.body.order_id);\r\n return t.get(docRef).then(doc =&gt; {\r\n console.log(doc, \&#x27;=&gt;\&#x27;, doc);\r\n var state = doc.data().order_state\r\n // Only process the order if we haven\&#x27;t already\r\n if (state == &quot;payment_not_processed&quot;) {\r\n // Do payment stuff, e.g. debit account from another Firestore document\r\n // ...\r\n //\r\n state = &quot;payment_successful&quot;\r\n t.update(docRef, {order_state: state})\r\n res.status(200).send(state);\r\n return\r\n }\r\n res.status(200).send(&quot;request ignored, state already: &quot; + state);\r\n });\r\n }).then(result =&gt; {\r\n console.log(\&#x27;Transaction result: \&#x27;, result);\r\n });\r\n } catch (e) {\r\n console.log(\&#x27;Transaction failure:\&#x27;, e);\r\n } \r\n});&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8110610&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><code style="vertical-align: baseline;">package.json</code></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;{\r\n &quot;dependencies&quot;: {\r\n &quot;@google-cloud/functions-framework&quot;: &quot;^3.3.0&quot;,\r\n &quot;@google-cloud/firestore&quot;: &quot;^7.6.0&quot;\r\n }\r\n}&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc81107f0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The key takeaway is that all payment processing work occurs within a </span><strong style="vertical-align: baseline;">transaction</strong><span style="vertical-align: baseline;">, making all actions idempotent. The code within the transaction might run multiple times due to Workflows retries, but it’s only committed once. </span></p>
|
||
<h2><strong style="vertical-align: baseline;">What about HTTP callbacks, Pub/Sub, Cloud Tasks?</strong></h2>
|
||
<p><span style="vertical-align: baseline;">So far, we’ve talked about how to make website-to-workflow and Workflows to Cloud Functions requests, exactly once. There are other ways of invoking or resuming Workflows such as </span><a href="https://cloud.google.com/blog/topics/developers-practitioners/introducing-workflows-callbacks"><span style="text-decoration: underline; vertical-align: baseline;">HTTP callbacks</span></a><span style="vertical-align: baseline;">, Pub/Sub messages or Cloud Tasks. How do you make those requests exactly once? Let’s take a look.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Callbacks</strong></h3>
|
||
<p><span style="vertical-align: baseline;">The good news is that Workflows HTTP callbacks are fully idempotent by default. It’s safe to retry a callback if it fails. For example:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;- createCallbackStep:\r\n call: events.create_callback_endpoint\r\n args:\r\n http_callback_method: &quot;POST&quot;\r\n result: callback_details\r\n- sendOutURL:\r\n call: http.post\r\n args:\r\n url: &quot;https://your-endpoint.com/foo&quot;\r\n body:\r\n callback_to_use: ${callback_details.url}\r\n...\r\n- callbackWaitStep:\r\n call: events.await_callback\r\n args:\r\n callback: ${callback_details}&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8110220&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Let’s assume that the first callback returns an error to the external caller. Based on the error, the caller might not know if the workflow callback was received, and should retry the callback. On the second callback, the caller will receive one of the following HTTP status codes:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">429</strong><span style="vertical-align: baseline;"> indicates that the first callback was received successfully. The second callback is rejected by the workflow.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">200</strong><span style="vertical-align: baseline;"> indicates that the second callback was received successfully. The first callback was either never received, or was received and processed successfully. If the latter, the second callback is not processed because </span><code style="vertical-align: baseline;">await_callback </code><span style="vertical-align: baseline;">is called only once. The second callback is discarded at the end of the workflow.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">404</strong><span style="vertical-align: baseline;"> indicates that a callback is not available. Either the first callback was received and processed and the workflow has completed, or the workflow is not running (and has failed, for example). To confirm this, you’ll need to send an API request to query the workflow execution state.</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">For more details, see </span><a href="https://cloud.google.com/workflows/docs/creating-callback-endpoints#invoke-once"><span style="text-decoration: underline; vertical-align: baseline;">Invoke a workflow exactly once using callbacks</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Pub/Sub messages </strong></h3>
|
||
<p><span style="vertical-align: baseline;">When using Pub/Sub to </span><a href="https://cloud.google.com/workflows/docs/trigger-workflow-eventarc"><span style="text-decoration: underline; vertical-align: baseline;">trigger</span></a><span style="vertical-align: baseline;"> a new workflow execution, Pub/Sub uses </span><strong style="vertical-align: baseline;">at-least-once</strong><span style="vertical-align: baseline;"> delivery with Workflows, and will retry on any delivery failure. Pub/Sub messages are automatically deduplicated. You don’t need to worry about duplicate deliveries in that time window (24 hours).</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Cloud Tasks</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Cloud Tasks is commonly used to </span><a href="https://cloud.google.com/workflows/docs/tutorials/buffer-workflows-executions"><span style="text-decoration: underline; vertical-align: baseline;">buffer workflow executions</span></a><span style="vertical-align: baseline;"> and provides </span><strong style="vertical-align: baseline;">at-least-once </strong><span style="vertical-align: baseline;">delivery but it </span><strong style="vertical-align: baseline;">doesn’t</strong><span style="vertical-align: baseline;"> have message deduplication. Workflow handlers </span><a href="https://cloud.google.com/tasks/docs/dual-overview"><span style="text-decoration: underline; vertical-align: baseline;">should be idempotent</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h2><strong style="vertical-align: baseline;">Conclusion</strong></h2>
|
||
<p><span style="vertical-align: baseline;">Exactly-once request processing is a hard problem. In this blog post, we’ve outlined some scenarios where you might need exactly-once request processing when you’re using Workflows. We also provided some ideas on how you can implement it. The exact solution will depend on the actual use case and the services involved.</span></p></div></description><pubDate>Fri, 03 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/application-development/using-single-execution-calls-with-workflows/</guid><category>Developers & Practitioners</category><category>Application Development</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Making API calls exactly once when using Workflows</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/application-development/using-single-execution-calls-with-workflows/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Randy Spruyt</name><title>Workflows Team Lead</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Mete Atamel</name><title>Cloud Developer Advocate</title><department></department><company></company></author></item><item><title>Scalable multi-tenancy management with Config Sync and team scopes</title><link>https://cloud.google.com/blog/products/containers-kubernetes/how-to-use-config-sync-team-scopes/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Ensuring application and service teams have the resources they need is crucial for platform administrators. </span><a href="https://cloud.google.com/anthos/fleet-management/docs/team-management"><span style="text-decoration: underline; vertical-align: baseline;">Fleet team management</span></a><span style="vertical-align: baseline;"> features in Google Kubernetes Engine (GKE) make this easier, allowing each team to function as a separate “tenant” within a fleet. In conjunction with </span><a href="https://cloud.google.com/anthos-config-management/docs/config-sync-overview"><span style="text-decoration: underline; vertical-align: baseline;">Config Sync</span></a><span style="vertical-align: baseline;">, a GitOps service in GKE, platform administrators can streamline resource management for their teams across the fleet.</span></p>
|
||
<p><span style="vertical-align: baseline;">Specifically, with Config Sync team scopes, platform admins can define fleet-wide and team-specific cluster configurations such as resource quotas and network policies, allowing each application team to manage their own workloads within designated namespaces across clusters.</span></p>
|
||
<p><span style="vertical-align: baseline;">Let's walk through a few scenarios.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Separating resources for frontend and backend teams</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Let's say you need to provision resources for frontend and backend teams, each requiring their own tenant space. Using team scopes and fleet namespaces, you can control which teams access specific namespaces on specific member clusters.</span></p>
|
||
<p><span style="vertical-align: baseline;">For example, the backend team might access their </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">shoestore</code><span style="vertical-align: baseline;"> namespaces on </span><code style="vertical-align: baseline;">us-east-cluster</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">us-west-cluster</code><span style="vertical-align: baseline;"> clusters, while the frontend team has their </span><code style="vertical-align: baseline;">frontend-a</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">frontend-b</code><span style="vertical-align: baseline;"> namespaces on all three member clusters.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_intro.max-1000x1000.jpg"
|
||
|
||
alt="1 - intro">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Unlocking Dynamic Resource Provisioning with Config Sync<br/></strong><span style="vertical-align: baseline;">You can enable Config Sync by default at the fleet level using Terraform. Here’s a sample Terraform configuration:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;resource &quot;google_gke_hub_feature&quot; &quot;feature&quot; {\r\n name = &quot;configmanagement&quot;\r\n location = &quot;global&quot;\r\n provider = google\r\n fleet_default_member_config {\r\n configmanagement {\r\n config_sync {\r\n source_format = &quot;unstructured&quot;\r\n git {\r\n sync_repo = &quot;https://github.com/GoogleCloudPlatform/anthos-config-management-samples&quot;\r\n sync_branch = &quot;main&quot;\r\n policy_dir = &quot;fleet-tenancy/config&quot;\r\n secret_type = &quot;none&quot;\r\n }\r\n }\r\n }\r\n }\r\n}&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc89155b0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="font-style: italic; vertical-align: baseline;">Note: Fleet defaults are only applied to new clusters created in the fleet.</span></p>
|
||
<p><span style="vertical-align: baseline;">This Terraform configuration enables Config Sync as a </span><a href="https://cloud.google.com/anthos/fleet-management/docs/manage-features#configure_fleet-level_defaults"><span style="text-decoration: underline; vertical-align: baseline;">default fleet-level feature</span></a><span style="vertical-align: baseline;">. It installs Config Sync and instructs it to fetch Kubernetes manifests from a </span><a href="https://github.com/GoogleCloudPlatform/anthos-config-management-samples/tree/main/fleet-tenancy/config" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Git repository</span></a><span style="vertical-align: baseline;"> (specifically, the “main” branch and the “fleet-tenancy/config” folder). This configuration automatically applies to all clusters subsequently created within the fleet. This approach offers a powerful way of configuring manifests across fleet clusters without the need for manual installation and configuration on individual clusters.</span></p>
|
||
<p><span style="vertical-align: baseline;">Now that you’ve configured Config Sync as a default fleet setting, you might want to sync specific Kubernetes resources to designated namespaces and clusters for each team. Integrating Config Sync with team scopes streamlines this process.</span></p>
|
||
<p><strong style="vertical-align: baseline;">Setting team scope</strong><span style="vertical-align: baseline;"> <br/></span><span style="vertical-align: baseline;">Following this example, let’s assume you want to apply a different network policy for the backend team compared to the frontend team. Fleet team management features simplify the process of provisioning and managing infrastructure resources for individual teams, treating each team as a separate “tenant” within the fleet. </span></p>
|
||
<p><span style="vertical-align: baseline;">To manage separate tenancy, as shown in the above team scope diagram, first </span><a href="https://cloud.google.com/anthos/fleet-management/docs/setup-teams"><span style="text-decoration: underline; vertical-align: baseline;">set up team scopes</span></a><span style="vertical-align: baseline;"> for the backend and frontend teams. This involves defining fleet-level namespaces and adding fleet member clusters to each team scope.</span></p>
|
||
<p><span style="vertical-align: baseline;">Now, let's dive into those Kubernetes manifests that Config Sync syncs into the clusters.</span></p>
|
||
<p><strong style="vertical-align: baseline;">Applying team scope in Config Sync<br/></strong><span style="vertical-align: baseline;">Each </span><a href="https://cloud.google.com/anthos/fleet-management/docs/team-management#fleet_team_management_overview"><span style="text-decoration: underline; vertical-align: baseline;">fleet namespace</span></a><span style="vertical-align: baseline;"> in the cluster is automatically labeled with </span><code style="vertical-align: baseline;">fleet.gke.io/fleet-scope: &lt;scope name&gt;</code><span style="vertical-align: baseline;">. For example, the backend team scope contains the fleet namespaces </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">shoestore</code><span style="vertical-align: baseline;">, both labeled with </span><code style="vertical-align: baseline;">fleet.gke.io/fleet-scope: backend</code><span style="vertical-align: baseline;">. </span></p>
|
||
<p><span style="vertical-align: baseline;">Config Sync's </span><a href="https://cloud.google.com/anthos-config-management/docs/how-to/namespace-scoped-objects#namespaceselectors_in_unstructured_repositories"><code style="text-decoration: underline; vertical-align: baseline;">NamespaceSelector</code></a><span style="vertical-align: baseline;"> utilizes this label to target specific namespaces within a team scope. Here's the configuration for the backend team:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;apiVersion: configmanagement.gke.io/v1\r\nkind: NamespaceSelector\r\nmetadata:\r\n name: backend-scope\r\nspec:\r\n mode: dynamic\r\n selector:\r\n matchLabels:\r\n fleet.gke.io/fleet-scope: backend&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8915970&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Applying NetworkPolicies for the backend team<br/></strong><span style="vertical-align: baseline;">By annotating resources with </span><code style="vertical-align: baseline;">configmanagement.gke.io/namespace-selector: &lt;NamespaceSelector name&gt;</code><span style="vertical-align: baseline;">, they're automatically applied to the right namespaces. Here’s the </span><code style="vertical-align: baseline;">NetworkPolicy</code><span style="vertical-align: baseline;"> of the backend team:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;apiVersion: networking.k8s.io/v1\r\nkind: NetworkPolicy\r\nmetadata:\r\n name: be-deny-all\r\n annotations:\r\n configmanagement.gke.io/namespace-selector: backend-scope\r\nspec:\r\n ingress:\r\n - from:\r\n - podSelector: {}\r\n podSelector:\r\n matchLabels: null&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8915160&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">This </span><code style="vertical-align: baseline;">NetworkPolicy</code><span style="vertical-align: baseline;"> is automatically provisioned in the backend team's </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">shoestore</code><span style="vertical-align: baseline;"> namespaces, adapting to fleet changes like adding or removing namespaces and member clusters.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_-_add_networkpolicy.max-1000x1000.jpg"
|
||
|
||
alt="2 - add networkpolicy">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Extending the concept: ResourceQuotas for the frontend team<br/></strong><span style="vertical-align: baseline;">Here's how a </span><code style="vertical-align: baseline;">ResourceQuota</code><span style="vertical-align: baseline;"> is dynamically applied to the frontend team's namespaces:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;apiVersion: configmanagement.gke.io/v1\r\nkind: NamespaceSelector\r\nmetadata:\r\n name: frontend-scope\r\nspec:\r\n mode: dynamic\r\n selector:\r\n matchLabels:\r\n fleet.gke.io/fleet-scope: frontend\r\n---\r\nkind: ResourceQuota\r\napiVersion: v1\r\nmetadata:\r\n name: fe-quota\r\n annotations:\r\n configmanagement.gke.io/namespace-selector: frontend-scope\r\nspec:\r\n hard:\r\n persistentvolumeclaims: &quot;6&quot;&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8915310&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Similarly, this </span><code style="vertical-align: baseline;">ResourceQuota</code><span style="vertical-align: baseline;"> targets the frontend team's </span><code style="vertical-align: baseline;">frontend-a</code><span style="vertical-align: baseline;"> and </span><code style="vertical-align: baseline;">frontend-b</code><span style="vertical-align: baseline;"> namespaces, dynamically adjusting as the fleet's namespaces and member clusters evolve.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/3_-_add_resourcequota.max-1000x1000.jpg"
|
||
|
||
alt="3 - add resourcequota">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Delegating resource management with Config Sync: Empowering the backend team<br/></strong><span style="vertical-align: baseline;">To allow the backend team to manage their own resources within their designated </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> namespace, you can use Config Sync's </span><a href="https://cloud.google.com/anthos-config-management/docs/reference/rootsync-reposync-fields"><code style="text-decoration: underline; vertical-align: baseline;">RepoSync</code></a><span style="vertical-align: baseline;">, and a slightly different </span><code style="vertical-align: baseline;">NamespaceSelector</code><span style="vertical-align: baseline;">.</span></p>
|
||
<p><strong style="vertical-align: baseline;">Targeting a specific fleet namespace<br/></strong><span style="vertical-align: baseline;">To zero in on the backend team's </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> namespace, the following </span><code style="vertical-align: baseline;">NamespaceSelector</code><span style="vertical-align: baseline;"> targets both the team scope and the namespace name by labels:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;apiVersion: configmanagement.gke.io/v1\r\nkind: NamespaceSelector\r\nmetadata:\r\n name: backend-bookstore\r\nspec:\r\n mode: dynamic\r\n selector:\r\n matchLabels:\r\n fleet.gke.io/fleet-scope: backend\r\n kubernetes.io/metadata.name: bookstore&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8915280&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><strong style="vertical-align: baseline;">Introducing RepoSync<br/></strong><span style="vertical-align: baseline;">Another Config Sync feature is </span><code style="vertical-align: baseline;">RepoSync</code><span style="vertical-align: baseline;">, which lets you delegate resource management within a specific namespace. For security reasons, </span><code style="vertical-align: baseline;">RepoSync</code><span style="vertical-align: baseline;"> has no default access; you must explicitly grant the necessary RBAC permissions to the namespace.</span></p>
|
||
<p><span style="vertical-align: baseline;">Leveraging the </span><code style="vertical-align: baseline;">NamespaceSelector</code><span style="vertical-align: baseline;">, the following </span><code style="vertical-align: baseline;">RepoSync</code><span style="vertical-align: baseline;"> resource and its respective </span><code style="vertical-align: baseline;">RoleBinding</code><span style="vertical-align: baseline;"> can be applied dynamically to all </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> namespaces across the backend team's member clusters. The </span><code style="vertical-align: baseline;">RepoSync</code><span style="vertical-align: baseline;"> points it to a repository owned by the backend team:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;kind: RepoSync\r\napiVersion: configsync.gke.io/v1beta1\r\nmetadata:\r\n name: repo-sync\r\n annotations:\r\n configmanagement.gke.io/namespace-selector: backend-bookstore\r\nspec:\r\n sourceFormat: unstructured\r\n git:\r\n repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples\r\n branch: main\r\n dir: fleet-tenancy/teams/backend/bookstore\r\n auth: none\r\n---\r\nkind: RoleBinding\r\napiVersion: rbac.authorization.k8s.io/v1\r\nmetadata:\r\n name: be-bookstore\r\n annotations:\r\n configmanagement.gke.io/namespace-selector: backend-bookstore\r\nsubjects:\r\n- kind: ServiceAccount\r\n name: ns-reconciler-bookstore\r\n namespace: config-management-system\r\nroleRef:\r\n kind: ClusterRole\r\n name: admin\r\n apiGroup: rbac.authorization.k8s.io&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8915be0&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="font-style: italic; vertical-align: baseline;">Note: The </span><code style="font-style: italic; vertical-align: baseline;">.spec.git</code><span style="font-style: italic; vertical-align: baseline;"> section would reference the backend team's repository.</span></p>
|
||
<p><span style="vertical-align: baseline;">The backend team’s repository contains a </span><code style="vertical-align: baseline;">ConfigMap</code><span style="vertical-align: baseline;">. Config Sync ensures that the </span><code style="vertical-align: baseline;">ConfigMap</code><span style="vertical-align: baseline;"> is applied to the </span><code style="vertical-align: baseline;">bookstore</code><span style="vertical-align: baseline;"> namespaces across all backend team’s member clusters, supporting a GitOps approach to management.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Easier cross-team resource management</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Managing resources across multiple teams within a fleet of clusters can be complex. Google Cloud's fleet team management features, combined with Config Sync, provide an effective solution to streamline this process.</span></p>
|
||
<p><span style="vertical-align: baseline;">In this blog, we explored a scenario with frontend and backend teams, each requiring their own tenant spaces and resources (</span><code style="vertical-align: baseline;">NetworkPolicies</code><span style="vertical-align: baseline;">, </span><code style="vertical-align: baseline;">ResourceQuotas</code><span style="vertical-align: baseline;">, </span><code style="vertical-align: baseline;">RepoSync</code><span style="vertical-align: baseline;">). Using Config Sync in conjunction with the fleet management features, we automated the provisioning of these resources, helping to ensure a consistent and scalable setup.</span></p>
|
||
<p><strong style="vertical-align: baseline;">Next steps</strong></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Learn how to </span><a href="https://cloud.google.com/anthos-config-management/docs/how-to/fleet-tenancy"><span style="text-decoration: underline; vertical-align: baseline;">use Config Sync to sync Kubernetes resources to team scopes and namespaces</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">To experiment with this setup, visit the </span><a href="https://github.com/GoogleCloudPlatform/anthos-config-management-samples/tree/main/fleet-tenancy" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">example repository</span></a><span style="vertical-align: baseline;">. Config Sync configuration settings are located within the </span><a href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/gke_hub_feature#nested_config_sync" rel="noopener" target="_blank"><code style="text-decoration: underline; vertical-align: baseline;">config_sync</code><span style="text-decoration: underline; vertical-align: baseline;"> block</span></a><span style="vertical-align: baseline;"> of the Terraform </span><code style="vertical-align: baseline;">google_gke_hub_feature</code><span style="vertical-align: baseline;"> resource.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">For simplicity, this example uses a public Git repository. To use a private repository, </span><a href="https://cloud.google.com/anthos-config-management/docs/how-to/installing-config-sync#git-creds-secret"><span style="text-decoration: underline; vertical-align: baseline;">create a Secret</span></a><span style="vertical-align: baseline;"> in each cluster to store authentication credentials.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">To learn more about Config Sync, see </span><a href="https://cloud.google.com/anthos-config-management/docs/config-sync-overview"><span style="text-decoration: underline; vertical-align: baseline;">Config Sync overview</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">To learn more about fleets, see </span><a href="https://cloud.google.com/kubernetes-engine/fleet-management/docs" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;"><span style="text-decoration: underline; vertical-align: baseline;">Fleet management overview</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
</li>
|
||
</ul></div></description><pubDate>Fri, 03 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/containers-kubernetes/how-to-use-config-sync-team-scopes/</guid><category>Management Tools</category><category>DevOps & SRE</category><category>Containers & Kubernetes</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Scalable multi-tenancy management with Config Sync and team scopes</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/containers-kubernetes/how-to-use-config-sync-team-scopes/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Janet Kuo</name><title>Staff Software Engineer, Google Cloud</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Kavitha Gowda</name><title>Product Manager, Google Cloud</title><department></department><company></company></author></item><item><title>Simplifying data modeling and schema generation in BigQuery using multi-modal LLMs</title><link>https://cloud.google.com/blog/products/data-analytics/how-to-use-an-llm-to-create-data-schemas-in-bigquery/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The intricate hierarchical data structures in data warehouses and lakes sourced from diverse origins can make data modeling a protracted and error-prone process. To quickly adapt and create data models that meet evolving business requirements without having to rework them excessively, you need data models that are flexible, modular and adaptable enough to accommodate many requirements. This requires advanced technologies, proficient personnel, and robust methodologies.</span></p>
|
||
<p><span style="vertical-align: baseline;">The advancements in generative AI offer numerous opportunities to address these challenges. Multimodal large language models (LLMs) can analyze examples of data in the data lake, including text descriptions, code, and even images of existing databases. By understanding this data and its relationships, LLMs can suggest or even automatically generate schema layouts, simplifying the laborious process of implementing the data model within the database, so developers can focus on higher value data management tasks.</span></p>
|
||
<p><span style="vertical-align: baseline;">In this blog, we walk you through how to use multimodal LLMs in BigQuery to create a database schema. To do so, we’ll take a real-world example of entity relationship (ER) diagrams and examples of data definition languages (DDLs), and create a database schema in three steps. </span></p>
|
||
<p><span style="vertical-align: baseline;">For this demonstration, we will use Data Beans, a fictional technology company built on BigQuery that provides a SaaS platform to coffee sellers. Data Beans leverages BigQuery’s integration with Vertex AI to access Google AI models like Gemini Vision Pro 1.0 to analyze unstructured data and integrate it with structured data, while using BigQuery to help with data modeling and generating insights. </span></p>
|
||
<h3><strong style="vertical-align: baseline;">STEP1 : Create an entity relationship diagram </strong></h3>
|
||
<p><span style="vertical-align: baseline;">The first step is to create an ER diagram using your favorite modeling tool, or to take a screenshot of an existing ER diagram. The ER diagram can contain primary key and foreign key relationships, and will then be used as an input to the Gemini Vision Pro 1.0 model to create relevant BigQuery DDLs.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image1_DfJ9Xwj.max-1000x1000.png"
|
||
|
||
alt="image1">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">STEP2 : Create a prompt with the ER image as input</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Next, </span><span style="vertical-align: baseline;">to create the DDL statements in BigQuery, </span><span style="vertical-align: baseline;">w</span><span style="vertical-align: baseline;">rite a prompt to take an ER image as an input. The prompt should include detailed and relevant rules that the Gemini model should follow. In addition, make sure the prompt captures learnings from the previous iterations — in other words, be sure to update your prompt as you experiment and iterate it. These can be provided as examples to the model, for example a valid schema description for BigQuery. Providing a working example for the model to follow will help the model create a data definition DDL that follows your desired rules. </span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;## Prompt to guide the model\r\nllm_erd_prompt=f&quot;&quot;&quot;Use BigQuery SQL commands to create the following:\r\n- Create a new BigQuery schema named &quot;{dataset_id}&quot;.\r\n- Use only BigQuery data types. Double and triple check this since it causes a lot of errors.\r\n- Create the BigQuery DDLs for the attached ERD.\r\n- Create primary keys for each table using the ALTER command. Use the &quot;NOT ENFORCED&quot; keyword.\r\n- Create foreign keys for each table using the ALTER command. Use the &quot;NOT ENFORCED&quot; keyword.\r\n- For each field add an OPTIONS for the description.\r\n- Cluster the table by the primary key.\r\n- For columns that can be null do not add &quot;NULL&quot; to the created SQL statement. BigQuery leaves this blank.\r\n- All ALTER TABLE statements should be at the bottom of the generated script.\r\n- The ALTER TABLE statements should be ordered by the primary key statements and then the foreign key statements. Order matters!\r\n- Double check your work especially that you used ONLY BigQuery data types.\r\n\r\n\r\nPrevious Errors that have been generated by this script. Be sure to check your work to avoid encountering these.\r\n- Query error: Type not found: FLOAT at [6:12]\r\n- Query error: Table test.company does not have Primary Key constraints at [25:1]\r\n\r\n\r\n## Example for model to influence from\r\nExample:\r\nCREATE TABLE IF NOT EXISTS `{project_id}.{dataset_id}.customer`\r\n(\r\n customer_id INTEGER NOT NULL OPTIONS(description=&quot;Primary key. Customer table.&quot;),\r\n country_id INTEGER NOT NULL OPTIONS(description=&quot;Foreign key: Country table.&quot;),\r\n customer_llm_summary STRING NOT NULL OPTIONS(description=&quot;LLM generated summary of customer data.&quot;),\r\n customer_lifetime_value STRING NOT NULL OPTIONS(description=&quot;Total sales for this customer.&quot;),\r\n customer_cluster_id FLOAT NOT NULL OPTIONS(description=&quot;Clustering algorithm id.&quot;),\r\n customer_review_llm_summary STRING OPTIONS(description=&quot;LLM summary are all of the customer reviews.&quot;),\r\n customer_survey_llm_summary STRING OPTIONS(description=&quot;LLM summary are all of the customer surveys.&quot;)\r\n)\r\nCLUSTER BY customer_id;\r\n\r\n\r\nCREATE TABLE IF NOT EXISTS `{project_id}.{dataset_id}.country`\r\n(\r\ncountry_id INTEGER NOT NULL OPTIONS(description=&quot;Primary key. Country table.&quot;),\r\ncountry_name STRING NOT NULL OPTIONS(description=&quot;The name of the country.&quot;)\r\n)\r\nCLUSTER BY country_id;\r\n\r\n\r\n\r\n\r\nALTER TABLE `{project_id}.{dataset_id}.customer` ADD PRIMARY KEY (customer_id) NOT ENFORCED;\r\nALTER TABLE `{project_id}.{dataset_id}.country` ADD PRIMARY KEY (country_id) NOT ENFORCED;\r\n\r\n\r\nALTER TABLE `{project_id}.{dataset_id}.customer` ADD FOREIGN KEY (country_id) REFERENCES `{project_id}.{dataset_id}.country`(country_id) NOT ENFORCED;\r\n&quot;&quot;&quot;&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc9189100&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Now you have an image of an ER diagram to present to your LLM.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">STEP 3: Call the Gemini Pro 1.0 Vision model </strong></h3>
|
||
<p><span style="vertical-align: baseline;">After creating a prompt in Step 2, you are now ready to call the Gemini Pro 1.0 Vision model to generate the output by using the image of your ER diagram as an input (left side of Figure 1). You can do this in a number of ways — either directly from Colab notebooks using Python, or through BigQuery ML, leveraging its integration with Vertex AI: </span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;imageBase64 = convert_png_to_base64(menu_erd_filename)\r\n\r\n\r\nllm_response = GeminiProVisionLLM(llm_erd_prompt, imageBase64, temperature=.2, topP=1, topK=32)&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc9189190&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Conclusions and resources</strong></h3>
|
||
<p><span style="vertical-align: baseline;">In this demonstration, we saw how the multimodal Gemini model can streamline the creation of data and schemas. And while manually writing prompts is fine, it can be a daunting task when you need to do it at enterprise scale to create thousands of assets such as DDLs. Leveraging the above process, you can parameterize and automate prompt generation, dramatically speeding up the workflow and providing consistency across thousands of generated artifacts. You can find the complete Colab Enterprise notebook source code </span><a href="https://github.com/GoogleCloudPlatform/data-beans/blob/main/colab-enterprise/gen-ai-demo/Menu-Synthetic-Data-Generation-GenAI.ipynb" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">here</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<p><span style="vertical-align: baseline;">BigQuery ML includes many new features to let you use Gemini Pro capabilities; for more, please see the </span><a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-text"><span style="text-decoration: underline; vertical-align: baseline;">documentation</span></a><span style="vertical-align: baseline;">.</span><span style="vertical-align: baseline;"> Then, check out </span><span style="vertical-align: baseline;">this </span><a href="https://www.brighttalk.com/webcast/20069/600727?utm_source=GoogleCloud&amp;utm_medium=brighttalk&amp;utm_campaign=600727" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">tutorial</span></a><span style="vertical-align: baseline;"> to learn how to apply Google's models to your data, deploy models, and operationalize ML workflows — all without ever moving your data from BigQuery</span><span style="vertical-align: baseline;">. Finally, </span><span style="vertical-align: baseline;">for a behind-the-scenes look on how we made this demo, watch this video on </span><a href="https://www.youtube.com/watch?v=7zmPRPhWSGA" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">how to build an end-to-end data analytics and AI application</span></a><span style="vertical-align: baseline;"> using advanced models like Gemini directly from BigQuery.</span></p>
|
||
<hr/>
|
||
<p><sup><span style="font-style: italic; vertical-align: baseline;">Googlers Luis Velasco, Navjot Singh, Skander Larbi and Manoj Gunti contributed to this blog post. Many Googlers contributed to make these features a reality</span></sup></p></div></description><pubDate>Fri, 03 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/data-analytics/how-to-use-an-llm-to-create-data-schemas-in-bigquery/</guid><category>AI & Machine Learning</category><category>Data Analytics</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Simplifying data modeling and schema generation in BigQuery using multi-modal LLMs</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/data-analytics/how-to-use-an-llm-to-create-data-schemas-in-bigquery/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Adam Paternostro</name><title>Technical Lead, Google Cloud</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Firat Tekiner</name><title>Senior Staff Product Manager, Google Cloud</title><department></department><company></company></author></item><item><title>Introducing Dataflux Dataset for Cloud Storage to accelerate PyTorch AI training</title><link>https://cloud.google.com/blog/products/ai-machine-learning/introducing-new-pytorch-dataflux-dataset-abstraction/</link><description><div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Introduction</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Machine learning (ML) models thrive on massive datasets, and fast data loading is key for cost-effective ML training. We recently launched a PyTorch Dataset abstraction, the </span><a href="https://github.com/GoogleCloudPlatform/dataflux-pytorch" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Dataflux Dataset</span></a><span style="vertical-align: baseline;">, for accelerating data loading from Google’s Cloud Storage. Dataflux provides up to 3.5x faster training times compared to fsspec, with small files. </span></p>
|
||
<p><span style="vertical-align: baseline;">Today’s launch builds upon Google’s commitment to open standards that spans over two decades of OSS contributions like TensorFlow, JAX, TFX, MLIR, KubeFlow, and Kubernetes, as well as sponsorship for critical OSS data science initiatives like Project Jupyter and NumFOCUS. </span></p>
|
||
<p><span style="vertical-align: baseline;">We also validated the Dataflux Dataset on </span><a href="https://github.com/argonne-lcf/dlio_benchmark" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Deep Learning IO (DLIO) benchmarks</span></a><span style="vertical-align: baseline;"> and realized similar performance gains, even with larger files. Due to this broad performance boost, we recommend using Dataflux Dataset over other libraries or direct Cloud Storage API calls for training workflows.</span></p>
|
||
<p><span style="vertical-align: baseline;">Key Dataflux Dataset features include:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Direct Cloud Storage integration:</strong><span style="vertical-align: baseline;"> Eliminate the need to download data locally first.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Performance optimization:</strong><span style="vertical-align: baseline;"> Achieve up to 3.5x faster training times, especially with small files. </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">PyTorch Dataset primitive:</strong><span style="vertical-align: baseline;"> Work seamlessly with familiar PyTorch concepts.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Checkpointing support:</strong><span style="vertical-align: baseline;"> Save and load model checkpoints directly to/from Cloud Storage.</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">Using Dataflux Datasets</strong></h3>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Prerequisites:</strong><span style="vertical-align: baseline;"> Python 3.8+</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Installation: </strong><strong style="vertical-align: baseline;">$ </strong><code style="vertical-align: baseline;">pip install gcs-torch-dataflux</code></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Authentication:</strong><span style="vertical-align: baseline;"> Use Google Cloud </span><a href="https://cloud.google.com/docs/authentication/provide-credentials-adc"><span style="text-decoration: underline; vertical-align: baseline;">application-default authentication</span></a></p>
|
||
</li>
|
||
</ol>
|
||
<p><strong style="vertical-align: baseline;">Example: Loading images for training</strong></p>
|
||
<p><span style="vertical-align: baseline;">There are only a few changes needed to enable the Dataflux Dataset. </span><span style="vertical-align: baseline;">If you’re using PyTorch and have data in Cloud Storage, you most likely have written your own Dataset implementation. The below snippet shows how easy it is to create a Dataflux Dataset. For further details, checkout our </span><a href="https://github.com/GoogleCloudPlatform/dataflux-pytorch/tree/main" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">GitHub</span></a><span style="vertical-align: baseline;"> page.</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;import numpy\r\nimport io\r\nfrom PIL import Image\r\nfrom dataflux_pytorch import dataflux_mapstyle_dataset\r\n\r\ndef transform(img_in_bytes): \r\n return numpy.asarray(\r\nImage.open(io.BytesIO(img_in_bytes)))\r\n\r\ndataset = dataflux_mapstyle_dataset.DatafluxMapStyleDataset(\r\n project_name=PROJECT_NAME,\r\n bucket_name=BUCKET_NAME,\r\n config=dataflux_mapstyle_dataset.Config(prefix=PREFIX),\r\n data_format_fn=transform,\r\n)\r\n\r\n# Use &quot;dataset&quot; as usual in your ML-Training loop in combination with PyTorch DataLoader.&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc84b2f70&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><strong style="vertical-align: baseline;">Under the hood</strong></h3>
|
||
<p><span style="vertical-align: baseline;">To achieve such significant performance gains for Dataflux, we addressed the data-loading performance bottlenecks in ML training workflows. In a training run, data is loaded in batches from storage, and after some processing, is sent from CPU to GPU for ML-Training computations. If reading and constructing a batch takes longer than GPU computation, then the GPU is effectively stalled and underutilized, leading to longer training times.</span></p>
|
||
<p><span style="vertical-align: baseline;">When data is in a cloud-based object storage system (like Google’s Cloud Storage), it takes longer to fetch the data than from a local disk, especially if the data is in small objects. This is due to time-to-first-byte latency. Once an object is ‘opened’ though, the cloud storage platform provides high throughput. In Dataflux, we employ a Cloud Storage feature called </span><a href="https://cloud.google.com/storage/docs/composing-objects"><span style="text-decoration: underline; vertical-align: baseline;">Compose Objects</span></a><span style="vertical-align: baseline;"> that can dynamically combine many smaller objects into a larger object. Then, instead of fetching (say) 1024 small objects (batch size), we only fetch 30 larger objects and download those to memory. The larger objects are then decomposed back to their individual smaller objects and served back as the dataset-samples. Any temporary composed objects created in the process are also cleaned up.</span></p>
|
||
<p><span style="vertical-align: baseline;">Another optimization that Dataflux Datasets employs is high-throughput parallel-listing, speeding up the initial metadata needed for the dataset. Dataflux uses a sophisticated algorithm called work-stealing to significantly speed up listings; with it, even the first AI training run, or “epoch,” is faster compared to Dataflux Datasets without parallel-listing, even on datasets that have tens of millions of objects. </span></p>
|
||
<p><span style="vertical-align: baseline;">Together, fast-listing and dynamic-composition help ensure that ML-training with Dataflux leads to minimal GPU stalls, leading to greatly reduced training time and increased accelerator utilization.</span></p>
|
||
<p><span style="vertical-align: baseline;">Fast-listing and dynamic-composition are part of the </span><a href="https://github.com/GoogleCloudPlatform/dataflux-client-python" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Dataflux Client Libraries</span></a><span style="vertical-align: baseline;"> and available on </span><a href="https://github.com/GoogleCloudPlatform/dataflux-pytorch/tree/main" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">GitHub</span></a><span style="vertical-align: baseline;">. Dataflux Dataset uses these client libraries under the hood.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Dataflux is available now</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Give the </span><a href="https://github.com/GoogleCloudPlatform/dataflux-pytorch/tree/main" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Dataflux Dataset for PyTorch</span></a><span style="vertical-align: baseline;"> (or the </span><a href="https://github.com/GoogleCloudPlatform/dataflux-client-python" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Dataflux Python client library</span></a><span style="vertical-align: baseline;"> if writing your own ML training dataset code) a try and </span><a href="mailto:dataflux-customer-support@google.com"><span style="text-decoration: underline; vertical-align: baseline;">let us know</span></a><span style="vertical-align: baseline;"> how it boosts your workflows! </span></p>
|
||
<p><span style="vertical-align: baseline;">You can learn more about this and our other storage AI related capabilities from our Google Cloud Next ‘24 recorded session “How to define a storage infrastructure for AI and analytical workloads”</span></p></div>
|
||
<div class="block-video">
|
||
|
||
|
||
|
||
<div class="article-module article-video ">
|
||
<figure>
|
||
<a class="h-c-video h-c-video--marquee"
|
||
href="https://youtube.com/watch?v=A4daQj9tnWk"
|
||
data-glue-modal-trigger="uni-modal-A4daQj9tnWk-"
|
||
data-glue-modal-disabled-on-mobile="true">
|
||
|
||
|
||
<img src="//img.youtube.com/vi/A4daQj9tnWk/maxresdefault.jpg"
|
||
alt="How to define a storage infrastructure for AI and analytical workloads"/>
|
||
|
||
<svg role="img" class="h-c-video__play h-c-icon h-c-icon--color-white">
|
||
<use xlink:href="#mi-youtube-icon"></use>
|
||
</svg>
|
||
</a>
|
||
|
||
|
||
</figure>
|
||
</div>
|
||
|
||
<div class="h-c-modal--video"
|
||
data-glue-modal="uni-modal-A4daQj9tnWk-"
|
||
data-glue-modal-close-label="Close Dialog">
|
||
<a class="glue-yt-video"
|
||
data-glue-yt-video-autoplay="true"
|
||
data-glue-yt-video-height="99%"
|
||
data-glue-yt-video-vid="A4daQj9tnWk"
|
||
data-glue-yt-video-width="100%"
|
||
href="https://youtube.com/watch?v=A4daQj9tnWk"
|
||
ng-cloak>
|
||
</a>
|
||
</div>
|
||
|
||
</div></description><pubDate>Thu, 02 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/ai-machine-learning/introducing-new-pytorch-dataflux-dataset-abstraction/</guid><category>Developers & Practitioners</category><category>AI & Machine Learning</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Introducing Dataflux Dataset for Cloud Storage to accelerate PyTorch AI training</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/ai-machine-learning/introducing-new-pytorch-dataflux-dataset-abstraction/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Mayur Deshpande</name><title>Staff Software Engineer</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Abhishek Lal</name><title>Product Manager, Google Cloud</title><department></department><company></company></author></item><item><title>Private networking patterns to Vertex AI workloads</title><link>https://cloud.google.com/blog/products/networking/private-connectivity-to-vertex-workloads/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">As enterprise strategic use cases for AI adoption increases, secure and reliable connectivity is more crucial than ever. Get ready to explore several private connections options for your Vertex AI workloads! In this blog, we'll dive into the existing options and reveal the services to get you connected on your AI journey.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Connectivity matrix </span></h3>
|
||
<p><a href="https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform"><span style="text-decoration: underline; vertical-align: baseline;">Vertex AI</span></a><span style="vertical-align: baseline;"> is a suite of products which provide different AI workloads that provide varying functionalities. The default method to access Vertex AI API’s are public which is the case with Google APIs in general. </span><span style="vertical-align: baseline;">Depending on your architecture you may have the requirement to access your API’s privately, because of security and enterprise governance, which means traffic does not travel over the internet to the public address of the API. In these cases there are several options which we will explore later in the blog but it will vary depending on the Vertex AI product you are connecting to. The image below shows the connectivity matrix for </span><a href="https://cloud.google.com/vertex-ai/docs/general/netsec-overview#support-table"><span style="text-decoration: underline; vertical-align: baseline;">accessing Vertex AI from on-premises and multicloud</span></a><span style="vertical-align: baseline;">.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1-matrix.max-1000x1000.png"
|
||
|
||
alt="1-matrix">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><span style="vertical-align: baseline;">Options</span></h3>
|
||
<p><span style="vertical-align: baseline;">As you can see from the previous matrix image there are several methods in addition to the public internet. These include:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><a href="https://cloud.google.com/vpc/docs/private-service-connect"><strong style="text-decoration: underline; vertical-align: baseline;">Private Service Connect</strong></a><strong style="vertical-align: baseline;"> (PSC) for Google APIs</strong><span style="vertical-align: baseline;"> - Provides private access to Google APIs over hybrid networking or within a Virtual Private Cloud (VPC) using a customer-specified IP address(s) and DNS endpoint name that can be leveraged for one or more use cases.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><a href="https://cloud.google.com/vpc/docs/private-google-access"><strong style="text-decoration: underline; vertical-align: baseline;">Private Google Access</strong></a><span style="vertical-align: baseline;"> - Provides private access to Google APIs over hybrid networking or within a Virtual Private Cloud (VPC) using a Google defined subnet.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><a href="https://cloud.google.com/vpc/docs/private-services-access"><strong style="text-decoration: underline; vertical-align: baseline;">Private Service Access</strong></a><strong style="vertical-align: baseline;"> (PSA) </strong><span style="vertical-align: baseline;">- Google and other providers, collectively known as "service providers," can offer services hosted within a Google-managed VPC network. PSA enables you to define IP addresses for the managed services in addition to establishing VPC peering to access the internal IP addresses of these Google and third-party services over hybrid networking or within the VPC.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Private Service Connect endpoint</strong><span style="vertical-align: baseline;"> - Enables consumers to securely access managed services hosted by Google or other providers from within their own Virtual Private Cloud (VPC) network or via hybrid networking, eliminating the need to define the producer's VPC network. Communication with managed services is established through PSC endpoints or backends defined by the consumer's IP space, facilitating multi-tenancy to producer services across VPCs.</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><span style="vertical-align: baseline;">Example </span></h3>
|
||
<p><span style="vertical-align: baseline;">The following diagram shows a Vector Search architecture in which the Vector Search API is enabled and managed in a service project (appropriately named "serviceproject") as part of a </span><a href="https://cloud.google.com/vpc/docs/shared-vpc"><span style="vertical-align: baseline;">Shared VPC</span></a><span style="vertical-align: baseline;"> deployment. The Vector Search Compute Engine resources are deployed as a Google-managed Infrastructure-as-a-Service (IaaS) in the service producer's VPC network.</span></p>
|
||
<p><a href="https://cloud.google.com/vpc/docs/private-service-connect#endpoints"><span style="text-decoration: underline; vertical-align: baseline;">Private Service Connect endpoints</span></a><span style="vertical-align: baseline;"> are deployed in the consumer's VPC network (serviceproject) for index query, in addition to </span><a href="https://cloud.google.com/vertex-ai/docs/general/googleapi-access-methods#psc"><span style="text-decoration: underline; vertical-align: baseline;">Private Service Connect endpoints for Google APIs</span></a><span style="vertical-align: baseline;"> for private index creation deployed in the host project, the VPC where the cloud router resides. Both index creation and index query are accessible privately through hybrid networking or within the VPC. </span></p>
|
||
<p><span style="vertical-align: baseline;">If the organization requires public access to index query, you can leverage the same producer service as a Private Service Connect Network Endpoint Groups </span><a href="https://cloud.google.com/load-balancing/docs/negs#psc-neg"><span style="text-decoration: underline; vertical-align: baseline;">(PSC NEG) </span></a><span style="vertical-align: baseline;">backend to a External Load Balancer, thereby enabling public access to the endpoint while also providing WAF and DDoS capabilities when associated with </span><a href="https://cloud.google.com/armor/docs/rule-tuning"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Armor</span></a><span style="vertical-align: baseline;">.</span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2-vertex-psc.max-1000x1000.png"
|
||
|
||
alt="2-vertex-psc">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><h3><span style="vertical-align: baseline;">Get hands-on and learn more</span></h3>
|
||
<p><span style="vertical-align: baseline;">This topic is hot right now, and there are many approaches you can use. There are a few resources available that you can use to get some hands-on experience. Please check out the following tutorials.</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Tutorial</strong><span style="vertical-align: baseline;"> - </span><a href="https://cloud.google.com/vertex-ai/docs/general/vertex-psc-vector-search"><span style="text-decoration: underline; vertical-align: baseline;">Use Private Service Connect to access a Vector Search index from on-premises</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Tutorial</strong><span style="vertical-align: baseline;"> - </span><a href="https://cloud.google.com/vertex-ai/docs/general/vertex-psc-gen-ai"><span style="text-decoration: underline; vertical-align: baseline;">Use Private Service Connect to access Generative AI on Vertex AI from on-premises</span></a><span style="vertical-align: baseline;">. </span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Tutorial</strong><span style="vertical-align: baseline;"> - </span><a href="https://cloud.google.com/vertex-ai/docs/general/vertex-psc-googleapis"><span style="text-decoration: underline; vertical-align: baseline;">Use Private Service Connect to access Vertex AI online predictions from on-premises</span></a></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Tutorial </strong><span style="vertical-align: baseline;">- </span><a href="https://cloud.google.com/vertex-ai/docs/general/vertex-psc-batch-predictions"><span style="text-decoration: underline; vertical-align: baseline;">Use Private Service Connect to access Vertex AI batch predictions from on-premises</span></a></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">To learn out more or share a thought find me on </span><a href="https://www.linkedin.com/in/ammett/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Linkedin</span></a>.</p></div>
|
||
<div class="block-related_article_tout">
|
||
|
||
|
||
|
||
|
||
|
||
<div class="uni-related-article-tout h-c-page">
|
||
<section class="h-c-grid">
|
||
<a href="https://cloud.google.com/blog/products/networking/connect-google-cloud-to-on-prem-and-other-clouds/"
|
||
data-analytics='{
|
||
"event": "page interaction",
|
||
"category": "article lead",
|
||
"action": "related article - inline",
|
||
"label": "article: {slug}"
|
||
}'
|
||
class="uni-related-article-tout__wrapper h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
|
||
h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3 uni-click-tracker">
|
||
<div class="uni-related-article-tout__inner-wrapper">
|
||
<p class="uni-related-article-tout__eyebrow h-c-eyebrow">Related Article</p>
|
||
|
||
<div class="uni-related-article-tout__content-wrapper">
|
||
<div class="uni-related-article-tout__image-wrapper">
|
||
<div class="uni-related-article-tout__image" style="background-image: url('https://storage.googleapis.com/gweb-cloudblog-publish/images/networking_3Z7Xc6t.max-500x500.jpg')"></div>
|
||
</div>
|
||
<div class="uni-related-article-tout__content">
|
||
<h4 class="uni-related-article-tout__header h-has-bottom-margin">Cross-Cloud Network: Private, customizable and flexible networking</h4>
|
||
<p class="uni-related-article-tout__body">Explore the Cross-Cloud Network architecture ebook and learn how Google’s global scale network can support your enterprise multicloud and...</p>
|
||
<div class="cta module-cta h-c-copy uni-related-article-tout__cta muted">
|
||
<span class="nowrap">Read Article
|
||
<svg class="icon h-c-icon" role="presentation">
|
||
<use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#mi-arrow-forward"></use>
|
||
</svg>
|
||
</span>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</a>
|
||
</section>
|
||
</div>
|
||
|
||
</div></description><pubDate>Thu, 02 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/networking/private-connectivity-to-vertex-workloads/</guid><category>Hybrid & Multicloud</category><category>Developers & Practitioners</category><category>Networking</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/0-ai-hero.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Private networking patterns to Vertex AI workloads</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/0-ai-hero.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/networking/private-connectivity-to-vertex-workloads/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Ammett Williams</name><title>Developer Relations Engineer</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Deepak Michael</name><title>Networking Specialist Customer Engineer</title><department></department><company></company></author></item><item><title>RAG in production faster with Ray, LangChain and HuggingFace</title><link>https://cloud.google.com/blog/products/ai-machine-learning/rag-quickstart-with-ray-langchain-and-huggingface/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">We’re excited to announce the release of a </span><a href="https://console.cloud.google.com/marketplace/product/google/rag-on-gke" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">quickstart solution</span></a><span style="vertical-align: baseline;"> and </span><a href="https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/applications/rag#rag-on-gke-application" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">reference architecture</span></a><span style="vertical-align: baseline;"> for retrieval augmented generation (RAG) applications, designed to accelerate your journey to production. In this post, you’ll learn how to quickly deploy a complete RAG application on </span><a href="https://cloud.google.com/kubernetes-engine?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Google Kubernetes Engine</span></a><span style="vertical-align: baseline;"> (GKE), and </span><a href="https://cloud.google.com/sql/docs/postgres"><span style="text-decoration: underline; vertical-align: baseline;">Cloud SQL for PostgreSQL</span></a><span style="vertical-align: baseline;"> and </span><a href="https://github.com/pgvector/pgvector" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">pgvector</span></a><span style="vertical-align: baseline;">, using Ray, LangChain, and Hugging Face.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">What is RAG?</strong></h3>
|
||
<p><span style="vertical-align: baseline;">RAG can improve the outputs of foundation modes, such as large language models (LLMs), for a specific application. Rather than relying purely on knowledge developed during training, AI apps equipped for RAG can retrieve the information most relevant to a user’s prompt from an external knowledge base, then add that information to the prompt before sending it to the generative model. The knowledge base can come in various forms, such as a vector database, traditional search index, or relational database — and by accessing it, customer service chabots can look up help center articles, digital shopping assistants can tap into product catalogs and customer reviews, and AI-powered travel agents can deliver up-to-date flight and hotel information. </span></p></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_RAG_Conceptual_Diagram_FINAL.max-1000x1000.png"
|
||
|
||
alt="1 RAG Conceptual Diagram FINAL">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">LLMs rely on their training data, which can quickly fall out of date and may not include data relevant to the application’s domain. Re-training or fine-tuning an LLM to provide fresh, domain-specific data can be an expensive and complex process. RAG not only gives the LLM access to such data without training or-fine tuning. but can also guide an LLM toward factual responses, thereby reducing hallucinations and enabling applications to provide human-verifiable source material. </span></p>
|
||
<p><span style="vertical-align: baseline;">For more background on how RAG works, see our blog on </span><a href="https://cloud.google.com/blog/products/ai-machine-learning/context-aware-code-generation-rag-and-vertex-ai-codey-apis"><span style="text-decoration: underline; vertical-align: baseline;">context-aware code generation</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">AI Infrastructure for RAG</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Prior to the rise of Generative AI, a typical application architecture might involve a database, a set of microservices, and a frontend. Even the most basic RAG applications introduce new requirements for serving LLMs, processing, and retrieving unstructured data. To meet these requirements, customers need infrastructure that is optimized specifically for AI workloads. </span></p>
|
||
<p><span style="vertical-align: baseline;">Many customers choose to access AI infrastructure like TPUs and GPUs via a fully managed platform, such as </span><a href="https://cloud.google.com/vertex-ai?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Vertex AI</span></a><span style="vertical-align: baseline;">. Others, however, prefer to manage their own infrastructure on top of GKE while leveraging open-source frameworks and open models. This blog post is for the latter group. </span></p>
|
||
<p><span style="vertical-align: baseline;">Building an AI platform from scratch involves a number of key decisions, such as which frameworks to use for model serving, which machine shapes to use for inference, how to protect sensitive data, how to meet cost and performance requirements, and how to scale as traffic grows. Each decision involves many tradeoffs against a vast and fast-changing landscape of generative AI tools.</span></p>
|
||
<p><span style="vertical-align: baseline;">This is why we have developed a quickstart solution and reference architecture for RAG applications built on top of GKE, Cloud SQL, and open-source frameworks </span><a href="https://www.ray.io/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Ray</span></a><span style="vertical-align: baseline;">, </span><a href="https://www.langchain.com/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">LangChain</span></a><span style="vertical-align: baseline;"> and </span><a href="https://huggingface.co/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Hugging Face</span></a><span style="vertical-align: baseline;">. Our solution is designed to help you get started quickly and accelerate your journey to production with RAG best practices built-in from the start.</span></p>
|
||
<h3><strong style="vertical-align: baseline;">Benefits of RAG on GKE and Cloud SQL</strong></h3>
|
||
<p><span style="vertical-align: baseline;">GKE and Cloud SQL accelerate your journey to production in a variety of ways:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Load Data Fast</strong><span style="vertical-align: baseline;"> - Use </span><a href="https://docs.ray.io/en/latest/data/data.html" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Ray Data</span></a><span style="vertical-align: baseline;"> to seamlessly access data in parallel from your Ray cluster via GKE’s </span><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver"><span style="text-decoration: underline; vertical-align: baseline;">GCSFuse driver</span></a><span style="vertical-align: baseline;">. Efficiently load your embeddings into </span><a href="https://cloud.google.com/sql/docs/postgres"><span style="text-decoration: underline; vertical-align: baseline;">Cloud SQL for PostgreSQL</span></a><span style="vertical-align: baseline;"> and </span><a href="https://github.com/pgvector/pgvector" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">pgvector</span></a><span style="vertical-align: baseline;"> to perform low latency vector search at scale.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Fast deploy</strong><span style="vertical-align: baseline;"> - Quickly deploy Ray, </span><a href="https://jupyter.org/hub" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">JupyterHub</span></a><span style="vertical-align: baseline;">, and Hugging Face </span><a href="https://huggingface.co/docs/text-generation-inference/en/index" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Text Generation Inference</span></a><span style="vertical-align: baseline;"> (TGI) to your GKE cluster</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Security made simple</strong><span style="vertical-align: baseline;"> - Get </span><a href="https://cloud.google.com/blog/products/containers-kubernetes/move-in-ready-kubernetes-security-with-gke-autopilot"><span style="text-decoration: underline; vertical-align: baseline;">move-in ready Kubernetes security</span></a><span style="vertical-align: baseline;"> with </span><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview"><span style="text-decoration: underline; vertical-align: baseline;">GKE</span></a><span style="vertical-align: baseline;">. Filter out sensitive or toxic content using </span><a href="https://cloud.google.com/sensitive-data-protection/docs/sensitive-data-protection-overview"><span style="text-decoration: underline; vertical-align: baseline;">Sensitive Data Protection</span></a><span style="vertical-align: baseline;"> (SDP). Leverage Google-standard authentication with </span><a href="https://cloud.google.com/security/products/iap?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Identity-Aware Proxy</span></a><span style="vertical-align: baseline;"> so users can seamlessly connect to your LLM frontend and Jupyter notebooks.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Cost efficiency &amp; reduced management overhead</strong><span style="vertical-align: baseline;"> - GKE reduces cluster maintenance and makes it easy to take advantage of cost-saving measures like </span><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/spot-vms"><span style="text-decoration: underline; vertical-align: baseline;">spot nodes</span></a><span style="vertical-align: baseline;"> via YAML configuration.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Scalability -</strong><span style="vertical-align: baseline;"> GKE </span><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler"><span style="text-decoration: underline; vertical-align: baseline;">automatically</span></a><span style="vertical-align: baseline;"> provisions nodes as traffic grows, eliminating the need for manual configuration to scale up.</span></p>
|
||
</li>
|
||
</ul>
|
||
<h3><strong style="vertical-align: baseline;">Deploying RAG on GKE and Cloud SQL</strong></h3>
|
||
<p><span style="vertical-align: baseline;">Our end-to-end </span><a href="https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/applications/rag" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">RAG application</span></a><span style="vertical-align: baseline;"> and </span><a href="https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/applications/rag#rag-on-gke-application" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">reference architecture</span></a><span style="vertical-align: baseline;"> provide the following:</span></p>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Google Cloud project </strong><span style="vertical-align: baseline;">- configures your project with the needed prerequisites to run the RAG application, including a GKE Cluster and Cloud SQL for PostgreSQL and pgvector instance</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">AI frameworks</strong><span style="vertical-align: baseline;"> - deploys Ray, JupyterHub, and Hugging Face TGI to GKE</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">RAG Embedding Pipeline</strong><span style="vertical-align: baseline;"> - generates embeddings and populates the Cloud SQL for PostgreSQL and pgvector instance</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Example RAG Chatbot Application</strong><span style="vertical-align: baseline;"> - deploys a web-based RAG chatbot to GKE</span></p>
|
||
</li>
|
||
</ol></div>
|
||
<div class="block-image_full_width">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div class="article-module h-c-page">
|
||
<div class="h-c-grid">
|
||
|
||
|
||
<figure class="article-image--large
|
||
|
||
|
||
h-c-grid__col
|
||
h-c-grid__col--6 h-c-grid__col--offset-3
|
||
|
||
|
||
"
|
||
>
|
||
|
||
|
||
|
||
|
||
<img
|
||
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_RAG_High_Level_Architecture.max-1000x1000.png"
|
||
|
||
alt="2 RAG High Level Architecture">
|
||
|
||
</a>
|
||
|
||
</figure>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The example chatbot application provides a web interface where users can interact with an open source LLM. It leverages data loaded by the RAG data pipeline into Cloud SQL for PostgreSQL with pgvector, providing more comprehensive and informative responses to user queries. </span></p>
|
||
<p><span style="vertical-align: baseline;">Our end-to-end RAG solution serves as a starting point for further development, demonstrating the potential of this technology for a wide range of applications. By combining the power of RAG with the scalability and flexibility of GKE and Cloud SQL as well as security features of Google Cloud, developers can build powerful and versatile applications that can handle complex tasks and provide valuable insights. </span></p>
|
||
<p><span style="vertical-align: baseline;">We plan to evolve this solution over time, including the ability to add custom data sets, replace models, and update the dataset and vector database with new documents.</span></p>
|
||
<p><span style="vertical-align: baseline;">For more information, please check our </span><a href="https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/rag/README.md" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">README</span></a><span style="vertical-align: baseline;"> and </span><a href="https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/applications/rag" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">github instructions</span></a><span style="vertical-align: baseline;">, and </span><a href="https://cloud.google.com/architecture/rag-capable-gen-ai-app-using-gke"><span style="text-decoration: underline; vertical-align: baseline;">reference RAG architecture</span></a><span style="vertical-align: baseline;">. You can also </span><a href="https://www.youtube.com/watch?v=qwFCZKKFXd4" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">view our Google Cloud Next 2024 session</span></a><span style="vertical-align: baseline;"> discussing RAG.</span></p></div></description><pubDate>Thu, 02 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/ai-machine-learning/rag-quickstart-with-ray-langchain-and-huggingface/</guid><category>Containers & Kubernetes</category><category>AI & Machine Learning</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>RAG in production faster with Ray, LangChain and HuggingFace</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/ai-machine-learning/rag-quickstart-with-ray-langchain-and-huggingface/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Julie Amundson</name><title>Senior Staff Software Engineer</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Jason Soo Hoo</name><title>Software Engineering Manager</title><department></department><company></company></author></item><item><title>Sullivan County debuts generative AI chatbot, Saige, to answer constituent FAQs</title><link>https://cloud.google.com/blog/topics/public-sector/sullivan-county-debuts-generative-ai-chatbot-saige-to-answer-constituent-faqs/</link><description><div class="block-paragraph"><p data-block-key="6t5sl">Sullivan County, New York, is home to the Catskills Mountains. It’s a great place to live, play, and raise a family, and we have a robust group of visitors and tourists. We are also a fairly small county. Being on the cutting edge of technology isn’t typically something people associate with local governments, and I’m proud of Sullivan County for consistently innovating.</p><p data-block-key="5cguk">Traditionally, in offices like ours, our teams are routinely answering rote questions about operating hours or document filing processes. These calls take up hours of their day they’d like to spend helping solve more complex problems. In 2023, we introduced a virtual agent powered by <a href="https://cloud.google.com/dialogflow">Dialogflow</a> that has helped us quickly provide our constituents with answers to simple questions. We embedded the agent into our chatbot on the <a href="https://www.sullivanny.us/" target="_blank">county website</a> to support those seeking information about our County Clerk’s and Treasurer’s offices.</p><p data-block-key="enceg">With the chatbot’s support, we’ve seen a 62% reduction in call volume. This has allowed our teams to focus on resolving more complex issues. The success of that rollout attracted the attention of the other offices in the county, and we saw a chance to expand our chatbot’s capabilities.</p><h3 data-block-key="eeoeu"><b>Simplifying generative AI with Vertex AI to overhaul a chatbot in 10 weeks</b></h3><p data-block-key="fh15p">Although more than 40 offices in Sullivan County were excited to implement a chatbot, we were initially concerned about the time and effort it would take to tackle a project of that scale. Some leaders assumed that many chatbots would require manually combing through information on office websites and building chatbot workflows for each of them. Then, I was introduced to <a href="https://cloud.google.com/vertex-ai-search-and-conversation?hl=en">Vertex AI Agent Builder</a>. Vertex AI Agent Builder makes it easy, even for our nontechnical teams, to train machine learning models that grow and change over time. Since it can “learn,” we can create chatbots that scrape information from the respective office sites and dynamically determine the best answer to a question on any given day. We don’t have to design any of those flows manually.</p><p data-block-key="6an9e">My team worked with Google Cloud Premier Partner <a href="https://quantiphi.com/partners/google-cloud/" target="_blank">Quantiphi</a> to implement a generative AI-powered version of our virtual agent (who we call Saige), and their enthusiasm about the project was palpable. The project was completed and connected every department in the county in just ten weeks, thanks to the support from Quantiphi.</p><h3 data-block-key="14gjh"><b>Improving interactions with constituents as Saige learns over time</b></h3><p data-block-key="36rus">I grew up tinkering with computers in the 1980s, and I’ve been told my entire life that computers will make our lives easier, but training people on new technologies is often cumbersome and complicated. I believe that if a task using technology is more difficult or time-consuming than doing it manually, no one will adopt that new tool. With Vertex AI, I’m seeing technology make good on its promise to improve lives.</p><p data-block-key="dki4l">We’ve already seen Saige grow and evolve to make our jobs easier. As it gathered information from various department sites, it helped us identify sources that needed to be updated. If it responds to a question incorrectly, I can click “This was not helpful,” and ask the question again. Saige is able to search for and find new information immediately. This is especially valuable because information changes; holidays change operating hours or a file is updated, and our small teams don’t have time to manually update every chatbot workflow.</p><p data-block-key="3cqu9">As Saige’s knowledge repository grows, our offices continue to get more and more time back to manage more complex issues that require the face-to-face, human touch. Our offices get more efficient, people get answers faster, and every interaction we have with a constituent helps inform the next.</p><h3 data-block-key="9ieoq"><b>Setting new goals for an AI-powered Sullivan County</b></h3><p data-block-key="5br2l">Hearing about new experiences from our teams and the people of Sullivan County is one thing, and it’s another to be able to truly measure impact. We’ve implemented a <a href="https://cloud.google.com/looker?hl=en">Looker</a> dashboard that is helping us track exactly how much impact our chat features have on our offices. I can easily view total user sessions, success rates of chat interactions, and peak hours, so we can refine our support to best meet our community’s needs.</p><p data-block-key="duhoq">I can also see what questions people are asking, which helps me to understand what subjects are trending. Gathering experiential data alongside quantifiable data helps us offer better services and information about popular topics, in more prominent places, on our websites.</p><p data-block-key="4bmd7">When I think of the future of Sullivan County’s online resources, I think about what I would want as a constituent coming to our sites for information. As our chat function continues to improve over time, I’d love to offer additional services directly through chat, such as payments or form submissions. All of these tools can come together to further augment our amazing staff and provide Sullivan County constituents with the best possible service.</p><p data-block-key="5rhn3">Interested in seeing how the state of New York is transforming ? Learn more about <a href="https://papers.govtech.com/How-GenAI-is-Transforming-Public-Sector-Services-in-New-York-142886.html?" target="_blank">how gen AI is transforming public sector services in New York</a>.</p></div></description><pubDate>Thu, 02 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/public-sector/sullivan-county-debuts-generative-ai-chatbot-saige-to-answer-constituent-faqs/</guid><category>Public Sector</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/GCPNA009-002---Sullivan-County_siWv7DA.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Sullivan County debuts generative AI chatbot, Saige, to answer constituent FAQs</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/GCPNA009-002---Sullivan-County_siWv7DA.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/public-sector/sullivan-county-debuts-generative-ai-chatbot-saige-to-answer-constituent-faqs/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Dan Hust</name><title>Director of Communications</title><department></department><company>Sullivan County</company></author></item><item><title>AI can be the catalyst to reignite your digital transformation</title><link>https://cloud.google.com/blog/products/application-modernization/register-for-building-apps-in-an-ai-era-webinar/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Be honest with me: Is your “digital transformation” stuck? Did you start in earnest a few years back, and now you’re sitting on half-finished projects, uneven outcomes, and a distracted staff? It happens. Maintaining momentum for multi-year efforts isn’t easy. Especially efforts that are increasingly broad and complex! We need a catalyst to focus our efforts, motivate our teams, and simplify our work. </span><strong style="vertical-align: baseline;">Early signals tell us that generative AI is that missing catalyst, and Google Cloud is a unique partner for your journey.</strong></p>
|
||
<h3><span style="vertical-align: baseline;">How you got stuck</span></h3>
|
||
<p><span style="vertical-align: baseline;">Digital transformation means many things to many people. Is it about becoming more efficient? Upgrading tools? Delivering new digital products to customers? Adopting a data-driven strategy? Changing internal culture? </span><a href="https://cloud.google.com/blog/transform/the-meaning-of-digital-transformation-is-changing"><span style="text-decoration: underline; vertical-align: baseline;">All of those things</span></a><span style="vertical-align: baseline;">? We don’t always see a unifying purpose to these efforts that’s capable of rallying an organization. This lack of focus often results in a dizzying array of disparate projects. </span></p>
|
||
<p><span style="vertical-align: baseline;">Some of these projects are focused on new customer experiences. You might have tried launching new mobile or web experiences with solid, but not spectacular, results. There are always a handful of backend projects initiated to adopt public cloud, establish a real-time data infrastructure, set up developer-friendly application platforms, and upgrade security services. This inevitably sparks modernization programs to make data more accessible, apps more scalable, and infrastructure more automated. Smart companies complement these technology efforts with promises to invest in a company culture that </span><a href="https://dora.dev/devops-capabilities/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">embraces modern thinking and elite capabilities</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<p><span style="vertical-align: baseline;">None of those are bad things! But sometimes they come with unintended consequences:</span></p>
|
||
<ul>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">More complex infrastructure that straddles public cloud and private cloud</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Legacy systems straining under load and change rates that they weren’t designed for</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Regularly changing measures of success, from innovation to cost savings to optimization to efficiencies</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Team demotivation, as this growing bag of projects seems increasingly disconnected from measurable outcomes</span></p>
|
||
</li>
|
||
</ul>
|
||
<p><span style="vertical-align: baseline;">There’s a better way. In our experience, a corporate investment in generative AI brings focus, meaning, and acceleration to a host of important IT efforts.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Why generative AI catalyzes your team</span></h3>
|
||
<p><span style="vertical-align: baseline;">A business strategy with generative AI at the center benefits customers and employees. Why? For customers, it helps focus your attention on delivering more personalized and engaging experiences. There are </span><a href="https://cloud.google.com/transform/101-real-world-generative-ai-use-cases-from-industry-leaders"><span style="text-decoration: underline; vertical-align: baseline;">at least 101 examples of that</span></a><span style="vertical-align: baseline;">. For staff, it puts a spotlight on how everyone can use smarter tools to design, deliver, and operate products — whether those are data reports, software applications, or infrastructure platforms. Everyone gets to join in!</span></p>
|
||
<p><span style="vertical-align: baseline;">And that’s the thing. It’s not just about generative AI; it’s about what it takes to be </span><span style="font-style: italic; vertical-align: baseline;">good</span><span style="vertical-align: baseline;"> at generative AI. Everyone needs to come together to fully commit to excellence in five supporting areas (that were usually left half-finished during a classic digital transformation):</span></p>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Automate your infrastructure</strong><span style="vertical-align: baseline;">. Now’s the time to establish a full range of automation for provisioning, upgrading, and deleting all of the machinery that supports your (AI-hosting) infrastructure.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Upgrade your data platform</strong><span style="vertical-align: baseline;">. Your AI models won’t be any good without good data. Timely, accurate data is critical, and that means investing in flexible data pipelines, scalable databases, and a data warehouse that’s ready for AI.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Improve your developer experience</strong><span style="vertical-align: baseline;">. To build with AI, your developers need the tools, frameworks, and platform services that help them iterate quickly. It’s also time to finish those cultural upgrades that unleash your teams.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Modernize your security practices</strong><span style="vertical-align: baseline;">. Embracing generative AI requires a whole set of data, application, and infrastructure security considerations. You won’t deploy it if you don’t trust it. It’s key to make the necessary upgrades to your security posture.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">Finish your cloud migration</strong><span style="vertical-align: baseline;">. It’s going to be hard to maximize the value of generative AI outside of the public cloud. Places like Google Cloud are purpose-built to support the access to innovation, elasticity, and scale that are so important right now.</span></p>
|
||
</li>
|
||
</ol>
|
||
<h3><span style="vertical-align: baseline;">What you need to succeed</span></h3>
|
||
<p><span style="vertical-align: baseline;">Looking to avoid some of the challenges of past transformations? There’s more than one way to proceed with your generative AI strategy, but at Google Cloud, we see three crucial building blocks for your success.</span></p>
|
||
<p><span style="vertical-align: baseline;">You need </span><strong style="vertical-align: baseline;">proximity</strong><span style="vertical-align: baseline;">. Generative AI models and apps require proximity to dependent data. From AlloyDB to BigQuery, Google Cloud’s data services give you the speed, scale, and price performance to keep your AI-based systems grounded by your unique information. And especially now, you need </span><strong style="vertical-align: baseline;">proximity to expertise</strong><span style="vertical-align: baseline;"> for your journey. This is a period of excitement and change, so you want Google’s world-class team partnering with you to help you architect, deliver, and optimize your AI-based solutions.</span></p>
|
||
<p><span style="vertical-align: baseline;">You need an </span><strong style="vertical-align: baseline;">integrated AI platform</strong><span style="vertical-align: baseline;">. This isn’t the time for building out complex, brittle, do-it-yourself AI platforms. Too much is evolving too quickly. Buy innovation and flexibility, not complexity. Our </span><a href="https://cloud.google.com/blog/products/compute/whats-new-with-google-clouds-ai-hypercomputer-architecture"><span style="text-decoration: underline; vertical-align: baseline;">unique AI hypercomputer</span></a><span style="vertical-align: baseline;">, </span><a href="https://cloud.google.com/vertex-ai"><span style="text-decoration: underline; vertical-align: baseline;">Vertex AI platform for MLOps</span></a><span style="vertical-align: baseline;">, and </span><a href="https://cloud.google.com/products/gemini"><span style="text-decoration: underline; vertical-align: baseline;">Gemini for Google Cloud</span></a><span style="vertical-align: baseline;"> offer best-in-class vertical and horizontal integrations that help you build, run, and optimize better than anywhere else.</span></p>
|
||
<p><span style="vertical-align: baseline;">Finally, you need cross-organization </span><strong style="vertical-align: baseline;">productivity assistance</strong><span style="vertical-align: baseline;">. AI is not just about different output; it’s about a different way of working. </span><a href="https://workspace.google.com/solutions/ai/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Gemini for Google Workspace</span></a><span style="vertical-align: baseline;"> helps everyone be more creative and productive. </span><a href="https://cloud.google.com/products/gemini/code-assist"><span style="text-decoration: underline; vertical-align: baseline;">Gemini Code Assist</span></a><span style="vertical-align: baseline;"> gives software developers powerful tools for understanding and writing quality software. </span><a href="https://cloud.google.com/products/gemini/cloud-assist"><span style="text-decoration: underline; vertical-align: baseline;">Gemini Cloud Assist</span></a><span style="vertical-align: baseline;"> will bring game-changing AI assistance to teams that need to troubleshoot and optimize their cloud systems.</span></p>
|
||
<p><span style="vertical-align: baseline;">Ready to get unstuck? Register for our </span><a href="https://cloudonair.withgoogle.com/events/building-applications-in-the-ai-era?utm_source=google&amp;utm_medium=blog&amp;utm_campaign=FY24-Q2-global-Prod931-onlineevent-er-Building-Apps-in-the-AI-Era&amp;utm_content=launch_blog&amp;utm_term=-" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Building Apps in an AI Era webinar</span></a><span style="vertical-align: baseline;"> to learn more about how Google Cloud can help you innovate faster, deliver unparalleled customer experiences, and secure a lasting competitive advantage.</span></p></div></description><pubDate>Thu, 02 May 2024 13:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/application-modernization/register-for-building-apps-in-an-ai-era-webinar/</guid><category>AI & Machine Learning</category><category>Application Modernization</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>AI can be the catalyst to reignite your digital transformation</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/application-modernization/register-for-building-apps-in-an-ai-era-webinar/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Richard Seroter</name><title>Chief Evangelist, Google Cloud</title><department></department><company></company></author></item><item><title>Your modernization journey starts with the endpoint. A Forrester Consulting study shows why.</title><link>https://cloud.google.com/blog/products/chrome-enterprise/your-modernization-journey-starts-with-the-endpoint-a-forrester-consulting-study-shows-why/</link><description><div class="block-paragraph"><p data-block-key="axxn5">In today’s digital age, endpoints are a business requirement for collaborating with coworkers, engaging with customers, and building great products. However, with a rise in cyber attacks, increased scrutiny on cost, and pressure to innovate, IT departments require a new kind of endpoint that improves user experience, simplifies management, increases security and reduces costs—which are some of the key traits of what we refer to as the modern endpoint.</p></div>
|
||
<div class="block-paragraph"><p data-block-key="axxn5">We commissioned Forrester Consulting to survey 652 IT professionals to explore the meaning of a modern endpoint to IT departments, which Forrester further defines in the study as “a mix of multiple next-generation capabilities that center around artificial intelligence (AI), web-based applications, the cloud, and the integration of data.” <sup>1</sup></p></div>
|
||
<div class="block-paragraph"><p data-block-key="pogf7">The study found that IT leaders are “prioritizing initiatives that lead to a modern endpoint.” In particular, the study found that “IT leaders are prioritizing AI, web-based applications, and endpoint management in the cloud because these initiatives are core to a modern endpoint and will allow businesses to evolve with their employees and customers needs.”<sup>1</sup></p><p data-block-key="4uh1f">Specific to AI, the study found that IT respondents' “number-one priority over the next 12 months is to enable end users to take advantage of AI on the endpoint.”<sup>1</sup> The study talks specifically to productivity gains for IT staff, who can use AI to “automate repetitive tasks, analyze data from endpoint devices and plan maintenance, or analyze user behavior to create more personalized computing experiences.” <sup>1</sup></p></div>
|
||
<div class="block-paragraph"><p data-block-key="pogf7">Businesses starting their journey to a true modern endpoint should consider adopting ChromeOS. In this blog post, we’ll visit three of the four key findings from the Forrester Consulting study <a href="http://inthecloud.withgoogle.com/forrester-next-generation-endpoint-report/home.html" target="_blank">“Delivering the Next-Generation endpoint</a>,” and show how ChromeOS can help businesses realize their modern endpoint needs.</p></div>
|
||
<div class="block-paragraph"><h3 data-block-key="pogf7"><b>1. Endpoint security, management, and deployment are the top barriers to a modern endpoint.</b></h3></div>
|
||
<div class="block-paragraph"><p data-block-key="pogf7">In the study, Forrester consulting found that IT respondents indicated that they spent nearly half of their working hours on endpoint security (19%), management (15%), and deployment (14%).<sup>1</sup></p><p data-block-key="4qmts">ChromeOS can help IT departments reduce time securing, managing, and deploying devices. With regards to security, ChromeOS is built to be secure at every layer: at the lowest level, every ChromeOS device relies on Verified Boot which checks for tampering; second, every app and tab on ChromeOS is sandboxed, meaning each app has a clear perimeter in which it can operate; and third, by making the web the core application platform, apps are simply secure by design, with much more control over how they interact with powerful device features. This means that ransomware simply can't run on ChromeOS devices.*</p><p data-block-key="f99jj">The result? There has never been a successful ransomware or virus attack reported on ChromeOS devices—ever.*</p><p data-block-key="6t5mo">ChromeOS can be centrally managed alongside your other devices via the web-based Google Admin console. From there, IT administrators gain a comprehensive view, allowing them to monitor and manage devices, track application deployments and versions, and control device policies and settings at scale, with changes applied across the fleet in seconds. Additionally, admins can revoke user access and securely wipe data from devices when necessary.</p><p data-block-key="5lvbl">With zero-touch enrollment, it’s possible to deploy a fleet of ChromeOS devices without IT interaction. ChromeOS devices can be shipped directly to your end users, who can then get started in minutes—with policies, settings and apps all instantly applied on first boot as the user connects to the internet and signs in.</p></div>
|
||
<div class="block-paragraph"><h3 data-block-key="pogf7"><b>2. Web applications are the key to a modern endpoint.</b></h3></div>
|
||
<div class="block-paragraph"><p data-block-key="pogf7">While AI was identified as IT’s top priority, the study identified the second-highest priority over the next 12 months as the need to embrace more web-based applications, with 81% of respondents saying that “adopting web-based applications is part of their organization’s digital transformation goals.”<sup>1</sup> Forrester Consulting also found that 90% of surveyed IT respondents believe the future of end user computing is web-based and 78% of respondents indicated that companies that don’t embrace the web will be left behind.<sup>1</sup></p><p data-block-key="dk5q8">The study found that IT departments perceive web-based applications not only as collaboration drivers, but as beneficial for IT departments as well. Why? Quoted from the study, “When the majority of applications that employees use are web-based, it makes all other elements of endpoint management easier to achieve, from simplifying management and improving security to unlocking the power of AI.”<sup>1</sup> In addition to the security benefits mentioned previously, the web helps businesses streamline ecosystem support, access, and deployment.</p><p data-block-key="ahmuq">With <a href="http://support.google.com/chromebook/answer/9658361?hl=en&amp;co=GENIE.Platform%3DDesktop" target="_blank">progressive web apps</a>, key software partnerships featured in <a href="http://chromeos.google/resources/recommended/" target="_blank">Chrome Enterprise Recommended</a>, simple application management, and more, ChromeOS can be the catalyst for businesses to realize the benefits of the web.</p></div>
|
||
<div class="block-paragraph"><h3 data-block-key="pogf7"><b>3. Achieving a modern endpoint benefits the business.</b></h3></div>
|
||
<div class="block-paragraph"><p data-block-key="pogf7">Finally, let's talk about cost. Forrester Consulting found that moving towards a modern endpoint would reduce costs to the IT department by roughly 19%, with 57% believing it will help reduce costs for the IT department.<sup>1</sup> Modern endpoints help IT departments save beyond the endpoint—including infrastructure costs, security software, support costs, and more.</p><p data-block-key="cplpd">ChromeOS devices can help streamline IT, slashing overall costs beyond the hardware, saving businesses $463 per device on average according to IDC.<sup>2</sup> These savings carry over even when deploying <a href="http://www.google.com/chromebook/discover/chromebookplus/" target="_blank">Chromebook Plus</a>, a category of high performance laptops with powerful AI capabilities, and greater hardware specs at a greater value. Chromebook Plus comes with 10 years of automatic updates, staying secure and usable for even longer.</p><p data-block-key="a9jqt">Embracing a modern endpoint strategy with ChromeOS can solve the most pressing IT challenges faced today. Check out the <a href="http://inthecloud.withgoogle.com/forrester-next-generation-endpoint-report/home.html" target="_blank">Forrester Consulting Study</a> to learn more about the modern endpoint, or contact one of our team members to <a href="https://chromeos.google/contact/" target="_blank">learn more about ChromeOS.</a></p></div>
|
||
<div class="block-paragraph"><p data-block-key="d62p6"><i><sup>1</sup></i> <a href="http://inthecloud.withgoogle.com/forrester-next-generation-endpoint-report/home.html" target="_blank"><sup>Forrester Consulting 2024 Modern Endpoint Research, sponsored by Google, April 2024</sup></a></p><p data-block-key="6h6da"><i><sup>2</sup></i> <a href="http://services.google.com/fh/files/misc/idc_business_value_of_chrome_os_paper.pdf" target="_blank"><i><sup>IDC Business Value Paper, sponsored by Google, The Business Value of ChromeOS, doc #49920522, March 2024</sup></i></a></p><p data-block-key="etj2b"><i><sup>*As of May 2024 there has been no evidence of any documented, successful virus attack or ransomware attack on ChromeOS. Data based on ChromeOS monitoring of various national and internal databases.</sup></i></p></div></description><pubDate>Thu, 02 May 2024 09:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/chrome-enterprise/your-modernization-journey-starts-with-the-endpoint-a-forrester-consulting-study-shows-why/</guid><category>Chrome Enterprise</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/2023_Wagtail_hero_template_2436x1200_0mwtXaa.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Your modernization journey starts with the endpoint. A Forrester Consulting study shows why.</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/2023_Wagtail_hero_template_2436x1200_0mwtXaa.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/chrome-enterprise/your-modernization-journey-starts-with-the-endpoint-a-forrester-consulting-study-shows-why/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Rob Beard</name><title>Product Manager</title><department>ChromeOS</department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Max Kayen</name><title>Product Marketing Manager</title><department>ChromeOS</department><company></company></author></item><item><title>Enhancing iEEG seizure identification and similarity search with Google Cloud</title><link>https://cloud.google.com/blog/products/databases/neuropace-scales-ml-workloads-with-vertex-ai/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Globally, epilepsy affects approximately 50 million people. Located in Mountain View, CA, NeuroPace, Inc.<sup>1</sup></span><span style="vertical-align: baseline;"> is committed to transforming the lives of those living with epilepsy by reducing or eliminating their seizures. The company's RNS® System,<sup>2</sup></span><span style="vertical-align: baseline;"> a responsive neurostimulation device, monitors brain activity to detect seizure precursors and delivers targeted electrical stimulation to prevent seizures. This device also captures iEEG (intracranial electroencephalogram) data, with over 15 million recordings from over 5,000 patients collected to date, making it the largest collection of ambulatory iEEG records available. </span></p>
|
||
<p><span style="vertical-align: baseline;">NeuroPace's AI team has developed electrographic seizure classifier models using clinical trial data from the RNS System and has fine-tuned these models through </span><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10354337/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">transfer learning</span></a><span style="vertical-align: baseline;"> to identify seizure onset times. Previously, machine learning (“ML”) training was constrained by the limited number of Graphical Processing Units (GPUs) available in on-premises virtual machines (VMs), slowing down the optimization of models and training processes. NeuroPace tackled this challenge by scaling ML workloads with Google Cloud, moving away from on-premises VMs and utilizing </span><a href="https://cloud.google.com/vertex-ai/?hl=en"><span style="text-decoration: underline; vertical-align: baseline;">Vertex AI</span></a><span style="vertical-align: baseline;"> for more efficient training and hyperparameter tuning.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Leveraging Google Cloud AI infrastructure</span></h3>
|
||
<p><span style="vertical-align: baseline;">Google Cloud's technologies have significantly improved and accelerated NeuroPace’s ML training capabilities. Searching through more than a million iEEG records to identify similar ones, a task that previously took minutes to hours, can now be completed in milliseconds using Google's </span><a href="https://cloud.google.com/alloydb/ai"><span style="text-decoration: underline; vertical-align: baseline;">AlloyDB AI</span></a><span style="vertical-align: baseline;">, part of the </span><a href="https://cloud.google.com/alloydb"><span style="text-decoration: underline; vertical-align: baseline;">AlloyDB for PostgreSQL</span></a><span style="vertical-align: baseline;"> database. Further, the integration of Vertex AI, GPUs, Compute Engine, and Google Cloud Storage has revolutionized NeuroPace’s ML training processes, enhancing scalability, automation, and efficiency.</span></p>
|
||
<p><span style="vertical-align: baseline;">Vertex AI, Google Cloud’s AI development platform, supports the entire ML workflow, including data engineering, model training, deployment, and monitoring. This integration enables NeuroPace's AI team to use various GPUs for model training, with </span><a href="https://cloud.google.com/blog/products/compute/introducing-g2-vms-with-nvidia-l4-gpus"><span style="text-decoration: underline; vertical-align: baseline;">L4 GPUs</span></a><span style="vertical-align: baseline;"> offering better price-performance compared to on-premises resources. With it, they’ve developed a cloud-native ML training system, achieving desired scalability and efficiency through Vertex AI and GPUs.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Patient similarity search with AlloyDB AI</span></h3>
|
||
<p><span style="vertical-align: baseline;">Identifying similar electrophysiological features across epilepsy patients has the potential to aid in discovering effective treatment options. NeuroPace has conducted research studies to identify similar iEEG patterns within a large dataset of over 1 million time-series iEEG records, utilizing the built-in vector search capabilities in AlloyDB AI. By employing IVFFlat and HNSW indexing methods, searches for similar iEEG records in this dataset can now be executed in approximately ten milliseconds.</span></p>
|
||
<p><span style="vertical-align: baseline;">AlloyDB AI enables storing data embeddings in vector form directly in the database, facilitating easier and faster similarity searches compared to standard PostgreSQL. This eliminates the need for elaborate external processing pipelines.</span></p>
|
||
<h3><span style="vertical-align: baseline;">The next-generation disease management system</span></h3>
|
||
<p><span style="vertical-align: baseline;">Data captured from the NeuroPace RNS System may provide insights into seizure trends and triggers leading to optimizing and personalizing epilepsy treatment.<sup>3</sup></span><span style="vertical-align: baseline;"> The research efforts to integrate Google Cloud's infrastructure with NeuroPace's RNS System data are directed towards creating a sophisticated disease management system for epilepsy, emphasizing tailored treatments and enhanced patient well-being.</span></p>
|
||
<hr/>
|
||
<p><sup><em>1. <a href="https://www.neuropace.com/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">https://www.neuropace.com/</span></a></em></sup><br/><sup><em>2. <span style="vertical-align: baseline;">Rx Only. The RNS® System is an adjunctive therapy for adults with refractory, partial onset seizures with no more than 2 epileptogenic foci. See important safety information at </span><a href="http://www.neuropace.com/safety/" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">http://www.neuropace.com/safety/.</span></a></em></sup><br/><sup><em>3. <span style="vertical-align: baseline;">The RNS System does not currently incorporate functionality that is based upon or utilizes AI features.</span></em></sup></p></div></description><pubDate>Wed, 01 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/databases/neuropace-scales-ml-workloads-with-vertex-ai/</guid><category>Customers</category><category>Databases</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Enhancing iEEG seizure identification and similarity search with Google Cloud</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/databases/neuropace-scales-ml-workloads-with-vertex-ai/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Sharanya Desai, Ph.D.</name><title>Technical Fellow, Director of AI, NeuroPace, Inc.</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Iris Fang</name><title>Enterprise Customer Engineer, Google Cloud</title><department></department><company></company></author></item><item><title>Managing Cloud Storage soft delete at scale</title><link>https://cloud.google.com/blog/products/storage-data-transfer/understanding-cloud-storages-new-soft-delete-feature/</link><description><div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Have you ever accidentally deleted your data? Unfortunately, many of us have, which is why most operating systems on personal computers have a recycle bin / trash can where you can go to get your files back. On the enterprise side, these accidental deletions can be at a much larger scale – sometimes involving millions or even billions of objects. There is also the prospect of someone gaining unauthorized access to your data and either performing a ransomware attack to try to hold your data hostage or simply deleting it!</span></p>
|
||
<p><span style="vertical-align: baseline;">We recently launched </span><a href="https://cloud.google.com/storage/docs/soft-delete"><span style="text-decoration: underline; vertical-align: baseline;">soft delete</span></a><span style="vertical-align: baseline;"> for Cloud Storage, an important new data protection feature compatible with all existing Cloud Storage features and workloads. It offers improved protection against accidental and malicious data deletion by providing you with a way to retain and restore recently deleted data at enterprise scale. With soft delete in place, you may also find that your organization can move more quickly when “pruning” old data, knowing that soft delete provides an undo mechanism in case of any mistakes.</span></p>
|
||
<p><span style="vertical-align: baseline;">In this blog, we provide you with the tools and insights you need to optimize your soft delete settings, even at scale, so that you use soft delete to protect your data based on its business criticality.</span></p>
|
||
<h2><span style="vertical-align: baseline;">How does soft delete work and how is it billed?</span></h2>
|
||
<p><span style="vertical-align: baseline;">When soft delete is enabled, deleted objects are retained in a hidden soft-deleted state for the soft delete retention duration set on that bucket, instead of being permanently deleted. If you need any of the soft-deleted objects back, simply run a restore and they are copied back to live state.</span></p>
|
||
<p><span style="vertical-align: baseline;">We introduced soft delete with a seven-day retention duration enabled on all existing buckets and as the default for newly created buckets. Soft delete is on by default because accidental deletion events are unfortunately all too common and much of the data stored in Cloud Storage is business-critical in nature. In addition to the seven-day default, you can select any number of days between 7 and 90, or you can disable the feature entirely.</span></p>
|
||
<p><span style="vertical-align: baseline;">Soft delete usage is billed based on the storage class of the recently deleted objects. In many cases, this only increases bills by a few percentage points, which hopefully represents a good value for the amount of protection that soft delete provides. However, enabling soft delete on buckets that contain a large amount of short-lived (frequently deleted) data can result in large billing increases, since an object deleted after an hour would be billed for the one hour the object was live, plus seven days of soft delete usage.</span></p>
|
||
<h2><span style="vertical-align: baseline;">How valuable is your data?</span></h2>
|
||
<p><span style="vertical-align: baseline;">In order to get to a state where soft delete protects you from data deletion risks that have the lowest economical impact, we recommend that you ask yourself the following three questions:</span></p>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">How important is my organization’s data?</strong><span style="vertical-align: baseline;"> Are we storing temporary objects or media transcodes that could be easily regenerated if they were lost? Soft delete protection is unlikely to be worth it in these cases. Or are we storing data that would put my business and/or customer relationships at risk if it were lost? Soft delete could provide a vital level of protection here.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">What level of data protection do we already have?</strong><span style="vertical-align: baseline;"> If Cloud Storage has the only copy of your business-critical data, then soft delete protection would be much more important than if you were storing long-term backups of all your data in another Google Cloud region, on-prem, or with another cloud provider.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><strong style="vertical-align: baseline;">How much data protection can we afford?</strong><span style="vertical-align: baseline;"> Soft delete can be much less expensive than performing traditional enterprise backups, but can still have a significant impact on billing, depending mostly on your deletion rates. We recommend considering the cost of soft delete relative to your overall Google Cloud bill rather than only storage because it is protecting your business data relied on by your overall workloads. You may find that leaving soft delete enabled on all your buckets only has a single digit percentage impact on your cloud bill, which may be worth it given the protection it provides against both accidental and malicious deletion events.</span></p>
|
||
</li>
|
||
</ol>
|
||
<p><span style="vertical-align: baseline;">Once you have a good idea as to where and how much you want to use soft delete, the next steps depend on your architectural choices and the overall complexity of your organization’s cloud presence. For the rest of this blog, we’ll cover how to assess soft delete’s impact and act on that information, starting with bucket-level metrics, then acting on bucket-level settings within a project, using Terraform for management, and concluding with organizational-level management approaches.</span></p>
|
||
<h2><span style="vertical-align: baseline;">Assessing bucket-level impacts</span></h2>
|
||
<p><span style="vertical-align: baseline;">You can estimate bucket-level soft delete costs using </span><a href="https://cloud.google.com/monitoring/api/v3/metrics"><span style="text-decoration: underline; vertical-align: baseline;">Cloud Monitoring metrics</span></a><span style="vertical-align: baseline;"> and visualize them using the </span><a href="https://cloud.google.com/monitoring/charts/metrics-explorer"><span style="text-decoration: underline; vertical-align: baseline;">Metrics Explorer</span></a><span style="vertical-align: baseline;">. You might want to inspect a handful of buckets that are representative of different kinds of datasets to get a better idea of which ones are more and less expensive to protect with soft delete.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Storage metrics</span></h3>
|
||
<p><span style="vertical-align: baseline;">Recently, we introduced </span><a href="https://cloud.google.com/monitoring/api/metrics_gcp#gcp-storage"><span style="text-decoration: underline; vertical-align: baseline;">new storage metrics</span></a><span style="vertical-align: baseline;"> that allow you to break down the object counts, bytes, and byte seconds by storage class, and then further by live vs. noncurrent vs. soft-deleted vs. multipart. These breakdowns can be extremely useful even beyond any soft delete analysis you may want to perform. In addition, you can now inspect the deletion rate using the new deleted_bytes metric:</span></p>
|
||
<p><span style="vertical-align: baseline;">The </span><strong style="vertical-align: baseline;">storage/v2/deleted_bytes</strong><span style="vertical-align: baseline;"> metric is a delta count of deleted bytes per bucket, grouped by storage class. It can be used to estimate soft delete billing impact, even if soft delete is disabled or set to a different retention duration than the one being considered.</span></p>
|
||
<p><span style="vertical-align: baseline;">For example, the absolute cost of soft delete can be calculated as follows: Soft delete retention duration × deleted bytes × </span><a href="https://cloud.google.com/storage/pricing"><span style="text-decoration: underline; vertical-align: baseline;">storage price</span></a><span style="vertical-align: baseline;">. For example, the cost (assuming us-central1 and Standard storage) of enabling a 7-day soft delete policy with 100,000 GB of deletions during the course of a month is (7 / 30.4375 days) × 100,000 GB × $0.02/GB mo = $459.96 (where 30.4375 is the average number of days per month).</span></p>
|
||
<p><span style="vertical-align: baseline;">The relative cost of soft delete can also be calculated by comparing the </span><strong style="vertical-align: baseline;">storage/v2/deleted_bytes</strong><span style="vertical-align: baseline;"> metric to the existing </span><strong style="vertical-align: baseline;">storage/v2/total_byte_seconds</strong><span style="vertical-align: baseline;"> metric: soft delete retention duration × deleted bytes / total bytes. Continuing from the above example and given 1,000,000 GB-months of storage for the month, the relative cost of enabling soft delete in this case is: (7 / 30.4375 days) × 100,000 GB / 1,000,000 GB = 2.3% impact.</span></p>
|
||
<h2><span style="vertical-align: baseline;">Metrics Explorer</span></h2>
|
||
<p><span style="vertical-align: baseline;">You can use the </span><a href="https://cloud.google.com/monitoring/charts/metrics-explorer"><span style="text-decoration: underline; vertical-align: baseline;">Metrics Explorer</span></a><span style="vertical-align: baseline;"> to create charts that visualize estimated soft delete costs for a given bucket:</span></p>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">In the navigation panel of the Google Cloud console, select Monitoring, and then select Metrics explorer (</span><a href="https://pantheon.corp.google.com/monitoring/metrics-explorer" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">Go to Metrics explorer</span></a><span style="vertical-align: baseline;">).</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Verify that </span><a href="https://cloud.google.com/monitoring/mql"><span style="text-decoration: underline; vertical-align: baseline;">MQL</span></a><span style="vertical-align: baseline;"> is selected in the Language toggle.</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Enter the following query into the query editor:</span></p>
|
||
</li>
|
||
</ol></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &quot;{\r\n fetch gcs_bucket :: storage.googleapis.com/storage/v2/deleted_bytes\r\n | value val(0) * 604800.0&#x27;s&#x27;\r\n | group_by [resource.bucket_name, metric.storage_class], window(), .sum;\r\n fetch gcs_bucket :: storage.googleapis.com/storage/v2/total_byte_seconds\r\n | filter metric.type != &#x27;soft-deleted-object&#x27;\r\n | group_by [resource.bucket_name, metric.storage_class], window(1d), .mean\r\n | group_by [resource.bucket_name, metric.storage_class], window(), .sum\r\n}\r\n| every 30d\r\n| within 360d\r\n| ratio&quot;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc22f0070&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Note: This query assumes a 7-day (604,800 seconds) soft delete window.</span></p>
|
||
<h2><span style="vertical-align: baseline;">Taking action within a project</span></h2>
|
||
<p><span style="vertical-align: baseline;">If you are a storage administrator making decisions about soft delete settings within a project, you may want to go over your list of buckets manually and make decisions based on your business knowledge of what should be protected versus what can go without soft delete. For a larger number of buckets, you might choose to use the above metrics to generate a list of buckets that exceed a billing impact threshold (e.g. 20% impact) on all your buckets and then disable soft delete on those buckets.</span></p>
|
||
<p><span style="vertical-align: baseline;">To assist with this, we published a </span><a href="https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/storage/cost-analysis" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">soft delete billing impact Python script</span></a><span style="vertical-align: baseline;"> on Github that generates a list of buckets in a project that exceed the percentage of billing impact that you specify, factoring in the storage classes of objects inside a bucket. The script can also be used to update the soft delete policies based on a specified relative cost threshold.</span></p>
|
||
<p><span style="vertical-align: baseline;">We recommend you use the </span><a href="https://cloud.google.com/sdk/gcloud/"><span style="text-decoration: underline; vertical-align: baseline;">Google Cloud CLI</span></a><span style="vertical-align: baseline;"> to configure soft delete settings on one or more buckets within a project. After </span><a href="https://cloud.google.com/sdk/docs/install-sdk"><span style="text-decoration: underline; vertical-align: baseline;">installing</span></a><span style="vertical-align: baseline;"> and </span><a href="https://cloud.google.com/docs/authentication/gcloud"><span style="text-decoration: underline; vertical-align: baseline;">signing in</span></a><span style="vertical-align: baseline;">, the following </span><a href="https://cloud.google.com/sdk/gcloud/reference/storage"><span style="text-decoration: underline; vertical-align: baseline;">gcloud storage</span></a><span style="vertical-align: baseline;"> commands are examples of actions you may want to take to enable, update, or disable soft delete policies within a specified project:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;# Set your project ID\r\n$ gcloud config set project $MY_PROJECT_ID\r\n\r\n# Disable Soft Delete on one bucket\r\n$ gcloud storage buckets update --clear-soft-delete gs://example-bucket\r\n\r\n# Disable Soft Delete on a list of buckets\r\n$ cat buckets.txt | gcloud storage buckets update -I --clear-soft-delete\r\n\r\n# Disable Soft Delete on all buckets in the project\r\n$ gcloud storage buckets update --clear-soft-delete gs://*\r\n\r\n# Enable Soft Delete on one bucket\r\n$ gcloud storage buckets update --soft-delete-duration=7d gs://example-bucket\r\n\r\n# Enable Soft Delete on a list of buckets\r\n$ cat buckets.txt | gcloud storage buckets update -I --soft-delete-duration=7d\r\n\r\n# Enable Soft Delete on all buckets in the project with a 14-day retention duration\r\n$ gcloud storage buckets update --soft-delete-duration=14d gs://*&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc22f0340&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h2><span style="vertical-align: baseline;">Taking action with Terraform</span></h2>
|
||
<p><span style="vertical-align: baseline;">If you use an orchestration layer like </span><a href="https://cloud.google.com/docs/terraform"><span style="text-decoration: underline; vertical-align: baseline;">Terraform</span></a><span style="vertical-align: baseline;">, adapting to soft delete should be as simple as updating your templates and deciding on the soft delete retention duration for each workload. This could also involve creating new templates dedicated to short-lived data so that soft delete is disabled for buckets created from those templates. Once you’ve defined your settings, Terraform can update existing buckets to conform to the templates, and new buckets should be created with your intended settings.</span></p>
|
||
<p><span style="vertical-align: baseline;">With Terraform, the primary thing you need to do is to update your template(s) to include a soft delete policy. Here is an example of setting the soft delete retention duration to seven days (604800 seconds) in a </span><code style="vertical-align: baseline;">google_storage_bucket</code><span style="vertical-align: baseline;"> resource:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;resource &quot;google_storage_bucket&quot; &quot;bucket&quot; {\r\n name = &quot;example-bucket&quot;\r\n location = &quot;US&quot;\r\n …\r\n soft_delete_policy {\r\n retention_duration_seconds = 604800\r\n }\r\n ...\r\n}&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc22f0160&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">To disable soft delete instead, simply set </span><code style="vertical-align: baseline;">retention_duration_seconds = 0</code><span style="vertical-align: baseline;">.</span></p>
|
||
<p><span style="vertical-align: baseline;">For more information, please also see: </span><a href="https://cloud.google.com/storage/docs/terraform-create-bucket-upload-object"><span style="text-decoration: underline; vertical-align: baseline;">Use Terraform to create storage buckets and upload objects</span></a><span style="vertical-align: baseline;">.</span></p>
|
||
<h2><span style="vertical-align: baseline;">Taking action across a large organization</span></h2>
|
||
<p><span style="vertical-align: baseline;">If you work for a large enterprise with thousands of projects and millions of buckets and mostly don’t use an orchestration layer, then a manual approach is not realistic, and you will need to make decisions at scale. If this is your situation, we recommend that you first learn about the bucket-level metrics and how to take action within a project as described earlier. In this section, we’ll extend these techniques to the organization level. Again, we assume you have already installed an up-to-date version of the gcloud CLI which you will need for this section.</span></p>
|
||
<p><span style="vertical-align: baseline;">To implement a policy across even the most complex of organizations, you will likely need to approach it in three steps using the gcloud command line environment:</span></p>
|
||
<ol>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Obtain permissions: ensure you can list and change bucket-level settings across the organization</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Assess: decide on an impact threshold above which you will disable soft delete, and obtain a list of buckets surpassing that threshold</span></p>
|
||
</li>
|
||
<li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;">
|
||
<p role="presentation"><span style="vertical-align: baseline;">Act: disable soft delete on that list of buckets</span></p>
|
||
</li>
|
||
</ol>
|
||
<h3><span style="vertical-align: baseline;">Obtain permissions</span></h3>
|
||
<p><span style="vertical-align: baseline;">Before you can do anything, you will need to identify someone with sufficient access permissions to analyze and change bucket-level configurations across your organization. This could be an existing </span><a href="https://cloud.google.com/resource-manager/docs/creating-managing-organization"><span style="text-decoration: underline; vertical-align: baseline;">Organization Administrator</span></a><span style="vertical-align: baseline;">. Alternatively, your Organization Administrator could create a custom role and assign it to you or another administrator for the specific purpose of managing soft delete settings:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;gcloud iam roles create storageBucketsUpdate \\\r\n --organization=example-organization-id-1 \\\r\n --title=&quot;Storage bucket update&quot; \\\r\n --description=&quot;Grants permission to get, list, and update Storage buckets.&quot; \\\r\n --stage=GA \\\r\n --permissions=&quot;storage.buckets.get,storage.buckets.list,storage.buckets.update&quot;\r\n\r\ngcloud organization add-iam-policy-binding example-organization-id-1 \\\r\n --member=\&#x27;user:test-user@example.com\&#x27; \\\r\n --role=\&#x27;storageBucketsUpdate\&#x27; \\\r\n --condition=\&#x27;expression=request.time &lt; timestamp(&quot;2024-07-01T00:00:00Z&quot;),\\\r\n title=expires_2024_07_01,\\\r\n description=Expires at midnight on 2024-07-01\&#x27;&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc22f0b20&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Note that once everything is done, at the end of this process, and your buckets are all updated, the Organization Administrator could safely delete this custom role if there wasn’t an ongoing need for a role with continued access to these settings:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;gcloud iam roles delete storageBucketsUpdate --organization=example-organization-id-1&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc22f0400&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><h3><span style="vertical-align: baseline;">Assess</span></h3>
|
||
<p><span style="vertical-align: baseline;">Armed with the power to act on bucket-level configurations across your organization, you can apply the project-level analysis above to obtain a list of all buckets across your organization that exceed your chosen impact threshold. Alternatively, you might choose to apply a uniform setting like 0d or 14d across all buckets in your organization.</span></p>
|
||
<h3><span style="vertical-align: baseline;">Act</span></h3>
|
||
<p><span style="vertical-align: baseline;">To update the soft delete policy for all your buckets across all your projects, you can iterate through all your projects, making the appropriate changes to the buckets in each project. For example, the following command disables soft delete on all buckets across your organization:</span></p></div>
|
||
<div class="block-code"><dl>
|
||
<dt>code_block</dt>
|
||
<dd>&lt;ListValue: [StructValue([(&#x27;code&#x27;, &#x27;gcloud projects list --format=&quot;value(projectId)&quot; | while read project\r\ndo\r\n gcloud storage buckets update --project=$project --clear-soft-delete gs://*\r\ndone&#x27;), (&#x27;language&#x27;, &#x27;&#x27;), (&#x27;caption&#x27;, &lt;wagtail.rich_text.RichText object at 0x3eefc8d9cc10&gt;)])]&gt;</dd>
|
||
</dl></div>
|
||
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Alternatively, you can use the </span><code style="vertical-align: baseline;">filter</code><span style="vertical-align: baseline;"> option of </span><code style="vertical-align: baseline;">projects list</code><span style="vertical-align: baseline;"> to only target a subset of your projects. For example, you might want to update projects with a specific label (</span><code style="vertical-align: baseline;">--filter="labels.environment:prod"</code><span style="vertical-align: baseline;">) or with a certain parent (</span><code style="vertical-align: baseline;">--filter="parent.id:123456789"</code><span style="vertical-align: baseline;">).</span></p>
|
||
<p><span style="vertical-align: baseline;">As a best practice, we recommend that you consider replacing the per-project action above with a command that selectively disables soft delete on specific bucket IDs. For example, you could loop through your project list, running the </span><a href="https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/storage/cost-analysis" rel="noopener" target="_blank"><span style="text-decoration: underline; vertical-align: baseline;">soft delete billing impact Python script</span></a><span style="vertical-align: baseline;"> for each project to update your bucket settings according to a % impact threshold you select to get a much more tailored outcome.</span></p>
|
||
<h2><span style="vertical-align: baseline;">Summary</span></h2>
|
||
<p><span style="vertical-align: baseline;">By following the best practices in this blog and taking advantage of the available tooling and controls, we hope that you now feel more confident in your ability to protect your business-critical data with soft delete while simultaneously minimizing its billing impact.</span></p></div></description><pubDate>Wed, 01 May 2024 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/storage-data-transfer/understanding-cloud-storages-new-soft-delete-feature/</guid><category>Google Cloud Next</category><category>Storage & Data Transfer</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/optimization_2022.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Managing Cloud Storage soft delete at scale</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/optimization_2022.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/products/storage-data-transfer/understanding-cloud-storages-new-soft-delete-feature/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Geoffrey Noer</name><title>Group Product Manager</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Michael Roth</name><title>Software Engineer</title><department></department><company></company></author></item></channel></rss> |