Audit logging

This page describes the audit logs created by Cloud Search as part of Cloud Audit Logs.

Overview

Google Cloud services write audit logs to help you answer the questions, "Who did what, where, and when" within your resources. Your Cloud projects contain only the audit logs for resources that are directly within the project. Other entities, such as folders, organizations, and Cloud Billing accounts, contain the audit logs for the entity itself.

For a general overview of Cloud Audit Logs, see Cloud Audit Logs. For a deeper understanding of Cloud Audit Logs, review Understanding audit logs.

Cloud Audit Logs provides the following audit logs for each Cloud project, folder, and organization:

• Admin Activity audit logs containing entries corresponding to methods performing Admin write operations. The methods corresponding to Admin Activity:Admin write operations are covered in an upcoming Audited operations section.
• Data Access audit logs containing entries corresponding to methods performing Admin read, Data write, and Data read operations. The methods corresponding to Data Access:Admin read, Data Access:Data write, Data Access:Data read operations are covered in an upcoming Audited operations section.
• System Event audit logs
• Policy Denied audit logs

Cloud Search writes Admin Activity audit logs, which record operations that modify the configuration or metadata of a resource. You can't disable Admin Activity audit logs.

Only if explicitly enabled, Cloud Search writes Data Access audit logs. Data Access audit logs contain API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data.

Cloud Search doesn't write System Event audit logs.

Cloud Search doesn't write Policy Denied audit logs.

Audited operations

The following summarizes which API operations correspond to each audit log type in Cloud Search:

Audit logs category	Cloud Search operations
Admin Activity: Admin write	indexing.datasources.updateSchema indexing.datasources.deleteSchema settings.datasources.create settings.datasources.delete settings.datasources.update settings.searchapplications.create settings.searchapplications.delete settings.searchapplications.reset settings.searchapplications.update settings.updateCustomer cloudsearch.IdentitySourceService.create cloudsearch.IdentitySourceService.update cloudsearch.IdentitySourceService.delete
Data Access: Admin read	indexing.datasources.getSchema settings.datasources.get settings.datasources.list settings.searchapplications.get settings.searchapplications.list settings.getCustomer cloudsearch.IdentitySourceService.get cloudsearch.IdentitySourceService.list
Data Access: Data write	indexing.datasources.items.delete indexing.datasources.items.deleteQueueItems indexing.datasources.items.index indexing.datasources.items.poll indexing.datasources.items.push indexing.datasources.items.unreserve indexing.datasources.items.upload media.upload
Data Access: Data read	indexing.datasources.items.get indexing.datasources.items.list operations.get operations.list debug.datasources.items.checkAccess debug.datasources.items.searchByViewUrl stats.getIndex stats.getQuery stats.getSession stats.getUser stats.index.datasources.get stats.query.searchapplications.get stats.session.searchapplications.get stats.user.search applications.get debug.identitysources.items.listForunmappedidentity debug.identitysources.unmappedids.list debug.datasources.items.unmappedids.list query.sources.list query.suggest query.search stats.getSearchapplication

Audit log format

Audit log entries—which can be viewed in Cloud Logging using the Logs Explorer, the Cloud Logging API, or the gcloud command-line tool—include the following objects:

The log entry itself, which is an object of type LogEntry. Useful fields include the following:

• The logName contains the resource ID and audit log type.
• The resource contains the target of the audited operation.
• The timeStamp contains the time of the audited operation.
• The protoPayload contains the audited information.
• The audit logging data, which is an AuditLog object held in the protoPayload field of the log entry.

Optional service-specific audit information, which is a service-specific object. For earlier integrations, this object is held in the serviceData field of the AuditLog object; later integrations use the metadata field.

For other fields in these objects, and how to interpret them, review Understanding audit logs.

Log name

Cloud Audit Logs resource names indicate the Cloud project or other Google Cloud entity that owns the audit logs, and whether the log contains Admin Activity, Data Access, Policy Denied, or System Event audit logging data. For example, the following shows log names for project-level Admin Activity audit logs and an organization's Data Access audit logs. The variables denote project and organization identifiers.

projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity
organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Note: The part of the log name following /logs/ must be URL-encoded. The forward-slash character, /, must be written as %2F.

Service name

Cloud Search audit logs use the service name cloudsearch.googleapis.com.

Resource types

Cloud Search audit logs use the resource type audited_resource for all audit logs.

For a list of other resource types, see Monitored resource types.

Enable audit logging

By default, audit logging is disabled for Cloud Search API. To enable audit logging for Google Cloud Search:

1. (optional) If you haven't created a Google Cloud Platform project to store logs, refer to Configure access to the Google Cloud Search API.

2. Obtain the project ID for the Google Cloud where you want to store the logs. To learn how to obtain a project ID, refer to Identifying projects.

3. To enable audit logging for a specific API, you need to determine its log category to enable. For APIs and their corresponding categories, refer to Audited operations earlier in this document.

4. Use the updateCustomer() REST API method to update the auditLogSettings with the log categories to enable:

   1. Obtain an OAuth 2.0 access token from the Google Authorization Server. For information about obtaining the token, refer to step 2 of Using OAuth 2.0 to Access Google APIs. Use one of the following OAuth scopes while obtaining the access token:

      • https://www.googleapis.com/auth/cloud_search.settings.indexing
      • https://www.googleapis.com/auth/cloud_search.settings
      • https://www.googleapis.com/auth/cloud_search

   2. Run the following curl command.

   curl --request PATCH \
   'https://cloudsearch.googleapis.com/v1/settings/customer?updateMask=auditLoggingSettings&key=[YOUR_API_KEY]' \
   --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
   --header 'Content-Type: application/json' \
   --data '{"auditLoggingSettings": { "project": "projects/PROJECT_ID", "CATEGORY1": "true", "CATEGORY2": "true" } }'
   

   Where:

   • YOUR_ACCESS_TOKEN is OAuth 2.0 access token obtained in step 4a.
   • PROJECT_ID is the project ID obtained in step 2.
   • CATEGORY1, CATEGORY2, … are the categories you chose to enable in step 3. Valid values are logAdminReadActions, logDataWriteActions, and logDataReadActions. Admin write actions are enabled by default and cannot be disabled. If you want audit logging for query methods, you must enable the Data read category.

   After updating AuditLoggingSettings, further requests to Cloud Search API generate an audit log in the project ID specified in AuditLoggingSettings.

5. Audit logging for query methods requires the Data read category to be enabled (done in step 4). To enable audit logging for query methods (query.sources.list, query.suggest and query.search) follow these additional steps:

   1. For each search application for which you want to enable audit logging, retrieve the name. The name must be in the form of searchapplications/<search_application_id>.

   2. Use the name to call settings.searchapplications.update with enableAuditLog set to true.

6. To enable audit logging for calls from cloudsearch.google.com, ensure the Data read category is enabled (step 4). Additionally, perform step 5b with a name of searchapplications/default .

Once enabled, logs can be viewed in the Logs Explorer section of the Google Cloud Console. Use the following filter to view only Cloud Search audit logs:

protoPayload.serviceName="cloudsearch.googleapis.com"

For information about how to view logs, refer to Logs explorer overview.

Note: If you need to change the project used to store logs, follow steps 2-4 with the new project ID.

Audit log permissions

Identity and Access Management permissions and roles determine which audit logs you can view or export. Logs reside in Cloud projects and in some other entities including organizations, folders, and Cloud Billing accounts. For more information, see Understanding roles.

To view Admin Activity audit logs, you must have one of the following IAM roles in the project that contains your audit logs:

• Project Owner, Project Editor, or Project Viewer
• The Logging Logs Viewer role
• A custom IAM role with the logging.logEntries.list IAM permission

To view Data Access audit logs, you must have one of the following roles in the project that contains your audit logs:

• Project Owner
• Logging's Private Logs Viewer role
• A custom IAM role with the logging.privateLogEntries.list IAM permission

If you are using audit logs from a non-project entity, such as an organization, then change the Cloud project roles to suitable organization roles.

View logs

To find and view audit logs, you need to know the identifier of the Cloud project, folder, or organization for which you want to view audit logging information. You can further specify other indexed LogEntry fields, like resource.type; for details, review Build queries in the Logs Explorer.

The following are the audit log names; they include variables for the identifiers of the Cloud project, folder, or organization:

   projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity
   projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access
   projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event
   projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fpolicy

   folders/FOLDER_ID/logs/cloudaudit.googleapis.com%2Factivity
   folders/FOLDER_ID/logs/cloudaudit.googleapis.com%2Fdata_access
   folders/FOLDER_ID/logs/cloudaudit.googleapis.com%2Fsystem_event
   folders/FOLDER_ID/logs/cloudaudit.googleapis.com%2Fpolicy

   organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity
   organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access
   organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fsystem_event
   organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fpolicy

You have several options for viewing your audit log entries.

Note: The Cloud Console Logs Explorer currently supports viewing logs for Google Cloud projects only. To read log entries for a specified folder or organization, use the Cloud Logging API or the gcloud command-line tool{.

Console

You can use the Logs Explorer in the Cloud Console to retrieve your audit log entries for your Cloud project:

1. In the Cloud Console, go to the Logging > Logs Explorer page.

   Go to the Logs Explorer page

   Note: If you are using the Legacy Logs Viewer page, switch to the Logs Explorer page.

2. On the Logs Explorer page, select an existing Cloud project.

3. In the Query builder pane, do the following:

   • In Resource, select the Google Cloud resource type whose audit logs you want to see.

   • In Log name, select the audit log type that you want to see:

     • For Admin Activity audit logs, select activity.
     • For Data Access audit logs, select data_access.
     • For System Event audit logs, select system_event.
     • For Policy Denied audit logs, select policy.

   If you don't see these options, then there aren't any audit logs of that type available in the Cloud project.

   For more details about querying using the new Logs Explorer, see Build queries in the Logs Explorer.

gcloud

The gcloud provides a command-line interface to the Logging API. Supply a valid PROJECT_ID, FOLDER_ID, or ORGANIZATION_ID in each of the log names.

To read your Google Cloud project-level audit log entries, run the following command:

gcloud logging read "logName : projects/PROJECT_ID/logs/cloudaudit.googleapis.com" \
    --project=PROJECT_ID

To read your folder-level audit log entries, run the following command:

gcloud logging read "logName : folders/FOLDER_ID/logs/cloudaudit.googleapis.com" \
    --folder=FOLDER_ID

To read your organization-level audit log entries, run the following command:

gcloud logging read "logName : organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com" \
    --organization=ORGANIZATION_ID

For more information about using the gcloud tool, see gcloud logging read.

API

When building your queries, replace the variables with valid values, substitute the appropriate project-level, folder-level, or organization-level audit log name or identifiers as listed in the audit log names. For example, if your query includes a PROJECT_ID, then the project identifier you supply must refer to the currently selected Cloud project.

To use the Logging API to look at your audit log entries, do the following:

1. Go to the Try this API section in the documentation for the entries.list method.

2. Put the following into the Request body part of the Try this API form. Clicking on this prepopulated form automatically fills the request body, but you need to supply a valid PROJECT_ID in each of the log names.

   {
     "resourceNames": [
       "projects/PROJECT_ID"
     ],
     "pageSize": 5,
     "filter": "logName : projects/PROJECT_ID/logs/cloudaudit.googleapis.com"
   }
   

3. Click Execute.

For more details about querying, see Logging query language.

For a sample audit log entry and how to find the most important information in it, see Understanding audit logs.

Exporting audit logs

You can export audit logs in the same way that you export other kinds of logs. For details about how to export your logs, see Exporting logs. Here are some applications of exporting audit logs:

• To keep audit logs for a longer period of time or to use more powerful search capabilities, you can export copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub. Using Pub/Sub, you can export to other applications, other repositories, and to third parties.

• To manage your audit logs across an entire organization, you can create aggregated sinks that can export logs from any or all Cloud projects in the organization.

• If your enabled Data Access audit logs are pushing your Cloud projects over their logs allotments, you can export and exclude the Data Access audit logs from Logging. For details, see Excluding logs.

Pricing and retention

Cloud Logging does not charge you for audit logs that cannot be disabled, including all Admin Activity audit logs. Cloud Logging charges you for Data Access audit logs that you explicitly request.

For more information about audit logs pricing, see Google Cloud's operations suite pricing.

The storage duration associated with Cloud Search audit logs are:

• Admin Activity logs (or Admin Write) - These logs are retained for 400 days.
• Data Access logs (Admin Read, Data Write and Data Read) - These logs are retained for 30 days.

For more information about storage duration, see Logs retention periods.

Current limitations

Cloud Search audit logging has these current limitations:

• The size of log entry must be less than 512 KB. If the size grows beyond 512 KB, then the response is dropped from the log entry. If that doesn't reduce the size to 512 KB or lower, the request is dropped. Finally, if the size still exceeds 512 KB, the log entry is dropped.

• Response bodies aren't logged for list(), get() and suggest() methods. However, response statuses are available.

• Only Query API calls from cloudsearch.google.com (if enabled) and customer search applications are logged.

• For search() calls, only Query, RequestOptions and DataSourceRestriction are logged in the request. In the response, only the url and metadata (source and objectType) for each SearchResult are audited.

• Calls issued to the Cloud Search backend and corresponding to data export aren’t audited.