Skip to main content

Configuring a GCP Integration

Beverley Andalora
  • Updated

Before you can integrate Google Cloud Platform with Virtana Platform, you must properly configure a Google Cloud service account and role. You then copy the billing account ID and the names of the BigQuery dataset, the project, and the service account being used for the Virtana Platform integration and enter it in Virtana Platform.

To complete the integration, perform the following tasks:

  1. GCP Configuration Prerequisites

  2. Enter the GCP Values in Virtana Platform

Note

In all GCP configuration tasks, the steps performed in GCP were accurate at the time the tasks were written. If the GCP interface changes, some steps might be different than described.

GCP Configuration Prerequisites

Note

Support for GCP is currently an Early Access feature. It is available to all Virtana Cloud Cost Management customers. Contact your Account Representative or cloudcostsavings@virtana.com to enable GCP for your account.

There are several configuration tasks that you must have completed in GCP prior to integrating with Virtana Platform. The process outlined below is just an overview of what needs to be done, with some specific configuration selections required by Virtana. If you need further details about how to perform the required tasks, see the GCP website:

You must enable BigQuery export of your Cloud Billing data and create a Virtana-specific role before you integrate with Virtana Platform. Following is a list of the configuration actions that must be completed in GCP to enable data export. The time required to complete these tasks depends on how familiar you are with the GCP console, but generally takes less than an hour.

Tip

You can use an existing project, dataset, and service account if they meet the configuration requirements. However, Virtana recommends creating these resources new, to ensure they are configured as needed.

Important

To integrate with Virtana Platform, you must be managing your Google Cloud resources using projects.

Required Permissions

To enable and configure the export of Cloud Billing data to a BigQuery dataset, you need the following permissions:

  • Billing Account Administrator role for the target Cloud Billing account.

  • BigQuery User role for the Cloud project that contains the BigQuery dataset that will be used to store the Cloud Billing data.

  • You might also need project permissions.

    If you are a member of a Google Cloud Organization, you need the Project Creator role on the Organization or Folder, to create a new project.

For more information about Google Cloud permissions, see Google Cloud documentation.

Navigating the Google Cloud Console

The following tasks describe navigation from the Google Cloud Console navigation (hamburger) menu in the upper left of the UI. You can also navigate to console pages by using the search tool.

Tip

If a sub-navigation panel doesn't display the UI as it should, you can bring it back by logging out of GCP and logging back in.

Create a Cloud Billing Account

If you already have a billing account, skip this task.

Google Cloud recommends that you create a single central cloud billing account, rather than multiple accounts. Multiple accounts are supported, but can cause issues. See the GCP Cloud Billing documentation for details.

  1. Navigate to Billing > Billing Management > Account management.

    Navigate using the left navigation pane or by using Search.

  2. Click Create Account and complete the form.

  3. Save a copy of the Billing Account ID.

    You will need to enter this ID into the Integration Setup form in Virtana Platform.

Create a Project as Billing Account

This project will contain the BigQuery dataset that will be used to store the Cloud Billing data.

Important: The project you select or create must be linked to the same cloud billing account as the GCP projects that generate the cost and pricing data that you want to export.

  1. Navigate to IAM & Admin > Create a Project.

    Navigate using the left navigation pane or by using Search.

  2. Complete the form to create the project.

    Give it a name that is easily identifiable as a project for Virtana integration.

    The Project ID is auto-generated, but can be modified at this point. It cannot be changed later.

Add a Dataset to Enable the Export of Billing Data

This dataset will be used to store the Cloud Billing data.

  1. Navigate to Billing > Billing Export.

  2. Under Detailed Usage Cost click Edit Settings.

    Be sure to edit Detailed usage cost settings so that all required data is exported.

    vp-integ-gcp-prereqs-billing-export-detailed-usage.png

  3. Select the project you just created and in the Dataset field click Create New Dataset, complete the form, and save.

    Recommendation: Name the dataset so it's clearly identifiable as being the detailed usage dataset to be used for the integration with Virtana. If you enter the wrong name in Virtana Platform, the integration will fail.

    Important:

    • Ensure Enable Table Expiration is NOT enabled.

    • Under Advanced Options, ensure "Google-managed encryption key" is selected. Do not use the "Customer-managed key" option.

Create a Role

You must create a role that allows Virtana Platform to access BigQuery data.

  1. Navigate to IAM & Admin > Roles and ensure the project you just created is still selected in the project selector.

  2. Filter the table for "bigquery" and select the required roles.

    Roles Required for Virtana

    Select the following roles to apply to a new role for the Virtana integration.

    The custom role contains permissions to enable: 

    • Querying cost and utilization BigQuery table 

    • Access customer topology APIs

    • Access performance metrics

    BigQuery IAM role permissions

    Permissions are assigned by roles at organization, dataset, or table level.

    • Organization level roles

      Provide permissions to run BigQuery jobs and access BigQuery resources.

    • Dataset level roles

      Provide access to a specific dataset. They do not provide access to all of a project’s resources.

    • Table or view level roles

      Provide access to an individual resource within a dataset. Do not provide access to all of a dataset’s resources.

      These resources include table and views. These resources do not include routines or models.

    • Multiple roles can be applied, resulting in a union of permissions from these roles.

    Roles for querying a table

    Role

    Description

    roles/bigquery.dataViewer

    to read data and metadata from the table or view

    roles/bigquery.filteredDataViewer

    to read filtered filtered table data

    roles/bigquery.readSessionUser

    create and use read sessions

    roles/bigquery.user

    read dataset metadata and list tables in the dataset

    API IAM role permissions

    API access can be granted at organization level or at individual endpoints service level. 

    • Organization level roles

      Include endpoint portal admin roles, such as list, attach, and detach custom domains.

    • Endpoint service level roles 

      Contain permissions to access endpoint APIs.

    Roles for accessing customer APIs (without editing):

    Role

    Description

    roles/servicemanagement.serviceConsumer

    Permissions for a principal to view and enable the API in their own project

    roles/servicemanagement.serviceController

    Permissions to make calls to the check and report methods in the Service Infrastructure API during runtime

    Performance metrics IAM role permissions

    The roles below are required for monitoring and provide read and write permissions for metrics.

    Roles for metrics:

    Role

    Description

    roles/monitoring.viewer

    grants read only access to monitoring

    roles/monitoring.metricWriter

    grants writing monitoring data to metrics scope

    Tip

    As you are selecting roles, be careful to click only on the checkboxes. If you click elsewhere in the table it will deselect any items you already selected.

    vp-integ-gcp-prereqs-role-create.png

  3. Click Create Role From Selection and complete the required fields.

    Give the role an easily identifiable title, such as Virtana Integration.

  4. Click Add Permissions and add any remaining permissions that are needed.

    vp-integ-gcp-prereqs-role-create-form.png

  5. In the table of assigned permissions, deselect any permissions you don't want included.

    Be careful to click only on the checkboxes.

    Permissions Required for Virtana

    Following are lists of all roles required for the integration with Virtana Platform. Any permissions not in these lists can be removed from the Virtana role that you create in GCP.

    BigQuery permissions

    bigquery.bireservations.get

    bigquery.reservations.get

    bigquery.capacityCommitments.get

    bigquery.reservations.list

    bigquery.capacityCommitments.list

    bigquery.routines.get

    bigquery.config.get

    bigquery.routines.list

    bigquery.datasets.create

    bigquery.rowAccessPolicies.getFilteredData

    bigquery.datasets.get

    bigquery.savedqueries.get

    bigquery.datasets.getIamPolicy

    bigquery.savedqueries.list

    bigquery.jobs.create

    bigquery.tables.createSnapshot

    bigquery.jobs.list

    bigquery.tables.export

    bigquery.models.export

    bigquery.tables.get

    bigquery.models.getData

    bigquery.tables.getData

    bigquery.models.getMetadata

    bigquery.tables.getIamPolicy

    bigquery.models.list

    bigquery.tables.list

    bigquery.reservationAssignments.list

    bigquery.transfers.get

    bigquery.reservationAssignments.search

    bigquerymigration.translation.translate

    Performance permissions

    cloudnotifications.activities.list

    monitoring.notificationChannelDescriptors.list

    monitoring.alertPolicies.get

    monitoring.notificationChannels.get

    monitoring.alertPolicies.list

    monitoring.notificationChannels.list

    monitoring.dashboards.get

    monitoring.publicWidgets.get

    monitoring.dashboards.list

    monitoring.publicWidgets.list

    monitoring.groups.get

    monitoring.services.get

    monitoring.groups.list

    monitoring.services.list

    monitoring.metricDescriptors.create

    monitoring.slos.get

    monitoring.metricDescriptors.get

    monitoring.slos.list

    monitoring.metricDescriptors.list

    monitoring.timeSeries.create

    monitoring.monitoredResourceDescriptors.get

    monitoring.timeSeries.list

    monitoring.monitoredResourceDescriptors.list

    monitoring.uptimeCheckConfigs.get

    monitoring.notificationChannelDescriptors.get

    monitoring.uptimeCheckConfigs.list

    API permissions

    opsconfigmonitoring.resourceMetadata.list

    servicemanagement.services.get

    resourcemanager.projects.get

    servicemanagement.services.quota

    servicemanagement.services.bind

    servicemanagement.services.report

    servicemanagement.services.check

    stackdriver.projects.get

  6. Click Create.

    The new role displays in the Roles table.

Create a Service Account

You must create a service account, assign the role you created, and give the Virtana Platform GCP service account access to your service account.

  1. Navigate to IAM & Admin > Service Accounts.

    Ensure the project you just created is still selected in the project selector.

  2. Click Create a Service Account and enter the service account details.

    Recommendation: Name the account so it's clearly identifiable as the account for the Virtana Platform integration. If you enter the wrong name in Virtana Platform, the integration will fail.

  3. Click Create and Continue.

    A section displays for granting access.

  4. Click Select a role, and select the role you previously created.

    Example: Virtana Integration

  5. Click Continue and Done.

    You don't need to grant access to users.

    The new account displays in the service accounts list.

  6. Click the name of the new service account, click the Permissions tab, and click Grant Access.

    vp-integ-gcp-prereqs-servacct-roles.png

  7. Enter the principal name.

    Principal name: customer-access@app-customer-access.iam.gserviceaccount.com

    You must enter this exact name. This is the Virtana service account that will access the GCP service account you created.

    The name is not case-sensitive, so capital and lowercase letters are read the same way.

  8. Assign two roles, replacing Service Account Admin role:

    Role 1: Service Account Token Creator

    Role 2: Service Account User

    screenshot of the principal and role fields filled in

  9. Click Save and verify that the new principal displays in the list of principals.

    vp-integ-gcp-prereqs-servacct-principal.png
Query your data in BigQuery to ensure everything is working

Your data won't be accessible for anywhere from a few hours to 48 hours. Google Cloud provides sample data you can use if you want to test the connection immediately. Information about the sample data is available at the end of the GCP tutorial entitled, Analyze your Cloud Billing data with BigQuery.

  1. Navigate to BigQuery > SQL Workspace.

  2. Expand the billing account project and click on the associated dataset.

    You might see a resource named "gcp_billing_export_resource..." This is a service account that is owned and managed by GCP. Do not delete it.

  3. Open the Editor, input your query, and run it.

    See the GCP BigQuery documentation or the blog BigQuery explained: How to query your data for details.

Enter the GCP Values in Virtana Platform

You must connect your GCP account with Virtana Platform, so that Virtana Platform can access and process GCP data.

Note

Support for GCP is currently an Early Access feature. It is available to all Virtana Cloud Cost Management customers. Contact your Account Representative or cloudcostsavings@virtana.com to enable GCP for your account.

About This Task

You need to gather the billing account ID, BigQuery dataset name, project name, and service account ID from the Google Cloud Console and enter it in Virtana Platform. So both GCP and Virtana Platform need to be open in separate browser windows.

Prerequisites

You must have completed the tasks described in GCP Configuration Prerequisites.

Steps

  1. Open the Virtana Platform (VP) console and navigate to Settings > Integrations > Cloud Providers and click the GCP card or click Add Integration.

    The Setup New GCP Integration form displays.

    screenshot of the GCP integration setup form in Virtana Platform

  2. In a separate browser window, open the Google Cloud Console for GCP.

    The Explorer panel displays.

  3. (GCP) Select Billing from the Navigation hamburger menu, and then locate and copy the billing account ID.

    The billing account ID can be found on several different pages, including Billing management>Account management, Billing Account Overview, and My Billing Accounts.

    screenshot of 3 pages in GCP where the billing account ID can be located.

  4. (VP) Paste the copied ID into the Billing Account ID field in Virtana Platform.

  5. (GCP) Navigate to the BigQuery Explorer panel and expand the project that was created for the Virtana Platform integration.

    vp-integ-gcp-collect-info.png

  6. (GCP) Click on the name of the dataset you want to use for the Virtana Platform integration.

  7. (GCP) Under Dataset Info, click the Copy to Clipboard icon next to the Dataset ID.

  8. (VP) Paste the dataset name in the BigQuery Dataset... field and the project name in the Project that stores the dataset field in the Setup form in Virtana Platform.

  9. (GCP) Navigate to IAM & Admin > Service Accounts and copy the email for the service account to be used with Virtana Platform.

    vp-integ-gcp-setup-servacct.png

  10. (VP) Paste the service account email into the Service Account... field in Virtana Platform.

  11. Give the integration a descriptive name and click Save.

    The new integration displays on the Cloud Provider Integrations page.

    You can edit any integration by clicking the integration name in the table.

This completes the GCP integration setup.

Roles and Permissions Required for a GCP Account

Before integrating with Virtana Platform, you need to create a service account and an associated custom role with the required permissions. The tables below describe the roles and permissions you must assign to a custom role for the service account.

Permissions Required for Virtana

Following are lists of all roles required for the integration with Virtana Platform. Any permissions not in these lists can be removed from the Virtana role that you create in GCP.

BigQuery permissions

bigquery.bireservations.get

bigquery.reservations.get

bigquery.capacityCommitments.get

bigquery.reservations.list

bigquery.capacityCommitments.list

bigquery.routines.get

bigquery.config.get

bigquery.routines.list

bigquery.datasets.create

bigquery.rowAccessPolicies.getFilteredData

bigquery.datasets.get

bigquery.savedqueries.get

bigquery.datasets.getIamPolicy

bigquery.savedqueries.list

bigquery.jobs.create

bigquery.tables.createSnapshot

bigquery.jobs.list

bigquery.tables.export

bigquery.models.export

bigquery.tables.get

bigquery.models.getData

bigquery.tables.getData

bigquery.models.getMetadata

bigquery.tables.getIamPolicy

bigquery.models.list

bigquery.tables.list

bigquery.reservationAssignments.list

bigquery.transfers.get

bigquery.reservationAssignments.search

bigquerymigration.translation.translate

Performance permissions

cloudnotifications.activities.list

monitoring.notificationChannelDescriptors.list

monitoring.alertPolicies.get

monitoring.notificationChannels.get

monitoring.alertPolicies.list

monitoring.notificationChannels.list

monitoring.dashboards.get

monitoring.publicWidgets.get

monitoring.dashboards.list

monitoring.publicWidgets.list

monitoring.groups.get

monitoring.services.get

monitoring.groups.list

monitoring.services.list

monitoring.metricDescriptors.create

monitoring.slos.get

monitoring.metricDescriptors.get

monitoring.slos.list

monitoring.metricDescriptors.list

monitoring.timeSeries.create

monitoring.monitoredResourceDescriptors.get

monitoring.timeSeries.list

monitoring.monitoredResourceDescriptors.list

monitoring.uptimeCheckConfigs.get

monitoring.notificationChannelDescriptors.get

monitoring.uptimeCheckConfigs.list

API permissions

opsconfigmonitoring.resourceMetadata.list

servicemanagement.services.get

resourcemanager.projects.get

servicemanagement.services.quota

servicemanagement.services.bind

servicemanagement.services.report

servicemanagement.services.check

stackdriver.projects.get

Related articles

Comments

0 comments

Article is closed for comments.