BigQuery

This page will walk through the setup of BigQuery in K using the direct connect method.

Integration details

Step 1) Setup a Google Cloud Service Account

• Create a Service Account by going to the Google Cloud Admin or clicking on this link

  • Give the Service Account a name (e.g. KADA BQ Integration)

  • Select the Projects that include the BigQuery instance(s) that you want to catalog

  • Click Save
    
    

• Create a Service Token

  • Click on the Service Account

    Open

    

     

     

  • Select the Keys tab. Click on Create new key

    Open

    

     

  • Select the JSON option. After clicking ‘CREATE’, the JSON file will automatically download to your device. Provide this to the user(s) that will complete the next steps

    Open

    

     

• Add grants on the Service Account by going to IAM page or clicking on this link

  • Click on ADD

    Open

    

     

     

  • Add the Service Account to the ‘New principals’ field.

    Open

    

  • Grant the following roles this principal as shown in the following screenshot.

    • BigQuery Job User

    • BigQuery Metadata Viewer

    • BigQuery Read Session User

    • BigQuery Resource Viewer

  • Open

    

    Click SAVE

Step 2) Connecting K to BigQuery

• Select Platform Settings in the side bar

• In the pop-out side panel, under Integrations click on Sources

• Click Add Source and select BigQuery

  Open

  

• Select Direct Connect

• Fill in the Source Settings and click Save & Next

  • Name: The name you wish to give your BigQuery Service in K

  • Host: Add your BigQuery Host Name (e.g. cloud.google.com)

  • Region: Select the region your Service is located in (check with your admin if you are unsure)

• Add the Connection details and click Save & Next when connection is successful

  • Credentials: Copy the content of the Credentials.json created in Step 1

• Test your connection and click Save

• Select the Databases you wish to load into K and click Finish Setup

  • All databases will be listed. If you have a lot of databases this may take a few seconds to load

• Return to the Sources page and locate the new BigQuery source that you loaded

• Click on the clock icon to select Edit Schedule and set your preferred schedule for the BigQuery load

  Open

  

Step 3) Manually run an ad hoc load to test BigQuery setup

• Next to your new Source, click on the Run manual load icon

  Open

  

   

• Confirm how your want the source to be loaded

  Open

  

• After the source load is triggered, a pop up bar will appear taking you to the Monitor tab in the Batch Manager page. This is the usual page you visit to view the progress of source loads

  Open