AWS Glue Connection for BigQuery gives "SparkProperties is missing but it is required" and "secretId is not defined in the schema" when using Go SDK

2 weeks ago 22
ARTICLE AD BOX

I'm trying to programmatically create a native Google BigQuery connection in AWS Glue using the AWS SDK for Go v2 (github.com/aws/aws-sdk-go-v2/service/glue).

According to the AWS docs (Glue 4.0+ supports native BigQuery connections since late 2023), the connection should be minimal: set ConnectionType to BIGQUERY, provide the secret name via SECRET_ID in Properties, and not need SparkProperties (that's for custom/marketplace connectors).

However, every attempt results in InvalidInputException with conflicting schema validation errors:

InvalidInputException: secretId: is not defined in the schema and the schema does not allow additional properties, SparkProperties: is missing but it is required

I am trying out this code snippet

import ( "context" "fmt" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/glue" gluetypes "github.com/aws/aws-sdk-go-v2/service/glue/types" ) func main() { cfg, _ := config.LoadDefaultConfig(context.TODO()) client := glue.NewFromConfig(cfg) secretName := "my-bq-secret" // secret name input := &glue.CreateConnectionInput{ ConnectionInput: &gluetypes.ConnectionInput{ Name: aws.String("test-bq-native-conn"), Description: aws.String("Test native BigQuery connection"), ConnectionType: gluetypes.ConnectionTypeBigquery, ConnectionProperties: map[string]string{ "SECRET_ID": secretName, }, // SparkProperties omitted // PhysicalConnectionRequirements omitted for minimal test }, } _, err := client.CreateConnection(context.TODO(), input) if err != nil { fmt.Printf("Error: %v\n", err) } else { fmt.Println("Success") } }
Read Entire Article