Can I send a JSON object to BigQuery without stringifying it?

2 weeks ago 13
ARTICLE AD BOX

I am trying to figure out if there is a way to send a JSON object to a BigQuery table that has a column of type JSON. I know the current practice is to stringify the JSON and send it over which gets parsed and stored as JSON but Im trying to find a way to send an object.

So I know the following works:

type Message struct { stuff int64 `json:"stuff"` stuff2 string `json:"stuff2"` stuff3 string `json:"stuff3"` } stuff3Map := map[string]string{ "status": "ok", "response_time": "5", } stuff3JSON, err := json.Marshal(stuff3Map) if err != nil { log.Fatalf("Failed to marshal stuff: %v", err) } msg := Message{ stuff: 34, stuff2: "test", stuff3: string(stuff3JSON) }

However Id like to know if it would be possible to make something like the following work

type Message struct { stuff int64 `json:"stuff"` stuff2 string `json:"stuff2"` stuff3 map[string]string `json:"stuff3"` // or []byte or string } stuff3Map := map[string]string{ "status": "ok", "response_time": "5", } stuff3JSON, err := json.Marshal(stuff3Map) if err != nil { log.Fatalf("Failed to marshal stuff: %v", err) } msg := Message{ stuff: 34, stuff2: "test", stuff3: stuff3JSON }

I know that BigQuery does support it (as mentioned here https://docs.cloud.google.com/bigquery/docs/schemas#standard_sql_data_types) and also as mentioned here: https://docs.cloud.google.com/bigquery/docs/schemas#creating_a_JSON_schema_file I can do it by defining a JSON schema but the objects I want to send are not of predetermined structure.

From what I can get from the docs, its a client library issue more than a BigQuery issue but still.

Also this https://docs.cloud.google.com/bigquery/docs/nested-repeated#define_nested_and_repeated_columns exists but also needs predetermined fields.

Finally, the bigquery.JSONValue(mapData) type exists but is that what I need and is there no way without it ? Thank you in advance.

Read Entire Article