I do a lot of ETL/ELT. We have been exploring options for loading into BigQuery, and I'm curious to find what others are doing.
I see multiple products from Google, but not sure exactly where I should reach. Looks like there's overlap there.
Right now most of my data is batched loaded at a certain interval. Mostly hourly or daily.
Usually comes in via CSV's that we have to preprocess and MERGE, but other times we have to hit API's to extract data.
I have explored other things like Spark via Databricks and hosted like Rivery and Fivetran, but just trying to figure out what else can be done with native Google products.
Currently, most of my data is batch-loaded at regular intervals. Mostly hourly or daily. Usually comes in via CSVs that we have to preprocess and MERGE, but other times we have to hit API's to extract data. I have explored other options, such as Spark via Databricks and hosted services like Rivery and Fivetran, but I'm trying to figure out what else can be done with native Google products. So, how do you load? For anyone comparing native options (Storage Write API, Dataflow templates, BigQuery Transfer Service, Dataform, Cloud Composer, bq load from GCS), here’s a concise BigQuery load patterns guide that maps use-cases to the right tool.
