There is much interest from our customers to be able to create Replicate tasks using scripts and move out of the GUI.
The biggest pain point customers have reported is the manual effort in creating the tasks and selecting tables needed, using the UI.
One of our customers is dealing with about 900+ tables and 140+ tasks and they are very interested in such a feature.
A possible solution which would work is explained below.
1. A configuration file is created in a specified format. This file has the information needed to create a task i.e, task name, table name, endpoint names
2. Use a script / API to create the tasks in replicate using the above configuration file as input.
It would be a great feature for supporting our customers in automation and faster implementation. Requesting your review.
I think this is a great idea. We have 80 tasks replicating about 4000 tables and adding more every month. As we grow we need as much automation as possible. Perhaps we can define specific design patterns and use this to ensure consistency as opposed to manually creating via UI.
Hi! This can fits our use-case too, we will manage hundreds of tables too. By now we are planning to use the import function, importing a well-crafted JSON describer of the tables to be replicated, but something more API-oriented would be great.
We are also interested in new APIs to create and modify tasks and endpoints. Task creation/editing APIs will enable software/data engineers to use their native development tools to create more integrated data pipelines (including Replicate) with fewer design-time tools. In turn they will be able to build solutions more quickly and consistently when programmatic interfaces are fully available. As previously suggested, this becomes particularly evident when building for a large number of tasks and/or tasks having a large number of tables.
We would also be interested in such a feature. Currently we have 300+ tables, but we still would like to add them by a script instead of using the GUI. And the amount of tables will also grow for us later.
Since this request was created in 2020, is there any update on it?
I have not heard any update. We are now up to 1,200 tasks and replicating thousands of tables across over 40 servers (20 prod). I believe this feature would help our CI/CD process.
