idea Create tasks through API / Scripts in Suggest an Idea

Create tasks through API / Scripts

jjames — Mon, 15 Jun 2020 05:21:36 GMT

There is much interest from our customers to be able to create Replicate tasks using scripts and move out of the GUI.

The biggest pain point customers have reported is the manual effort in creating the tasks and selecting tables needed, using the UI.

One of our customers is dealing with about 900+ tables and 140+ tasks and they are very interested in such a feature.

A possible solution which would work is explained below.

1. A configuration file is created in a specified format. This file has the information needed to create a task i.e, task name, table name, endpoint names

2. Use a script / API to create the tasks in replicate using the above configuration file as input.

It would be a great feature for supporting our customers in automation and faster implementation. Requesting your review.

Re: Create tasks through API / Scripts

dtylerlivingston — Thu, 18 Jun 2020 20:32:07 GMT

I think this is a great idea. We have 80 tasks replicating about 4000 tables and adding more every month. As we grow we need as much automation as possible. Perhaps we can define specific design patterns and use this to ensure consistency as opposed to manually creating via UI.

Re: Create tasks through API / Scripts - Status changed to: Open - On Roadmap

Ola_Mayer — Wed, 24 Jun 2020 14:24:44 GMT

James,

Thank you for your feedback. This feature is already on our roadmap.

Can you please share with us (offline) the name of your customer?

Thank you ,

Ola Mayer

Re: Create tasks through API / Scripts

jjames — Thu, 25 Jun 2020 23:07:52 GMT

That is great news. Thanks Ola. I will DM the customer name.

Thanks,

Jijo James

Re: Create tasks through API / Scripts

mic_lau — Mon, 23 Nov 2020 15:47:30 GMT

Hi!
This can fits our use-case too, we will manage hundreds of tables too.
By now we are planning to use the import function, importing a well-crafted JSON describer of the tables to be replicated, but something more API-oriented would be great.

Re: Create tasks through API / Scripts

parviz_asoodehfard — Fri, 26 Mar 2021 19:22:02 GMT

Hi,

May I ask when we can expect this feature?

Re: Create tasks through API / Scripts

BradA — Thu, 05 Aug 2021 16:56:46 GMT

We are also interested in new APIs to create and modify tasks and endpoints. Task creation/editing APIs will enable software/data engineers to use their native development tools to create more integrated data pipelines (including Replicate) with fewer design-time tools. In turn they will be able to build solutions more quickly and consistently when programmatic interfaces are fully available. As previously suggested, this becomes particularly evident when building for a large number of tasks and/or tasks having a large number of tables.

Re: Create tasks through API / Scripts

AndreasFenske — Tue, 01 Mar 2022 07:55:25 GMT

We would also be interested in such a feature. Currently we have 300+ tables, but we still would like to add them by a script instead of using the GUI. And the amount of tables will also grow for us later.

Since this request was created in 2020, is there any update on it?

Re: Create tasks through API / Scripts

dtylerlivingston — Wed, 02 Mar 2022 00:22:48 GMT

I have not heard any update. We are now up to 1,200 tasks and replicating thousands of tables across over 40 servers (20 prod). I believe this feature would help our CI/CD process.

Re: Create tasks through API / Scripts

Nulee_Massaro — Thu, 26 May 2022 18:20:52 GMT

Just a quick update - this has been delayed but we are picking this up in our planning again. We will update you ASAP.

Re: Create tasks through API / Scripts

KellyIBT — Mon, 01 Aug 2022 06:04:41 GMT

Yes, Looking forward to an update on this please.

Re: Create tasks through API / Scripts

bobvecchione — Fri, 05 Aug 2022 12:23:22 GMT

Hi folks..

We have recently exposed API s in the May release to assist with DTAP such as updating parameters.

Our cloud solution is based on an API first approach and this request will met. I would be happy to provide more details.

While this request certainly makes sense for Enterprise Manager and we have added this to our priority list we don't have a date for delivery.

I will keep this thread updated.

Bob

Re: Create tasks through API / Scripts

dtylerlivingston — Fri, 05 Aug 2022 01:12:44 GMT

This is very intriguing and I look forward to reviewing the capabilities when released. The ability to script both creation and management of tasks will help us support Replicate better.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 02 Aug 2023 15:16:12 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Create tasks through API / Scripts - Status changed to: Closed - Archived

Ideation — Wed, 02 Aug 2023 15:16:14 GMT