LST storage paths

[Kerri]

Hello! 

We are in the process of reconfiguring tables from our existing tasks into tiers based on business needs. With the addition of new LSS and Snowflake tasks for Tier 2, we now have two LST connections using the same log stream storage path.
The Snowflake target is in a different schema so no issue there, but the source is still the same thus the reuse of the same log stream path. 

Example:
LST task for Tier 1 using storage path: /data/streams/LST_PORTFOLIO_DATABASE

LST task for Tier 2 also using storage path: /data/streams/LST_PORTFOLIO_DATABASE

Is this going to cause us any issues?

Thanks so much in advance,
Kerri

[narendersarva]

Hi @Kerri 

log stream storage paths must be unique per task

In Replicate, each log stream staging task writes metadata and change data to the log stream storage directory. Reusing the same storage path for multiple tasks that are independently configured and running can lead to file conflicts, metadata overwrites, and also data corruption in some cases.

Hope this helps.

Thanks
Naren

[Kerri]

Hello Naren,

Yes this does help. Does this mean we simply need the MYSQL administrator to provide us with a new storage path or can I just simply add Tier2 to my existing path and have it work? 

Thanks,
Kerri

[DesmondWOO]

Hi @Kerri ,

I'm not sure I fully understand your situation. If your source is the same, you can add the required tables in the staging task and then create a Tier 2 task to retrieve changes. For example:

LSS (Table1, Table2)
|--> Tier 1 (Table1)
|--> Tier 2 (Table2)

Regards,
Desmond

[Kerri]

Hello Desmond, 

The second question is specific to the LST endpoint connection and the and its storage path, not the LSS task. 
More specifically, Is this path generated at runtime, meaning I can simply add the Tier2 qualifier to the storage path to establish distinction between the two? Currently, the LST endpoint connection storage path is the same for both Tiers. 

Thanks,
Kerri

[DesmondWOO · Accepted solution]

Hi @Kerri ,

The structure of the storage path is as follows:

\<Staging Task Name>\LOG_STREAM\audit_service\<timestamp>\

So, as Naren mentioned previously, each LogStream Staging task must have its own storage path. If you add a new LogStream staging task, the storage path will be different.

If you want to add a new replication task to retrieve changes from the same source, you can create a new replication task and select the source endpoint that you are using in Tier 1.

If my response does not fully address your question, could you share a bit more detail or an example of what you're hoping to achieve?

Regards,
Desmond