topic How do I create a directory in HDFS with Talend? in Talend Studio

How do I create a directory in HDFS with Talend?

Anonymous — Sat, 16 Nov 2024 12:12:51 GMT

Noob question, so please don't laugh too hard.... :-)
I am creating a date/time based directory structure in HDFS via file insertion from Talend:
YYYY/MM/DD/HH/
So, when I insert files into HDFS, the necessary file structure is automatically created:
inserting file "YYYY/MM/DD/11/data.txt" creates directory "YYYY/MM/DD/11/"
EXCEPT when I happen to be inserting the file at the start of the hour (i.e. 11:00 on the dot).
I'm running into an issue where if I try to create the file at the start of the hour Talend names the file "YYYY/MM/DD/11" (where "11" is the file "data.txt" renamed for some reason) and not "YYYY/MM/DD/11/data.txt". Thus, all my following insert attempts for that hour fail because I am trying to write the files to a directory called "YYYY/MM/DD/11/".
The work around I found was to manually create the directory structure before Talend attempts to insert files.
How do I create a directory in HDFS with Talend so I can remove the manual step?

Re: How do I create a directory in HDFS with Talend?

Anonymous — Mon, 27 Aug 2012 03:31:36 GMT

>> The work around I found was to manually create the directory structure before Talend attempts to insert files.
For now, I have added the create the directory call (hadoop fs -mkdir ...) as a system call component in Talend,
I would like to know if there is a more elegant solution.....

Re: How do I create a directory in HDFS with Talend?

Anonymous — Mon, 27 Aug 2012 23:40:25 GMT

In the component properties, HDFS Directory, can't you just set up a context variable called context.datedDirectory and each time the job runs it get a different date from a Talend.GetDate function or a parsing of the files you are putting into is and talend will create the directory in HDFS based on this context variable's value each time the job runs?