https://community.qlik.com/t5/Talend-Studio/Using-Talend-to-crawl-a-website/m-p/2354084#M120128 I would use a regulary expression and filter the content of the first page for links. After collection all links you can iterate over them and so on. Wed, 03 Apr 2013 21:17:31 GMT Anonymous 2013-04-03T21:17:31Z https://community.qlik.com/t5/Talend-Studio/Using-Talend-to-crawl-a-website/m-p/2354083#M120127 Hi,<BR />My use case is like this:- I want to crawl a website say talend.com, and extract all the information on the website into Hadoop. After that I want to search for specific strings in the data, and use it populate Hive and create a report.<BR />I want to use Talend to populate data from a website and store it in Hadoop. I watched this video<BR /><A href="http://www.talend.com/resources/webinars/watch/215#validatewebinar" target="_blank" rel="nofollow noopener noreferrer">http://www.talend.com/resources/webinars/watch/215#validatewebinar</A><BR />Based on this when i use a t_FileFetch or t_HttpRequest and connect to a URI say "<A href="http://talend.com" target="_blank" rel="nofollow noopener noreferrer">http://talend.com</A>" - I only get the first page , which I can save in a file. How can I iterate over the entire contents of a directory- I need to know each distinct URL like talend.com/products etc. How can I iteratively fetch all files under a master URL. Wed, 03 Apr 2013 11:27:50 GMT https://community.qlik.com/t5/Talend-Studio/Using-Talend-to-crawl-a-website/m-p/2354083#M120127 _AnonymousUser 2013-04-03T11:27:50Z https://community.qlik.com/t5/Talend-Studio/Using-Talend-to-crawl-a-website/m-p/2354084#M120128 I would use a regulary expression and filter the content of the first page for links. After collection all links you can iterate over them and so on. Wed, 03 Apr 2013 21:17:31 GMT https://community.qlik.com/t5/Talend-Studio/Using-Talend-to-crawl-a-website/m-p/2354084#M120128 Anonymous 2013-04-03T21:17:31Z