How to Load data from web when data table is at multiple pages

[shekhar_analyti]

Hi All ,

How to load Election Result data from a website when the data table is scattered at multiple pages .

Website : http://eciresults.nic.in/StatewiseS24.htm

Data pattern

My requirement is that my qvw app should load data one by one from the above web data table till it reaches

end . Here its 41th page .

Thanks

[hector_munoz · Accepted solution]

Hi Shekhar,

In that web you have a URL parameter that identifies which partition of table shows:

1 -> http://eciresults.nic.in/StatewiseS24.htm

2 -> http://eciresults.nic.in/StatewiseS241.htm

3 -> http://eciresults.nic.in/StatewiseS242.htm

4 -> http://eciresults.nic.in/StatewiseS243.htm

5 -> http://eciresults.nic.in/StatewiseS244.htm

....

40 -> http://eciresults.nic.in/StatewiseS2439.htm

41 -> http://eciresults.nic.in/StatewiseS2440.htm

So, we could consider a rule like:

IF page = 1 THEN 24 ELSE (240 + page - 1)

, and create a for loop like:

FOR vsCounter = 1 TO 41

  LET vsPage = If($(vsCounter) = 1, 24, 240 + $(vsCounter) - 1);

  LET vsURL = 'http://eciresults.nic.in/StatewiseS' & '$(vsPage)' & '.htm';

  LOAD @1 AS Constituency,

  @2 AS [Const. No.],

  @3 AS [Leading Candidate],

  @4 AS [Leading Party],

  @5 AS [Trailing Candidate],

  @6 AS [Trailing Party],

  @7 AS Margin,

  @8 AS Status

  FROM $(vsURL) (html, codepage is 1252, no labels, table is @7)

  WHERE Recno() > 4;

NEXT

I think this should serve to you...

Regards,
H

[shekhar_analyti]

Hi Hector ,

Thanks for detailed information .

It will be great if you can help me with the approach of one more scenarios related with above :

1) If the count of number pages changes daily . For example , current day its 

41 -> http://eciresults.nic.in/StatewiseS2440.htm  , but next day it increases by two to become 43 (http://eciresults.nic.in/StatewiseS2442.htm) , and next day increase by 5 to become 48 (http://eciresults.nic.in/StatewiseS2447.htm )

Thanks & Regards

Shekhar

[hector_munoz]

Hi Shekthar,

You can use a trick that consists in read e.g. 100 pages and control reload breaks with ErrorMode variable:

FOR vsCounter = 1 TO 100

  LET vsPage = If($(vsCounter) = 1, 24, 240 + $(vsCounter) - 1);

  LET vsURL = 'http://eciresults.nic.in/StatewiseS' & '$(vsPage)' & '.htm';

  SET ErrorMode = 0;

  LOAD @1 AS Constituency,

  @2 AS [Const. No.],

  @3 AS [Leading Candidate],

  @4 AS [Leading Party],

  @5 AS [Trailing Candidate],

  @6 AS [Trailing Party],

  @7 AS Margin,

  @8 AS Status

  FROM $(vsURL) (html, codepage is 1252, no labels, table is @7)

  WHERE Recno() > 4;

  SET ErrorMode = 1;

NEXT

In the above example we try to read 100 pages. We disable reload break with ErrorMode = 0 begore we try to read the page of the iteration. After reading or trying to read it, web enable reload break again (ErrorMode = 1).

Can you try it?

Regards,
H