Page 2 of 2

Re: Automated download of pictures

PostPosted: Thu Mar 09, 2017 10:57 am
by Enrico Maria Giordano
nageswaragunupudi wrote:Thank you.
GETURLTOFILE() is working with http: but not https:


This is not true. It would work if tipssl were linked (and opensll DLL's were available).

nageswaragunupudi wrote:URLDOWNLOADTOFILE() is working well with all and is also much faster than GETURLTOFILE().


I was not aware the URLDOWNLOADTOFILE() is much faster than GETURLTOFILE(). Are you made any real test? Can you share the timing result?

nageswaragunupudi wrote:The actual reason for my testing is to find a faster alternative to the existing function WebPageContents(). Our requirement is to directly read the contents of the URL into memory.


I use this (but it needs opensll as well):

Code: Select all  Expand view
FUNCTION GETURL( cUrl )

    LOCAL cHtm := ""

    LOCAL oCli

    TRY
        oCli = TIPClientHttp():New( cUrl )

        IF EMPTY( oCli ) THEN BREAK

        oCli:nConnTimeout = -1

        IF !oCli:Open() THEN BREAK

        cHtm = oCli:ReadAll()

        IF EMPTY( oCli:cReply ) .OR. !( "OK" $ UPPER( oCli:cReply ) )
            cHtm = ""
        ENDIF
    CATCH
    END

    IF !EMPTY( oCli ) THEN oCli:Close()

    RETURN cHtm


EMG

Re: Automated download of pictures

PostPosted: Thu Mar 09, 2017 1:25 pm
by nageswaragunupudi
I was not aware the URLDOWNLOADTOFILE() is much faster than GETURLTOFILE(). Are you made any real test? Can you share the timing result?


I used the Tajmahal.jpg ( http not https )

GETURLTOFILE() --> 3.1 to 3.3 secs
Both
URLDOWNLOADTOFILE() and
MEMOWRIT( cFile, webpagecontents( curl ) ) --> 0.06 to 0.09 secs

Code: Select all  Expand view
function testweb4()

   local cURL := "http://cdn.history.com/sites/2/2015/04/hith-eiffel-tower-iStock_000016468972Large.jpg"
   local cFile := "test.jpg"
   local nSecs

   FERASE( cFile )
   URLDOWNLOADTOFILE( 0, cURL, cFile ) // Ignore time taken for the first time

   nSecs := SECONDS()
   GETURLTOFILE( cURL, cFile )
   ? SECONDS() - nSecs, File( cFile ) // --> 3.23

   nSecs := SECONDS()
   URLDOWNLOADTOFILE( 0, cURL, cFile )
   ? SECONDS() - nSecs, File( cFile ) // --> 0.07

   nSecs := SECONDS()
   MEMOWRIT( cFile, webpagecontents( curl ) )
   ? SECONDS() - nSecs, File( cFile )  // --> 0.07

   if File( cFile )
      XIMAGE( cFile )
   endif

return nil

Re: Automated download of pictures

PostPosted: Thu Mar 09, 2017 2:46 pm
by Enrico Maria Giordano
That's only a cache effect. Try calling this before download:

Code: Select all  Expand view
DLL FUNCTION DELETEURLCACHEENTRY( cUrl AS LPSTR ) AS BOOL;
    PASCAL FROM "DeleteUrlCacheEntryA" LIB "wininet.dll"


EMG

Re: Automated download of pictures

PostPosted: Fri Mar 10, 2017 5:32 pm
by nageswaragunupudi
Mr EMG

You are right.
Without the benefit of cache, GETURLTOFILE() is faster than the other two.
Another observation is that GETURLTOFILE() does not take advantage of cache and other two take advantage of cache if available

Re: Automated download of pictures

PostPosted: Fri Mar 10, 2017 6:09 pm
by Enrico Maria Giordano
nageswaragunupudi wrote:Mr EMG

You are right.
Without the benefit of cache, GETURLTOFILE() is faster than the other two.
Another observation is that GETURLTOFILE() does not take advantage of cache and other two take advantage of cache if available


That's true. But use of cache means that the file is not really downloaded, even if it's changed. So use it with caution.

EMG

Re: Automated download of pictures

PostPosted: Sat Mar 11, 2017 3:27 am
by nageswaragunupudi
That's true. But use of cache means that the file is not really downloaded, even if it's changed. So use it with caution.

Yes. Thanks