Automated download of pictures

[Marc Venken]

Hello,

I need to download pic's from a site. I don't have access to ftp, so I will loop all the pics into a browsers and need to hit CTRL-S + ENTER to download the pic.
since it will be lots of img. the window needs to close afther download, and the proces should go further.

In this sample, the loop will show the images, but I can't get the download the work automatic.

I don't know how I can execute a keyboard buffer when it is in the window of the picture.

Any Idea ?

Code: Select all  Expand view  RUN

#include "fivewin.ch"
#include "Hbgtinfo.ch"

function Main()

   local oWnd, oImage, oBar
   local aPics := { "","","" }

   aPics[1] = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Taj_Mahal_in_March_2004.jpg"
   aPics[2] = "http://cdn.history.com/sites/2/2015/04/hith-eiffel-tower-iStock_000016468972Large.jpg"
   aPics[3] = "https://cdn.sportdirect.com/resizer/500x500/108104-8200-01.jpg"

   DEFINE WINDOW oWnd TITLE "FWH : XIMAGE"

   website (aPics)

   ACTIVATE WINDOW oWnd CENTERED

return nil

function website(aPics)

   //DEFINE CLIPBOARD oWinfax
   //oWinfax:SetText( "Test" )
   //oWinfax:SetText( VK_CONTROL+ASC("S")+VK_RETURN )  // CTRL + S  and   RETURN
   //oWinfax:SetText( VK_CONTROL+VK_RETURN )  // CTRL + S  and   RETURN

   //HB_GTINFO( HB_GTI_CLIPBOARDDATA, "Hello!" )

   HB_GTINFO( HB_GTI_CLIPBOARDDATA, VK_CONTROL+ASC('S')+VK_RETURN )


   for i = 1 to len(aPics)

     cLink = "c:\program files\google\chrome\application\chrome.exe "+aPics[I]

    //WaitRun( cLink ) // Program stops until end

    WaitRun( cLink, .T. ) // Program stops until end

    //WaitRun( cLink, 0 )


    //__Keyboard(VK_CONTROL+ASC('S')+VK_RETURN)
    //__Keyboard(chr(VK_CONTROL)+"S"+chr(VK_RETURN))
    //__keyboard()

    //__Keyboard(chr(VK_CONTROL)+"V")


    //ShellExecute( 0, "open", oWinfax )
    //ShellExecute( 0, "open", HB_GTINFO( HB_GTI_CLIPBOARDDATA, VK_CONTROL+ASC('S')+VK_RETURN )  )

    //WaitRun( cLink, "READING FORUM PAGE", { || downloadpic() } )
    //WaitRun( cLink, "READING FORUM PAGE", { || downloadpic() } )

    //msgrun ( cLink )
    //MsgRun( cLink, "READING FORUM PAGE", { || downloadpic() } )

    //syswait(5)


   next
return NIL


   
 

[Enrico Maria Giordano]

This is what I'm using to download a file from a http url:

Code: Select all  Expand view  RUN

#command IF <condition> THEN <*statements*> => if <condition> ; <statements> ; end


FUNCTION GETURLTOFILE( cUrl, cFile, lAuth )

    LOCAL lOk := .F.

    LOCAL oCli

    DEFAULT lAuth := .F.

    TRY
        oCli = TIPClientHttp():New( cUrl )

        IF lAuth THEN oCli:UseBasicAuth()

        IF EMPTY( oCli ) THEN BREAK

        oCli:nConnTimeout = -1

        IF !oCli:Open() THEN BREAK

        DEFAULT cFile := oCli:oUrl:cFile

        IF !oCli:ReadToFile( cFile ) THEN BREAK

        lOk = !EMPTY( oCli:cReply ) .AND. "OK" $ UPPER( oCli:cReply )

        IF !lOk THEN FERASE( cFile )
    CATCH
    END

    IF !EMPTY( oCli ) THEN oCli:Close()

    RETURN lOk

EMG

[nageswaragunupudi]

The above posting is a great tutorial for me and it is a good learning for me.

At the same time, for the sake of interest, I would like to present some simple variations that are possible with the functionality that is already available in FWH.

Our starting point is this:

Code: Select all  Expand view  RUN

aPics := {  "https://upload.wikimedia.org/wikipedia/commons/c/c8/Taj_Mahal_in_March_2004.jpg", ;
            "http://cdn.history.com/sites/2/2015/04/hith-eiffel-tower-iStock_000016468972Large.jpg", ;
            "https://cdn.sportdirect.com/resizer/500x500/108104-8200-01.jpg" }

 

If our purpose is only to view them and all of them, here is a single liner:

Code: Select all  Expand view  RUN

XBROWSER aPics TITLE "WEB IMAGES" SETUP ( oBrw:nRowHeight := 200, oBrw:aCols[ 1 ]:nWidth := 300, ;
   oBrw:aCols[ 1 ]:cDataType := 'F', oBrw:aCols[ 1 ]:nDataBmpAlign := AL_CENTER )
 

This is not what exactly Mr Venken wanted. He wants to save these images on local hard disk.
Again a single line of code for reading the images from the web and saving them locally:

Code: Select all  Expand view  RUN

AEval( aPics, { |c,i,x| x := AfterAtNum( "/", c ), MEMOWRIT( x, WebPageContents( c ) ), aPics[ i ] := x } )
 

Now all the three images from the web are saved to our hard-disk with the same names.
We like to verify if the images are really downloaded.

Code: Select all  Expand view  RUN

XBROWSER aPics TITLE "DOWNLOADED IMAGES" COLUMNS 1,1 SETUP ( oBrw:cHeaders := { "FileName", "Image" }, ;
   oBrw:nRowHeight := 200, oBrw:aCols[ 2 ]:nWidth := 300, ;
   oBrw:aCols[ 2 ]:cDataType := 'F', oBrw:aCols[ 2 ]:nDataBmpAlign := AL_CENTER )
 

I hope this simplifies your work.

[Marc Venken]

Waawh. Again a very nice and quick solution.

I see also the very interesting oneliners of Xbrowser

I didn't find a topic where this onliner (FULL SETUP) is talked about.

I only use xbrowser("datafile") // always dbf files .. for quick browse and export to exel

or else I create a full

@ 330,10 XBROWSE oBrw SIZE 1300,250 .....


Thanks !!

Rick started a topic that FWH is more than Xbase. I have to agree.

Could this also be done in other languages with so less code ? I Don't think so.
Would I have a working solution in so less time ? I don't think so.

This forum is very usefull for people like me.

The strength of FWH could be in hands of the smaller companies, with just a few programmers. The point would be to introduce
FWH to the new generation....

Thanks again to all.

[Marc Venken]

This is what I'm using to download a file from a http url:

Code: Select all  Expand view  RUN

#command IF <condition> THEN <*statements*> => if <condition> ; <statements> ; end


FUNCTION GETURLTOFILE( cUrl, cFile, lAuth )

    LOCAL lOk := .F.

    LOCAL oCli

    DEFAULT lAuth := .F.

    TRY
        oCli = TIPClientHttp():New( cUrl )

        IF lAuth THEN oCli:UseBasicAuth()

        IF EMPTY( oCli ) THEN BREAK

        oCli:nConnTimeout = -1

        IF !oCli:Open() THEN BREAK

        DEFAULT cFile := oCli:oUrl:cFile

        IF !oCli:ReadToFile( cFile ) THEN BREAK

        lOk = !EMPTY( oCli:cReply ) .AND. "OK" $ UPPER( oCli:cReply )

        IF !lOk THEN FERASE( cFile )
    CATCH
    END

    IF !EMPTY( oCli ) THEN oCli:Close()

    RETURN lOk

EMG

Enrico,

I'm interested in this code, because it's different then FW code as I know.
Am I wrong? A quick search in the forum is giving zero result on this function.
I see in more of your postings this kind of programming.
Where is it from ?

Marc

[Enrico Maria Giordano]

It uses TIPClientHttp() that is an xHarbour class.

EMG

[Marc Venken]

AEval( aPics, { |c,i,x| x := AfterAtNum( "/", c ), MEMOWRIT( x, WebPageContents( c ) ), aPics[ i ] := x } )

I don't see the webpagecontents function.

I think that I'm missing several links to the functions library. Do they exist ?

I want to learn what is going on in this oneliner

AEval and DBEval are often used (not me, because i don't know it yet) but i'm sure that 30% of my code is replacebel with these functions.

This kind of code I use very often and is typical code in my case :

Code: Select all  Expand view  RUN

function updateNofoto()
   use master NEW
   select master
   SET ORDER TO TAG code
   use nofoto NEW
   nofoto->(dbgotop())
   do while !nofoto->(eof())
      if master->(dbseek(nofoto->id))
         nofoto->memotxt = master->memotxt
         nofoto->samenstel = master->kwaliteit
         nofoto->gewicht = master->grammage
         nofoto->geslacht = master->geslacht
         nofoto->mouwen = master->mouwen
      endif
      nofoto->(dbskip())
   enddo
   close all
return NIL
 

Is this code a good candidate for a eval function ? or is this code OK.

[Enrico Maria Giordano]

Programming is (among other things) a trade off between compactness and readability. Don't be tempted by excessive compactness and less readabilty. Oneliner instruction such that is a double cut knife. Use it with care.

EMG

[Marc Venken]

Strange,

The samples works ok (xbrowse and download) , but with the dbf data the Xbrowse only works, but the download part gives this error

There are 1600 records, all processed ok in the browse, none in the download.

As far as I can Google, it has something todo with the ole Open (maybe to many open connections? )
Maybe the program is to quick in order to proces the download rec by Rec ?
Did'nt find a solution.


(DOS Error -2147352571) WINOLE/1007 Argument error: OPEN

The code

Code: Select all  Expand view  RUN

function READHTTP()

   local oWnd, oImage, oBar
/*
      local aPics := {  "https://upload.wikimedia.org/wikipedia/commons/c/c8/Taj_Mahal_in_March_2004.jpg", ;
            "http://cdn.history.com/sites/2/2015/04/hith-eiffel-tower-iStock_000016468972Large.jpg", ;
            "https://cdn.sportdirect.com/resizer/500x500/108104-8200-01.jpg" }
 */


   use c:\programmas\fotoselect\lever\hummel NEW ALIAS bron
   bron->(DbGoTop())
   select bron
   index on bron->foto tag foto unique

   aPics:= FW_DbfToArray("foto")

   xbrowser(aPics)

   XBROWSER aPics TITLE "WEB IMAGES" SETUP ( oBrw:nRowHeight := 200, oBrw:aCols[ 1 ]:nWidth := 300, ;
     oBrw:aCols[ 1 ]:cDataType := 'F', oBrw:aCols[ 1 ]:nDataBmpAlign := AL_CENTER )

   AEval( aPics, { |c,i,x| x := AfterAtNum( "/", c ), MEMOWRIT( x, WebPageContents( c ) ), aPics[ i ] := x } )


return nil

 

[nageswaragunupudi]

(DOS Error -2147352571) WINOLE/1007 Argument error: OPEN

This means you have some invalid URLs in the list. You should make sure that these are all valid URLs.
We don't get a runtime error with XBrowse, because Xbrowse uses TRY/CATCH construct.

When you have the URL names in a DBF, it is not again necessary to read the names into an array to download.
You can as well do:

Code: Select all  Expand view  RUN

function downloadpics()

field FOTO
local aPics := {}
local cImage, cFile, cURL


USE C:\PROGRAMMAS\FOTOSELECT\LEVER\HUMMEL NEW ALIAS BRON
INDEX ON FOTO TAG FOTO UNIQUE
GO TOP

do while !Eof()
   cURL  := AllTrim( FOTO )
   if Lower( Left( cURL, 4 ) ) == "http"
      cFile    := AfterAtNum( "/", cURL )
      if !Empty( cFile )
         TRY
            cImage   := WebPageContents( cURL )
            if IsBinaryData( cImage )
               MEMOWRIT( cFile, cImage )
               AAdd( aPics, cFile )
            endif
         CATCH
         END
      endif
   endif
   SKIP
enddo

XBROWSER aPics TITLE "DOWNLOADED IMAGES" COLUMNS 1,1 SETUP ( oBrw:cHeaders := { "FileName", "Image" }, ;
   oBrw:nRowHeight := 200, oBrw:aCols[ 2 ]:nWidth := 300, ;
   oBrw:aCols[ 2 ]:cDataType := 'F', oBrw:aCols[ 2 ]:nDataBmpAlign := AL_CENTER )

return nil
 

[nageswaragunupudi]

AEval and DBEval are often used (not me, because i don't know it yet) but i'm sure that 30% of my code is replacebel with these functions.

There is no specific advantage of using AEVAL instead of for..next loop or DBEVAL instead of do while !eof() loop.
Experts like Mr Antonio and Mr EMG know the internals of (x)Harbour lot better than us. Mr Antonio once told me that for..next loop is more efficient than AEval().

Like Mr EMG said, readability of code is more important. Please write your code in such a way that you can understand it even after ten years. It is also desirable that even other programmers can read and understand your code.

I see also the very interesting oneliners of Xbrowser

We purposefully made XBROWSER so powerful. We did not intend it to be included in release versions, but to serve as a great tool for developers. While XBROWSER automatically builds all standard functionality of xbrowse, we can add additional functionality with SETUP clause.

With XBROWSER you can view "anything".
XBROWSER <cimagefile>
XBROWSER oDlg // works as an object inspector
etc.

I don't see the webpagecontents function.

\fwh\source\function\imgtxtio.prg

[nageswaragunupudi]

Mr EMG

I request you to help.
I tried this program using the function you have posted

Code: Select all  Expand view  RUN

function testweb4()

   local cUrl  := "https://upload.wikimedia.org/wikipedia/commons/c/c8/Taj_Mahal_in_March_2004.jpg"
   local cFile := "test.jpg"

   FERASE( cFile )
   GETURLTOFILE( cURL, cFile )
   ? File( cFile ) // --> .F.

return nil
 

The file is not created i.e., not downloaded.
Can you please guide me what I am doing wrong?

[Enrico Maria Giordano]

Mr. Rao,

probably the problem is https protocol. You have to use xHarbour tipssl.lib. Sorry, I'm not familiar with it (never used). You'll need of openssl DLLs too.

EMG

[Enrico Maria Giordano]

Try this, it works fine here:

Code: Select all  Expand view  RUN

#include "Fivewin.ch"


FUNCTION MAIN()

    ? URLDOWNLOADTOFILE( 0, "https://upload.wikimedia.org/wikipedia/commons/c/c8/Taj_Mahal_in_March_2004.jpg", "test.jpg" )

    RETURN NIL


DLL FUNCTION URLDOWNLOADTOFILE( pCaller AS LONG, cUrl AS LPSTR, cFileName AS LPSTR, nReserved AS DWORD, nFnCB AS LONG ) AS LONG;
    PASCAL FROM "URLDownloadToFileA" LIB "urlmon.dll"

EMG

[nageswaragunupudi]

Thank you.
GETURLTOFILE() is working with http: but not https:
URLDOWNLOADTOFILE() is working well with all and is also much faster than GETURLTOFILE().

The actual reason for my testing is to find a faster alternative to the existing function WebPageContents(). Our requirement is to directly read the contents of the URL into memory.