Seeking tool that download all files from internet directory

Seeking tool that download all files from internet directory

Michael Faerber

Member #4,800

July 2004

Hi! I have found a specific site (http://fun.barnal.de/videos) and want to download all files from this page to view them offline. Question is: how?

I tried it with HTTrack, but I can't resume the download after I have stopped it.

Do you know a tool (must be available for Linux) that could do this for me?

--
"The basic of informatics is Microsoft Office." - An informatics teacher in our school
"Do you know Linux?" "Linux? Isn't that something for visually impaired people?"

gnolam

Member #2,030

March 2002

Wget.

--
Move to the Democratic People's Republic of Vivendi Universal (formerly known as Sweden) - officially democracy- and privacy-free since 2008-06-18!

kentl

Member #2,905

November 2002

If you use FireFox you could try the "Download them all" extension. I'm not sure about the name, it's popular so you'll find it.

BAF

Member #2,981

December 2002

downTHEMall is the name of the extension... at least the one I have.

BAF.zone | SantaHack!

Kitty Cat

Member #2,815

October 2002

man wget said:

o   Retrieve the first two levels of wuarchive.wustl.edu, saving them to /tmp.

           wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/

--
"Do not meddle in the affairs of cats, for they are subtle and will pee on your computer." -- Bruce Graham

ReyBrujo

Moderator

January 2001

wget or httrack

--
RB
光子「あたしただ…奪う側に回ろうと思っただけよ」
Mitsuko's last words, Battle Royale

miran

Member #2,407

June 2002

I used to use httrack. It was teh awesome.

--
sig used to be here

Michael Faerber

Member #4,800

July 2004

Problem with wget is that I cannot resume the download after I have stopped it. HTTrack seemed to offer that option, but it seemed to random - sometimes it worked, sometimes it started redownloading the whole page again.

DownThemAll however seems to work fine! So, if nobody proposes a better program, I will use this.

--
"The basic of informatics is Microsoft Office." - An informatics teacher in our school
"Do you know Linux?" "Linux? Isn't that something for visually impaired people?"

gnolam

Member #2,030

March 2002

-c resumes partially downloaded files. -N makes sure only new files get downloaded. What's the problem?

--
Move to the Democratic People's Republic of Vivendi Universal (formerly known as Sweden) - officially democracy- and privacy-free since 2008-06-18!

Evert

Member #794

November 2000

I second wget.

Mars

Member #971

February 2001

"DownThemAll!" is really quite nice for in-Firefox use.

--
This posting is a natural product. The slight variations in spelling and grammar enhance its individual character and beauty and in no way are to be considered flaws or defects.

Michael Faerber

Member #4,800

July 2004

Hey, gnolam, you really helped me with your "-c" option. I suppose I have to read the man pages more often.

So I'll use wget now! Thanks for your help!

--
"The basic of informatics is Microsoft Office." - An informatics teacher in our school
"Do you know Linux?" "Linux? Isn't that something for visually impaired people?"

jhuuskon

Member #302

April 2000

Now if only someone made a frontend for wget that would make it actually behave like it should. Wget is another fine example of opensource at its prime: Needlessly complicated, poorly documented and it doesn't work like it's supposed to.

You don't deserve my sig.

Evert

Member #794

November 2000

Quote:

Care to elaborate?
I've used it without any problems for weeks before even looking at the manpage (which I needed only when I wanted to make a local copy of a website). The manpage itself is detailed and list all options very clearly.

jhuuskon

Member #302

April 2000

I tried numerous times to downlad an image gallery (a html page that links to the jpegs). However, it only downloads the index page and stops regardless of recursion options specified. Another funky thing, even when i tell wget to retain only donwloaded jpegs, it keeps the index even though i told it to retain only jpegs.

The help file (yes i've tried it in windows) lists all options allright, but the explanations are arbitrary at best and the examples are, while well demonstrating the flexibility of Wget, totally useless from a practical point of view.

You don't deserve my sig.

Kitty Cat

Member #2,815

October 2002

man wget said:

o   You have a file that contains the URLs you want to download?  Use the -i
    switch:

           wget -i <file>

man wget also said:

-F
--force-html
    When input is read from a file, force it to be treated as an HTML file.  This
    enables you to retrieve relative links from existing HTML files on your local
    disk, by adding "<base href="url">" to HTML, or using the --base command-line
    option.

-B URL
--base=URL
    Prepends URL to relative links read from the file specified with the -i option.

If the images are all the same extension and in the same directory on the site:

Quote:

o   You want to download all the GIFs from a directory on an HTTP server.  You
    tried wget http://www.server.com/dir/*.gif, but that didn't work because HTTP
    retrieval does not support globbing.  In that case, use:

            wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

    More verbose, but the effect is the same.  -r l1 means to retrieve recur
    sively, with maximum depth of 1.  -no-parent means that references to the par
    ent directory are ignored, and -A.gif means to download only the GIF files.  -A
    "*.gif" would have worked too.

That method won't work if the site has a robots.txt file set up, though.

--
"Do not meddle in the affairs of cats, for they are subtle and will pee on your computer." -- Bruce Graham

jhuuskon

Member #302

April 2000

Didn't you think i tried that? Just didn't work. I even forged the user agent and told it to ignore robots.txt but to no avail.

You don't deserve my sig.