有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?


当前回答

import java.io.*;
import java.net.*;

public class filedown {
    public static void download(String address, String localFileName) {
        OutputStream out = null;
        URLConnection conn = null;
        InputStream in = null;

        try {
            URL url = new URL(address);
            out = new BufferedOutputStream(new FileOutputStream(localFileName));
            conn = url.openConnection();
            in = conn.getInputStream();
            byte[] buffer = new byte[1024];

            int numRead;
            long numWritten = 0;

            while ((numRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, numRead);
                numWritten += numRead;
            }

            System.out.println(localFileName + "\t" + numWritten);
        } 
        catch (Exception exception) { 
            exception.printStackTrace();
        } 
        finally {
            try {
                if (in != null) {
                    in.close();
                }
                if (out != null) {
                    out.close();
                }
            } 
            catch (IOException ioe) {
            }
        }
    }

    public static void download(String address) {
        int lastSlashIndex = address.lastIndexOf('/');
        if (lastSlashIndex >= 0 &&
        lastSlashIndex < address.length() - 1) {
            download(address, (new URL(address)).getFile());
        } 
        else {
            System.err.println("Could not figure out local file name for "+address);
        }
    }

    public static void main(String[] args) {
        for (int i = 0; i < args.length; i++) {
            download(args[i]);
        }
    }
}

其他回答

这里有许多优雅而有效的答案。但是简洁会让我们失去一些有用的信息。特别是,人们通常不希望将连接错误视为异常,并且可能希望以不同的方式处理某些与网络相关的错误—例如,决定是否应该重试下载。

下面是一个方法,它不会为网络错误抛出异常(仅用于真正异常的问题,如url格式错误或写入文件的问题)

/**
 * Downloads from a (http/https) URL and saves to a file. 
 * Does not consider a connection error an Exception. Instead it returns:
 *  
 *    0=ok  
 *    1=connection interrupted, timeout (but something was read)
 *    2=not found (FileNotFoundException) (404) 
 *    3=server error (500...) 
 *    4=could not connect: connection timeout (no internet?) java.net.SocketTimeoutException
 *    5=could not connect: (server down?) java.net.ConnectException
 *    6=could not resolve host (bad host, or no internet - no dns)
 * 
 * @param file File to write. Parent directory will be created if necessary
 * @param url  http/https url to connect
 * @param secsConnectTimeout Seconds to wait for connection establishment
 * @param secsReadTimeout Read timeout in seconds - trasmission will abort if it freezes more than this 
 * @return See above
 * @throws IOException Only if URL is malformed or if could not create the file
 */
public static int saveUrl(final Path file, final URL url, 
  int secsConnectTimeout, int secsReadTimeout) throws IOException {
    Files.createDirectories(file.getParent()); // make sure parent dir exists , this can throw exception
    URLConnection conn = url.openConnection(); // can throw exception if bad url
    if( secsConnectTimeout > 0 ) conn.setConnectTimeout(secsConnectTimeout * 1000);
    if( secsReadTimeout > 0 ) conn.setReadTimeout(secsReadTimeout * 1000);
    int ret = 0;
    boolean somethingRead = false;
    try (InputStream is = conn.getInputStream()) {
        try (BufferedInputStream in = new BufferedInputStream(is); OutputStream fout = Files
                .newOutputStream(file)) {
            final byte data[] = new byte[8192];
            int count;
            while((count = in.read(data)) > 0) {
                somethingRead = true;
                fout.write(data, 0, count);
            }
        }
    } catch(java.io.IOException e) { 
        int httpcode = 999;
        try {
            httpcode = ((HttpURLConnection) conn).getResponseCode();
        } catch(Exception ee) {}
        if( somethingRead && e instanceof java.net.SocketTimeoutException ) ret = 1;
        else if( e instanceof FileNotFoundException && httpcode >= 400 && httpcode < 500 ) ret = 2; 
        else if( httpcode >= 400 && httpcode < 600 ) ret = 3; 
        else if( e instanceof java.net.SocketTimeoutException ) ret = 4; 
        else if( e instanceof java.net.ConnectException ) ret = 5; 
        else if( e instanceof java.net.UnknownHostException ) ret = 6;  
        else throw e;
    }
    return ret;
}

简单使用有一个问题:

org.apache.commons.io.FileUtils.copyURLToFile(URL, File)

如果你需要下载和保存非常大的文件,或者在一般情况下,如果你需要自动重试以防连接断开。

在这种情况下,我建议使用Apache HttpClient以及org.apache.commons.io.FileUtils。例如:

GetMethod method = new GetMethod(resource_url);
try {
    int statusCode = client.executeMethod(method);
    if (statusCode != HttpStatus.SC_OK) {
        logger.error("Get method failed: " + method.getStatusLine());
    }
    org.apache.commons.io.FileUtils.copyInputStreamToFile(
        method.getResponseBodyAsStream(), new File(resource_file));
    } catch (HttpException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
    method.releaseConnection();
}

你可以在一行中使用netloader for Java:

new NetFile(new File("my/zips/1.zip"), "https://example.com/example.zip", -1).load(); // Returns true if succeed, otherwise false.

如果你使用代理,你可以在Java程序中设置代理,如下所示:

Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("https.proxyHost", "HTTPS proxy of your org");
systemSettings.put("https.proxyPort", "8080");

如果您没有使用代理,请不要在代码中包含上述代码行。完整的工作代码下载文件时,你是一个代理。

public static void main(String[] args) throws IOException {
    String url = "https://raw.githubusercontent.com/bpjoshi/fxservice/master/src/test/java/com/bpjoshi/fxservice/api/TradeControllerTest.java";
    OutputStream outStream = null;
    URLConnection connection = null;
    InputStream is = null;
    File targetFile = null;
    URL server = null;

    // Setting up proxies
    Properties systemSettings = System.getProperties();
        systemSettings.put("proxySet", "true");
        systemSettings.put("https.proxyHost", "HTTPS proxy of my organisation");
        systemSettings.put("https.proxyPort", "8080");
        // The same way we could also set proxy for HTTP
        System.setProperty("java.net.useSystemProxies", "true");
        // Code to fetch file
    try {
        server = new URL(url);
        connection = server.openConnection();
        is = connection.getInputStream();
        byte[] buffer = new byte[is.available()];
        is.read(buffer);

        targetFile = new File("src/main/resources/targetFile.java");
        outStream = new FileOutputStream(targetFile);
        outStream.write(buffer);
    } catch (MalformedURLException e) {
        System.out.println("THE URL IS NOT CORRECT ");
        e.printStackTrace();
    } catch (IOException e) {
        System.out.println("I/O exception");
        e.printStackTrace();
    }
    finally{
        if(outStream != null)
            outStream.close();
    }
}

总结(并以某种方式润色和更新)之前的答案。以下三种方法实际上是等效的。(我添加了明确的超时,因为我认为这是必须的。没有人希望下载在连接丢失时永远冻结。)

public static void saveUrl1(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (BufferedInputStream in = new BufferedInputStream(
         streamFromUrl(url, secsConnectTimeout,secsReadTimeout));
         OutputStream fout = Files.newOutputStream(file)) {

            final byte data[] = new byte[8192];
            int count;
            while((count = in.read(data)) > 0)
                fout.write(data, 0, count);
        }
}

public static void saveUrl2(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (ReadableByteChannel rbc = Channels.newChannel(
             streamFromUrl(url, secsConnectTimeout, secsReadTimeout)
        );
        FileChannel channel = FileChannel.open(file,
             StandardOpenOption.CREATE,
             StandardOpenOption.TRUNCATE_EXISTING,
             StandardOpenOption.WRITE)
        ) {

        channel.transferFrom(rbc, 0, Long.MAX_VALUE);
    }
}

public static void saveUrl3(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (InputStream in = streamFromUrl(url, secsConnectTimeout,secsReadTimeout) ) {
        Files.copy(in, file, StandardCopyOption.REPLACE_EXISTING);
    }
}

public static InputStream streamFromUrl(URL url,int secsConnectTimeout,int secsReadTimeout) throws IOException {
    URLConnection conn = url.openConnection();
    if(secsConnectTimeout>0)
        conn.setConnectTimeout(secsConnectTimeout*1000);
    if(secsReadTimeout>0)
        conn.setReadTimeout(secsReadTimeout*1000);
    return conn.getInputStream();
}

我没有发现明显的差异,在我看来都是对的。它们既安全又高效。(速度的差异似乎无关紧要——我从本地服务器写入180 MB到SSD磁盘的时间大约在1.2到1.5秒之间波动)。它们不需要外部库。所有这些都可以使用任意大小和(根据我的经验)HTTP重定向。

此外,如果没有找到资源(通常是404错误),所有抛出FileNotFoundException,如果DNS解析失败则抛出java.net.UnknownHostException;其他IOException对应传输过程中的错误。