有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?


当前回答

下载一个文件需要你阅读它。无论哪种方式,您都必须以某种方式查看该文件。而不是逐行,你可以从流中逐字节读取:

BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
byte data[] = new byte[1024];
int count;
while((count = in.read(data, 0, 1024)) != -1)
{
    out.write(data, 0, count);
}

其他回答

当使用Java 7+时,使用以下方法从Internet下载文件并将其保存到某个目录:

private static Path download(String sourceURL, String targetDirectory) throws IOException
{
    URL url = new URL(sourceURL);
    String fileName = sourceURL.substring(sourceURL.lastIndexOf('/') + 1, sourceURL.length());
    Path targetPath = new File(targetDirectory + File.separator + fileName).toPath();
    Files.copy(url.openStream(), targetPath, StandardCopyOption.REPLACE_EXISTING);

    return targetPath;
}

文档在这里。

总结(并以某种方式润色和更新)之前的答案。以下三种方法实际上是等效的。(我添加了明确的超时,因为我认为这是必须的。没有人希望下载在连接丢失时永远冻结。)

public static void saveUrl1(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (BufferedInputStream in = new BufferedInputStream(
         streamFromUrl(url, secsConnectTimeout,secsReadTimeout));
         OutputStream fout = Files.newOutputStream(file)) {

            final byte data[] = new byte[8192];
            int count;
            while((count = in.read(data)) > 0)
                fout.write(data, 0, count);
        }
}

public static void saveUrl2(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (ReadableByteChannel rbc = Channels.newChannel(
             streamFromUrl(url, secsConnectTimeout, secsReadTimeout)
        );
        FileChannel channel = FileChannel.open(file,
             StandardOpenOption.CREATE,
             StandardOpenOption.TRUNCATE_EXISTING,
             StandardOpenOption.WRITE)
        ) {

        channel.transferFrom(rbc, 0, Long.MAX_VALUE);
    }
}

public static void saveUrl3(final Path file, final URL url,
    int secsConnectTimeout, int secsReadTimeout))
    throws MalformedURLException, IOException {

    // Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
    try (InputStream in = streamFromUrl(url, secsConnectTimeout,secsReadTimeout) ) {
        Files.copy(in, file, StandardCopyOption.REPLACE_EXISTING);
    }
}

public static InputStream streamFromUrl(URL url,int secsConnectTimeout,int secsReadTimeout) throws IOException {
    URLConnection conn = url.openConnection();
    if(secsConnectTimeout>0)
        conn.setConnectTimeout(secsConnectTimeout*1000);
    if(secsReadTimeout>0)
        conn.setReadTimeout(secsReadTimeout*1000);
    return conn.getInputStream();
}

我没有发现明显的差异,在我看来都是对的。它们既安全又高效。(速度的差异似乎无关紧要——我从本地服务器写入180 MB到SSD磁盘的时间大约在1.2到1.5秒之间波动)。它们不需要外部库。所有这些都可以使用任意大小和(根据我的经验)HTTP重定向。

此外,如果没有找到资源(通常是404错误),所有抛出FileNotFoundException,如果DNS解析失败则抛出java.net.UnknownHostException;其他IOException对应传输过程中的错误。

第一种方法采用新通道

ReadableByteChannel aq = Channels.newChannel(new url("https//asd/abc.txt").openStream());
FileOutputStream fileOS = new FileOutputStream("C:Users/local/abc.txt")
FileChannel writech = fileOS.getChannel();

第二种方法使用FileUtils

FileUtils.copyURLToFile(new url("https//asd/abc.txt", new local file on system("C":/Users/system/abc.txt"));

第三种使用方法

InputStream xy = new ("https//asd/abc.txt").openStream();

这就是我们如何通过使用基本的Java代码和其他第三方库来下载文件。这些只是作为快速参考。请用谷歌以上关键词获取详细信息及其他选项。

import java.io.*;
import java.net.*;

public class filedown {
    public static void download(String address, String localFileName) {
        OutputStream out = null;
        URLConnection conn = null;
        InputStream in = null;

        try {
            URL url = new URL(address);
            out = new BufferedOutputStream(new FileOutputStream(localFileName));
            conn = url.openConnection();
            in = conn.getInputStream();
            byte[] buffer = new byte[1024];

            int numRead;
            long numWritten = 0;

            while ((numRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, numRead);
                numWritten += numRead;
            }

            System.out.println(localFileName + "\t" + numWritten);
        } 
        catch (Exception exception) { 
            exception.printStackTrace();
        } 
        finally {
            try {
                if (in != null) {
                    in.close();
                }
                if (out != null) {
                    out.close();
                }
            } 
            catch (IOException ioe) {
            }
        }
    }

    public static void download(String address) {
        int lastSlashIndex = address.lastIndexOf('/');
        if (lastSlashIndex >= 0 &&
        lastSlashIndex < address.length() - 1) {
            download(address, (new URL(address)).getFile());
        } 
        else {
            System.err.println("Could not figure out local file name for "+address);
        }
    }

    public static void main(String[] args) {
        for (int i = 0; i < args.length; i++) {
            download(args[i]);
        }
    }
}

下载一个文件需要你阅读它。无论哪种方式,您都必须以某种方式查看该文件。而不是逐行,你可以从流中逐字节读取:

BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
byte data[] = new byte[1024];
int count;
while((count = in.read(data, 0, 1024)) != -1)
{
    out.write(data, 0, count);
}