有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?


当前回答

import java.io.*;
import java.net.*;

public class filedown {
    public static void download(String address, String localFileName) {
        OutputStream out = null;
        URLConnection conn = null;
        InputStream in = null;

        try {
            URL url = new URL(address);
            out = new BufferedOutputStream(new FileOutputStream(localFileName));
            conn = url.openConnection();
            in = conn.getInputStream();
            byte[] buffer = new byte[1024];

            int numRead;
            long numWritten = 0;

            while ((numRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, numRead);
                numWritten += numRead;
            }

            System.out.println(localFileName + "\t" + numWritten);
        } 
        catch (Exception exception) { 
            exception.printStackTrace();
        } 
        finally {
            try {
                if (in != null) {
                    in.close();
                }
                if (out != null) {
                    out.close();
                }
            } 
            catch (IOException ioe) {
            }
        }
    }

    public static void download(String address) {
        int lastSlashIndex = address.lastIndexOf('/');
        if (lastSlashIndex >= 0 &&
        lastSlashIndex < address.length() - 1) {
            download(address, (new URL(address)).getFile());
        } 
        else {
            System.err.println("Could not figure out local file name for "+address);
        }
    }

    public static void main(String[] args) {
        for (int i = 0; i < args.length; i++) {
            download(args[i]);
        }
    }
}

其他回答

在underscore-java库中有一个方法U.fetch(url)。

文件pom.xml:

<dependency>
  <groupId>com.github.javadev</groupId>
  <artifactId>underscore</artifactId>
  <version>1.84</version>
</dependency>

代码示例:

import com.github.underscore.U;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class Download {
    public static void main(String[] args) throws IOException {
        Files.write(Paths.get("data.bin"),
            U.fetch("https://stackoverflow.com/questions"
                + "/921262/how-to-download-and-save-a-file-from-internet-using-java").blob());
    }
}

下载一个文件需要你阅读它。无论哪种方式,您都必须以某种方式查看该文件。而不是逐行,你可以从流中逐字节读取:

BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
byte data[] = new byte[1024];
int count;
while((count = in.read(data, 0, 1024)) != -1)
{
    out.write(data, 0, count);
}

如果你使用代理,你可以在Java程序中设置代理,如下所示:

Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("https.proxyHost", "HTTPS proxy of your org");
systemSettings.put("https.proxyPort", "8080");

如果您没有使用代理,请不要在代码中包含上述代码行。完整的工作代码下载文件时,你是一个代理。

public static void main(String[] args) throws IOException {
    String url = "https://raw.githubusercontent.com/bpjoshi/fxservice/master/src/test/java/com/bpjoshi/fxservice/api/TradeControllerTest.java";
    OutputStream outStream = null;
    URLConnection connection = null;
    InputStream is = null;
    File targetFile = null;
    URL server = null;

    // Setting up proxies
    Properties systemSettings = System.getProperties();
        systemSettings.put("proxySet", "true");
        systemSettings.put("https.proxyHost", "HTTPS proxy of my organisation");
        systemSettings.put("https.proxyPort", "8080");
        // The same way we could also set proxy for HTTP
        System.setProperty("java.net.useSystemProxies", "true");
        // Code to fetch file
    try {
        server = new URL(url);
        connection = server.openConnection();
        is = connection.getInputStream();
        byte[] buffer = new byte[is.available()];
        is.read(buffer);

        targetFile = new File("src/main/resources/targetFile.java");
        outStream = new FileOutputStream(targetFile);
        outStream.write(buffer);
    } catch (MalformedURLException e) {
        System.out.println("THE URL IS NOT CORRECT ");
        e.printStackTrace();
    } catch (IOException e) {
        System.out.println("I/O exception");
        e.printStackTrace();
    }
    finally{
        if(outStream != null)
            outStream.close();
    }
}
public class DownloadManager {

    static String urls = "[WEBSITE NAME]";

    public static void main(String[] args) throws IOException{
        URL url = verify(urls);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        InputStream in = null;
        String filename = url.getFile();
        filename = filename.substring(filename.lastIndexOf('/') + 1);
        FileOutputStream out = new FileOutputStream("C:\\Java2_programiranje/Network/DownloadTest1/Project/Output" + File.separator + filename);
        in = connection.getInputStream();
        int read = -1;
        byte[] buffer = new byte[4096];
        while((read = in.read(buffer)) != -1){
            out.write(buffer, 0, read);
            System.out.println("[SYSTEM/INFO]: Downloading file...");
        }
        in.close();
        out.close();
        System.out.println("[SYSTEM/INFO]: File Downloaded!");
    }
    private static URL verify(String url){
        if(!url.toLowerCase().startsWith("http://")) {
            return null;
        }
        URL verifyUrl = null;

        try{
            verifyUrl = new URL(url);
        }catch(Exception e){
            e.printStackTrace();
        }
        return verifyUrl;
    }
}

可以使用Apache的HttpComponents而不是Commons IO来下载文件。这段代码允许您根据URL在Java中下载文件,并将其保存到特定的目的地。

public static boolean saveFile(URL fileURL, String fileSavePath) {

    boolean isSucceed = true;

    CloseableHttpClient httpClient = HttpClients.createDefault();

    HttpGet httpGet = new HttpGet(fileURL.toString());
    httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0");
    httpGet.addHeader("Referer", "https://www.google.com");

    try {
        CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
        HttpEntity fileEntity = httpResponse.getEntity();

        if (fileEntity != null) {
            FileUtils.copyInputStreamToFile(fileEntity.getContent(), new File(fileSavePath));
        }

    } catch (IOException e) {
        isSucceed = false;
    }

    httpGet.releaseConnection();

    return isSucceed;
}

与单行代码相比:

FileUtils.copyURLToFile(fileURL, new File(fileSavePath),
                        URLS_FETCH_TIMEOUT, URLS_FETCH_TIMEOUT);

这段代码将使您对进程有更多的控制,不仅可以指定超时,还可以指定User-Agent和Referer值,这对许多网站来说都是至关重要的。