有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?


当前回答

这可以读取互联网上的文件,并将其写入文件。

import java.net.URL;
import java.io.FileOutputStream;
import java.io.File;

public class Download {
    public static void main(String[] args) throws Exception {
         URL url = new URL("https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png");  // Input URL
         FileOutputStream out = new FileOutputStream(new File("out.png"));  // Output file
         out.write(url.openStream().readAllBytes());
         out.close();
    }
}

其他回答

在underscore-java库中有一个方法U.fetch(url)。

文件pom.xml:

<dependency>
  <groupId>com.github.javadev</groupId>
  <artifactId>underscore</artifactId>
  <version>1.84</version>
</dependency>

代码示例:

import com.github.underscore.U;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class Download {
    public static void main(String[] args) throws IOException {
        Files.write(Paths.get("data.bin"),
            U.fetch("https://stackoverflow.com/questions"
                + "/921262/how-to-download-and-save-a-file-from-internet-using-java").blob());
    }
}

在java.net.http.HttpClient上使用授权的解决方案:

HttpClient client = HttpClient.newHttpClient();

HttpRequest request = HttpRequest.newBuilder()
        .GET()
        .header("Accept", "application/json")
        // .header("Authorization", "Basic ci5raG9kemhhZXY6NDdiYdfjlmNUM=") if you need
        .uri(URI.create("https://jira.google.ru/secure/attachment/234096/screenshot-1.png"))
        .build();

HttpResponse<InputStream> response = client.send(request, HttpResponse.BodyHandlers.ofInputStream());

try (InputStream in = response.body()) {
    Files.copy(in, Paths.get(target + "screenshot-1.png"), StandardCopyOption.REPLACE_EXISTING);
}

下载一个文件需要你阅读它。无论哪种方式,您都必须以某种方式查看该文件。而不是逐行,你可以从流中逐字节读取:

BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
byte data[] = new byte[1024];
int count;
while((count = in.read(data, 0, 1024)) != -1)
{
    out.write(data, 0, count);
}

下面是用Java代码从网上下载电影的示例代码:

URL url = new
URL("http://103.66.178.220/ftp/HDD2/Hindi%20Movies/2018/Hichki%202018.mkv");
    BufferedInputStream bufferedInputStream = new  BufferedInputStream(url.openStream());
    FileOutputStream stream = new FileOutputStream("/home/sachin/Desktop/test.mkv");

    int count = 0;
    byte[] b1 = new byte[100];

    while((count = bufferedInputStream.read(b1)) != -1) {
        System.out.println("b1:" + b1 + ">>" + count + ">> KB downloaded:" + new File("/home/sachin/Desktop/test.mkv").length()/1024);
        stream.write(b1, 0, count);
    }

试试Java NIO:

URL website = new URL("http://www.website.com/information.asp");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("information.html");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);

使用transferFrom()可能比从源通道读取并写入此通道的简单循环更有效。许多操作系统可以直接将字节从源通道传输到文件系统缓存中,而不需要实际复制它们。

点击这里查看更多信息。

注意:transferFrom中的第三个参数是传输的最大字节数。整数。MAX_VALUE将传输最多2^31字节,长。MAX_VALUE最多允许2^63字节(比现有的任何文件都大)。