有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?
当前回答
可以使用Apache的HttpComponents而不是Commons IO来下载文件。这段代码允许您根据URL在Java中下载文件,并将其保存到特定的目的地。
public static boolean saveFile(URL fileURL, String fileSavePath) {
boolean isSucceed = true;
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet(fileURL.toString());
httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0");
httpGet.addHeader("Referer", "https://www.google.com");
try {
CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
HttpEntity fileEntity = httpResponse.getEntity();
if (fileEntity != null) {
FileUtils.copyInputStreamToFile(fileEntity.getContent(), new File(fileSavePath));
}
} catch (IOException e) {
isSucceed = false;
}
httpGet.releaseConnection();
return isSucceed;
}
与单行代码相比:
FileUtils.copyURLToFile(fileURL, new File(fileSavePath),
URLS_FETCH_TIMEOUT, URLS_FETCH_TIMEOUT);
这段代码将使您对进程有更多的控制,不仅可以指定超时,还可以指定User-Agent和Referer值,这对许多网站来说都是至关重要的。
其他回答
这个答案几乎和选中的答案完全一样,但是有两个增强:它是一个方法,它关闭了FileOutputStream对象:
public static void downloadFileFromURL(String urlString, File destination) {
try {
URL website = new URL(urlString);
ReadableByteChannel rbc;
rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(destination);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
fos.close();
rbc.close();
} catch (IOException e) {
e.printStackTrace();
}
}
下载一个文件需要你阅读它。无论哪种方式,您都必须以某种方式查看该文件。而不是逐行,你可以从流中逐字节读取:
BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
byte data[] = new byte[1024];
int count;
while((count = in.read(data, 0, 1024)) != -1)
{
out.write(data, 0, count);
}
如果你使用代理,你可以在Java程序中设置代理,如下所示:
Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("https.proxyHost", "HTTPS proxy of your org");
systemSettings.put("https.proxyPort", "8080");
如果您没有使用代理,请不要在代码中包含上述代码行。完整的工作代码下载文件时,你是一个代理。
public static void main(String[] args) throws IOException {
String url = "https://raw.githubusercontent.com/bpjoshi/fxservice/master/src/test/java/com/bpjoshi/fxservice/api/TradeControllerTest.java";
OutputStream outStream = null;
URLConnection connection = null;
InputStream is = null;
File targetFile = null;
URL server = null;
// Setting up proxies
Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("https.proxyHost", "HTTPS proxy of my organisation");
systemSettings.put("https.proxyPort", "8080");
// The same way we could also set proxy for HTTP
System.setProperty("java.net.useSystemProxies", "true");
// Code to fetch file
try {
server = new URL(url);
connection = server.openConnection();
is = connection.getInputStream();
byte[] buffer = new byte[is.available()];
is.read(buffer);
targetFile = new File("src/main/resources/targetFile.java");
outStream = new FileOutputStream(targetFile);
outStream.write(buffer);
} catch (MalformedURLException e) {
System.out.println("THE URL IS NOT CORRECT ");
e.printStackTrace();
} catch (IOException e) {
System.out.println("I/O exception");
e.printStackTrace();
}
finally{
if(outStream != null)
outStream.close();
}
}
第一种方法采用新通道
ReadableByteChannel aq = Channels.newChannel(new url("https//asd/abc.txt").openStream());
FileOutputStream fileOS = new FileOutputStream("C:Users/local/abc.txt")
FileChannel writech = fileOS.getChannel();
第二种方法使用FileUtils
FileUtils.copyURLToFile(new url("https//asd/abc.txt", new local file on system("C":/Users/system/abc.txt"));
第三种使用方法
InputStream xy = new ("https//asd/abc.txt").openStream();
这就是我们如何通过使用基本的Java代码和其他第三方库来下载文件。这些只是作为快速参考。请用谷歌以上关键词获取详细信息及其他选项。
在java.net.http.HttpClient上使用授权的解决方案:
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.GET()
.header("Accept", "application/json")
// .header("Authorization", "Basic ci5raG9kemhhZXY6NDdiYdfjlmNUM=") if you need
.uri(URI.create("https://jira.google.ru/secure/attachment/234096/screenshot-1.png"))
.build();
HttpResponse<InputStream> response = client.send(request, HttpResponse.BodyHandlers.ofInputStream());
try (InputStream in = response.body()) {
Files.copy(in, Paths.get(target + "screenshot-1.png"), StandardCopyOption.REPLACE_EXISTING);
}
推荐文章
- 如何在java中格式化持续时间?(如格式H:MM:SS)
- urlencoder .encode(字符串)已弃用,我应该使用什么代替?
- javax.transaction.Transactional vs . org.springframework.transaction.annotation.Transactional
- Java 8接口方法中不允许“同步”的原因是什么?
- 如何找到Java堆大小和内存使用(Linux)?
- 使用Enum实现单例(Java)
- RabbitMQ与通道和连接之间的关系
- buildSessionFactory()配置方法在Hibernate中已弃用?
- Spring MVC -如何获得所有的请求参数在一个地图在Spring控制器?
- 如何在Java中按两个字段排序?
- 文件之间的差异。路径中的分隔符和斜杠
- 在方法参数中使用NotNull注释
- Spring MVC中处理可选参数的@RequestParam
- Tomcat:如何查找正在运行的Tomcat版本?
- 使用node.js下载图像