有一个在线文件(如http://www.example.com/information.asp),我需要抓取并保存到一个目录。我知道有几种逐行抓取和读取在线文件(url)的方法,但是否有一种方法可以使用Java下载并保存文件?
当前回答
总结(并以某种方式润色和更新)之前的答案。以下三种方法实际上是等效的。(我添加了明确的超时,因为我认为这是必须的。没有人希望下载在连接丢失时永远冻结。)
public static void saveUrl1(final Path file, final URL url,
int secsConnectTimeout, int secsReadTimeout))
throws MalformedURLException, IOException {
// Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
try (BufferedInputStream in = new BufferedInputStream(
streamFromUrl(url, secsConnectTimeout,secsReadTimeout));
OutputStream fout = Files.newOutputStream(file)) {
final byte data[] = new byte[8192];
int count;
while((count = in.read(data)) > 0)
fout.write(data, 0, count);
}
}
public static void saveUrl2(final Path file, final URL url,
int secsConnectTimeout, int secsReadTimeout))
throws MalformedURLException, IOException {
// Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
try (ReadableByteChannel rbc = Channels.newChannel(
streamFromUrl(url, secsConnectTimeout, secsReadTimeout)
);
FileChannel channel = FileChannel.open(file,
StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING,
StandardOpenOption.WRITE)
) {
channel.transferFrom(rbc, 0, Long.MAX_VALUE);
}
}
public static void saveUrl3(final Path file, final URL url,
int secsConnectTimeout, int secsReadTimeout))
throws MalformedURLException, IOException {
// Files.createDirectories(file.getParent()); // Optional, make sure parent directory exists
try (InputStream in = streamFromUrl(url, secsConnectTimeout,secsReadTimeout) ) {
Files.copy(in, file, StandardCopyOption.REPLACE_EXISTING);
}
}
public static InputStream streamFromUrl(URL url,int secsConnectTimeout,int secsReadTimeout) throws IOException {
URLConnection conn = url.openConnection();
if(secsConnectTimeout>0)
conn.setConnectTimeout(secsConnectTimeout*1000);
if(secsReadTimeout>0)
conn.setReadTimeout(secsReadTimeout*1000);
return conn.getInputStream();
}
我没有发现明显的差异,在我看来都是对的。它们既安全又高效。(速度的差异似乎无关紧要——我从本地服务器写入180 MB到SSD磁盘的时间大约在1.2到1.5秒之间波动)。它们不需要外部库。所有这些都可以使用任意大小和(根据我的经验)HTTP重定向。
此外,如果没有找到资源(通常是404错误),所有抛出FileNotFoundException,如果DNS解析失败则抛出java.net.UnknownHostException;其他IOException对应传输过程中的错误。
其他回答
这是另一个基于Brian Risk的答案的Java 7变体,使用了try-with语句:
public static void downloadFileFromURL(String urlString, File destination) throws Throwable {
URL website = new URL(urlString);
try(
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(destination);
) {
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
}
}
public class DownloadManager {
static String urls = "[WEBSITE NAME]";
public static void main(String[] args) throws IOException{
URL url = verify(urls);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
InputStream in = null;
String filename = url.getFile();
filename = filename.substring(filename.lastIndexOf('/') + 1);
FileOutputStream out = new FileOutputStream("C:\\Java2_programiranje/Network/DownloadTest1/Project/Output" + File.separator + filename);
in = connection.getInputStream();
int read = -1;
byte[] buffer = new byte[4096];
while((read = in.read(buffer)) != -1){
out.write(buffer, 0, read);
System.out.println("[SYSTEM/INFO]: Downloading file...");
}
in.close();
out.close();
System.out.println("[SYSTEM/INFO]: File Downloaded!");
}
private static URL verify(String url){
if(!url.toLowerCase().startsWith("http://")) {
return null;
}
URL verifyUrl = null;
try{
verifyUrl = new URL(url);
}catch(Exception e){
e.printStackTrace();
}
return verifyUrl;
}
}
public void saveUrl(final String filename, final String urlString)
throws MalformedURLException, IOException {
BufferedInputStream in = null;
FileOutputStream fout = null;
try {
in = new BufferedInputStream(new URL(urlString).openStream());
fout = new FileOutputStream(filename);
final byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1) {
fout.write(data, 0, count);
}
} finally {
if (in != null) {
in.close();
}
if (fout != null) {
fout.close();
}
}
}
您将需要处理异常,可能是该方法的外部异常。
更简单的非阻塞I/O用法:
URL website = new URL("http://www.website.com/information.asp");
try (InputStream in = website.openStream()) {
Files.copy(in, target, StandardCopyOption.REPLACE_EXISTING);
}
这个答案几乎和选中的答案完全一样,但是有两个增强:它是一个方法,它关闭了FileOutputStream对象:
public static void downloadFileFromURL(String urlString, File destination) {
try {
URL website = new URL(urlString);
ReadableByteChannel rbc;
rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(destination);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
fos.close();
rbc.close();
} catch (IOException e) {
e.printStackTrace();
}
}
推荐文章
- codestyle;把javadoc放在注释之前还是之后?
- 如何在Spring中定义List bean ?
- 将Set<T>转换为List<T>的最简洁的方法
- 在JavaScript中,什么相当于Java的Thread.sleep() ?
- 使用Java重命名文件
- URL从Java中的类路径加载资源
- .toArray(new MyClass[0]) or .toArray(new MyClass[myList.size()])?
- Hibernate中不同的保存方法之间有什么区别?
- Java 8流和数组操作
- Java Regex捕获组
- Openssl不被视为内部或外部命令
- 如何添加自定义方法到Spring Data JPA
- 如何在Ubuntu中设置Java环境路径
- 无法执行dex:在Eclipse中超过GC开销限制
- 有人能解释一下JPA和Hibernate中的mappedBy吗?