这里我有从几个不同的导航站点爬取的所有站点，其中一些是重复的，我的意思是，例如：
http://www.hao123.com/index.htm 和 http://www.hao123.com
这是两个内容相同的网站，当然还有其他情况，比如缺少斜线。通过单独使用 URL，我仍然将它们视为两个不同的站点。
我的问题是：是否有任何有效的方法可以将它们识别为一个站点？谢谢！
【问题讨论】：
标签： java web-crawler
据我所知，没有万无一失的方法。
话虽如此，一种方法可能是从每个 URL 加载内容，然后将 Levenshtein 距离算法应用于同一域名下的所有页面。然后，您可以设置一个阈值，以确定内容在被认为相同之前的“相似程度”（好像内容略有变化，我想大部分内容仍然相同。）页面长度的 10%可能是该值的一个很好的起点。
这可能会相对较慢，具体取决于您拥有的网站数量，但会考虑到每次加载时内容的细微差异，而简单的哈希或长度计算不会。
为了使这一点更可靠，您可以检查某些内容在您期望的负载中是否相同（或不同） - 例如页面标题。
【讨论】：
使用正则解析出域名
示例 sn-p：
String a = "http://www.google.com";
String tempString = a.substring(a.indexOf(".")+1, a.length()); // gets rid of everything before the first dot
String domainString = tempString.substring(0, tempString.indexOf(".")); // grabs everything before the second dot
System.out.println(domainString);
输出google
编辑：
这是一个示例独立演示，可以处理更复杂的域结构并提取单个组件。
您可以在以下源代码的 main 方法中添加更多域测试用例来测试各种域，但目前它正在测试以下域：
http://www.google.com/
ftp://www.google.com
http://google.com/
google.com
localhost:80
这是来源（请原谅我的懒惰意大利面）：
package domain.parser.test;
public class Parseromatic {
public static void main(String[] args) {
Parseromatic parser = new Parseromatic();
parser.extract("http://www.google.com/");
parser.extract("ftp://www.google.com");
parser.extract("http://google.com/");
parser.extract("google.com");
parser.extract("localhost:80");
}
public void extract(String a){
if(a.contains(".")){ // Initial outOfBounds proof check in cases like (http://localhost:80)
String leadingString = a.substring(0, a.indexOf(".")); // First portion of the URL
boolean hasProto = protocol(leadingString);
// Now lets grab the rest
String trailingString = a.substring(a.indexOf(".")+1, a.length());
// Check if it contains a forward-slash
if(trailingString.contains("/")){
// We snip out everything before the slash
String middleString = snipOffPages(trailingString);
// Now we're only left with the domain related things
// Check if subdomain was left in the leadingString
if(middleString.contains(".")){
// Yep so lets deal with that
if(hasProto){ // If it had a protocol
System.out.println("Subdomain: "+leadingString.substring(leadingString.indexOf("://")+3, leadingString.length()));
} else { // If it didn't have a protocol
System.out.println("Subdomain: "+leadingString);
}
// Now let's split up the rest
String[] split1 = middleString.split("\\.");
System.out.println("Domain: "+split1[0]);
// Check for port
if (split1[1].contains(":")){
// Assuming port is specified
String[] split2 = split1[1].split(":");
System.out.println("Top-Domain: "+split2[0]);
System.out.println("Port: "+split2[1]);
} else {
// Assuming no port specified
System.out.println("Top-Domain: "+split1[1]);
System.out.println("Port: N/A");
}
} else {
// No subdomain was present
System.out.println("Subdomain: N/A");
if(hasProto){ // If it had a protocol
System.out.println("Domain: "+leadingString.substring(leadingString.indexOf("://")+3, leadingString.length()));
} else { // If it didn't have a protocol
System.out.println("Domain: "+leadingString);
}
// Check for port
if (middleString.contains(":")){
// Assuming port is specified
String[] split2 = middleString.split(":");
System.out.println("Top-Domain: "+split2[0]);
System.out.println("Port: "+split2[1]);
} else {
// Assuming no port specified
System.out.println("Top-Domain: "+middleString);
System.out.println("Port: N/A");
}
}
} else { // We assume it only contains domain related things
if(trailingString.contains(".")){
// Yep so lets deal with that
if(hasProto){ // If it had a protocol
System.out.println("Subdomain: "+leadingString.substring(leadingString.indexOf("://")+3, leadingString.length()));
} else { // If it didn't have a protocol
System.out.println("Subdomain: "+leadingString);
}
// Now let's split up the rest
String[] split1 = trailingString.split("\\.");
System.out.println("Domain: "+split1[0]);
// Check for port
if (split1[1].contains(":")){
// Assuming port is specified
String[] split2 = split1[1].split(":");
System.out.println("Top-Domain: "+split2[0]);
System.out.println("Port: "+split2[1]);
} else {
// Assuming no port specified
System.out.println("Top-Domain: "+split1[1]);
System.out.println("Port: N/A");
}
} else {
// No subdomain was present
System.out.println("Subdomain: N/A");
if(hasProto){ // If it had a protocol
System.out.println("Domain: "+leadingString.substring(leadingString.indexOf("://")+3, leadingString.length()));
} else { // If it didn't have a protocol
System.out.println("Domain: "+leadingString);
}
// Check for port
if (trailingString.contains(":")){
// Assuming port is specified
String[] split2 = trailingString.split(":");
System.out.println("Top-Domain: "+split2[0]);
System.out.println("Port: "+split2[1]);
} else {
// Assuming no port specified
System.out.println("Top-Domain: "+trailingString);
System.out.println("Port: N/A");
}
}
}
} else {
// Assuming only one level exists
boolean hasProto = protocol(a);
// Check if protocol was present
if(hasProto){
String noProto = a.substring(a.indexOf("://")+3, a.length());
// If some pages or something is specified
if(noProto.contains("/")){
noProto = snipOffPages(noProto);
}
// Check for port
if(noProto.contains(":")){
String[] split1 = noProto.split(":");
System.out.println("Subdomain: N/A");
System.out.println("Domain: "+split1[0]);
System.out.println("Top-Domain: N/A");
System.out.println("Port: "+split1[1]);
} else {
System.out.println("Subdomain: N/A");
System.out.println("Domain: "+noProto);
System.out.println("Top-Domain: N/A");
System.out.println("Port: N/A");
}
} else {
// If some pages or something is specified
if(a.contains("/")){
a = snipOffPages(a);
}
// Check for port
if(a.contains(":")){
String[] split1 = a.split(":");
System.out.println("Subdomain: N/A");
System.out.println("Domain: "+split1[0]);
System.out.println("Top-Domain: N/A");
System.out.println("Port: "+split1[1]);
} else {
System.out.println("Subdomain: N/A");
System.out.println("Domain: "+a);
System.out.println("Top-Domain: N/A");
System.out.println("Port: N/A");
}
}
}
System.out.println(); // Cosmetic empty line, can ignore
}
public String snipOffPages(String a){
return a.substring(0,a.indexOf("/"));
}
public boolean protocol(String a) {
// Protocol extraction
if(a.contains("://")){ // Check for existance of protocol declaration
String protocolString = a.substring(0, a.indexOf("://"));
System.out.println("Protocol: "+protocolString);
return true;
}
else {
System.out.println("Protocol: N/A");
return false;
}
}
}
对于上面的指定域，它会输出：
Protocol: http
Subdomain: www
Domain: google
Top-Domain: com
Port: N/A
Protocol: ftp
Subdomain: www
Domain: google
Top-Domain: com
Port: N/A
Protocol: http
Subdomain: N/A
Domain: google
Top-Domain: com
Port: N/A
Protocol: N/A
Subdomain: N/A
Domain: google
Top-Domain: com
Port: N/A
Protocol: N/A
Subdomain: N/A
Domain: localhost
Top-Domain: N/A
Port: 80
【讨论】：
最好的方法可能是使用正则表达式来获取域名并保留所有域名的列表。每当您检查新的 URL 时，也要对照您的“已访问”域名列表进行检查。 这是一个关于如何获取域名的老问题：
【讨论】：