如果您对测量两段文本的语义相似性更感兴趣，我建议您看看这个gitlab项目。你可以把它作为服务器运行，也有一个预先构建的模型，你可以很容易地使用它来测量两段文本的相似性;尽管它主要用于测量两个句子的相似度，但你仍然可以在你的情况下使用它。它是用java编写的，但您可以将其作为RESTful服务运行。

另一个选择是DKPro Similarity，这是一个库，有各种算法来测量文本的相似性。然而，它也是用java编写的。

代码示例:

// this similarity measure is defined in the dkpro.similarity.algorithms.lexical-asl package
// you need to add that to your .pom to make that example work
// there are some examples that should work out of the box in dkpro.similarity.example-gpl 
TextSimilarityMeasure measure = new WordNGramJaccardMeasure(3);    // Use word trigrams

String[] tokens1 = "This is a short example text .".split(" ");   
String[] tokens2 = "A short example text could look like that .".split(" ");

double score = measure.getSimilarity(tokens1, tokens2);

System.out.println("Similarity: " + score);

Generally a cosine similarity between two documents is used as a similarity measure of documents. In Java, you can use Lucene (if your collection is pretty large) or LingPipe to do this. The basic concept would be to count the terms in every document and calculate the dot product of the term vectors. The libraries do provide several improvements over this general approach, e.g. using inverse document frequencies and calculating tf-idf vectors. If you are looking to do something copmlex, LingPipe also provides methods to calculate LSA similarity between documents which gives better results than cosine similarity. For Python, you can use NLTK.

这里有一个小应用程序让你开始…

import difflib as dl

a = file('file').read()
b = file('file1').read()

sim = dl.get_close_matches

s = 0
wa = a.split()
wb = b.split()

for i in wa:
    if sim(i, wb):
        s += 1

n = float(s) / float(len(wa))
print '%d%% similarity' % int(n * 100)

句法相似性 有3种简单的方法来检测相似性。

Word2Vec 手套 Tfidf或countvectorizer

语义相似性 可以使用BERT嵌入和尝试不同的词池策略来获得文档嵌入，然后在文档嵌入上应用余弦相似度。

一种先进的方法是利用BERT分数来获得相似度。

研究论文链接:https://arxiv.org/abs/1904.09675

如果您对测量两段文本的语义相似性更感兴趣，我建议您看看这个gitlab项目。你可以把它作为服务器运行，也有一个预先构建的模型，你可以很容易地使用它来测量两段文本的相似性;尽管它主要用于测量两个句子的相似度，但你仍然可以在你的情况下使用它。它是用java编写的，但您可以将其作为RESTful服务运行。

另一个选择是DKPro Similarity，这是一个库，有各种算法来测量文本的相似性。然而，它也是用java编写的。

代码示例:

// this similarity measure is defined in the dkpro.similarity.algorithms.lexical-asl package
// you need to add that to your .pom to make that example work
// there are some examples that should work out of the box in dkpro.similarity.example-gpl 
TextSimilarityMeasure measure = new WordNGramJaccardMeasure(3);    // Use word trigrams

String[] tokens1 = "This is a short example text .".split(" ");   
String[] tokens2 = "A short example text could look like that .".split(" ");

double score = measure.getSimilarity(tokens1, tokens2);

System.out.println("Similarity: " + score);

为了用更少的数据集找到句子的相似性，并获得更高的精度，你可以使用下面的python包，它使用预训练的BERT模型，

pip install similar-sentences