为了简单起见，我化简了向量a和b:

Let :
    a : [1, 1, 0]
    b : [1, 0, 1]

那么余弦相似度(Theta)

 (Theta) = (1*1 + 1*0 + 0*1)/sqrt((1^2 + 1^2))* sqrt((1^2 + 1^2)) = 1/2 = 0.5

cos0.5的逆是60度。

import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;

/**
 * 
* @author Xiao Ma
* mail : 409791952@qq.com
*
*/
  public class SimilarityUtil {

public static double consineTextSimilarity(String[] left, String[] right) {
    Map<String, Integer> leftWordCountMap = new HashMap<String, Integer>();
    Map<String, Integer> rightWordCountMap = new HashMap<String, Integer>();
    Set<String> uniqueSet = new HashSet<String>();
    Integer temp = null;
    for (String leftWord : left) {
        temp = leftWordCountMap.get(leftWord);
        if (temp == null) {
            leftWordCountMap.put(leftWord, 1);
            uniqueSet.add(leftWord);
        } else {
            leftWordCountMap.put(leftWord, temp + 1);
        }
    }
    for (String rightWord : right) {
        temp = rightWordCountMap.get(rightWord);
        if (temp == null) {
            rightWordCountMap.put(rightWord, 1);
            uniqueSet.add(rightWord);
        } else {
            rightWordCountMap.put(rightWord, temp + 1);
        }
    }
    int[] leftVector = new int[uniqueSet.size()];
    int[] rightVector = new int[uniqueSet.size()];
    int index = 0;
    Integer tempCount = 0;
    for (String uniqueWord : uniqueSet) {
        tempCount = leftWordCountMap.get(uniqueWord);
        leftVector[index] = tempCount == null ? 0 : tempCount;
        tempCount = rightWordCountMap.get(uniqueWord);
        rightVector[index] = tempCount == null ? 0 : tempCount;
        index++;
    }
    return consineVectorSimilarity(leftVector, rightVector);
}

/**
 * The resulting similarity ranges from −1 meaning exactly opposite, to 1
 * meaning exactly the same, with 0 usually indicating independence, and
 * in-between values indicating intermediate similarity or dissimilarity.
 * 
 * For text matching, the attribute vectors A and B are usually the term
 * frequency vectors of the documents. The cosine similarity can be seen as
 * a method of normalizing document length during comparison.
 * 
 * In the case of information retrieval, the cosine similarity of two
 * documents will range from 0 to 1, since the term frequencies (tf-idf
 * weights) cannot be negative. The angle between two term frequency vectors
 * cannot be greater than 90°.
 * 
 * @param leftVector
 * @param rightVector
 * @return
 */
private static double consineVectorSimilarity(int[] leftVector,
        int[] rightVector) {
    if (leftVector.length != rightVector.length)
        return 1;
    double dotProduct = 0;
    double leftNorm = 0;
    double rightNorm = 0;
    for (int i = 0; i < leftVector.length; i++) {
        dotProduct += leftVector[i] * rightVector[i];
        leftNorm += leftVector[i] * leftVector[i];
        rightNorm += rightVector[i] * rightVector[i];
    }

    double result = dotProduct
            / (Math.sqrt(leftNorm) * Math.sqrt(rightNorm));
    return result;
}

public static void main(String[] args) {
    String left[] = { "Julie", "loves", "me", "more", "than", "Linda",
            "loves", "me" };
    String right[] = { "Jane", "likes", "me", "more", "than", "Julie",
            "loves", "me" };
    System.out.println(consineTextSimilarity(left,right));
}
}

下面是一个简单的计算余弦相似度的Python代码:

import math

def dot_prod(v1, v2):
    ret = 0
    for i in range(len(v1)):
        ret += v1[i] * v2[i]
    return ret

def magnitude(v):
    ret = 0
    for i in v:
        ret += i**2
    return math.sqrt(ret)

def cos_sim(v1, v2):
    return (dot_prod(v1, v2)) / (magnitude(v1) * magnitude(v2))

这段Python代码是我实现算法的快速而肮脏的尝试:

import math
from collections import Counter

def build_vector(iterable1, iterable2):
    counter1 = Counter(iterable1)
    counter2 = Counter(iterable2)
    all_items = set(counter1.keys()).union(set(counter2.keys()))
    vector1 = [counter1[k] for k in all_items]
    vector2 = [counter2[k] for k in all_items]
    return vector1, vector2

def cosim(v1, v2):
    dot_product = sum(n1 * n2 for n1, n2 in zip(v1, v2) )
    magnitude1 = math.sqrt(sum(n ** 2 for n in v1))
    magnitude2 = math.sqrt(sum(n ** 2 for n in v2))
    return dot_product / (magnitude1 * magnitude2)


l1 = "Julie loves me more than Linda loves me".split()
l2 = "Jane likes me more than Julie loves me or".split()


v1, v2 = build_vector(l1, l2)
print(cosim(v1, v2))

让我试着用Python代码和一些图形数学公式来解释这一点。

假设我们的代码中有两个非常短的文本:

texts = ["I am a boy", "I am a girl"] 

我们想要比较下面的查询文本，看看查询与上面的文本有多接近，使用快速余弦相似度评分:

query = ["I am a boy scout"]

我们应该如何计算余弦相似度分数?首先，让我们用Python为这些文本构建一个tfidf矩阵:

from sklearn.feature_extraction.text import TfidfVectorizer
    
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(texts)

接下来，让我们检查tfidf矩阵及其词汇表的值:

print(tfidf_matrix.toarray())

# output 
array([[0.57973867, 0.81480247, 0.        ],
       [0.57973867, 0.        , 0.81480247]])

这里，我们得到一个tfidf矩阵，tfidf值为2 x 3，或者2个文档/文本x 3个术语。这是我们的tfidf文档术语矩阵。让我们通过调用vectorizer.vocabulary_来看看这3项是什么

print(vectorizer.vocabulary_)

# output
{'am': 0, 'boy': 1, 'girl': 2}

这告诉我们，tfidf矩阵中的3项是“am”，“boy”和“girl”。'am'在第0列，'boy'在第1列，'girl'在第2列。术语“I”和“a”已被矢量器删除，因为它们是停止词。

现在我们有了tfidf矩阵，我们想要比较我们的查询文本和我们的文本，看看我们的查询有多接近我们的文本。为此，我们可以计算查询的余弦相似度分数与文本的tfidf矩阵。但首先，我们需要计算查询的tfidf:

query = ["I am a boy scout"]

query_tfidf = vectorizer.transform([query])
print(query_tfidf.toarray())

#output
array([[0.57973867, 0.81480247, 0.        ]])

Here, we computed the tfidf of our query. Our query_tfidf has a vector of tfidf values [0.57973867, 0.81480247, 0. ], which we will use to compute our cosine similarity multiplication scores. If I am not mistaken, the query_tfidf values or vectorizer.transform([query]) values are derived by just selecting the row or document from tfidf_matrix that has the most word matching with the query. For example, row 1 or document/text 1 of the tfidf_matrix has the most word matching with the query text which contains "am" (0.57973867) and "boy" (0.81480247), hence row 1 of the tfidf_matrix of [0.57973867, 0.81480247, 0. ] values are selected to be the values for query_tfidf. (Note: If someone could help further explain this that would be good)

在计算query_tfidf之后，我们现在可以将query_tfidf向量与文本tfidf_matrix矩阵相乘或点积，以获得余弦相似度分数。

回想一下，余弦相似度分数或公式等于以下:

cosine similarity score = (A . B) / ||A|| ||B||

这里，A =我们的query_tfidf向量，B =我们的tfidf_matrix的每一行

注意:A。B = A * B^T，或者A点积B = A乘以B转置。

了解了公式之后，让我们手动计算query_tfidf的余弦相似度分数，然后将我们的答案与sklearn提供的值进行比较。度量cosine_similarity函数。让我们手动计算:

query_tfidf_arr = query_tfidf.toarray()
tfidf_matrix_arr = tfidf_matrix.toarray()

cosine_similarity_1 = np.dot(query_tfidf_arr, tfidf_matrix_arr[0].T) / 
  (np.linalg.norm(query_tfidf_arr) * np.linalg.norm(tfidf_matrix_arr[0])) 
cosine_similarity_2 = np.dot(query_tfidf_arr, tfidf_matrix_arr[1].T) / 
  (np.linalg.norm(query_tfidf_arr) * np.linalg.norm(tfidf_matrix_arr[1]))

manual_cosine_similarities = [cosine_similarity_1[0], cosine_similarity_2[0]]
print(manual_cosine_similarities)    

#output
[1.0, 0.33609692727625745]

我们手动计算的余弦相似度分数给出了[1.0,0.33609692727625745]的值。让我们用sklearn提供的答案值来检查手动计算的余弦相似度分数。度量cosine_similarity函数:

from sklearn.metrics.pairwise import cosine_similarity

function_cosine_similarities = cosine_similarity(query_tfidf, tfidf_matrix)
print(function_cosine_similarities)

#output
array([[1.0        , 0.33609693]])

输出值都是相同的!手动计算余弦相似度值与函数计算余弦相似度值相同!

因此，这个简单的解释说明了余弦相似值是如何计算的。希望这个解释对你有帮助。

两个向量A和B存在于二维空间或三维空间中，它们之间的夹角为cos相似度。

如果角度更大(可以达到最大180度)，即Cos 180=-1，最小角度为0度。cos0 =1意味着向量是对齐的，因此向量是相似的。

cos 90=0(这足以得出向量A和B根本不相似，因为距离不能为负，余弦值将在0到1之间。因此，更多的角度意味着降低相似性(视觉化也有意义)