w-shingling

In natural language processing a w-shingling is a set of unique shingles (therefore n-grams) each of which is composed of contiguous subsequences of tokens within a document, which can then be used to ascertain the similarity between documents.

The symbol w denotes the quantity of tokens in each shingle selected, or solved for.

The document, "a rose is a rose is a rose" can therefore be maximally tokenized as follows: The set of all contiguous sequences of 4 tokens (Thus 4=n, thus 4-grams) is For a given shingle size, the degree to which two documents A and B resemble each other can be expressed as the ratio of the magnitudes of their shinglings' intersection and union, or where |A| is the size of set A.

The resemblance is a number in the range [0,1], where 1 indicates that two documents are identical.

This definition is identical with the Jaccard coefficient describing similarity and diversity of sample sets.