Distance Analysis of English Texts. I. Shakespeare's Sonnets.

Distances between identical symbols in information strings (biological, language, computer programs (*.exe files) are described with a different precision with four distributions: Exponential, Weibull, lognormal and negative binomial. The correlations are sometimes highly significant. Here are analyzed distances between signs in the Shakespeare's Sonnets. Some distance tests revealed specific formal features of Sonnets.

Statistical properties of information distributions, especially their extreme skewness, raised the notion of their specificity (Haitun 1982a, b, c). Determining frequencies of symbols or words was a time consuming task suitable for shortening unbearably long time periods (Yule 1944).

These linguistic studies had some pragmatical value, too: Learning of languages starting with the most frequent words and phrases and attribution of texts to authors.

The inverse function to frequencies are distances between identical counted objects.

Distances between identical symbols exist in all information strings with any number of symbols or their k-tuples (words). Their manual counting was even more troublesome than counting words. Therefore such studies were made only for neighbor symbols where the local transitivity (frequencies of 2-tuples, e. g. ab) was studied for example by Harary and Paper (1957).

Time intervals between consecutive patent applications of patentees (Kunz 1987), and time intervals between consecutive publications (Kunz 1993) were determined for some small samples.

A stochastical generation of a string of m repeatings of an alphabet of n symbols is conventionally modeled by tossing a dice with n-sides.

A coin is the first nontrivial model of the dice with two sides. When a coin is tossed, there appear differently long sequences, when one result prevails. The distribution of sequences between successive events (head or tail) in all possible runs is known as the negative binomial distribution.

The negative binomial distribution is the inverse to the binomial distribution. It evaluates frequencies of distances between consecutive binary symbols in their strings.

This distribution was a statistical curiosity till some decades ago since its evaluation was a rather difficult task (Irwing 1963), because its distribution function does not exist in a closed form. Now it is included in standard statistical software program packages (STATGRAPHICS, Statistical Graphics Corporation).

The distances between symbols in a codone and English and Czech text were analyzed by Kunz and Rádl (1998). I then analyzed the distances between numerals in the first 10000 digits of the number e and the distances in the artificial codone based on the number e (Kunz 2000).

The purpose of this study is at first, to determine statistical properties of Shakespeare's sonnets, and to gain some knowledge, how the poet used the laguage, and then, to find, if distance analysis can reveal some differences between prosaic and poetic texts.

The Shakespeare's sonnets were obtained as an ASCII file on Internet (Project Gutenberg). Their numbering and dividing rows were stripped, as well as doubled or tripled spacebars, using MS-Word. After these formal corrections, the file contains 93772 signs including spaces, 76092 signs without spaces in 2155 lines and 17582 words.

It means that the mean length of a word is 4.327 signs (including apostrophes and punctuation marks), the mean length of a verse is 43.51 signs (with spaces), and 35.31 signs (without spaces), and/or 8.159 words.

After this, the distances were determined by a program elaborated by Rádl. The string is at first indexed with the position index i (i going from 1 to m) of each individual symbol in the string, and then the differences of these position indexes are determined. The differences are the topological distances between the same symbols. The sets of these values were evaluated by different statistical tests. The program counting distances counts all signs, including spacebar, return, and punctuation marks.

Replacing the numbering of sonnets by an sign, the length of sonnets can be determined as the distances between these signs. The results are tabulated as follows:

Table 1

The normal distribution. Mean: 649.47, standard deviation 22.1.

The length of sonnets is slightly bimodal, between the central narrow peak and the second one a walley exists. The differences of the length in this area including about one half of all sonnets are about 9 signs in 14 verses, it means about 2 words.

Then the distances between the individual letters were determined, at first separately the lower case and the upper case (when enough occurrences available), than taken together.

From all available implemented distributions, only four distributions gave significant results, the exponential distribution, the Weibull distribution, the lognormal distribution, and the negative binomial distribution, as before.

The actual values (mean, standard deviation, skewness, kurtosis, distribution parameters etc.) were determined only in some instances.

The distances between the consecutive spacebars greater than 1 determine the number of words of the length corresponding to the distance minus one. There exists 17680 spacebars after corrections. This is a somewhat different number compared with the direct counting of words. The results are tabulated as follows. Cumulating frequencies of shorter distances, improved in some cases the fit, since bellow it the counts are scattered, and the differences can balance themselves.

The distribution of length of words seems to have the lognormal shape, but this guess was not tested.

Notes to some results:

The distribution of one letter words is poorly correlated by the lognormal distribution. There are two peeks and one walley. The main peak between distances 51-60 is slightly above length of one verse (21 occurrences against 12 expected). It alone makes 45.9 % of the chi-square test value.

Table 3

The distribution of distances between words of length 4. The first half.

Chisquare = 26.3937 with 18 degree of freedom. Significance level = 0.09109

The chisquare value is rather low. But inspecting its constituents, we see that there are only 49 distances 10 and 11 words against 77.3 expected and 25 distance 14 words against 15.8 expected. These two differences make only one percent of all occurrences of words but 66.3 % of the chisquare value.

Table 4

The distribution of distances between words of length 4. The second half.

Chisquare = 12.2746 with 19 degrees of freedom. Sig. level = 0.873556.

Chisquare is almost excellent. Inspecting its constituents, we see again that there is only 25 distances 11 against 35.8 expected and 16 distances 16 against 10.1 expected. This difference makes less than one percent of all occurrences.

When consistency of both parts is tested by the two sample analysis, the zero hypothesis shall not be rejected.

The distribution of five letter words is poor due to their low repeating within distance one (273 occurrences against 368.3 expected). This makes 36.8 % of the chi-square test value. There is a peak within the distance 2 (377 occurrences against 318.3 expected). This makes 16.1 % of the poor chi-square test value. Other deviations are minor.

The exponential distribution of six letter words is poor due to their low repeating within distance one (140 occurrences against 163.8 expected). This makes 17.4 % of the chi-square test value. There is a peak within distances 6-13 (639 occurrences against 588.6 expected). This makes 22.2 % of the poor chi-square test value.

The distribution of seven letter words, similarly as next odd words is described by the Weilbull distribution. The correlation is poor. There exist two shortages, within distance one 27 (19 occurrences against 27.2 expected), and then within distances 80-92 (6 occurrences against 11.9 expected). They together make 65.4 % of the chi-square test value.

The distribution of eight letter words, as well as ten letter words is again the negative binomial distribution. The shortage of these words within distances 133-146 (1 occurrence against 5.8 expected) contributes 34 % of the chi-square test value. The tail is longer (14 occurrences against 8.8 expected). This makes 26.1 % of the poor chi-square test value.

The distribution of nine letter words is fairly correlated. The shortage of these words within distances 133-146 (1 occurrence against 5.8 expected) contributes 34 % of the chi-square test value.

The distribution of ten letter words is fairly correlated, too. The shortage of these words within distances 51-75 (16 occurrence against 26.2 expected) contributes one half of the chi-square test value.

The distributions of longer words are good correlated, or the tests failed due to few data.

Distances between punctuation marks show the length of sentences or clauses.

Here is given the interesting result with the distribution of the points:

Table 5

Chisquare = 246.117 with 17 d.f. Sig. level = 0

Chisquare = 279.572 with 14 d.f. Sig. level = 0

The results for all letters are presented in the form of the table, where the frequencies of all symbols are given and the significance of the performed chi-square tests. Then the commentaries to all symbols of the alphabet are given.

Notes:

EX = exponential distribution

WE = Weibull distribution

L N = lognormal distribution

NB = negative binomial distribution

* = the test was not made, since not enough of data

Statistic = XX, the chi-square test

The Weibull distribution is the best one only in the case of the letter m. The lognormal distribution correlates 4 cases of capital letters I, N, and S, and both case of the letter j (there are only two capital J). The exponential distribution is the best in the most performed tests, and the negative binomial distribution in 9 cases.

The fit varied from the excellent, for example f with the chi-square value 0.994, to practically zero values, as at the most frequent vowels.

The differences between experimental and calculated values were usually great at the shortest distances (1 till 10). Adjusting the lowest possible value to greater distances by pooling these distances increased the significance of the chi-square tests in some cases. The significance improved dramatically sometimes, see below.

Now, the commentaries to the individual letters follow.

A

The capital case A frequency allowed the separate test. The result with the exponential distribution is good, even if there is to high repeating within one verse distances (90 occurrences against 75.8 expected). This makes 39.5 % of the chi-square test value.

The distribution of distances between lower case a and both case (a + A) seems to be the exponential, at least their tails fit. The distribution cannot be satisfactorily described by a simple function due to fluctuations of frequencies between odd and even distances. This feature can be documented by pooling the lower distances between both case (a + A):

The fluctuations between odd and even distances are not constant.

Correlating observed frequencies of the same classes of distances of one part against the same of the other part gives fairly linear plot (due to the span of values in the logarithmic scale). The two way sample analysis shows that the parts are from one whole.

B

The distribution of distances between upper case B is exponential. There is a peak within distances 438-655 which corresponds to 10-12 verses (90 occurrences against 70 expected). This makes nearly two thirds of the chi-square test value.

The exponential distribution of distances between lower case b is worsened by a peak within distances 277-337 (9 occurrences against 22.4 expected). This makes more than one third of the chi-square test value. There are too few doubled bb (5 occurrences against 12.5 expected), which contributes 19.1 % of the chi-square test value, and to many occurrences ( 256 against 224.6 expected) within distances 32-62. This again makes 18.7 % of the chi-square test value. The combination of both cases changed the form of the distribution to Weilbull. There are too few doubled bb (5 occurrences against 11.9 expected), which contributes 24.1 % of the chi-square test value. Other deviations have a minor weight.

C

The distribution of this letter varies between exponential (b chi-square value 0.3726, b+ B chi-square value 0.5223), and negative binomial (b chi-square value 0.3580, b+ B chi-square value 0.4790). In both cases, there is a shortage in the range 143-169 (34 occurrences against 48.8 expected, or 26 occurrences against 38.8 expected ). This makes 29.3 %, and 24.8 % of the chi-square test value, respectively.

D

Here also the negative binomial or exponential distribution were applicable, with many deviations. There are too few doubled dd (18 occurrences against 79.2 [d] or 81.4 [d + D] expected), which contributes 62.4 % or 63.6 % of the chi-square test value, respectively. The exponential distribution is fair over the limit 30 (the chi-square test value 0.168).

E

There are relatively few E comparing with the number of e.

The distribution of distances between lower case e and both case (e + E) seems to be the negative binomial, at least their tails fit. The distribution cannot be satisfactorily described by a simple function due to fluctuations of frequencies between odd and even distances. This feature can be documented by pooling the lower distances between e:

both case (a + A):

The fluctuations between odd and even distances are not constant.

Correlating observed frequencies of the same classes of distances of one part against the same of the other part gives fairly linear plot (due to the span of values in the logarithmic scale). The two way sample analysis shows that the parts can not always be considered as parts from one whole.

The first part corresponds to the third part, and correlates badly with the second, and fourth part, too.

The first part corresponds to the second and fourth parts, and correlates badly with the second, and fifth part. As an example, the output of the test between part 1 and 3 is given:

The correlation of the second half is good, only when the lower limit is set over 29:

The exponential distribution of both case l + L is distorted by many double ll, which again makes more than 90 % of the total very high chi-square test value. The correlation of the first half is good, when the lower limit is set over 12:

The correlation of the second half is good, only when the lower limit is set over 29:

The consistency of both parts is good (l1/l2) = 0.5984, (l + L1)/(l + L2) = 0.6922. Even the comparisons of (l1/l + L1) = 0.6785, (l2/l + L2) = 0.5816 are acceptable.

M

This letter is correlated at best with the Weilbull distribution. Even the negative binomial distribution is acceptable (the chi-square test value 0.582). The doubled mm fit excellently with the negative binomial distribution but they form a peak in the Weilbull distribution (42 occurrences against 30.7 expected). This makes more than one third of the chi-square test value. There is a shortage of distances 100-113 (both models, 50 occurrences against 63.9 (WE) or 61.5 (NB) expected). This makes one quarter of the chi-square test value (WE). The combination with the upper case M decreased weights of these fluctuations somewhat.

N

The distribution of n and (n + N) was divided into two parts, which were slightly different. The negative binomial distribution of this letter is distorted by the shortage of doubled nn (only 11 % of expected, which makes two thirds of the chi-square test value). The chi-square test values improved by pooling lower distances differently: 1. part of n over 20 = 0.917, 2. part of n over 24 = 0.706, 1. part of (n + N) over 25 = 0.925, 2. part (n + N) over 24 = 0.938.

O

The lognormal distribution of capital O has a peak between 29-41 verses (20 occurrences against 14.3 expected). This makes 73.3 % of the chi-square test value. The distribution of o and (o + O) was again divided into two parts, which were slightly different. The negative binomial distribution of this letter is distorted by the shortage of doubled oo (only 51.8 % of expected, which makes one half of the chi-square test value). The chi-square test values improved by pooling lower distances differently: 1. part of o over 21 = 0.216, 2. part of n over 12 = 0.356, 1. part of (o + O) over 29 = 0.101, 2. part (o + O) over 11 = 0.472.

P

The negative binomial distribution of this letter is distorted by the surplus of doubled pp (44 occurrences against 10.4 expected), which makes 92.7 % of the chi-square test value. The chi-square test values improved by forced lower limit 10 to 0.296. The tail over 161 is almost perfect.

Q

The exponential distribution gives no opportunity for some comments.

R

The distribution of r and (r + R) was divided into two parts, which were slightly different. The negative binomial distribution of this letter is distorted by the shortage of doubled rr (only 24.8 % of expected, which makes about one half of the chi-square test value). The chi-square test values improved by pooling lower distances differently: 1. part of r over 30 = 0.241, 2. part of r over 42 = 0.671, 1. part of (r + R) over 30 = 0.186, 2. part (r + R) over 40 = 0.541.

S

The lognormal distribution of the capital S is good, the significance of the chi-square test is distorted by a peak within distances of 10-14 verses (24 occurrences against 19.4 expected). This contributes one quarter to the chi-square test value. The distribution of the lower case s an d (s + S) was divided into two parts, which were slightly different. The negative binomial distribution of this letter is distorted significantly by the shortage of doubled ss only in the 1. part (91 occurrences against 125.7 expected). This makes 15.2 % of the chi-square test value. The sign s appears less than expected within distances 2-6 (801 occurrences against 874 expected), and more than expected within distances 6-12 (933 occurrences against 835.9 expected). The chi-square test value improved by pooling lower distances: 1. part of s over 30 = 0.167, 2. part of s over 30 = 0.660. Here appeared a shortage of the distances 49-53 (52 occurrences against 78.6 expected). This makes 75.9 % of the chi-square test value. The distribution of the both case s +S is similarly the negative binomial one, fair with pooled lower distances: 1. part of s over 20 = 0.521, 2. part of s over 30 = 0.525.

T

The exponential distribution of the capital T has a peak within distances 157-260 (103 occurrences against 84.9 expected). This contributes 23.7 % of the chi-square test value. The distribution is then shorter than expected (9 occurrences over 523 against 21.6 expected). This makes 46 % of the chi-square test value. The distribution of the lower case t is the negative binomial one. When divided into three parts, all parts show the shortage of doubled tt (11-17 % of expected), 66-71 % of the chi-square test value. The first part fitted excellently over 13, the significance chi-square test value is 0.919, the tail over 25 of the second part gives a fair chi-square test value 0.509, whereas the same tail the third part has the chi-square test value only 0.0019.

The distribution of both (t + T) is the negative binomial one. All three parts show the shortage of doublets Tt + tt (9.8-15 % of expected), 61.8-71.9 % of the chi-square test value. The first part fitted good over 12, the significance chi-square test value is 0.798, the tail over 25 of the second part gives a fair chi-square test value 0.529, whereas the same tail the third part has the chi-square test value only 0.003, and it is better correlated as the negative binomial distribution.

U

There are no doubled uu (0 occurrence against 55.7 expected). This makes 73.3 % of the chi-square test value of the exponential distribution. When the lower limit is set to 20, the chi-square test value is improved to 0.263. The exponential distribution of both (u + U), divided into three parts, correlates differently, again. The first part fitted poorly over 20, the significance chi-square test value is 0.105, the tail over 20 of the second part gives a good chi-square test value 0.747, whereas the tail over 13 the third part has the chi-square test value 0.512.

V

The exponential distribution has a shortage of distances till 32 (223 occurrences against 256.8 expected). This contributes 39 % of the chi-square test value. Then there follows a peak within distances 33-76 (368 occurrences against 312.5 expected). This contributes 33.8 % of the chi-square test value. The tail over 50 fits good with the chi-square test value 0.402.

W

The exponential distribution of the upper case W gives a good fit. There is a peak of the distances 113-224 (69 occurrences against 55.6 expected). This alone makes 45.4 % of the chi-square test value. The exponential distribution of the lower case gives an acceptable fit over 10 (the chi-square test value 0.350). There are no doubled ww (59 % of the chi-square test value). Combined (w + W) improved somewhat the fit, the absence ww makes 61.9 % of the chi-square test value, since the sample is greater. There is a shortage of the distances 118-131 (about 3 verses, 28 occurrences against 45.1 expected). This makes 10.6 % of the chi-square test value. Over 15 the chi-square test value is 0.462.

X

The exponential distribution is almost perfect.

Y

The exponential distribution of the upper case Y gives a good fit. It somewhat improves the very poor lognormal distribution of the lower There is a long peak within distances 74-117 (272 occurrences against 200.2 expected). This alone makes 48.2 %of the chi-square test value. The lognormal distribution of this letter is shorter than expected (25 occurrences over 205 against 52.6 expected). This makes 33.5 % of the chi-square test value.

Z

The exponential distribution is almost perfect.

I tried to find also the distribution of distances between words or groups of signs. As an example, the frequency of All (10), all (121) and *all (as call, shall etc., 209 occurrences). The distribution of distances between the determiner all is the Weilbull one, the chi-square test value is 0.448 with 121 occurrences.

The corrections (removing off superfluous spacebars) in some cases worsened the fits, when compared with preliminary tests made with the raw text, as if the writer's errors were a part of the scheme leading to some distribution of distances between symbols.

In verses, repeating of some letters in some intervals is intentional, since they form rhymes. But in statistics, this feature is blurred by their occurrences within verses. The verse structure of the text revealed itself in the use of points.

To high repeating of the capital A within one verse distances (90 occurrences against 75.8 expected) is due mostly to the sonnet number 66, where 11 verses start with "And". This starting "And" repeats in other sonnets, too, and in combination with other starting A makes the peak. This distortion must be considered as intentional.

Some distributions of distances between consonants are highly regular, especially their tails, if the low distances inside words are pooled. They are described with a different precision with four distributions: exponential, Weibull, lognormal and negative binomial. Sometimes it is rather difficult to decide which distribution is the better one for fitting.

The splitting of statistics of some frequent letters, which was a necessity due to the insufficient memory of the used software, showed new possibilities of the distance analysis.

Since there are statistically significant differences between the parts, it seems, that Sonnets are not a single work, but a collection of sonnets including different parts. No attempt was made to synchronize a statistical analysis with a subject and stylistical analysis.

If the results are compared with published example (Kunz @ Rádl, 1998) of a scientific paper, than there can be observed some differences. In both cases, the vowels, except u, are poorly fitted. In both cases, letter f gave nearly ideal fit.

Consonants with the worser fit in the Sonnets are: b, c, d, , g, h, k, l, v, and w. Consonants with the better fit in the Sonnets are: m, x, and z. Since there are only few data for study, it can be only speculated, if it is the caused by the different use of these consonants in rhymes, which could produce observed peaks and fluctuations.

It can be concluded, that the analysis of distances between lexical units in text could become an useful method of text analysis.

Haitun, S. D. (1982a) Stationary Scientometric Distributions I: Different Approximations. Scientometrics, , 4, 525.

Haitun, S. D. (1982b) Stationary Scientometric Distributions II: Non Gaussian Nature of Scientific Activities. Scientometrics, 4, 89 - 101.

Haitun, S. D. (1982c) Stationary Scientometric Distributions III: The Role of the Zipf Distribution. Scientometrics, 5, 375 - 395.

Harary, F.; Paper, H. H. (1957) Toward a General Calculus of Phonemic Distribution, Language, 33, 143 -- 169.

Irwing, J. O. (1963) The Place of Mathematics in Medical and Biological Statistics, J. Royal. Statistical Soc. A, 126, 1 - 45.

Kunz, M. (1987) Time Spectra of Patent Information, Scientometrics, 11, 163 - 173.

Kunz, M. (1993) About metrics of bibliometrics, J. Chem. Inform. Comput.

Sci., 33, 193 – 196.

Kunz, M. ; Rádl, Z. (1998) Distribution of Distances in Information Strings, J. Chem. Inform. Comput. Sci., 38, 374-378.