|
This article is cited in 3 scientific papers (total in 3 papers)
Probability theory and mathematical statistics
A statistical test for the Zipf's law by deviations from the Heaps' law
M. G. Chebuninab, A. P. Kovalevskiicb a Sobolev Institute of Mathematics,
4, Koptyuga ave.,
Novosibirsk, 630090, Russia
b Novosibirsk State University, 1, Pirogova str., Novosibirsk, 630090, Russia
c Novosibirsk State Technical University, 20, K. Marksa ave., 630073, Novosibirsk, Russia
Abstract:
We explore a probabilistic model of an artistic text: words of the text are chosen independently of each other
in accordance with a discrete probability distribution on an infinite dictionary. The words are enumerated 1, 2, $\ldots$,
and the probability of appearing the $i$'th word is asymptotically a power function.
Bahadur proved that in this case the number of different words as a function of the length of the text, again,
asymptotically behaves like a power function.
On the other hand, in the applied statistics community there are statements known as the Zipf’s and Heaps’ laws that are supported by empirical observations.
We highlight the links between Bahadur results and Zipf's/Heaps' laws, and
introduce and analyse a corresponding statistical test.
Keywords:
Zipf's law, Heaps' law, weak convergence.
Received September 24, 2019, published December 4, 2019
Citation:
M. G. Chebunin, A. P. Kovalevskii, “A statistical test for the Zipf's law by deviations from the Heaps' law”, Sib. Èlektron. Mat. Izv., 16 (2019), 1822–1832
Linking options:
https://www.mathnet.ru/eng/semr1170 https://www.mathnet.ru/eng/semr/v16/p1822
|
Statistics & downloads: |
Abstract page: | 291 | Full-text PDF : | 155 | References: | 33 |
|