A Universal Random Coding Ensemble for Sample-Wise Lossy Compression

Abstract

We propose a universal ensemble for the random selection of rate–distortion codes which is asymptotically optimal in a sample-wise sense. According to this ensemble, each reproduction vector, $\widehat{\mathit{x}}$, is selected independently at random under the probability distribution that is proportional to $2^{-LZ(\widehat{\mathit{x}})}$, where $LZ(\widehat{\mathit{x}})$ is the code length of $\widehat{\mathit{x}}$ pertaining to the 1978 version of the Lempel–Ziv (LZ) algorithm. We show that, with high probability, the resulting codebook gives rise to an asymptotically optimal variable-rate lossy compression scheme under an arbitrary distortion measure, in the sense that a matching converse theorem also holds. According to the converse theorem, even if the decoder knew the ℓ-th order type of source vector in advance (ℓ being a large but fixed positive integer), the performance of the above-mentioned code could not have been improved essentially for the vast majority of codewords pertaining to source vectors in the same type. Finally, we present a discussion of our results, which includes among other things, a clear indication that our coding scheme outperforms the one that selects the reproduction vector with the shortest LZ code length among all vectors that are within the allowed distortion from the source vector.

1. Introduction

We revisit the well-known problem of lossy source coding for finite-alphabet sequences with respect to a certain distortion measure ([1]; [2], Chapter 10; [3], Chapter 9; [4]; [5], Chapters 7 and 8). More concretely, our focus is on d-semifaithful codes, namely, variable–rate codes that meet a certain distortion constraint for every source sequence (and not only in expectation). As is very well known [1], the rate–distortion function quantifies the minimum achievable expected coding rate for a given memoryless source and distortion measure.

During several past decades, many research efforts were motivated by the fact that the source statistics are rarely (if not never) known in practice, and were, therefore, directed to the quest for universal coding schemes, namely, coding schemes which do not depend on the unknown statistics, but nevertheless, approach the lower bound (i.e., the entropy in lossless compression, or the rate–distortion function in the lossy case) asymptotically, as the block length grows without bound. We next provide a very brief (and non-comprehensive) review of some of the relevant earlier work.

In lossless compression, the theory of universal source coding is very well developed and mature. Davisson’s work [6] concerning universal coding redundancies has established the concepts of weak universality and strong universality (vanishing maximin and minimax redundancies, respectively), and has characterized the connection to the capacity of the ‘channel’ defined by the family of conditional distributions of the data to be compressed given the index (or parameter) of the source in the class [7,8,9]. For many of the frequently encountered parametric classes of sources, the minimum achievable redundancy of universal codes is well known to be dominated by $\frac{klogn}{2n}$, where k is the number of degrees of freedom of the parameter, and n is the block length [10,11,12,13]. A central idea that arises from Davisson’s theory is to construct a Shannon code pertaining to the probability distribution of the data vector with respect to a mixture (with a certain prior function) of all sources in the class. Rissanen, who was the inventor of the minimum description length (MDL) principle [14], has proved in [15] a converse to a coding theorem, which asserts that asymptotically, no universal code can achieve redundancy below $(1-ϵ)\frac{klogn}{2n}$, with the possible exception of sources from a subset of the parameter space, whose volume tends to zero as $n\to \infty$ for every positive $ϵ$. Merhav and Feder [16] have generalized this result to more general classes of sources, with the term $\frac{klogn}{2n}$ substituted by the capacity of the above-mentioned ‘channel’. Further developments, including more refined redundancy analyses, have been carried out in later studies.

In the wider realm of universal lossy compression, the theory is, unfortunately, not as sharp and well developed as in the lossless setting. We confine our attention, in this work, to d-semifaithful codes [17], namely, codes that satisfy the distortion requirement with probability one. Zhang, Yang, and Wei [18] have proved that, unlike in lossless compression, in the lossy case, even if the source statistics are known perfectly, it is impossible to achieve redundancy below $\frac{logn}{2n}$, but $\frac{logn}{n}$ is achievable. Not knowing the source conveys the price of enlarging the multiplicative constant in front of $\frac{logn}{n}$. Indeed, Yu and Speed [19] have established weak universality with a constant that grows with the cardinalities of the alphabets of the source and the reconstruction [20]. Ornstein and Shields [17] have considered universal d-semifaithful coding for stationary and ergodic sources with respect to the Hamming distortion measure, and established convergence with probability one to the rate–distortion function. Kontoyiannis [21] had several interesting findings. The first is a certain central limit theorem (CLT), with an $O(1/\sqrt{n})$ redundancy term, whose coefficient is described as a limiting Gaussian random variable with some constant variance. The second is the so-called law of iterated logarithm (LIL), with redundancy proportional to $\sqrt{\frac{log(logn)}{n}}$ infinitely often with probability one. One of the counter-intuitive conclusions from [21] is that universality is priceless under those CLT and LIL criteria. In [22], many of the findings are based on the observation that optimal compression can be characterized in terms of the negative logarithm of the probability of a sphere of radius $nD$ around the source vector with respect to the distortion measure, where D is the allowed per-letter distortion. In the same article, they proposed also the ensemble of random coding with respect to a probability distribution given by a mixture of all distributions in a certain class. In two recent articles, Mahmood and Wagner [23,24] have studied d-semifaithful codes that are strongly universal with respect to both the source and the distortion function. The redundancy rates in [23] behave like $\frac{logn}{n}$ with different multiplicative constants. Other interesting results concerning a special distortion measure appear in [25].

A parallel line of research work on universal lossless and lossy compression, which was pioneered by Ziv, pertains to the individual sequence approach. According to this approach, there are no assumptions on the statistical properties of the source. The source sequence to be compressed is considered an arbitrary deterministic (individual) sequence, but limitations are imposed on the encoder and/or the decoder to be implementable by finite-state machines. This includes, first and foremost, the celebrated Lempel–Ziv (LZ) algorithm [26,27,28], as well as further developments that extend the scope to lossy compression with and without side information [29,30], as well as to joint source–channel coding [31,32]. In the lossless case, the article [33] provides an individual sequence analogue of the above-mentioned result due to Rissanen, where the expression $\frac{klogn}{2n}$ continues to designate the best achievable redundancy, but the main term of the compression ratio there is the empirical entropy of the source vector instead of the ordinary entropy of the probabilistic setting. The converse bound of [33] applies to the vast majority of source sequences within each type, and the vast majority of types (in analogy to the vast majority of the parameter space in Rissanen’s framework). In a way, this kind of a converse result still contains some flavor of the probabilistic setting, because arguing that the number of exceptional typical sequences is relatively small is actually equivalent to imposing a uniform distribution across the type and asserting that the induced probability of violating the bound is small. The achievability result of [33], on the other hand, holds pointwise for every sequence. A similar comment applies to [34], where asymptotically pointwise lossy compression was established with respect to first-order statistics (i.e., “memoryless” statistics) with an emphasis on distortion universality, similar to in [23,24].

A similar kind of a mix between the individual sequence setting and the probabilistic setting applies to this paper as well, in the context of universal rate–distortion coding, but here, just like in [34], there is no limitation to finite-state encoders/decoders as in [33]. In particular, our converse theorem asserts that given an arbitrary variable-rate code, and given an arbitrary distortion function within a certain wide class, the vast majority of reproduction vectors that represent source sequences of a given type (of any fixed order), must have a code length that is essentially at least as large as the negative logarithm of the probability of a ball with normalized radius D (D being the allowed per-letter distortion) centered at the given source sequence. The probability of this ball is taken with respect to a universal distribution that is proportional to $2^{-LZ(\widehat{\mathit{x}})}$, where $LZ(\widehat{\mathit{x}})$ is the code length of LZ encoding of the reproduction vector, $\widehat{\mathit{x}}$.

The emphasis on the word “majority” in the previous paragraph (which was also mentioned earlier) calls for an explanation: it should be understood that in the absence of limitations on the encoding memory resources (such as the finite-state machine model described above), there cannot exist any non-trivial lower bound that applies to each and every given individual sequence. The reason is simple: given a particular individual source sequence, one can always tailor an encoder that compresses that sequence to a single bit (even losslessly), for example, by assigning the bit ‘0’ to be the compressed representation of the given sequence and by appending the bit ‘1’ as a header to the uncompressed binary representation of any other source sequence. For the given individual sequence, the compression ratio would then be $1/n$, which tends to zero as n grows without bound (of course, this encoder would be very poor for many other source sequences). It is, therefore, unavoidable that any non-trivial lower bound must apply collectively over some set of source sequences, allowing at least some minority of exceptional sequences that may violate the bound.

We also present a matching achievability result, asserting that for every source sequence, this code length is essentially achievable by random coding, using a universal ensemble of codes, which is defined by independent random selection, where each codeword is drawn under the above-described universal probability distribution.

While the achievability result in [34] was pointwise as well, it was tailored to a memoryless structure, in the sense that it was given in terms of the rate–distortion function of the first-order empirical distribution, which is blind to any empirical dependencies and repetitive patterns within the source sequence. In this paper, we both extend the scope to general individual sequences beyond the memoryless statistics and extend the allowable class of distortion measures. In terms of the technical aspects, the proof of the achievability result is similar to a parallel proof in [34], but the novelty lies considerably more in the converse theorem and its proof, which is very different from the one in [34] because here there is no information-theoretic single-letter expression for the best achievable asymptotic performance.

Our main novel messages in this work can be summarized as follows.

• 1.. We essentially derive an asymptotic formula for the rate–distortion function of an individual sequence. It is given by the normalized negative logarithm of the universal probability of a sphere of radius $nD$ (in the distortion metric sense) around the given source sequence.

• 2.. We establish the universality of the LZ probability distribution, not only in the context of lossless compression, but also for lossy compression, in the role of a universal distribution for independent random selection of a rate–distortion codebook.

• 3.. The expression of this rate–distortion function is intimately related to a converse result due to Kontoyiannis and Zhang [22] (in terms of the probability of a sphere), but it is more explicit in a certain sense, that will be discussed in Section 5.

• 4.. We provide some intuitive insight regarding this rate–distortion function by demonstrating the analogy to the ordinary rate–distortion function (see Section 5).

• 5.. We show that our achievability scheme outperforms the (seemingly) natural extension of LZ compression to the lossy case, where one seeks, within a sphere of radius $nD$, the reproduction vector whose LZ code length is minimal (see Section 5).

The outline of this paper is as follows. In Section 2, we establish the notation conventions, define a few terms and quantities, and provide some background. In Section 3, we present the converse theorem and its proof. In Section 4, we present the achievability theorem and prove it. Finally, in Section 5, we summarize the paper and discuss our results.

2. Notation, Definitions, and Background

Throughout the paper, random variables will be denoted by capital letters, specific values they may take will be denoted by the corresponding lower case letters, and their alphabets will be denoted by calligraphic letters. Random vectors and their realizations will be denoted, respectively, by capital letters and the corresponding lower case letters, both in a bold face font. Their alphabets will be superscripted by their dimensions. The source vector of length n, $(x_1,\dots ,x_n)$, with components, $x_i$, $i=1,\dots ,n$, from a finite alphabet, $\mathcal{X}$, will be denoted by $\mathit{x}$. The set of all such n-vectors will be denoted by ${\mathcal{X}}^n$, which is the nth-order Cartesian power of $\mathcal{X}$. Likewise, a reproduction vector of length n, $({\widehat{x}}_1,\dots ,{\widehat{x}}_n)$, with components, ${\widehat{x}}_i$, $i=1,\dots ,n$, from a finite alphabet, $\widehat{\mathcal{X}}$, will be denoted by $\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n$. We denote the cardinalities of $\mathcal{X}$ and $\widehat{\mathcal{X}}$ by J and K, respectively.

For $i\le j$, the notation $x_i^j$ will be used to denote the substring $(x_i,x_{i+1},\dots ,x_j)$. Probability distributions will be denoted by the letter P or Q with possible subscripts, depending on the context. The probability of an event $\mathcal{E}$ will be denoted by $Pr\{\mathcal{E}\}$, and the expectation operator with respect to a probability distribution P will be denoted by $\mathbb{E}\{·\}$. For two positive sequences, $a_n$ and $b_n$, the notation $a_n\stackrel{·}{=}{b}_n$ will stand for equality in the exponential scale, that is, ${lim}_{n\to \infty }\frac{1}{n}log\frac{a_n}{b_n}=0$. Similarly, $a_n\stackrel{·}{\le }{b}_n$ means that ${lim\; sup}_{n\to \infty }\frac{1}{n}log\frac{a_n}{b_n}\le 0$, and so on. The indicator function of an event $\mathcal{E}$ will be denoted by $\mathcal{I}\{E\}$. The notation ${\left[x\right]}_{+}$ will stand for $max\{0,x\}$. The logarithmic function, $logx$, will be understood to be defined to the base 2. Logarithms to the base e will be denoted by ln.

Let ℓ be a positive integer that divides n. The ℓth-order empirical distribution of $\mathit{x}\in {\mathcal{X}}^n$, which will be denoted by ${\widehat{P}}_{\mathit{x}}^{\ell }$, is the vector of relative frequencies $\{{\widehat{P}}_{\mathit{x}}^{\ell }(a^{\ell }),a^{\ell }\in {\mathcal{X}}^{\ell }\}$, where

(1)${\widehat{P}}_{\mathit{x}}^{\ell }(a^{\ell })=\frac{\ell }{n}{\displaystyle \sum _{i=0}^{n/\ell -1}}\mathcal{I}\{x_{i\ell +1}^{(i+1)\ell }=a^{\ell }\}.$

The set of all ℓth-order empirical distributions of sequences in ${\mathcal{X}}^n$ will be denoted by ${\mathcal{P}}_n^{\ell }$. For $P^{\ell }\in {\mathcal{P}}_n^{\ell }$, the type class $\{\mathit{x}\in {\mathcal{X}}^n:{\widehat{P}}_{\mathit{x}}^{\ell }=P^{\ell }\}$ will be denoted by ${\mathcal{T}}_n(P^{\ell })$. Likewise, ${\mathcal{T}}_n(Q^{\ell })$ will denote $\{\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n:{\widehat{P}}_{\widehat{\mathit{x}}}^{\ell }=Q^{\ell }\}$, where ${\widehat{P}}_{\widehat{\mathit{x}}}^{\ell }$ is the ℓth-order empirical distribution of $\widehat{\mathit{x}}$. Finally, ${\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^{\ell }$ will denote the ℓth-order joint empirical distribution of $(\mathit{x},\widehat{\mathit{x}})$, i.e.,

(2)${\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^{\ell }(a^{\ell },b^{\ell })=\frac{\ell }{n}{\displaystyle \sum _{i=0}^{n/\ell -1}}\mathcal{I}\{x_{i\ell +1}^{(i+1)\ell }=a^{\ell },{\widehat{x}}_{i\ell +1}^{(i+1)\ell }=b^{\ell }\},(a^{\ell },b^{\ell })\in {\mathcal{X}}^{\ell }\times {\widehat{\mathcal{X}}}^{\ell }.$

For a given positive integer n, a distortion function, d, is a function from ${\mathcal{X}}^n\times {\widehat{\mathcal{X}}}^n$ into ${\mathbb{R}}^{+}$. In the two main parts of this paper, different assumptions will be imposed on the distortion function.

• 1.. For the achievability theorem, the distortion function can be completely arbitrary.

• 2.. For the converse theorem, we assume that $d(\mathit{x},\widehat{\mathit{x}})$ depends on $\mathit{x}$ and $\widehat{\mathit{x}}$ only via their first-order joint empirical distribution, ${\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^1$, and that for a given such distribution, it grows linearly in n, that is, $d(\mathit{x},\widehat{\mathit{x}})=n\rho ({\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^1)$, where the function $\rho$ is independent of n.

Regarding item 2, additive distortion measures, which obviously comply with the requirement, are given by linear functionals of ${\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^1$. However, here arbitrary non-linear functionals are allowed as well.

A rate–distortion block code of length n is a mapping, ${\varphi }_n:{\mathcal{X}}^n\to {\mathcal{B}}_n$, ${\mathcal{B}}_n\subset {\{0,1\}}^{*}$, that maps the space of source vectors of length n, ${\mathcal{X}}^n$, into a set, ${\mathcal{B}}_n$, of variable-length compressed bit strings. The decoder is a mapping ${\psi }_n:{\mathcal{B}}^n\to {\mathcal{C}}_n\subseteq {\widehat{\mathcal{X}}}^n$ that maps the set of compressed variable-length binary strings into a reproduction codebook ${\mathcal{C}}^n$. A block code is called d-semifaithful if for every $\mathit{x}\in {\mathcal{X}}^n$, $d(\mathit{x},{\psi }_n({\varphi }_n(\mathit{x})))\le nD$. The code length for $\mathit{x}$, denoted $L(\mathit{x})$, is the number of bits of ${\varphi }_n(\mathit{x})$. Since $L(\mathit{x})$ depends on $\mathit{x}$ only via ${\varphi }_n(\mathit{x})$, we will also denote it sometimes as $L({\varphi }_n(\mathit{x}))$ or by $L(\widehat{\mathit{x}})$ ($\widehat{\mathit{x}}$ being the reproduction vector pertaining to ${\varphi }_n(\mathit{x})$), with a slight abuse of notation. For the converse theorem, we assume that correspondence between ${\mathcal{B}}_n$ and ${\mathcal{C}}_n$ is one-to-one. For the achievability theorem, we consider prefix-free codes. Accordingly, the encoder can equivalently be presented as a cascade of a reproduction encoder (also known as a vector quantizer), which maps ${\mathcal{X}}^n$ into ${\mathcal{C}}_n$, followed by an entropy coder, which maps ${\mathcal{C}}_n$ into ${\mathcal{B}}_n$ with no additional loss of information.

For the purpose of presenting both the converse theorem and the achievability theorem, we need to recall a few terms and facts concerning the 1978 version of the LZ algorithm (also known as the LZ78 algorithm) [27]. The incremental parsing procedure of the LZ78 algorithm is a procedure of sequentially parsing a vector $\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n$ such that each new phrase is the shortest string that has not been encountered before as a parsed phrase, with the possible exception of the last phrase which might be incomplete. For example, the incremental parsing of the vector $\widehat{\mathit{x}}=\mathrm{a}\mathrm{b}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{a}\mathrm{a}\mathrm{b}\mathrm{b}\mathrm{a}\mathrm{a}\mathrm{a}\mathrm{b}\mathrm{a}\mathrm{a}$ is $\mathrm{a},\mathrm{b},\mathrm{b}\mathrm{a},\mathrm{b}\mathrm{a}\mathrm{a},\mathrm{b}\mathrm{b},\mathrm{a}\mathrm{a},\mathrm{a}\mathrm{b},\mathrm{a}\mathrm{a}$. Let $c(\widehat{\mathit{x}})$ denote the number of phrases in $\widehat{\mathit{x}}$ resulting from the incremental parsing procedure. Let $LZ(\widehat{\mathit{x}})$ denote the length of the LZ78 binary compressed code for $\widehat{\mathit{x}}$. According to ([27], Theorem 2),

(3)$\begin{array}{ccc}\hfill LZ(\widehat{\mathit{x}})& \le & [c(\widehat{\mathit{x}})+1]log\{2K[c(\widehat{\mathit{x}})+1]\}\hfill \\ & =& c(\widehat{\mathit{x}})log[c(\widehat{\mathit{x}})+1]+c(\widehat{\mathit{x}})log(2J)+log\{2K[c(\widehat{\mathit{x}})+1]\}\hfill \\ & =& c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})+c(\widehat{\mathit{x}})log\left[1+\frac{1}{c(\widehat{\mathit{x}})}\right]+c(\widehat{\mathit{x}})log(2K)+log\{2K[c(\widehat{\mathit{x}})+1]\}\hfill \\ & \le & c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})+loge+\frac{n(logK)log(2K)}{(1-{ϵ}_n)logn}+log\left[2K(n+1)\right]\hfill \\ & \stackrel{▵}{=}& c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})+n·ϵ(n),\hfill \end{array}$

where we remember that K is the cardinality of $\widehat{\mathcal{X}}$, and where $ϵ(n)$ clearly tends to zero as $n\to \infty$, at the rate of $1/logn$. We next define a universal probability distribution (see also [35,36]):

(4)$U(\widehat{\mathit{x}})=\frac{2^{-LZ(\widehat{\mathit{x}})}}{{\sum }_{{\widehat{\mathit{x}}}^{\prime }\in {\widehat{\mathcal{X}}}^n}{2}^{-LZ({\widehat{\mathit{x}}}^{\prime })}},\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n.$

Finally, we define the D-sphere around $\mathit{x}$ as

(5)$\mathcal{S}(\mathit{x},D)=\{\widehat{\mathit{x}}:d(\mathit{x},\widehat{\mathit{x}})\le nD\},$

and

(6)$U\left[\mathcal{S}(\mathit{x},D)\right]={\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}U(\widehat{\mathit{x}}).$

For later use, we also define

(7)$\widehat{\mathcal{S}}(\widehat{\mathit{x}},D)=\{\mathit{x}:d(\mathit{x},\widehat{\mathit{x}})\le nD\}.$

Our purpose is to derive upper and lower bounds on the smallest achievable code length, $L(\mathit{x})$, for d-semifaithful block codes of length n, and individual sequences, $\{\mathit{x}\}$, from a given ℓth-order-type class, ${\mathcal{T}}_n(P^{\ell })$. As will be seen shortly, in both the converse and the achievability theorems, the main term of the bound on the length function will be $-log(U[\mathcal{S}(\mathit{x},D)])$.

3. The Converse Theorem

The following converse theorem asserts that even if the type class of the source vector was known to the decoder ahead of time, the code length could not be much smaller than $-log(U[\mathcal{S}(\mathit{x},D)])$ for the vast majority of the codewords pertaining to that type.

Theorem 1.

Let ℓ be a positive integer that divides n and let ${\widehat{P}}^{\ell }$ be an arbitrary empirical distribution pertaining to a certain type class, ${\mathcal{T}}_n({\widehat{P}}^{\ell })$, of source sequences in ${\mathcal{X}}^n$. Let d be a distortion function that depends on $(\mathit{x},\widehat{\mathit{x}})$ only via ${\widehat{P}}_{\mathit{x}\widehat{\mathit{x}}}^1$. Then, for every d-semifaithful variable-length block code, with one-to-one correspondence between ${\mathcal{B}}_n$ and ${\mathcal{C}}_n$, and for every $ϵ>0$, the following lower bound applies to a fraction of at least $(1-2n^{-ϵ})$ of the codewords, $\{{\varphi }_n(\mathit{x}),\mathit{x}\in {\mathcal{T}}_n({\widehat{P}}^{\ell })\}$:

(8) $L({\varphi }_n(\mathit{x}))\ge -log(U\left[\mathcal{S}(\mathit{x},D)\right])-n{\Delta }_n(\ell )-ϵlogn,$

where ${\Delta }_n(\ell )$ has the property ${lim}_{n\to \infty }{\Delta }_n(\ell )=1/\ell$.

As a technical note, observe that ${\Delta }_n(\ell )$ can be made small when ℓ is chosen to be large, as ${\Delta }_n(\ell )$ behaves like $1/\ell$ for fixed ℓ and large n. This suggests that the theorem is meaningful mainly when ℓ is appreciably large, which is not surprising, because the larger ℓ is, the better one can exploit empirical dependencies within the source sequence. Enlarging ℓ also increases the asymptotic lower bound and, thus, makes it tighter. An alternative approach would be to let $\ell ={\ell }_n$ grow with n, but sufficiently slowly such that ${\Delta }_n({\ell }_n)\to 0$ as $n\to \infty$.

The remaining part of this section is devoted to the proof of Theorem 1. The proof is based on two main steps whose aim is to bypass the need for the traditional information-theoretic expression of the rate–distortion function. The first step is based on a simple counting argument, displayed in Equations (9)–(14) below, which leads to the identity (15) and, more specifically, to (16). Equation (16) is pivotal to the proof because it establishes exact equality between the sphere-covering ratio (on the left-hand side) and a quantity (on the right-hand side) that can be interpreted as one over the probability of falling into an $nD$-sphere around $\mathit{x}$ under random coding with respect to the uniform distribution over the type class, ${\mathcal{T}}_n(Q^{\ell })$. The sphere-covering ratio on the left-hand side belongs to the converse, and the expression on the right-hand side is strongly related to achievability (see Theorem 2 below), as it directly supports random coding. Therefore, these identities create the basic link between the converse and the direct theorems. The second step is to replace the uniform distribution over ${\mathcal{T}}_n(Q^{\ell })$ by the universal LZ distribution, $U(\mathit{x})\propto 2^{-LZ(\widehat{\mathit{x}})}$, and thereby relax the dependence upon $P^{\ell }$ (or $Q^{\ell }$), as well as on the parameter ℓ, without sacrificing the asymptotic performance. This proof is entirely different from the one in [34].

Proof. 

We first establish a relationship that will be used later on. For two given types ${\mathcal{T}}_n(P^{\ell })\subset {\mathcal{X}}^n$ and ${\mathcal{T}}_n(Q^{\ell })\subset {\widehat{\mathcal{X}}}^n$, consider the quantity,

(9)$N(D)={\displaystyle \sum _{\mathit{x},\widehat{\mathit{x}}}}\mathcal{I}\{\mathit{x}\in {\mathcal{T}}_n(P^{\ell }),\widehat{\mathit{x}}\in {\mathcal{T}}_n(Q^{\ell }),d(\mathit{x},\widehat{\mathit{x}})\le nD\}.$

We can evaluate $N(D)$ in two ways. The first is as follows:

(10)$\begin{array}{ccc}\hfill N(D)& =& {\displaystyle \sum _{\mathit{x}\in {\mathcal{T}}_n(P^{\ell })}}|T_n(Q^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|\hfill \end{array}$

(11)$\begin{array}{ccc}& =& |{\mathcal{T}}_n(P^{\ell })|·|{\mathcal{T}}_n(Q^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|,\hfill \end{array}$

where the second equality is since $|{\mathcal{T}}_n(Q^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|$ is the same for all $\mathit{x}\in {\mathcal{T}}_n(P^{\ell })$, due to the permutation-invariance assumption on the distortion function. By the same token, we can also express $N(D)$ in the following manner:

(12)$\begin{array}{ccc}\hfill N(D)& =& {\displaystyle \sum _{\widehat{\mathit{x}}\in {\mathcal{T}}_n(Q^{\ell })}}|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|\hfill \end{array}$

(13)$\begin{array}{ccc}& =& |{\mathcal{T}}_n(Q^{\ell })|·|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|,\hfill \end{array}$

which follows from the same consideration by symmetry. It follows then that

(14)$|{\mathcal{T}}_n(P^{\ell })|·|{\mathcal{T}}_n(Q^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|=\left|{\mathcal{T}}_n(Q^{\ell })\right|·|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|,$

or, equivalently,

(15)$\frac{|{\mathcal{T}}_n(P^{\ell })|}{|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|}=\frac{|{\mathcal{T}}_n(Q^{\ell })|}{|{\mathcal{T}}_n(Q^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|}.$

Now, let $Q_{*}^{\ell }$ be the type of $\widehat{\mathit{x}}$ that maximizes $|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|$. Then, the last equation implies that

(16)$\frac{|{\mathcal{T}}_n(P^{\ell })|}{{max}_{\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n}|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)|}=\frac{|{\mathcal{T}}_n(Q_{*}^{\ell })|}{|{\mathcal{T}}_n(Q_{*}^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|},\forall \mathit{x}\in {\mathcal{T}}_n(P^{\ell }).$

This relationship will be used shortly.

Let $P^{\ell }\in {\mathcal{P}}_n^{\ell }$ be given. Any d-semifaithful code must fully cover the type class ${\mathcal{T}}_n(P^{\ell })$ with spheres of radius $nD$ (henceforth, referred to as D-spheres) centered at the various codewords. Let ${\widehat{\mathit{x}}}_1,\dots ,{\widehat{\mathit{x}}}_M\in {\widehat{\mathcal{X}}}^n$ be M codewords. The number of members of ${\mathcal{T}}_n(P^{\ell })$ that are covered by ${\widehat{\mathit{x}}}_1,\dots ,{\widehat{\mathit{x}}}_M\in {\widehat{\mathcal{X}}}^n$ is upper-bounded as follows.

(17)$\begin{array}{ccc}\hfill G& =& \left|{\displaystyle \bigcup _{i=1}^M}\left[{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}({\widehat{\mathit{x}}}_i,D)\}\right]\right|\hfill \\ & \le & {\displaystyle \sum _{i=1}^M}|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}({\widehat{\mathit{x}}}_i,D)\}|\hfill \\ & \le & M·{\displaystyle \underset{\widehat{\mathit{x}}\in \widehat{{\mathcal{X}}^n}}{max}}|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)\}|,\hfill \end{array}$

and so, the necessary condition for complete covering, which is $G\ge |{\mathcal{T}}_n(P^{\ell })|$, amounts to

(18)$\begin{array}{ccc}\hfill M& \ge & \frac{|{\mathcal{T}}_n(P^{\ell })|}{{max}_{\widehat{\mathit{x}}\in \widehat{{\mathcal{X}}^n}}|{\mathcal{T}}_n(P^{\ell })\bigcap \widehat{\mathcal{S}}(\widehat{\mathit{x}},D)\}|}\hfill \\ & =& \frac{|{\mathcal{T}}_n(Q_{*}^{\ell })|}{|{\mathcal{T}}_n(Q_{*}^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|}\hfill \\ & \stackrel{▵}{=}& M_0,\hfill \end{array}$

where the second line is from (16). Consider now a variable-length code with a codebook of size M. Let $L(\widehat{\mathit{x}})$ denote the length (in bits) of the compressed binary string that represents $\widehat{\mathit{x}}$. The number of codewords with $L(\widehat{\mathit{x}})\le logM-ϵlogn$ is upper-bounded as follows:

(19)$\begin{array}{ccc}\hfill |\{\widehat{\mathit{x}}\in {\mathcal{C}}_n:L(\widehat{\mathit{x}})\le logM-ϵlogn\}|& =& {\displaystyle \sum _{k=1}^{logM-ϵlogn}}\left|\{\widehat{\mathit{x}}:L(\widehat{\mathit{x}})=k\}\right|\hfill \\ & \le & {\displaystyle \sum _{k=1}^{logM-ϵlogn}}{2}^k\hfill \\ & =& 2^{logM-ϵlogn+1}-1\hfill \\ & <& 2n^{-ϵ}M,\hfill \end{array}$

where in the first inequality we have used the assumed one-to-one property of the mapping between the reproduction codewords and their variable-length compressed binary representations. It follows then, that for at least $M(1-2n^{-ϵ})$ out of the M codewords in ${\mathcal{C}}^n$ (that is, the vast majority of codewords), we have

(20)$\begin{array}{ccc}\hfill L({\varphi }_n(\mathit{x}))& \ge & logM-ϵlogn\hfill \\ & \ge & log{M}_0-ϵlogn\hfill \\ & =& -log\left(\frac{|{\mathcal{T}}_n(Q_{*}^{\ell })\bigcap \mathcal{S}(\mathit{x},D)|}{|{\mathcal{T}}_n(Q_{*}^{\ell })|}\right)-ϵlogn\hfill \\ & =& -log\left[{\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}{U}_{Q_{*}}(\widehat{\mathit{x}})\right]-ϵlogn,\hfill \end{array}$

where $U_{Q_{*}}$ is the uniform probability distribution across the type class $Q_{*}^{\ell }$, i.e.,

(21)$U_{Q_{*}}(\widehat{\mathit{x}})=\left\{\begin{array}{cc}\frac{1}{|{\mathcal{T}}_n(Q_{*}^{\ell })|}\hfill & \widehat{\mathit{x}}\in {\mathcal{T}}_n(Q_{*}^{\ell })\hfill \\ 0\hfill & \mathrm{elsewhere}\hfill \end{array}\right.$

We now argue that for every $\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n$

(22)$U_{Q_{*}}(\widehat{\mathit{x}})\le {exp}_2\{-LZ(\widehat{\mathit{x}})+n{\Delta }_n(\ell )\}.$

For $\widehat{\mathit{x}}\notin {\mathcal{T}}_n(Q_{*}^{\ell })$, this is trivial as the left-hand side is equal to zero. For $\widehat{\mathit{x}}\in {\mathcal{T}}_n(Q_{*}^{\ell })$, we have the following consideration. In ([37], Equation (30)), it is shown (with somewhat different notation) that a certain quantity called the (unnormalized) s-state complexity [27] of $\widehat{\mathit{x}}$, $n{\rho }_s(\widehat{\mathit{x}})$, for $s=S(\ell )\stackrel{▵}{=}(K^{\ell +1}-1)/(K-1)$, is upper-bounded by

(23)$n{\rho }_s(\widehat{\mathit{x}})\le \frac{n}{\ell }[H_{Q_{*}}({\widehat{X}}^{\ell })+1],$

where

(24)$H_{Q_{*}}({\widehat{X}}^{\ell })=-{\displaystyle \sum _{b^{\ell }\in {\widehat{\mathcal{X}}}^{\ell }}}{Q}_{*}^{\ell }(b^{\ell })log{Q}_{*}^{\ell }(b^{\ell }).$

On the other hand, $n{\rho }_s(\widehat{\mathit{x}})$ is lower-bounded (see [37], Equation (32)) by

(25)$n{\rho }_s(\widehat{\mathit{x}})\ge c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})-\frac{nlog(4S^2(\ell ))logK}{(1-{ϵ}_n)logn}-S^2(\ell )log(4S^2(\ell )),$

where ${ϵ}_n\to 0$, and so,

(26)$\frac{n}{\ell }[H_{Q_{*}}({\widehat{X}}^{\ell })+1]\ge c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})-\frac{nlog(4S^2(\ell ))logK}{(1-{ϵ}_n)logn}-S^2(\ell )log(4S^2(\ell )).$

Combining this with the inequality ([38], p. 17, Lemma 2.3),

(27)$|{\mathcal{T}}_n(Q_{*}^{\ell })|\ge {\left(\frac{n}{\ell }+1\right)}^{-K^{\ell }}·2^{n{H}_{Q_{*}}({\widehat{X}}^{\ell })/\ell },$

we have

(28)$\begin{array}{ccc}\hfill log|{\mathcal{T}}_n(Q_{*}^{\ell })|& \ge & c(\widehat{\mathit{x}})logc(\widehat{\mathit{x}})-n{\delta }_n(\ell )-K^{\ell }log\left(\frac{n}{\ell }+1\right)\hfill \\ & \ge & LZ(\widehat{\mathit{x}})-nϵ(n)-n{\delta }_n(\ell )-K^{\ell }log\left(\frac{n}{\ell }+1\right)\hfill \\ & \stackrel{▵}{=}& LZ(\widehat{\mathit{x}})-n{\Delta }_n(\ell ),\hfill \end{array}$

where

(29)${\delta }_n(\ell )=\frac{log\left[4S^2(\ell )\right]logK}{(1-{ϵ}_n)logn}+\frac{S^2(\ell )log\left[4S^2(\ell )\right]}{n}+\frac{1}{\ell },$

and where the second inequality in (28) follows from (3). The last line of (28) is equivalent to (22). It follows then, that for at least $M(1-2·n^{-ϵ})$ out of the M codewords in ${\mathcal{C}}^n$,

(30)$\begin{array}{ccc}\hfill L({\varphi }_n(\mathit{x}))& \ge & -log\left[{\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}{2}^{-LZ(\widehat{\mathit{x}})}\right]-n{\Delta }_n(\ell )-ϵlogn\hfill \\ & =& -log\left[{\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}\frac{2^{-LZ(\widehat{\mathit{x}})}}{{\sum }_{{\widehat{\mathit{x}}}^{\prime }\in {\widehat{\mathcal{X}}}^n}{2}^{-LZ({\widehat{\mathit{x}}}^{\prime })}}\right]-\hfill \\ & & log\left({\displaystyle \sum _{\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n}}{2}^{-LZ(\widehat{\mathit{x}})}\right)-n{\Delta }_n(\ell )-ϵlogn\hfill \\ & \ge & -log(U\left[\mathcal{S}(\mathit{x},D)\right])-n{\Delta }_n(\ell )-ϵlogn,\hfill \end{array}$

where in the last step we have applied Kraft’s inequality to the LZ code-length function. This completes the proof of Theorem 1.   □

4. The Achievability Theorem

The lower bound of Theorem 1 naturally suggests achievability using the universal distribution, U, for random selection of the various codewords. The basic idea is quite standard and simple: the quantity $U[\mathcal{S}(\mathit{x},D)]$ is the probability that a single randomly chosen reproduction vector, drawn under U, would fall within distance $nD$ from the source vector, $\mathit{x}$. If all reproduction vectors are drawn independently under U, then the typical number of such random selections that it takes before one sees the first one in $\mathcal{S}(\mathit{x},D)$ is of the exponential order of $1/U[\mathcal{S}(\mathit{x},D)]$. Given that the codebook is revealed to both the encoder and decoder, once it has been selected, the encoder merely needs to transmit the index of the first such reproduction vector within the codebook, and the description length of that index can be made essentially as small as $log\{1/U[\mathcal{S}(\mathit{x},D)]\}=-log(U[\mathcal{S}(\mathit{x},D)])$. We use this simple idea to prove achievability for an arbitrary distortion measure. The proof is similar to the parallel proof in [34], and it is presented here mainly for completeness.

The achievability theorem is the following.

Theorem 2.

Let $d:{\mathcal{X}}^n\times {\widehat{\mathcal{X}}}^n\to {\mathbb{R}}^{+}$ be an arbitrary distortion function. Then, for every $ϵ>0$, there exists a sequence of d-semifaithful, variable-length block codes of block length n, such that for every $\mathit{x}\in {\mathcal{X}}^n$, the code length for $\mathit{x}$ is upper-bounded by

(31) $L(\mathit{x})\le -log(U\left[\mathcal{S}(\mathit{x},D)\right])+(2+ϵ)logn+c+{\delta }_n,$

where $c>0$ is a constant and ${\delta }_n=O(n{J}^n{e}^{-n^{1+ϵ}})$.

Proof. 

The proof is based on the following simple well-known fact: Given a source vector $\mathit{x}\in {\mathcal{X}}^n$ and a codebook, ${\mathcal{C}}_n$, let $I(\mathit{x})$ denote the index, i, of the first vector, ${\widehat{\mathit{x}}}_i$, such that $d(\mathit{x},{\widehat{\mathit{x}}}_i)\le nD$, namely, ${\widehat{\mathit{x}}}_i\in \mathcal{S}(\mathit{x},D)$. If all reproduction vectors are drawn independently under U, then, for every positive integer, N:

(32)$Pr\{I(\mathit{x})>N\}={(1-U\left[\mathcal{S}(\mathit{x},D)\right])}^N=exp\{Nln(1-U\left[\mathcal{S}(\mathit{x},D)\right]\}\le exp\{-N·U\left[\mathcal{S}(\mathit{x},D)\right]\},$

and so, if $N=N_n=e^{{\lambda }_n}/U\left[\mathcal{S}(\mathit{x},D)\right]$, for some arbitrary positive sequence, $\{{\lambda }_n\}$, that tends to infinity, then

(33)$Pr\{I(\mathit{x})>N_n\}\le exp\{-e^{{\lambda }_n}\}.$

This fact will be used few times in this section.   □

For later use, we also need the following uniform lower bound to $U[\mathcal{S}(\mathit{x},D)]$: for a given $\mathit{x}$, let ${\widehat{\mathit{x}}}_0\in {\widehat{\mathcal{X}}}^n$ denote an arbitrary reproduction vector within $\mathcal{S}(\mathit{x},D)$. Then,

$\begin{array}{}(34)& \hfill U[\mathcal{S}(\mathit{x},D)]& \ge & U({\widehat{\mathit{x}}}_0)\hfill (35)& & =& \frac{2^{-LZ({\widehat{\mathit{x}}}_0)}}{{\sum }_{\widehat{\mathit{x}}\in {\widehat{\mathcal{X}}}^n}{2}^{-LZ(\widehat{\mathit{x}})}}\hfill (36)& & \ge & 2^{-LZ({\widehat{\mathit{x}}}_0)}.\hfill \end{array}$

Next, observe that $LZ({\widehat{\mathit{x}}}_0)$ is maximized by the K-ary extension of the counting sequence ([27], p. 532), which is defined as follows: for $i=1,2,\dots ,m$ (m – positive integer), let $u(i)$ denote the K-ary string of length $i{K}^i$ that lists, say, in lexicographic order, all the $K^i$ words from ${\widehat{\mathcal{X}}}^i$, and let ${\widehat{\mathit{x}}}_0=(u(1)u(2)\dots u(m))$, whose length is

(37)$\begin{array}{ccc}\hfill n& =& {\displaystyle \sum _{i=1}^m}i{K}^i\hfill \\ & =& K·{\displaystyle \sum _{i=1}^m}i{K}^{i-1}\hfill \\ & =& K·\frac{\partial }{\partial K}\left({\displaystyle \sum _{i=1}^m}{K}^i\right)\hfill \\ & =& K·\frac{\partial }{\partial K}\left(\frac{K^{m+1}-K}{K-1}\right)\hfill \\ & =& \frac{K}{{(K-1)}^2}[m{K}^{m+1}-(m+1)K^m+1].\hfill \end{array}$

The LZ incremental parsing of ${\widehat{\mathit{x}}}_0$, which is exactly $(u(1),u(2),\dots ,u(m))$, yields

(38)$c({\widehat{\mathit{x}}}_0)={\displaystyle \sum _{i=1}^m}{K}^i=\frac{K^{m+1}-K}{K-1},$

and so, considering Equation (3), it follows that $LZ({\widehat{\mathit{x}}}_0)\le (1+{ϵ}_n)nlogK$ for some ${ϵ}_n\to 0$ as $n\to \infty$. It follows then, that

(39)$U\left[\mathcal{S}(\mathit{x},D)\right]\ge 2^{-n(1+{ϵ}_n)logK}.$

Remark 1.

For an alternative to this upper bound on the LZ code length, one can slightly modify the LZ algorithm as follows: if $LZ(\widehat{\mathit{x}})\le nlogK$, use the LZ algorithm as usual, otherwise, send $\widehat{\mathit{x}}$ uncompressed using $nlogK$ bits. To distinguish between the two modes of operation, append a flag bit to indicate whether or not the data are LZ-compressed. The modified code length would then be $L{Z}^{\prime }(\widehat{\mathit{x}})=min\{LZ(\widehat{\mathit{x}}),nlogK\}+1$. Now, replace $LZ(\widehat{\mathit{x}})$ by $L{Z}^{\prime }(\widehat{\mathit{x}})$ in all places throughout this paper, including the definition of U.

Consider now an independent random selection of all reproduction vectors to form a codebook, ${\mathcal{C}}_n$, of size $M=A^n$ ($A>K$) codewords, ${\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_2,\dots ,{\widehat{\mathit{x}}}_M$, according to U. Once the codebook ${\mathcal{C}}_n=\{{\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_2,\dots ,{\widehat{\mathit{x}}}_M\}$ has been drawn, it is revealed to both the encoder and the decoder. Consider next the following encoder. As defined before, let $I(\mathit{x})$ be defined as the index of the first codeword that falls within $\mathcal{S}(\mathit{x},D)$, but now, with the small modification that if none of the $A^n$ codewords fall in $\mathcal{S}(\mathit{x},D)$, then we define $I(\mathit{x})=A^n$ anyway (and then the encoding fails). Next, we define the following probability distribution over the positive integers, $1,2,\dots ,A^n$:

(40)$u\left[i\right]=\frac{1/i}{{\sum }_{k=1}^{A^n}1/k},i=1,2,\dots ,A^n.$

Given $\mathit{x}$, the encoder finds $I(\mathit{x})$ and encodes it using a variable-rate lossless code with the length function (in bits, and ignoring the integer length constraint)

(41)$\begin{array}{ccc}\hfill L(\mathit{x})& =& -logu[I(\mathit{x})]\hfill \\ & \le & logI(\mathit{x})+log\left({\displaystyle \sum _{k=1}^{A^n}}\frac{1}{k}\right)\hfill \\ & \le & logI(\mathit{x})+log(ln{A}^n+1)\hfill \\ & =& logI(\mathit{x})+log(nlnA+1)\hfill \\ & \le & logI(\mathit{x})+logn+c,\hfill \end{array}$

where $c=log(lnA+1)$. It follows that the expected codeword length for $\mathit{x}\in {\mathcal{X}}^n$ (with respect to the randomness of the code) is upper-bounded by

(42)$\begin{array}{ccc}\hfill \mathbb{E}\{L(\mathit{x})\}& \le & \mathbb{E}\{logI(\mathit{x})\}+logn+c\hfill \\ & \le & log\mathbb{E}\{I(\mathit{x})\}+logn+c\hfill \\ & =& log\left({\displaystyle \sum _{k=1}^{A^n}}k·{\left(1-U[\mathcal{S}(\mathit{x},D)]\right)}^{k-1}·U\left[\mathcal{S}(\mathit{x},D)\right]+A^n·{\left(1-U[\mathcal{S}(\mathit{x},D)]\right)}^{A^n}\right)+logn+c\hfill \\ & =& log\left({\displaystyle \sum _{k=1}^{\infty }}min\{k,A^n\}·{\left(1-U[\mathcal{S}(\mathit{x},D)]\right)}^{k-1}·U\left[\mathcal{S}(\mathit{x},D)\right]\right)+logn+c\hfill \\ & \le & log\left\{{\displaystyle \sum _{k=1}^{\infty }}k·{\left(1-U[\mathcal{S}(\mathit{x},D)]\right)}^{k-1}·U\left[\mathcal{S}(\mathit{x},D)\right]\right\}+logn+c\hfill \\ & =& log\left(\frac{1}{U[\mathcal{S}(\mathit{x},D)]}\right)+logn+c,\hfill \end{array}$

and we denote

(43)$L^{+}(\mathit{x})\stackrel{▵}{=}log\left(\frac{1}{U[\mathcal{S}(\mathit{x},D)]}\right)+logn+c.$

Consider now the quantity

(44)$\begin{array}{ccc}\hfill E_n& \stackrel{▵}{=}& \mathbb{E}\{max({\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\mathcal{I}\{d(\mathit{x},\widehat{\mathit{X}})>nD\},\hfill \\ & & {\left[{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left(L(\mathit{x})-L^{+}(\mathit{x})-(1+ϵ)logn\right)\right]}_{+})\},\hfill \end{array}$

where the expectation is with respect to the randomness of the code ${\mathcal{C}}_n$. If $E_n$ can be upper-bounded by ${\delta }_n$, which tends to zero as $n\to \infty$, this will imply that there must exist a code for which both

(45)${\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\mathcal{I}\{d(\mathit{x},\widehat{\mathit{x}})>nD\}\le {\delta }_n$

and

(46)${\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left(L(\mathit{x})-L^{+}(\mathit{x})-(1+ϵ)logn\right)\le {\delta }_n$

at the same time. Observe that since the left-hand side of (45) is either zero or one, then if we know that it must be less than ${\delta }_n\to 0$ for some codebook ${\mathcal{C}}_n$, it means that it must vanish as soon as n is large enough such that ${\delta }_n<1$, namely, $d(\mathit{x},\widehat{\mathit{x}})\le nD$ for all $\mathit{x}$, in other words, the code is d-semifaithful. Furthermore, by (46), for the same codebook, we must have

(47)$L(\mathit{x})\le L^{+}(\mathit{x})+(1+ϵ)logn+{\delta }_n\mathit{x}\in {\mathcal{X}}^n,$

and ${\delta }_n$ adds a negligible redundancy term.

To prove that $E_n\to 0$, we first use the simple fact that the maximum of two non-negative numbers is upper-bounded by their sum, i.e.,

(48)$\begin{array}{ccc}\hfill E_n& \le & \mathbb{E}\left\{{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\mathcal{I}\{d(\mathit{x},\widehat{\mathit{X}})>nD\}\right\}+\hfill \\ & & \mathbb{E}\left\{{\left[{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left(L(\mathit{x})-L^{+}(\mathit{x})-(1+ϵ)logn)\right)\right]}_{+}\right\},\hfill \end{array}$

and, therefore, it is sufficient to prove that each one of these terms tends to zero. As for the first term, we have:

(49)$\begin{array}{ccc}\hfill \mathbb{E}\left\{{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\mathcal{I}\{d(\mathit{x},\widehat{\mathit{X}})>nD\}\right\}& \le & \mathbb{E}\left\{{\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}\mathcal{I}\{d(\mathit{x},\widehat{\mathit{X}})>nD\}\right\}\hfill \\ & =& {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}\mathbb{E}\left\{\mathcal{I}\{d(\mathit{x},\widehat{\mathit{X}})>nD\}\right\}\hfill \\ & =& {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}Pr\{d(\mathit{x},\widehat{\mathit{X}})>nD\}\hfill \\ & =& {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}{\left(1-U[\mathcal{S}(\mathit{x},D)]\right)}^{A^n}\hfill \\ & \le & {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}exp\left\{-A^{n}U\left[\mathcal{S}(\mathit{x},D)\right]\right\}\hfill \\ & \stackrel{(a)}{\le }& {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}_n}}exp\left\{-exp\left\{n\left[lnA-(1+{ϵ}_n)lnK\right]\right\}\right\}\hfill \\ & \le & J^{n}exp\left(-exp\left\{n\left[lnA-(1+{ϵ}_n)lnK\right]\right\}\right\},\hfill \end{array}$

where in (a) we have used (39). This quantity decays double-exponentially rapidly as $n\to \infty$ since we have assumed $A>K$.

As for the second term of (48), we have:

(50)$\begin{array}{ccc}& & \mathbb{E}\left\{{\left[{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left(L(\mathit{x})-L^{+}(\mathit{x})-(1+ϵ)logn\right)\right]}_{+}\right\}\hfill \\ & \stackrel{(a)}{\le }& \mathbb{E}\left\{{\left[{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left(logI(\mathit{x})-log\frac{1}{U[\mathcal{S}(\mathit{x},D)]}-(1+ϵ)logn\right)\right]}_{+}\right\}\hfill \\ & =& {\int }_0^{\infty }Pr\left\{{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left[logI(\mathit{x})-log\frac{1}{U[\mathcal{S}(\mathit{x},D)]}-(1+ϵ)logn)\right]\ge s\right\}ds\hfill \\ & =& {\int }_0^{nlogA}Pr\left\{{\displaystyle \underset{\mathit{x}\in {\mathcal{X}}^n}{max}}\left[logI(\mathit{x})-log\frac{1}{U[\mathcal{S}(\mathit{x},D)]}-(1+ϵ)logn\right]\ge s\right\}ds\hfill \\ & =& {\int }_0^{nlogA}Pr\left[{\displaystyle \bigcup _{\mathit{x}\in {\mathcal{X}}^n}}\left\{I(\mathit{x})\ge \frac{2^{(1+ϵ)logn+s}}{U[\mathcal{S}(\mathit{x},D)]}\right\}\right]ds\hfill \\ & \le & {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}{\int }_0^{nlogA}Pr\left\{I(\mathit{x})\ge \frac{2^{(1+ϵ)logn+s}}{U[\mathcal{S}(\mathit{x},D)]}\right\}ds\hfill \\ & \stackrel{(b)}{\le }& {\displaystyle \sum _{\mathit{x}\in {\mathcal{X}}^n}}{\int }_0^{nlogA}exp\{-2^s{n}^{1+ϵ}\}ds\hfill \\ & \le & J^n·(nlogA)·exp\{-n^{1+ϵ}\},\hfill \end{array}$

where in (a) we have used (41) and (43), and in (b) we have used (33). The right-most side of this chain of inequalities clearly decays as well when n grows without bound. This completes the proof.

5. Summary and Discussion

By deriving asymptotically matching upper and lower bounds, we have established the quantity $-\frac{1}{n}log(U\left[\mathcal{S}(\mathit{x},D)\right])$ as having the significance of an empirical rate–distortion function for individual sequences. While this quantity is not easy to calculate for large n, the operative meaning of our results is that we propose a universal ensemble for rate–distortion coding. According to this ensemble, the codewords are drawn independently under the probability distribution that is proportional to $2^{-LZ(\widehat{\mathit{x}})}$.

There are several observations, insights, and perspectives that should be addressed.

Relation to earlier converse bounds. The converse bound is given in terms of the probability of a sphere of radius $nD$ around the source vector $\mathit{x}$, under the universal distribution, U, defined in (4). This is intimately related to a converse result due to Kontoyiannis and Zhang ([22], Theorem 1, part i), which states that for any d-semifaithful code, there exists a probability distribution Q on ${\widehat{\mathcal{X}}}^n$ such that $L(\mathit{x})\ge -log(Q[\mathcal{S}(\mathit{x},D)])$ for all $\mathit{x}$ (see also [21]). Here, upon giving up any claims on a minority of the codewords pertaining to a given type class, we derived a lower bound of essentially the same form with the benefit of specifying a concrete choice of the distribution Q, i.e., we propose $Q=U$, the universal distribution (unlike the distribution in [22] (Section III.A), which is proportional to $2^{-L(\widehat{\mathit{x}})}$ across the codebook).

Interpretation of the main term of the bound. First, observe that in the special case of lossless compression ($D=0$), the expression $-log(U[\mathcal{S}(\mathit{x},D)])$ boils down to $LZ(\mathit{x})$, as expected. In the lossy case, since $LZ(\widehat{\mathit{x}})$ is essentially bounded by a linear function of n (see (39)), we can approximate the main term as follows:

(51)$\begin{array}{ccc}\hfill -log(U[\mathcal{S}(\mathit{x},D)])& \le & -log\left({\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}{2}^{-LZ(\widehat{\mathit{x}})}\right)\hfill \\ & =& -log\left({\displaystyle \sum _{L\ge 1}}{2}^{-L}·|\{\widehat{\mathit{x}}:LZ(\widehat{\mathit{x}})=L\}\bigcap \mathcal{S}(\mathit{x},D)|\right)\hfill \\ & \approx & {\displaystyle \underset{L\ge 1}{min}}\left\{L-log|\{\widehat{\mathit{x}}:LZ(\widehat{\mathit{x}})=L\}\bigcap \mathcal{S}(\mathit{x},D)|\right\}.\hfill \end{array}$

This expression, when normalized by n, can be viewed as a certain extension of the rate–distortion function, from the memoryless case to the general case, in the following sense. For a memoryless source P, the rate–distortion function has the following representation, which is parallel to (51):

(52)$R(D)={\displaystyle \underset{P_{\widehat{X}}}{min}}\left[H(\widehat{X})-{\displaystyle \underset{\{P_{X|\widehat{X}}:\mathbb{E}d(X,\widehat{X})\le D,P_X=P\}}{max}}H(\widehat{X}|X)\right],$

where the maximum over the empty set is understood to be $-\infty$. Indeed, if we replace U by the the uniform distribution across the first-order type pertaining to the optimal $P_{\widehat{X}}$, this is the corresponding single-letter expression of $-log(P_{\widehat{X}}\left[\mathcal{S}(\mathit{x},D)\right])$ that is obtained using the method in [38].

Comparing to the LZ description length of the most compressible $\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)$. Since our achievable bound involves LZ compression, it is interesting to compare it to the conceptually simple coding scheme that encodes $\mathit{x}$ by the vector $\widehat{\mathit{x}}$ that minimizes $LZ(\widehat{\mathit{x}})$ within $\mathcal{S}(\mathit{x},D)$. Consider the following chain of equalities and inequalities:

(53)$\begin{array}{ccc}\hfill {\displaystyle \underset{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}{min}}LZ(\widehat{\mathit{x}})& =& -log\left({\displaystyle \underset{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}{max}}{2}^{-LZ(\widehat{\mathit{x}})}\right)\hfill \\ & \ge & -log\left({\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}{2}^{-LZ(\widehat{\mathit{x}})}\right)\hfill \\ & \ge & -log\left({\displaystyle \sum _{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}}\frac{2^{-LZ(\widehat{\mathit{x}})}}{{\sum }_{{\widehat{\mathit{x}}}^{\prime }\in {\widehat{\mathcal{X}}}^n}{2}^{-LZ({\widehat{\mathit{x}}}^{\prime })}}\right)\hfill \\ & =& -log(U[\mathcal{S}(\mathit{x},D)]),\hfill \end{array}$

which means that the performance of our proposed scheme is never worse (and conceivably often much better) than that of selecting the vector $\widehat{\mathit{x}}$ with the smallest $LZ(\widehat{\mathit{x}})$ among all reproduction vectors in $\mathcal{S}(\mathit{x},D)$. The reason for the superiority of the proposed scheme is that it takes advantage of the fact that $\widehat{\mathit{x}}$ cannot be any vector in ${\widehat{\mathcal{X}}}^n$, but it must be a member of the codebook, ${\mathcal{C}}_n$, i.e., one of the possible outputs of a vector quantizer. On the other hand, in view of [27], ${min}_{\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)}LZ(\widehat{\mathit{x}})$ is essentially achievable upon compressing the output of a certain reproduction encoder (or vector quantizer) using a finite-state encoder, but a finite-state machine does not have enough memory resources to take advantage of the fact that vectors outside ${\mathcal{C}}_n$ cannot be encountered by the encoder. Another interesting comparison between the two schemes is in terms of computational complexity. While in our scheme, the encoder has to carry out typically about $1/U[\mathcal{S}(\mathit{x},D)]$ distortion calculations before finding the first $\widehat{\mathit{x}}\in \mathcal{S}(\mathit{x},D)$, in the alternative scheme the number of calculations is $|\mathcal{S}(\mathit{x},D)|$. The former is a decreasing function of D, whereas the latter is an increasing function of D. Therefore, in terms of computational complexity, the preference between the two schemes might depend on D. Specifically, for an additive distortion measure, it is easy to see that

(54)$\frac{1}{U[\mathcal{S}(\mathit{x},D)]}\stackrel{·}{\le }{exp}_2\{nR(D,P_{\mathit{x}}^1)\}$

and, by the method of types [38]:

(55)$\left|\mathcal{S}(\mathit{x},D)\right|\stackrel{·}{=}{exp}_2\{nE(D,P_{\mathit{x}}^1))\}\stackrel{▵}{=}{exp}_2[max\{H(\widehat{X}|X),\mathbb{E}d(X,\widehat{X})\le D,P_X=P_{\mathit{x}}^1\}].$

Therefore, whenever D is large enough such that $R(D,P_{\mathit{x}}^1)<E(D,P_{\mathit{x}}^1))$, it is guaranteed that the coding scheme proposed here is computationally less demanding than the alternative scheme of minimizing $LZ(\widehat{\mathit{x}})$ across $\mathcal{S}(\mathit{x},D)$.

Implementation of the random coding distribution. The universal random coding distribution is not difficult to implement. One way to achieve this is by feeding the LZ decoder with a sequence of purely random bits (fair coin tosses) until we have obtained n symbols at the decoder output. The details can be found in [36].

Universality with respect to the distortion measure. As mentioned in the Introduction, in [23,24,34], there are results on the existence of rate–distortion codes that are universal, not only in terms of the source, but also in the sense of the distortion measure. Since the proof of our achievability scheme is very similar to that of [34], it is possible to extend the achievability proof here too, so as to make our code distortion-universal for a wide class of distortion measures. This can be carried out by redefining $E_n$ to include the maximization of both terms over a dense grid of distortion functions, as was performed in [34]. We opted not to include this in the present paper since it is straightforward given the results we already have here and in [34].

Institutional Review Board Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The author declares no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

References

1. Berger, T. Rate Distortion Theory—A Mathematical Basis for Data Compression; Prentice-Hall Inc.: Englewood Cliffs, NJ, USA, 1971.

2. Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley & Sons: Hoboken, NJ, USA, 2006.

3. Gallager, R.G. Information Theory and Reliable Communication; John Wiley & Sons: New York, NY, USA, 1968.

4. Gray, R.M. Source Coding Theory; Kluwer Academic Publishers: Boston, MA, USA, 1990.

5. Viterbi, A.J.; Omura, J.K. Principles of Digital Communication and Coding; McGraw-Hill Inc.: New York, NY, USA, 1979.

6. Davisson, L.D. Universal noiseless coding. IEEE Trans. Inf. Theory; 1973; 19, pp. 783-795. [DOI: https://dx.doi.org/10.1109/TIT.1973.1055092]

7. Gallager, R.G. Source Coding with Side Information and Universal Coding; LIDS-P-937 Massachusetts Institute of Technology: Cambridge, MA, USA, 1976.

8. Ryabko, B. Coding of a source with unknown but ordered probabilities. Probl. Inf. Transm.; 1979; 15, pp. 134-138.

9. Davisson, L.D.; Leon-Garcia, A. A source matching approach to finding minimax codes. IEEE Trans. Inf. Theory; 1980; 26, pp. 166-174. [DOI: https://dx.doi.org/10.1109/TIT.1980.1056167]

10. Krichevsky, R.E.; Trofimov, R.K. The performance of universal encoding. IEEE Trans. Inf. Theory; 1981; 27, pp. 199-207. [DOI: https://dx.doi.org/10.1109/TIT.1981.1056331]

11. Shtar’kov, Y.M. Universal sequential coding of single messages. Probl. Inf. Transm.; 1987; 23, pp. 175-186.

12. Barron, A.R.; Rissanen, J.; Yu, B. The minimum description length principle in coding and modeling. IEEE Trans. Inf. Theory; 1998; 44, pp. 2734-2760. [DOI: https://dx.doi.org/10.1109/18.720554]

13. Yang, Y.; Barron, A.R. Information-theoretic determination of minimax rates of convergence. Ann. Stat.; 1999; 27, pp. 1564-1599. [DOI: https://dx.doi.org/10.1214/aos/1017939142]

14. Rissanen, J. Modeling by shortest data description. Automatica; 1978; 14, pp. 465-471. [DOI: https://dx.doi.org/10.1016/0005-1098(78)90005-5]

15. Rissanen, J. Universal coding, information, prediction, and estimation. IEEE Trans. Inf. Theory; 1984; 30, pp. 629-636. [DOI: https://dx.doi.org/10.1109/TIT.1984.1056936]

16. Merhav, N.; Feder, M. A strong version of the redundancy–capacity theorem of universal coding. IEEE Trans. Inf. Theory; 1995; 41, pp. 714-722. [DOI: https://dx.doi.org/10.1109/18.382017]

17. Ornstein, D.S.; Shields, P.C. Universal almost sure data compression. Ann. Probab.; 1990; 18, pp. 441-452. [DOI: https://dx.doi.org/10.1214/aop/1176990840]

18. Zhang, Z.; Yang, E.-H.; Wei, V. The redundancy of source coding with a fidelity criterion. I. known statistics. IEEE Trans. Inf. Theory; 1997; 43, pp. 71-91. [DOI: https://dx.doi.org/10.1109/18.567651]

19. Yu, B.; Speed, T. A rate of convergence result for a universal d-semifaithful code. IEEE Trans. Inf. Theory; 1993; 39, pp. 813-820. [DOI: https://dx.doi.org/10.1109/18.256490]

20. Silva, J.F.; Piantanida, P. On universal d-semifaithful coding for memoryless sources with infinite alphabets. IEEE Trans. Inf. Theory; 2022; 68, pp. 2782-2800. [DOI: https://dx.doi.org/10.1109/TIT.2021.3134891]

21. Kontoyiannis, I. Pointwise redundancy in lossy data compression and universal lossy data compression. IEEE Trans. Inf. Theory; 2000; 46, pp. 136-152. [DOI: https://dx.doi.org/10.1109/18.817514]

22. Kontoyiannis, I.; Zhang, J. Arbitrary source models and Bayesian codebooks in rate-distortion theory. IEEE Trans. Inf. Theory; 2002; 48, pp. 2276-2290. [DOI: https://dx.doi.org/10.1109/TIT.2002.800493]

23. Mahmood, A.; Wagner, A.B. Lossy compression with universal distortion. IEEE Trans. Inf. Theory; 2023; 69, pp. 3525-3543. [DOI: https://dx.doi.org/10.1109/TIT.2023.3247601]

24. Mahmood, A.; Wagner, A.B. Minimax rate-distortion. arXiv; 2022; arXiv: 2202.04481

25. Sholomov, L.A. Measure of information in fuzzy and partially defined data. Dokl. Math.; 2006; 74, pp. 775-779. [DOI: https://dx.doi.org/10.1134/S1064562406050401]

26. Ziv, J. Coding theorems for individual sequences. IEEE Trans. Inf. Theory; 1978; 24, pp. 405-412. [DOI: https://dx.doi.org/10.1109/TIT.1978.1055911]

27. Ziv, J.; Lempel, A. Compression of individual sequences via variable-rate coding. IEEE Trans. Inf. Theory; 1978; 24, pp. 530-536. [DOI: https://dx.doi.org/10.1109/TIT.1978.1055934]

28. Potapov, V.N. Redundancy estimates for the Lempel-Ziv algorithm of data compression. Discret. Appl. Math.; 2004; 135, pp. 245-254. [DOI: https://dx.doi.org/10.1016/S0166-218X(02)00308-6]

29. Merhav, N.; Ziv, J. On the Wyner-Ziv problem for individual sequences. IEEE Trans. Inf. Theory; 2006; 52, pp. 867-873. [DOI: https://dx.doi.org/10.1109/TIT.2005.864434]

30. Ziv, J. Fixed-rate encoding of individual sequences with side information. IEEE Trans. Inf. Theory; 1984; 30, pp. 348-452. [DOI: https://dx.doi.org/10.1109/TIT.1984.1056878]

31. Merhav, N. Finite-state source-channel coding for individual source sequences with source side information at the decoder. IEEE Trans. Inf. Theory; 2022; 68, pp. 1532-1544. [DOI: https://dx.doi.org/10.1109/TIT.2021.3133422]

32. Ziv, J. Distortion-rate theory for individual sequences. IEEE Trans. Inf. Theory; 1980; 26, pp. 137-143. [DOI: https://dx.doi.org/10.1109/TIT.1980.1056164]

33. Weinberger, M.J.; Merhav, N.; Feder, M. Optimal sequential probability assignment for individual sequences. IEEE Trans. Inf. Theory; 1994; 40, pp. 384-396. [DOI: https://dx.doi.org/10.1109/18.312161]

34. Merhav, N. D-semifaithful codes that are universal over both memoryless sources and distortion measures. IEEE Trans. Inf. Theory; 2023; 69, pp. 4746-4757. [DOI: https://dx.doi.org/10.1109/TIT.2023.3253133]

35. Cohen, A.; Merhav, N. Universal randomized guessing subjected to distortion. IEEE Trans. Inf. Theory; 2022; 68, pp. 7714-7734. [DOI: https://dx.doi.org/10.1109/TIT.2022.3194073]

36. Merhav, N.; Cohen, A. Universal randomized guessing with application to asynchronous decentralized brute–force attacks. IEEE Trans. Inf. Theory; 2020; 66, pp. 114-129. [DOI: https://dx.doi.org/10.1109/TIT.2019.2920538]

37. Merhav, N. Guessing individual sequences: Generating randomized guesses using finite-state machines. IEEE Trans. Inf. Theory; 2020; 66, pp. 2912-2920. [DOI: https://dx.doi.org/10.1109/TIT.2019.2946303]

38. Csiszár, I.; Körner, J. Information Theory—Coding Theorems for Discrete Memoryless Systems; 2nd ed. Cambridge University Press: Cambridge, UK, 2011.