(Warning: These materials may be subject to lots of typos and errors. We are grateful if you could spot errors and leave suggestions in the comments, or contact the author at yjhan@stanford.edu.)
This lecture continues the introduction of information-theoretic lower bounds without formally developing tools. The emphasis of this lecture is that, the dependence of the fundamental limits on the problem structure may be very subtle, where the results may change significantly even with tiny changes in the problem setting. Hence, one should be very careful when applying some known lower bounds to similar problems, and it is vital to fully understand the specific problem structure. We will use a specific example of communication complexity in the theoretical computer science literatue to illustrate this phenomenon.
1. Equality Evaluation: Deterministic Communication Complexity
Alice and Bob hold vectors respectively, and they would like to check whether
or not. However, they live very far from each other, and some limited resources of communication are available. What is the minimum number of communication bits to perform this task reliably, in the sense that they always find the correct answer for any pairs
? In other words, what is the deterministic communication complexity of the distributed computing problem for the equality function
?
To answer this question, we need to formally define all possible communication protocols. Here we define a general class of blackboard communication protocols.
Definition 1 (Blackboard Communication Protocol) A blackboard communication protocol over domain
with range
is a binary tree where each internal node
is labeled by a function
or by a function
, and each leaf is labeled by an element
.
The value of the protocol on input
is the label of the leaf reached by starting from the root and walking on the tree using the following rule: at each internal node labeled by
, walk left if
and right if
; at each internal node labeled by
, walk left if
and right if
. The cost of the protocol is the height of the tree.
Intuitively speaking, there is a blackboard in front of both Alice and Bob in a blackboard communication protocol on which they can write binary bits, and this process continues sequentially where later bits can depend on the entire history. The formal definition above simply uses a binary tree to keep track of the history, and uses functions to illustrate who will be writing on the blackboard at stage
. The communication cost of a given protocol is simply the maximum length of the message written on the blackboard, where the maximum is taken over all possible input pairs
.
Now the question we are asking is: among all blackboard communication protocols which always give the correct evaluation of the equality function, what is the minimum communication cost? This is also known as the deterministic communication complexity, where “deterministic” here means zero-error.
To prove a lower bound on the deterministic complexity, we need to make the following key observation:
Lemma 2 (Copy-paste Property) If a blackboard communication protocol arrives at the same leaf node
on both inputs
and
, then on input
the protocol also arrives at
.
Proof: For each internal nodes labeled by
on the path from the root to
, the condition ensures that
. Similarly,
if the internal node is labeled by
. Then the result is immediate.
The above copy-paste property shows the following limitation of any blackboard communication protocols: the set of all input pairs arriving at any leaf node forms a rectangle, i.e., takes the form of
. Hence, if the value of the function to be computed is not a constant on large rectangles, each leaf node will need to take care of small rectangles and more leaves are required. This is the key insight of the following theorem.
Theorem 3 (Log-rank Inequality) Let
be a binary matrix defined as
for all
. Then the deterministic communication complexity of computing
is at least
, where the linear rank is understood over
.
Proof: For any leaf node which outputs
, define
. Then clearly
. By the sub-additivity of rank, we have
However, by Lemma 2, we have for any leaf, so inequality (1) implies that the number of leaf nodes is at least
, and we are done.
Applying Theorem 3 to the equality function, the matrix is a
identity matrix, and therefore the deterministic communication complexity is at least
. This is essentially tight: Alice can communicate the entire vector
to Bob using
bits and Bob then sends the 1-bit evaluation of the equality function back to Alice. Hence, the deterministic communication complexity of equality evaluation is
.
2. Equality Evaluation: Randomized Private-coin Communication Complexity
Next we consider a slight variant of the previous problem: what is the communication complexity if Alice and Bob can tolerate some maximum probability of error on all inputs? This quantity is called the randomized communication complexity. Here to introduce randomness, we assume that both Alice and Bob have access to some (private, not shared) random number generators. We also assume that
is a small constant (say,
) and suppress the dependence of the communication complexity on
.
A striking result is that, if we allow some small probability of error (even of the order ), the randomized communication complexity drops from
to
. Here is a communication protocol: by Bertrand’s postulate there exists some prime number
. Alice chooses such a
and
, evaluates the polynomial
and sends to Bob using
bits, where
. Bob then evaluates
in (2) and outputs
. Clearly, if
then Bob always outputs
correctly. If
, the map
is a non-zero polynomial with degree at most
, so it has at most
zeros in
. Consequently, Bob makes mistakes with probability at most
for any
.
The previous upper bound shows that Theorem 3 no longer holds for the randomized communication complexity. In fact, a careful inspection of the proof shows that the zero-error property is crucial for Theorem 3 to hold. However, we may still perform a deterministic simulation of any randomized protocol and reduce to the previous case.
Theorem 4 Let
and
be the deterministic and randomized private-coin communication complexity of computing
, respectively. Then
Proof: Given a randomized protocol, Alice can transmit the probabilities of arriving at all leaf nodes based on her input
. Given these probabilities, Bob can privately evaluate the probabilities of arriving at each leaf based on the inputs
(which is made possible by the product law of independent events thanks to the private-coin assumption). Then Bob sums up probabilities for leaf nodes which output
and
, respectively, and takes the majority vote. Since the error probability of the randomized protocol is at most
, the final output given by the majority vote is always correct. The proof of (3) is completed by noting that these probabilities sent by Alice can be made within precision
without hurting the majority vote.
Applying Theorem 4 to the equality function , we immediately obtain that
. Hence, the randomized private-coin communication complexity of equality evaluation is
.
3. Equality Evaluation: Randomized Public-coin Communication Complexity
We ask again the same question in the previous section, but now Alice and Bob have shared randomness. Careful readers may have noticed a subtle point used in the proof of Theorem 4, i.e., the identity for events
given by independent private randomness. Hence, the
lower bound does not directly apply to the public-coin scenario, and the public-coin protocols may perform much better. This is indeed the case – the randomized public-coin communication complexity is actually
.
There is nothing to prove for the lower bound. For the upper bound, here is a simple procotol: both Alice and Bob draw (the same collection of) independent vectors . Then Alice sends an
-bit vector
to Bob, and Bob claims that
if and only if
for all
. Clearly, this protocol errs with probability at most
on all inputs. Hence, the randomized public-coin communication complexity of equality evaluation is
.
4. Equality Evalution: Randomized One-round Private-coin Communication Complexity
Now we consider a final variant of the distributed equality evaluation problem, where we assume a randomized private-coin protocol but restrict the protocol to be only one-round. Recall that the previous blackboard communication protocol allows for multiple rounds of interaction, i.e., Alice may provide feedbacks to the messages transmitted by Bob, and vice versa. In the one-round scenario, Alice and Bob can only send -bit messages once to a central referee, and the target is to minimize the communication cost
while controlling the maximum error probability below
. Then what is the communication complexity in this scenario?
Surprisingly, the communication complexity is , which is different from all previous ones. We first prove this lower bound. Notice that the one-round protocol has a more concise representation than a binary tree, i.e., any randomized strategy of Alice or Bob may be written as stochastic mappings from
to
. In particular, the copy-paste property in Lemma 2 becomes insufficient for our purposes. Let
be the transition matrices of the stochastic mappings of Alice and Bob, respectively, and
be the matrix with
being the probability that the referee outputs
given messages
. Now
denotes the probability of outputting
on inputs
, the correctness property ensures that
where denotes the
identity matrix, and
denotes the
norm of vectors. Hence, (4) implies that
is an approximate non-negative matrix factorization of the identity matrix, and
is the approximate non-negative rank of the identity matrix. The following result in linear algebra provides a lower bound on
.
Theorem 5 Let
. If there exist stochastic matrices
and a non-negative matrix
such that
, then
The proof of Theorem 5 is quite involved, and we refer interested readers to the reference cited in bibliographic notes. Now is an immediate consequence of (4) and Theorem 5.
To show the upper bound , the idea is to simulate a randomized protocol by deterministic ones. Recall that in the previous section, there is a randomized public-coin protocol which requires
bits of communication. Let
be
independent deterministic realizations of the randomized protocol with maximum error probability
, then by Hoeffding’s inequality, for any fixed input
we have
Hence, by (5) and the union bound over possible inputs, for fixed
we may simply choose
such that the average probability of error of
deterministic protocols on each inputs is at most
.
Now the private-coin protocol is as follows: put the above deterministic protocols in a
matrix. Alice draws
, runs all protocols in
-th row and sends all outputs to the referee (together with
). Similarly, Bob draws
and runs all protocols in
-th column. The central referee simply looks at the protocol at the
-th entry, whose probability of error is exactly the average probability of error of these
deterministic protocols, which is at most
. Meanwhile, the communication complexity of this protocol is
, as desired.
5. Bibliographic Notes
Communication complexity is a central topic in theoretical computer science. The blackboard communication protocol is proposed by Yao in 1979, where some central tools (including Theorems 3 and 4) are available in the excellent book by Kushilevitz and Nisan (1996). The proof of Theorem 5 is given by Newman and Szegedy (1996), and the simulation idea is due to Newman (1991).
- Andrew Chi-Chih Yao, Some complexity questions related to distributive computing (preliminary report). In Proceedings of the eleventh annual ACM symposium on Theory of computing, pages 209–213. ACM, 1979.
- Eyal Kushilevitz and Noam Nisan, Communication complexity. Cambridge University Press. 1996.
- Ilan Newman and Mario Szegedy, Public vs private coin flips in one round communication games, In Proceedings of the 28th annual ACM symposium on Theory of computing, pages 561–570. ACM, 1996.
- Ilan Newman, Private vs. common random bits in communication complexity. Information processing letters 39.2 (1991): 67–71.
One thought on “Lecture 2: Importance of Problem Structures”