Graduate Algorithms

This lecture covered the first few sections of Chapter 7 from the textbook. We learnt how to define a flow network, the notion of a maximum flow, how to construct an augmenting path, and the Ford Fulkerson algorithm.

Today, we’ll cover linear programming, which apart from being one of the most important questions in operations research, is also a tool for generating lower bounds for NP-hard problems automatically. We’ll see how LPs can help us get a(nother) 2-approximation for Vertex Cover.

Today’s lecture will continue our study of approximations. This time we will consider the travelling salesman problem in metric spaces  (Section 7.7 of these notes), and arbitrarily good approximations for KNAPSACK (KT 11).

In this lecture, we covered the simple 2-approximation for computing the vertex cover. In class, we proved the approximation ratio based on a simple local argument: namely that for each pair of vertices the algorithm chooses, OPT must pick at least one of them. We also looked at the problem of minimizing makespan which has the distinction of yielding one of the very first approximation algorithms, a 2-1/m approximation due to Ron Graham in 1966.

In both of the above cases, we emphasized the need for a lower bound on OPT, so as to reason about the approximation ratio guaranteed by our method. We were able to construct such a lower bound directly by inspection of the input or the algorithm. We then moved onto the problem of SET COVER, where such a lower bound is much harder to come by, and is constructed bit by bit from the steps of the algorithm.

The treatment of the SET COVER analysis in class differs slightly from the textbook. These notes are a useful guide to the method of analysis we used in class.

I’ll be using Chapter 11 from the textbook as a guide for our journey through approximation algorithms. However, I’d like to set the foundations with some basic material that I’ll draw from other sources. Since I don’t have comprehensive lecture notes yet, here are some links, with page numbers to look at:

• Rajeev Motwani’s (old) lecture notes on approximation algorithms. The notes are old, but the fundamentals haven’t really changed. I’ll be drawing on material from Chapter 1 (pages 1-33: the definitions, and the impossibility results).
• Sariel Har-Peled’s notes on approximation algorithms. The relevant material is in pages 1-2 (the greedy approximation algorithm for VERTEX COVER)

Wednesday’s lecture will be on the dreaded topic of NP-hardness. Although the concept was first established as recently as 1971, when the Cook-Levin theorem showed the existence of an NP-complete problem, there are innumerable NP-Complete problems lurking out there in the wild, no matter what field of research you go into (let alone in computer science). According to Christos Papadimitrious, 6000 new NP-Completeness results are proved each year. Here are some examples, taken from Kevin Wayne, of NP-Complete problems in the wild. Where possible, I’ve linked to actual documentation of this fact.

• Aerospace engineering: optimal mesh partitioning for finite elements.
• Biology: protein folding.
• Chemical engineering: heat exchanger network synthesis.
• Civil engineering: equilibrium of urban traffic flow.
• Economics: computation of arbitrage in financial markets with friction.
• Electrical engineering: VLSI layout.
• Environmental engineering: optimal placement of contaminant sensors.
• Financial engineering: find minimum risk portfolio of given return.
• Game theory: find Nash equilibrium that maximizes social welfare.
• Genomics: phylogeny reconstruction.
• Mechanical engineering: structure of turbulence in sheared flows.
• Medicine: reconstructing 3-D shape from biplane angiocardiogram.
• Operations research: optimal resource allocation.
• Physics: partition function of 3-D Ising model in statistical mechanics.
• Politics: Shapley-Shubik voting power.
• Pop culture: Minesweeper consistency, and Tetris.
• Statistics: optimal experimental design.

All of which goes to say, it’s important to understand NP-hardness, because the odds are high that you’ll bump into an NP-hard problem sooner rather than later.

Update: Scott’s question in class was: how do we express the operation of sorting as a decision problem in the form described ?

Answer: One way of doing this is to define the language SORT as the set of all pairs ($latex \sigma,\mu$), where $latex \sigma=(x_1,x_2,\ldots{x_n})$, $latex \mu=(y_1,y_2,\ldots{y_n})$, and $latex (\sigma,\mu)$ is in SORT if $latex \mu$ equals $latex \sigma$ in sorted order. Notice that you can check membership in $latex O(n\log{n})$ by sorting $latex \sigma$ in $latex O(n\log{n})$ time and verifying that it equals $latex \mu$ in linear time, but it’s not clear that you can do anything any faster. You can check that $latex \mu$ is sorted in linear time, but how do you check that it contains exactly the same multi-set of elements as $latex \sigma$ ?

On Monday, we’ll cover the various algorithms for computing a minimum spanning tree of a graph. A very nice applet that demonstrates both Prim’s and Kruskal’s algorithm for the MST can be found here.

Note: Monday’s lecture will be in MEB 3147, instead of MEB 3105. This is a one-time shift, only for this Monday’s lecture.

Some useful links:

• A proof of Cayley’s theorem by Edsger Dijkstra.
• Priority queue applet (as implemented via a heap or array)
• The Union Find data structure (two applets)

Slides for today’s lecture are a lightly modified version of Kevin Wayne’s slides (pdf) accompanying the textbook.

Today’s lecture covered the basics of dynamic programming, from Ch. 6 of KT. We looked at weighted interval scheduling, curve fitting, and SUBSET-SUM, and examined how to obtain solutions via dynamic programming for each of these problems.

The Wikipedia reference article on dynamic programming describes some of the history of the method, as well as providing examples of where it occurs out in the wild. Some of the more well known examples include

• BLAST, the genomic sequence analyzer
• The Viterbi algorithm for calculating likely paths in Markov chains
• The Duckworth/Lewis method for prorating required targets in one-day cricket games when rain intervenes.

The subtlety of why the SUBSET-SUM dynamic program does not run in polynomial time will be addressed in more detail when we cover NP-hardness. Remember though that “n”, the size of the input, is most precisely modelled (at least for complexity-theoretic purposes) as the number of bits needed to write down the input in a “reasonably compact” encoding. I could obviously pad any input string with 0s and 1s to make it seem longer, hence the “reasonably compact” requirement.

For example, consider a graph on n vertices and m edges. Since there are n vertices, we need $latex \log{n}$ bits to describe each vertex, and $latex 2log{n}$ bits to describe each edge. Summing this up, we need a total of $latex n\log{n}+2m\log{n}$ bits, which is $latex O((n+m)\log{n}$. This is what we expect, except for the extra $latex \log{n}$ factor.

It turns out that in “normal” graph algorithms, we cheat a little by assuming that we can perform operations on vertices (like comparing two vertices) in constant time, which would contradict the fact that a vertex representation takes $latex \log{n}$ bits. This is one aspect of the RAM (Random Access Machine) model of computation, where we assume that operations on single cells of memory can be performed in constant time, regardless of the size of the values. What this means is that instead of viewing the space needed by a vertex to be $latex \log{n}$ bits, we can pretend that it uses a constant number of bits, in which case the total number of bits needed to write down a graph reduces to $latex O(n+m)$. For almost all algorithms that we analyze, we implicitly use the RAM model of computation. This has its own problems: we can solve NP-hard problems in polynomial time in a RAM model (!) augmented with bitwise operators, but as long as you aren’t doing bizarre things with bits and encodings of numbers, you’re safe.

Numbers however are trickier. As I explained in class, the space needed to store a value W is $latex \log{W}$, and so any algorithm that takes W as input and runs in time super polynomial in $latex \log{W}$ does not run in polynomial time in the size of the input.

The content of this lecture is drawn from multiple sources, and so I’m providing a handout here.

WARNING: this is a nontrivial example of divide-and-conquer and may require thought-like substances.

Update: The recursive method of splitting a polynomial into even and odd parts works for any set of evaluation points. So why do we need to use the complex roots of unity ? The answer is in the Halving Lemma. It allows us to reduce the number of evaluations needed by half at each recursive step. If this were not the case, we’d be evaluating the same number of points at each recursive level, and if we solve the resulting recurrence, we obtain an algorithm that needs $latex n^2$ time to run. I’ve added a small section explaining this at the end of Section 4.4.3 in the notes.

In Monday’s lecture, we will cover material from Ch. 5 of the textbook. Specifically, we will examine the role of divide-and-conquer in designing efficient algorithms for mergesort (5.1), multiplying numbers (5.5), and computing the closest pair of a set of points in the plane (5.4).