This rather wonderful blog post on debugging a horrible memory management issue with MySQL databases illustrates one of the incredibly complex subtleties that the shared memory model presents.
Our systems already have nonuniform memory architectures, and will increasingly and more severely (e.g. no cache coherency) in the future. If you want to add more processors, it’s unavoidable due to the von Neumann Bottleneck.
I don’t think there’s any way for the shared-memory model to account for this, without just turning into the message-passing model. And if you don’t account for it, insidious problems like the “swap insanity” one begin to show up, on top of sacrificing performance.
Another day, I might pontificate more about this, but today, I just wanted to pass along that interesting link, since I thought it was a wonderful example.