However Hadoop and Datarush offer models which may, at least for the time being, be an acceptable way of programming of large, multi-core systems for a certain (albeit fairly large and well-heeled) sector of the market. Pervasive’s Datarush doesn’t presuppose much about the underlying architecture at all, just that it can run Java. Hadoop is slightly different but makes little assumption about the underlying hardware. Both are in fact highly viable approaches to the issue of building high throughput, data intensive, commercial applications.

What they aren’t is true programming systems.

The issue with most of the technologies cited by Denis is that they are largely for specialists. They are very popular in a variety of incarnations in areas such as HPC (where we are talking about specialised areas such as automotive and aeronautic engineering, big science and similar areas); where they have largely failed to penetrate is mainstream applications. It is because of the expediencies of the hardware industry that processor architectures have been dominated by shared memory architectures. However most manufacturers are looking at shared-memory and some have developed or acquired substantial expertise in that direction.

For technical reasons it seems unlikely that the shared memory scenario can persist and that ultimately shared-nothing will come to dominate. When that happens environments like the Datarush libraries and Hadoop will still be useful but there will be a need for technologies that will bring current technologies and their derivatives, most likely the latter, to the general purpose user.

Is your topic limited to large dataset problems using a shared-nothing parallel model implemented using a Map-Reduce technique? If so, then Google MapReduce, Greenplum, GridGain, Hive, Holumbus, Mars, Misco, Plasma MapReduce, Qizmt, Sector/Sphere, and Twister are suitable comparisons.