Showing posts with label parallel computing. Show all posts
                                  Showing posts with label parallel computing. Show all posts

                                  Wednesday, August 17, 2011

                                  Parallelism On The Cloud And Polygot Programmers

                                  I am very passionate about the idea of giving developers the control over parallelism without them having to deal with the underlying execution semantics of their code.

                                  The programming languages and the constructs, today, are designed to provide abstraction, but they are not designed to estimate the computational complexity and dependencies. The frameworks such as MapReduce is designed not to have any dependencies between the computing units, but that's not true for the majority of the code. It is also not trivial to rewrite existing code to leverage parallelism. As, with the cloud, when the parallel computing continues to be a norm rather than an exception, the current programs are not going to run any faster. In fact, they will be relatively slower compared to other programs that would leverage parallel computation. Robert Harper, a Professor of Computer Science at Carnegie Mellon University recently wrote an excellent blog post - parallelism is not concurrency. I would encourage you to spend a few minutes to read that. I have quoted a couple of excerpts from that post.

                                  "what is needed is a language-based model of computation in which we assign costs to the steps of the program we actually write, not the one it (allegedly) compiles into. Moreover, in the parallel setting we wish to think in terms of dependencies among computations, rather than the exact order in which they are to be executed. This allows us to factor out the properties of the target platform, such as the number, p, of processing units available, and instead write the program in an intrinsically parallel manner, and let the compiler and run-time system (that is, the semantics of the language) sort out how to schedule it onto a parallel fabric."

                                  The post argues that language-based optimization is far better than machine-based optimization. There's an argument that the machine knows better than a developer what runs faster and what the code depends upon. This is why, for relational databases, the SQL optimizers have moved from rule-based to cost-based. The developers used to write rules inside a SQL statement to instruct the optimizer, but now the developers focus on writing a good SQL query and an optimizer picks a plan to execute the query based on the cost of various alternatives. This machine-based optimization argument quickly falls apart when you want to introduce language-based parallelism that can be specified by a developer in a scale-out situations where it's not a good idea to depend on a machine-based optimization. The cloud is designed based on this very principle. It doesn't optimize things for you, but it has native support for you to introduce deterministic parallelism through functional programming.

                                  "Just as abstract languages allow us to think in terms of data structures such as trees or lists as forms of value and not bother about how to “schedule” the data structure into a sequence of words in memory, so a parallel language should allow us to think in terms of the dependencies among the phases of a large computation and not bother about how to schedule the workload onto processors. Storage management is to abstract values as scheduling is to deterministic parallelism."

                                  As far as the cloud computing goes, we're barely scratching the surface of what's possible. It's absolutely absurd to assume that the polygot programmers will stick to one programming model and learn to spot difference between parallelism and concurrency. The language constructs, annotations, and runtime need to evolve to help the programmers automate most of these tasks to write cloud-native code. These will also be the core tenants of any new programming languages and frameworks. There's also a significant opportunity to move the existing legacy code in the cloud if people can figure out a way to break it down for computational purposes without changing it i.e. using annotations, aspects etc. The next step would be to simplify the design-deploy-maintain life cycle on the cloud. If you're reading this, it's a multi-billion dollars opportunity. Imagine, if you could turn your implementation-specific concurrent access to resources into abstract deterministic parallelism, you can indeed leverage the scale-out properties of cloud fairly easily since that's the guiding principle behind the cloud.

                                  There are other examples you would see where people are moving away from an implementation-centric approach to an abstraction that is closer to the developers and end users. The most important shift that I have seen is from files to documents. People want to work on documents; files are just an instantiation of how things are done. Google Docs and iPad are great examples that are document-centric and not file-centric.

                                  Photo courtesy: bass_nroll on Flickr

                                  Friday, September 12, 2008

                                  Google Chrome Design Principles

                                  Many of you would have read the Google Chrome comic-strip and also would have test driven the browser. I have been following few blog posts that have been discussing the technical and business impact but let's take a moment and look at some of the fundamental architectural design principles behind this browser and its impact on the ecosystem of web developers.
                                  • Embrace uncertainty and chaos: Google does not expect people to play nice. There are billions of pages with unique code and rendering all of them perfectly is not what Google is after. Instead Chrome puts people in charge of shutting down pages (applications) that do not behave. Empowering people to pick what they want and allow them to filter out the bad experience is a great design approach.
                                  • Support the journey from pages to applications to the cloud: Google embraced the fact that the web is transitioning from pages to applications. Google took an application-centric approach to design the core architecture of Chrome and turned it into a gateway to the cloud and yet maintained the tab metaphor to help users transition through this journey.
                                  • Scale through parallelism: Chrome's architecture makes each application a separate process. This architecture would allow Chrome to better tap into the multi-core architecture if it gets enough help from an underlying operating system. Not choosing a multi-threaded architecture reinforces the fact that parallelism on the multi-core is the only way to scale. I see an opportunity in designing a multi-core adaptation layer for Chrome to improve process-context switching since it still relies on a scheduler to get access to a CPU core.
                                  • Don't change developers' behavior: JavaScript still dominates the web design. Instead of asking developers to code differently Google actually accelerated Javascript via their V8 virtual machine. One of the major adoption challenges of parallel computing is to compose applications to utilize the multi-core architecture. This composition requires developers to acquire and apply new skill set to write code differently.
                                  • Practice traditional wisdom: Java introduced a really good garbage collector that was part of the core language from day one and did not require developers to explicitly manage memory. Java also had a sandbox model for the Applets (client-side runtime) that made Applets secured. Google recognized this traditional wisdom and applied the same concepts to Javascript to make Chrome secured and memory-efficient.
                                  • Growing up as an organization: The Chrome team collaborated with Android to pick up webkit and did not build one on their own (actually this is not a common thing at Google). They used their existing search infrastructure to find the most relevant pages and tested Chrome against them. This makes it a good 80-20 browser (80% of the people always visit the same 20% of the pages). This approach demonstrates a high degree of cross-pollination. Google is growing up as an organization!

                                  Thursday, April 10, 2008

                                  How is rise of massive parallel processing capabilities changing the dynamics of computing

                                  The rise of massive parallel processing capabilities that multi-core architecture brings in has plentiful opportunities and challenges. The recent $20m investment by Microsoft and Intel into the parallel computing research has created quite a buzz around this topic. How well we can utilize the multi-core architecture without rewriting any of the applications is a major challenge ahead since most of the applications are not designed to leverage the potential of multi cores to its fullest extent. The current multi-threaded applications leverage parallel processing capabilities to some extent but these threads do not scale beyond few cores and hence the current applications won't run any slower on more cores but they will run relatively slower compared to the new applications that can leverage this parallel processing architecture. The best way to cease an opportunity of utilizing the multi cores is to have a single theaded application seamlessly utilizing the potential of a multi-core architecture. This does require significant work to rewrite the algorithms and middleware but this is essentially a long tail and has significant business value proposition.

                                  The very concept of single-threaded application relying on concurrency at the algorithms level is going to challenge many developers since this is fundamentally a different way of looking at the problem. The algorithm design approaches have been changing to make algorithms explicitly aware of the available multi cores so that the algorithm can dispatch data on a random core without worrying about the issues related to multi-threading such as deadlocks, locking etc and let threads communicate with each other using asynchronous messages without sharing any common state. The algorithms always work better if it knows more about the data. If this were not to be the case, the brute-force algorithm would be the best one to solve any problem. More you know about the data, you can fine tune the algorithm and that would help you discover some good insights and that would further tighten the algorithm. The increasingly efficient processing capabilities could help deal with a large set of data early on without investing too much time upfront in an algorithm design to discard certain data. When you add the abundant main memory into this mix it has profound implications since the algorithms that were originally designed to access data from a disk are not efficient any more now that the data that they need is always available in the main addressable memory with different data structures and indexes. The cryptography algorithms have been designed to make sure that the attack cannot be completed in a reasonable amount of time given the plentiful resources. We should look at these algorithm design principles to do the opposite - how we can replicate the reverse semantics in other domains to actually make use of massive parallel processing capabilities.

                                  I foresee an innovation opportunity in new functional languages such as Erlang and Haskell or new runtime for current dynamic and object-oriented languages to tap into this potential. The companies such as RapidMind and also indicate growing interest in this area. Initially the cheap storage, followed by massive parallel processing capabilities, and main-memory computing is going to change the computing dynamics and redefine many things in coming years. We are ripe for the change. The whole is greater than sum of its parts. Yes, indeed.

                                  Friday, June 1, 2007

                                  Moore's law for software

                                  Software design has strange relationship with the computing resources. If the resources are low it is difficult to design and if the resources are in abundance it is a challenge to utilize them. It is rather odd to ask the designers and developers to have , but this is true and it is happening.

                                  The immense computing resources have opened up a lot of opportunities for the designers and developers to design agile and highly interactive web interfaces by tapping into this computing cloud. Effective resource utilization by software is by far lagging the fast growing computing resources. Google has successfully demonstrated a link between the humongous cloud infrastructure and the applications that effectively use these resources. Gmail and Google Maps are examples of agile and highly interactive interfaces that consumes heavy resources. Google's
                                  MapReduce is an example of effective utilization of the computing resources by designing the search to use heavy parallelization. One of the challenges that designers face these days is to be able construct an application from an interaction perspective such that it can actually use the available resources effectively to provide better use experience. Traditionally the performance tuning is all about fixing software to perform faster without adding extra computing resources. The designers and developers now have a challenge to actually use the resources. The cloud computing is going to be more and more relevant as various organizations catch up on Web 2.0 and Enterprise 2.0. Google, Yahoo, Salesforce, and Microsoft are betting on their huge infrastructure that can deliver the juice required for their applications. Cloud computing is not just about hardware - it is about the scale of computing and the infrastructure that is required to get to that scale such as physical location, energy and cooling requirements, dark fiber etc.

                                  Not every single piece of code in software can be parallelized. Developers hit a set of serial tasks in the code flow for many dynamic conditions. Semantic search is a classic example that has many challenges to use parallel computing resources since you do end up serializing certain tasks due to the dynamic nature of many semantic search engines and their abilities of natural language processing. Cognitive algorithms are not the same as statistical or relevancy algorithms and require a radically different design approach to effectively utilize the available resources.

                                  Intel has been pushing the industry to improve performance on its multi core CPU . Microsoft recently announced an initiative to redesign the next Windows for multiple cores. The design is not just about one, two, or three cores. The resources are going to increase at much faster pace and software designers and developers are late to react and follow this massive computing. Computing in a cloud requires a completely different kind of approach in software design and there are some great opportunities to innovate around it.


                                                                  Mobile phone



                                                                  Buy a car