当前位置：首页 >> JAVA与XML >> Review of "The Petstore Revisited: J2EE vs .NET Application Server Performance Benchmark" @ JDJ

Review of "The Petstore Revisited: J2EE vs .NET Application Server Performance Benchmark" @ JDJ -

The Middleware Company (TMC) recently published a benchmark report that compares the Sun J2EE PetStore with a functionally equivalent version created using .NET technologies. The J2EE PetStore version was improved from the original code by TMC employees, while the .NET version was written and optimized by Microsoft employees.

As this summary will show, the report is seriously flawed. One of the big points of the report was that the initial benchmarks and comparisons were done with a version of the J2EE PetStore that was "not properly optimized for performance and not meant to be benchmarked." This report supposedly fixes this since the new implementation has been "extensively optimized for performance and scalability," even claiming that they have rebuilt the PetStore "fully optimizing it for performance."

The Entity Beans
Let's start with the obvious: the use of BMP entity beans. BMP entity beans are a part of EJB's legacy, and were particularly useful in the beginning since there were no advanced CMP implementations around, and semantics and features varied greatly between implementations. In particular, BMP entity beans have something known as the 1+n loading problem, whereby doing a finder call results in one SQL call for the actual query, then one SQL call for each entity bean that was found. CMP entity beans don't have this problem since read-ahead can be used in order to eagerly load the beans.

The choice to use BMP entity beans is a factor that on its own seriously impacts the performance of the application. The claim was that the J2EE PetStore now was "fully optimized for performance," but this clearly shows that it was not.

Garbage Collection
Furthermore, the report authors note in many places that garbage collection is an important source of slowdowns in J2EE, and that concurrent garbage collection should be used if possible. While I agree that garbage collection is an important thing to consider, it should be noted that concurrent garbage collection is only preferred if it's important that the end user does not perceive any slowdowns. As is noted in Sun's own VM documentation, concurrent garbage collection yields a lower throughput than regular garbage collection, so in a benchmark you would assume that concurrent garbage collection should not be turned on.

One result of the conclusion that avoiding GC is a good thing is, of course, that you should try to minimize the number of objects created as much as possible. However, the J2EE PetStore is seriously misdesigned in this regard in one particular place: the use of value objects. If an entity bean has not changed, then two calls to the value object method (e.g., getProduct() in ProductEJB) should result in the creation of one value object that's then cached for subsequent calls. This is particularly true when read-only state is used (as in this case), and this is also, after all, how the .NET code works.

Caching
While the J2EE PetStore code goes through some serious hoops to store parts of a category product list in the user session, the .NET code simply caches the entire category product listing in an application-level cache. Read the previous sentence again. I couldn't believe my eyes when I saw this difference between the two implementations. While the J2EE code is doing finders, getting value objects, putting stuff into users' sessions, the .NET code just fetches the objects from the Cache object (that is, as long as it doesn't need to ask the business/data object Product for a category-specific listing during the cache construction). This makes the comparison between the two extremely unfair. This, along with the above mentioned 1+n problem, could easily explain the majority of the differences between the J2EE PetStore and the .NET PetStore.

Transactional Settings
We read "in the revised Middle Java PetStore 2.0, all transaction boundaries were revised to minimize the number of transactions." Presumably, this should mean that there's a minimal amount of transactions being started during the execution of the J2EE PetStore. But is this the case?

In the XML descriptor for CatalogEJB (shoppingcart_ejb.xml) we find that all methods run as TX_REQUIRED, i.e., transactions are started for each call. But is this necessary? Some of them deal with returning sets of products and categories. Those items are read-only, so dealing with them should not involve transactions. But, in the "fully optimized" J2EE PetStore, where "all transaction boundaries were revised to minimize the number of transactions," it seems this does not apply.

SQL
One serious issue that dramatically added to the Line Count metric was the absurd absence of PreparedStatement. All SQL queries were built by hand, string concatenating.

Conclusion
As you can see, the J2EE implementation never stood a chance against Microsoft's .NET implementation. There were clearly fundamental 101 errors that make the report completely null and void. You can read my full analysis at http://dreambean.com/petstore.html.