|
ObjectWatch Newsletter # 42 Feel free to forward this newsletter, but don't make any changes. Thanks. There are now over 17,000 subscribers to this newsletter and over 68,000 readers! For your own free email subscription, send mail to sub@objectwatch.com saying: subscribe your-name, your-email For example, subscribe William Shakespeare,william@objectwatch.com To unsubscribe, send mail to sub@objectwatch.com saying unsubscribe your-name, your e-mail CONTENTS
QUOTATION OF THE MONTH:J2EE WEB SERVICE STANDARDS" Many observers assume that J2EE includes support for Web services, but an examination of the 600-plus pages of the J2EE specification reveals that Web services is never specifically mentioned. Those who wish to architect solutions that include both J2EE and Web Services will have to look outside the specification for guidance." The Software Developers Web Services Report, November 2002, by Warren Keuffel The winner of the Quotation of the Month Contest is Jon R. Gribskov who gets a choice of a personally autographed copy of Roger Sessions's book, "COM+ and the Battle for the Middle Tier", Roger Sessions's upcoming book, "Software Fortresses; Modeling Enterprise Architectures", or the ObjectWatch BBQ apron. Send in your nominations for Quotation of the Month to roger@objectwatch.com, and you too can get a personally autographed book/apron! MAIN ARTICLE:J2EE VERSUS .NET; THE LATEST BENCHMARKSpeed, to put it simply, is superfluous. Easy to say, sitting at Starbucks, surveying the diurnal drift to Dell, IBM, and other destinations downstream. Were I sitting in one of those malodorous automobiles waggling their way like overweight fish down Highway 183, I might not share this view. But I am not. I am comfortably sipping my doppio macchiato. And from this vantage point, it is clear. Nobody at Starbucks cares about speed. I don't even care about speed. There are many things I do care about: the height of the foam, the temperature of the espresso, the availability of my favorite chair. But whether it takes my personal barista, KC, one minute or five to make my doppio? Not an issue. I'll be sitting in the sun over there, thank you, just yell when it is ready. Not only do I not care about speed when drinking doppios, I don't care about it when building fortresses. Software fortresses, that is. How fast a software fortress processes a request is almost never an issue. Most competent fortress architects can design a system that is "fast enough." There are many other problems that might conspire to debilitate a software fortress, but speed is rarely one of them. For those readers not up to speed (no pun intended) on the idea of software fortresses, you might want to checkout "The Software Fortress Model" in The ObjectWatch Newsletter #40 http://www.objectwatch.com/issue_40.htm). Given that I have so little interest in speed, why, you might ask, am I dedicating a whole article to a recent performance benchmark comparing J2EE to .NET? Especially one that is only peripherally related to software fortresses? It turns out that this benchmark has many important lessons that are relevant to not only software fortresses, but to all other approaches to building enterprise systems. The benchmark in question is called "J2EE and .NET Application Server and Web Services Benchmark" produced by a company called The Middleware Company. The Middleware Company (TMC) is hardly an unbiased observer of the J2EE/.NET wars. TMC is headed by Ed Roman who has written several important books about Java. TMC makes its money selling J2EE training and consulting. TMC is the owner of TheServerSide.com, a premier web site for information on J2EE. Still, despite TMC's heavy J2EE partiality, the TMC benchmark answers some important questions. Let me start by describing the overall benchmark. The benchmark is based on a contrived application known as Java Pet Store. This application was first introduced by Sun as a design pattern for how to build a highly scalable enterprise caliber web application. Within six months of the release of Java Pet Store, Microsoft released a functionally equivalent version known as .NET Pet Store and claimed much better performance. The Java community disputed this, claiming that the original Java Pet Store was not as optimized as it could have been, despite the fact that Sun touted this application as a role model for enterprise systems. TMC decided to redo the Java benchmark, throwing at it every possible optimization of J2EE. Keep in mind that TMC is the leading company offering architect level training in J2EE. If anybody can optimize a J2EE application, TMC can. TMC's ability to optimize the Java Pet Store application was impressive. Their rewritten J2EE code performed 17 times better than the original blueprint code issued by Sun. TMC does not have the same level of expertise in .NET as it does in J2EE. For this reason they approached Microsoft and offered to let them optimize the .NET Pet Store application for this benchmark. Had they not done so, the .NET community would have quite reasonably cried "foul!" The TMC benchmark considered two J2EE products identified only as Application Server A and Application Server B. The reason, TMC said, for doing this was because of licensing restrictions. Apparently the major J2EE vendors include a licensing restriction that prohibits anybody from using their product in a public benchmark, a prohibition that is not included in the .NET licensing. Based on an analysis of the vendor specific portions of the publicly available TMC benchmark code, the Java community has overwhelmingly concluded that Application Server A is BEA's WebLogic and Application Server B is IBM's WebSphere. This is consistent with what little TMC did say about products A and B: "the products chosen represent widely used, state-of-the art, commercially marketed enterprise J2EE application servers." WebLogic and WebSphere are, in my experience, the only widely used implementations of J2EE within enterprise applications. In this article, I will assume that the Java community is correct and that product A and B are WebLogic and WebSphere respectively. If either vendor cares to deny this, let me know and I will correct this in a future newsletter. The TMC benchmark tested four scenarios, each looking at different aspects of a large commercial system. I will identify these scenarios by the acronyms BB (browser-based), DTP (distributed transaction processing), DSOAP (Direct SOAP), and ISOAP (Internet SOAP). Briefly, I'll describe each scenario. For more information, refer to the benchmark itself. BB (browser-based) consists of two types of browser based clients in an equal mix. The first type of client is a read-only Pet Store visitor. In a software fortress architecture, this type of user is creating workload directed primarily at the presentation fortress. The second type of client is making Pet Store purchases. This adds slightly more backend load and most of that read/write. In a software fortress architecture this adds more involvement by a business application fortress, but still mostly targets the presentation fortress. DTP (distributed transaction processing) mimics a workload almost exclusively in the traditional middle tier, and specifically, a middle tier processing distributed transactions. In a software fortress architecture this corresponds to a workload directed at a business application fortress whose data strongbox spans two discrete databases. DSOAP (Direct SOAP) mimics two different enterprise systems communicating with each other over SOAP. In a software fortress architecture, this corresponds to a business application fortress accessed through a heterogeneous synchronous drawbridge. ISOAP (Internet SOAP) mimics a SOAP request coming in over the Internet (rather than the intranet, as implied in DSOAP). In a software fortress architecture, this corresponds to workload targeting a web service fortress with minor backend work being done by a business application fortress and a homogeneous synchronous drawbridge connecting the two. To summarize which software fortress artifacts are primarily exercised by which scenarios, we have:
This is a nice mix. It exercises large chunks of a typical enterprise software fortress architecture. The major missing piece is the heterogeneous asynchronous drawbridge. The backend database for all scenarios was SQLServer for .NET and Oracle for J2EE. This is certainly fair for WebLogic, which frequently uses Oracle, and for .NET, which almost exclusively uses SQLServer. It may not be as fair to WebSphere, which would more commonly use DB2. The benchmark calls itself "J2EE and .NET Application Server and Web Services Benchmark" but in fact, there are few conclusions we can draw about J2EE as a whole. The benchmark only considered two J2EE systems and those two systems that it did consider had wildly different results. It is quite likely that, had a third J2EE system been included, its results would have been very different from either of the two J2EE systems that were included. A better interpretation of the benchmark is that of a comparison between WebLogic, WebSphere, and .NET. Remember, however, that the supposition that WebLogic and WebSphere were benchmarked at all is supported only by strong circumstantial evidence. Not only did the benchmark limit the number of J2EE vendors, it also limited the number of operating systems. Neither WebLogic nor WebSphere were benchmarked on anything other than Windows and Linux. There is one benchmark result for which WebLogic or WebSphere might have done better had it been run on a mainframe system, which I will point out as I go along. Limiting the operating systems, of course, had less impact on .NET, which only runs on Windows. There are four questions that we can now reasonably answer, given the information in the TMC benchmark. These questions are:
I'll go through these one by one. LINUX VERSUS WINDOWSTMC tested both WebSphere and WebLogic on both Windows 2000 and RedHat Linux 7.2. TMC found that WebLogic performed "noticeably" better on Windows. For WebSphere, TMC found that the operating system made no difference to performance. To simplify the benchmarks, TMC used Windows for WebSphere as well. While the choice between Linux and Windows may have been irrelevant to performance, it is definitely relevant to overall system cost. Windows, for the server machine configuration used in the TMC benchmark, costs $5,990. Linux costs nothing. Surely Linux is cheaper. Isn't it? TMC broke down the cost of the server machine into three components: the hardware, the operating system (Windows or Linux), and the infrastructure (.NET, WebLogic, or WebSphere). Since the hardware was the same in all benchmarks, the differentiating costs were due to the operating system and the infrastructure. It turns out that the cost of the operating system is relatively insignificant in the overall server costs. Of the total WebLogic server cost of $76,990, only $5,990 was attributable to Windows. Of the total WebSphere server cost of $84,990, again, only $5,990 was for Windows. In neither case was the cost of Windows more than 8% of the total server cost. However the use of Linux does have one dramatic cost consequence. It eliminates the possibility of using .NET as the application server infrastructure. Since Linux does not have a comparable infrastructure, the use of Linux thus dictates the use of either WebSphere or WebLogic. While the cost of the operating system is relatively insignificant in the overall server cost equation, the cost of the infrastructure is not at all insignificant. WebLogic costs $40,000. WebSphere costs $48,000. This is as much as the server hardware. .NET, in contrast, cost nothing. It is part of the Windows operating system. The bottom line is that in order to save $5,990 of operating system costs, one must lock oneself into a "free" operating system that will require an additional $40-48,000 for server infrastructure! TMC did not include the cost of the database in their benchmark cost analysis. This was a rather odd decision, given that the database is just as critical to this benchmark as is the operating system or the infrastructure. Had TMC included the cost of the database, the cost analysis would have further favored .NET, given that SQLServer is much less expensive than either Oracle or DB2. Given all of this, it is easy to see why IBM is pushing Linux so hard. Don't give Microsoft $5,990 for Windows, IBM says, take Linux for free! IBM does not mention that once you have committed to Linux, you will have no choice but to spend $48,000 on IBM's infrastructure and yet more money on IBM's expensive DB2 database. You can kiss the free .NET infrastructure and the low cost SQLServer database goodbye. They will never again be an option for you. IBM's product line will be safe. BUILD COSTThe cost of actually building a system on WebLogic, WebSphere, or .NET is dominated by programming costs. Programming cost is dominated by a single measure, lines of code (LOC). A platform that requires more LOCs is going to be a more expensive to program than a platform than requires fewer LOCs. In my experience, a very good programmer will average at most 100 lines of debugged code per day. Assuming a salary/benefits cost of $100,000 per year, and 220 work days per year, we can predict a cost of (100,000/(220*100)), or $4.54 per LOC. How many LOCs does it take to build a WebLogic, WebSphere, or .NET system? In this area, there was no reported difference between WebLogic or WebSphere and there would be no reason to expect a difference between Linux and Windows. There was, however, a big difference in LOC between the two J2EE derivative systems and .NET. According to TMC, the total LOC count for the J2EE systems was 14,004. Keep in mind that this code was scrubbed by TMC, one of the leading Java training companies in the world. The .NET system required a relatively paltry 2,096 LOC. Most managers care more about dollars than LOC. Fortunately, we can translate from one to the other. Multiply LOC by $4.54 and you get dollars. The J2EE systems, therefore, cost $63,578 to program. The .NET system costs $9,516 to program. The J2EE systems cost $54,062 more to build than the equivalent .NET system. Using this benchmark as a cost predictor, we can assume that it will cost more than 6 times as much to build a system in J2EE than it will to build the same system in .NET. In reality, the build cost difference will be even higher for several reasons. First, maintenance costs is also related to LOC, so we can expect much higher maintenance costs for the J2EE system compared to the .NET system. Second, J2EE requires the use of Java whereas .NET can be done in Visual Basic. The general rule of thumb seems to be that experienced Java programmers are 30% more expensive than similarly experienced Visual Basic programmers. Third, the programming of your WebSphere application is, for many companies, not going to be done by $100,000/year Java programmers at all, but by much more expensive IBM consultants. This last point is one more reason IBM, which derives a huge amount of its revenue from consulting services, is hot on Linux. Free operating system. Expensive consulting. ADMINISTRATION COSTThe TMC benchmark gives us a starting point for estimating administration costs. While the benchmark doesn't measure administration costs directly, it does measure the cost of optimization, that is, how many hours were required to make the various systems perform as well as possible. The .NET system required a total of two person-weeks to fully optimize. The optimizations included database tuning and changes to a few defaults on the presentation side. According to TMC, "The optimizations made were minimal changes from default .NET Framework settings." The two J2EE systems each required ten person-weeks to fully optimize. As TMC notes in the benchmark, "The process of tuning and optimizing the configurations for both J2EE application servers was non-trivial..." The optimizations included choosing and installing a Java Runtime Environment, choosing a JDBC driver, fine-tuning the throttling of concurrent requests, configuring the number of concurrent connections and threads ("a process that proved tricky"), configuring the number of server processes, and generally balancing trade-offs in the run-time garbage collection. Based on the difference in ten person-weeks versus two person-weeks, we can predict that either a WebLogic or WebSphere system will cost 5 times as much to administer as will a similar .NET system. USER LOADUp until now, I have just been looking at the overall system cost. In every category of system cost, .NET leads. But what do you get for your money? Nobody cares if .NET is cheaper if .NET can't handle the system load. The results were clear. In every test and every server configuration, .NET outran the J2EE systems. I will just discuss the 4 CPU server configuration results, since this is the configuration used to calculate total server costs that I discussed earlier. The performance results, however, are similar for all configurations. The most relevant BB benchmark was one that tested delivery of web pages containing typical product catalog images. In this test, the 4 CPU WebLogic system was able to support a system load mimicking about 3500 concurrent users. WebSphere was able to support a fraction of that, approximately 1000 concurrent users. .NET, on the other hand, was able to support more than both of these systems put together, approximately 6500 concurrent users. When the beta version of .NET 1.1 was tested, this 6500 user support shot up to over 9,000. The DTP benchmark was very interesting. Here WebLogic maxed out at 59 transactions per second. WebSphere could not even sustain 18 transactions per second. According to TMC, "J2EE Application Server B was unable to sustain peaks throughput for more than 2 hours on the distributed transaction benchmark. It was unable to sustain any throughput beyond 4 hours, destabilizing over this period of time to the point of failure." And this after ten person-weeks of expert optimization! No wonder the IBM licensing agreements specifically prohibit public benchmark disclosure! As in the BB benchmark, .NET could handle more than both J2EE systems combined: 79 transactions per second (117 transactions per second with the .NET 1.1 beta). For DSOAP, WebLogic and WebSphere maxed out at between 1500 and 2,000 concurrent users. .NET maxed out at 6,000 concurrent users, almost twice the combined load of both WebLogic and WebSphere. This increased to 7500 with the 1.1 beta. For ISOAP, WebLogic and WebSphere were almost identical at about 2400 users each and .NET at slightly over 3000 increasing to over 6,000 on the 1.1 beta. CRITICISMS OF BENCHMARKDespite the great value of the TMC benchmark, there are four areas in which TMC could have done a better job. First, TMC could have included mainframe operating systems. I understand that this would have been more expensive, but doing so would have likely benefited both WebLogic and WebSphere in the user load benchmark. It is almost certain, however, that this increase in user load would have come at a substantially higher per user cost. Second, TMC should have allowed WebSphere to use DB2, its preferred database. This would have given WebSphere the best possible chance of success and might have ameliorated WebSphere's embarrassing showing in the DTP segment of the benchmark. Third, speaking of databases, TMC should have included the cost of the database in its server cost calculations. Doing so would have further benefited the value proposition of .NET. Fourth, TMC should have disclosed the fact that Microsoft gave financial support for the testing of .NET, support that was not given by either of the J2EE vendors. There is no conflict, in my mind, with such support. Virtually every industry benchmark is supported by vendors and the lack of financial support by either IBM or BEA is more damning than the fact that Microsoft WAS willing to financially participate. However, TMC should have made this support (and lack of support) clear in the benchmark results. That this did not come out until intense questioning by the Java community undermined the credibility of the benchmark, at least in the eyes of those who were not happy with the results. SUMMARY OF RESULTSAdmittedly, the TMC benchmark is not perfect, but no benchmark is. Overall, we owe TMC a debt of gratitude for a difficult job well done. The following conclusions are, I believe, strongly supported by the TMC benchmark:
So how does all this relate to me, sitting here, drinking my second doppio macchiato of the day? It tells me nothing about how fast I will get my doppio. Remember, speed is superfluous. Whether Starbucks produces my drink using WebLogic, WebSphere, or .NET, I will get my doppio in about the same amount of time. But from Starbuck's perspective, they can make a lot more doppios for a lot less cost for a lot more people using .NET than they can using either WebLogic or WebSphere. This is true even if they run their espresso machine on Linux. All of this is apparent from the TMC benchmark. One more thing is apparent. At $1.85 per doppio, nobody is passing on that cost savings to me. - Roger Sessions The TMC benchmark can be downloaded from http://www.middleware-company.com/j2eedotnetbench/ For the reaction to the benchmark from the Java community, see http://www.theserverside.com/home/thread.jsp?thread_id=9797 For a FAQ on the benchmark from TMC, see OBJECTWATCH SERVICESObjectWatch and Roger Sessions can help your company build successful enterprise caliber systems. We offer training at all levels in the software fortress methodology, architectural reviews and consulting services. For more information on how we can help you, see our web site at www.objectwatch.com or contact janet@objectwatch.com. We INVENTED the software fortress methodology. Let us help you use this methodology to create a secure, manageable, scalable, and flexible enterprise architecture. LETTERS (AND NON-LETTERS) FROM VERY IMPORTANT PEOPLEThe Chair of the OASIS BTP Technical Committee responds to The ObjectWatch Newsletter #41 In The ObjectWatch Newsletter #41, I described two competing standards for transactions spanning web services. One, called BTP, is from the OASIS consortium. The other, called WS-T, is from Microsoft, IBM, and BEA. In my article, I described the three problems I see with WS-T:
I sent the article to each of the authors of WS-T: Felipe Cabrera (Microsoft), George Copeland (Microsoft), Bill Cox (BEA), Tom Freund (IBM), Johannes Klein (Microsoft), Tony Storey (IBM), and Statish Thatte (Microsoft). I asked if any of them would be willing to defend the choices they had made in these three areas. Not one of them was willing to do so. I also criticized WS-T's competition, BTP. The main problem I see with BTP is an overly complex approach to transactions that it calls "cohesions". I also sent a copy of the article to the BTP team, inviting them to respond. Unlike the WS-T team, the BTP team, represented by William Z. Pope (Choreology, Ltd) did respond. Pope is the Chair of the OASIS BTP Technical Committee that is responsible for the BTP proposal. I am happy to print Pope's letter in its entirety. I regret that I do not have a similarly lucid response from the WS-T team. Should I ever receive one, I will include it in a future newsletter. Pope's discussion of "the perfect breakfast" and breakfast tacos relates to the analogy for loosely-coupled transactions I used in my original article. ................. Hi Roger, I'm the chair of the OASIS BTP technical committee and wanted to comment on your article on BTP versus WS-T. Specifically to describe the difference between cohesions and atoms in BTP. Side note this was not the easiest route for the TC to take, many members were opposed to including cohesions because of their 'newness' and the increased complexity in the specification. In the end the TC agreed that the additional capability was worth while. BTP atomic transactions are exactly as you describe, participants either all succeed or all fail together. This is comfortable for anyone who is familiar with traditional transaction processing. The core of BTP cohesions is to allow for more flexibility in the controlling application. In the quest for your perfect breakfast you can decide that it is essential to have the breakfast taco from El Arroya and the doppio macchiato from Starbucks but if Burger King can't make the fries you still want the rest of it so you can have a pretty good breakfast. To do this you define a cohesive transaction with two required participants and one desired participant. This allows you to conduct the prior negotiation as was done in your core example but with something between the all-or-nothing semantics of the pure atomic transaction. Once there is a system that allows for the maybe case, the controlling application (Bill's breakfast coordination service) can still get you the best breakfast available under the rules you defined. These rules can then be expanded to define alternate services, like fries from Wendy's, when the Burger King has none available. The coordination service checks all your primaries first, just as it did before, but then tries to recover from failures when it is possible/permissible to do so. At least abstractly, cohesive transactions evolve into atomic transactions when it becomes time to complete. Once all of the elements that are going to participate have been signed up the decision is committed across that set of participants. If the required elements are present then all of the elements are confirmed along with the desired elements that were selected (Burger King fries today). If required elements are missing (no tacos left) then all the other elements are cancelled. Hope this explanation is helpful, bill William Z. Pope
Legal NoticesThe ObjectWatch Newsletter does not rent out its subscription list. This newsletter is Copyright (c) 2002 by ObjectWatch, Inc., Austin, Texas. All rights are reserved, except that it may be freely redistributed provided that it is redistributed in its entirety, and that absolutely no changes are made in any way, including the removal of these legal notices. ObjectWatch is a registered trademark (r) of ObjectWatch, Inc., Austin, Texas. Software Fortresses, Software Fortress Architectures, and the Software Fortress Model are trademarks (tm) of ObjectWatch, Inc., Austin Texas. All other trademarks are owned by their respective companies.
|