By Michael Webb, Editor-in-Chief, MikeWebb@CompuServe.COM
This article is a discussion of the results listed in the Performance Numbers article. The sheer number of tests performed made it a very long article, so I decided to include this description separately. If you haven't seen the numbers yet, you might want to take a look at the other article before proceeding.
When I was putting these tests together, I essentially looked around to see what types of software I had that would provide good indications of an Amiga's performance capabilities. Fortunately, I've always been curious about benchmarks, and also use various pieces of software that require a great deal of computing power.
I ultimately decided on SysInfo, SysSpeed, and XOpa for providing the raw benchmark scores, and Scenery Animator and CygnusEd to show two sides of an Amiga's performance in more "real-life" settings. (Please see the Performance Numbers article for the names of the authors of these programs.)
It is important to keep in mind that indications of a computer's performance will vary greatly with application, and two programs that supposedly measure the same quantity can produce vastly disparate results. Much is dependent upon the methods used for evaluation, but since this is generally all determined by the programmer, we, the users, really don't know what's going on inside each program. So one shouldn't expect data from different programs to agree. Results for a single program run on various system configurations can, however, be regarded as a basis for fairly accurate comparison.
Anyway, I ran an array of tests with each benchmarker, I used Scenery Animator to render the same image under different circumstances and parameters, and I performed a simple but massive search-and-replace with CygnusEd. I tried to show how different performance-influencing factors within an Amiga will show up in different ways under different tests.
Finally, I emphasize that these results are not to be taken as scientific, empirical data. They were not performed under perfect conditions. For instance, to be as exact as possible, I would have had to perform a fresh OS install on each machine, and have nothing extra running in the background while performing tests. Instead, I used my existing A500-based and A4000T systems, and put together boot disks for the plain A500 and A1200. In many cases, various things, even some full-blown applications, were running in the background. But because the Amiga is so outstanding at multitasking, doing so (as long as the applications were "idling," as they were at the time) generally had no noticeable effect on results. So I didn't bother to go to extremes, since I probably would have needed a timekeeping device more accurate than my watch to reflect any differences anyway.
I selected my test systems in more or less the same way in which I selected the software. I looked around and used what was there, and my modest Amiga collection (some parts, some computers) yielded four machines I considered distinct enough to produce interesting results.
My A500 is actually two Amigas in one, since the GVP A530 Turbo (a combination 40-MHz 68030/68882/8 MB fast RAM/SCSI/hard disk/etc. device) is such a significant part of the system. Note that you shouldn't consider it a complete model for all 68030 Amigas (particularly the A3000(T)), because while the A530's 68030 is on a high-speed 32-bit bus, it must interface in an asynchronous manner to the original 16-bit Amiga chip RAM/custom chip bus. This probably impacts graphics speed more than anything else. Therefore, for many purposes, this does make a good model 68030 system. But just keep in mind that it is, after all, an A500 at heart.
And when the A530 is disabled, that A500 functions on its own, taking me back in most ways to 1987 with a 7.14-MHz 68000 (differences include 2 MB chip RAM, display enhancer, ECS, and AmigaOS 3.1). This provided the really, really low end for the tests. But due largely to the Amiga's special OS and hardware design, a 68000 Amiga is not always as horribly slow as one might expect, as we shall see from some of the numbers compiled in these tests.
In my case, "Amiga collection" does not imply anything terribly diverse. I had three A500's and an A1000 a few months ago, and from the standpoint of a performance test, these were four of the same machine (until you take the A530 into consideration). But then I got the A4000T, of course, and very recently, we also came into possession of an A1200. I thought the A1200 would be particularly interesting, because its 68020 is a good deal slower than the 68030 in the A530, but its AGA chipset and 32-bit chip RAM pathway are significantly faster than the A500's ECS and asynchronous 16-bit access to chip RAM. So this would provide an interesting comparison, revolving around the question, "On an Amiga, what influences performance more? CPU, or chipset?" The answer, as it turns out, appears to vary depending on how you use the machine.
Of course, the star of the whole show here is the A4000T, and so I was sure to include it in the tests, rounding out the high-end.
So what's missing? I would have liked to have been able to try 68000- and 68020-based Amigas with some fast RAM, since forcing the processor to use chip RAM only can really slow things down under the right circumstances. Also, as you may have noticed, the 68040 was absent from these tests. Not coincidentally, I own neither a 68000/68020 Amiga with fast RAM, nor a 68040 Amiga. In many cases, however, chip-RAM-only testing probably didn't affect results to a great degree, and you can generally think of the 68040 as a two-to-three-times-slower 68060.
Testing with SysInfo is fairly straightforward. There has been grumbling throughout the Amiga community about the reliability of this benchmarker over the years, but as with everything of this type, its results are usually good enough for comparisons involving this one program on different systems. For those not familiar with SysInfo, its results include Dhrystones (one of many "performance units" used in evaluations), followed by the ratio of the given machine's processing speed to those of various other Amigas. For instance, the A4000T's rating of 63.59 versus an A600 indicates that it performs SysInfo's test 63.59 times more quickly than an A600 does. MIPS and MFLOPS are very common, simple tests of how many millions of instructions per second the CPU and FPU, respectively, can execute. "Chip Speed vs A600" probably means CPU bandwidth into chip RAM, using the A600 as a reference point. The comments are a more light-hearted criterion coded into SysInfo (and keep in mind it was written before the 68060 became so widely available, hence the rather extreme comment for the A4000T). The final item lists hard disk data transfer rates. Interestingly, this test reveals the effect of additional SCSI devices on the performance of the chain overall, as the A4000T's disk speed decreased from 5.4 to some 3.7 MB/second when my external CD-ROM and SyQuest drives were running.
SysSpeed is a MUI application that runs a wide variety of tests, and allows direct comparison among different systems by use of "modules," essentially lists of results that can be loaded into SysSpeed alongside the current system's numbers.
The first round of tests focuses on memory performance, which uses a somewhat cryptic notation. As far as I know, "b" means "byte," "w" stands for "word," and "l" represents "longword." These are just different-sized pieces of data on the order of 8 bits, 16 bits, etc. The A4000T was the overwhelming winner in many of these tests, but in operations involving chip RAM, the A1200 often kept up, and sometimes surpassed it. The A500/A530, with its terribly slow CPU interface to chip RAM, suffered most in such tests, while turning in very respectable numbers in the fast RAM category.
Drive tests required a hard disk, which eliminated the A500. For some reason, SysSpeed didn't like the A1200's hard disk, and I had to fill that column with N/A's too. So only the A500/A530 and A4000T showed up in this category, with the latter far ahead of the former.
For both Intuition and Graphics tests, I used NTSC Low Resolution 320x200 in order to see what the CPU's and blitters could crank out when there was as little bus contention as possible. In these tests, the interesting interaction of CPU and chipset played a very significant role; for some tests, the processor did all the work, but AGA's improved bandwidth left more time for the A1200 and A4000T to actually do this. In other cases, the Amiga blitter and other custom hardware is clearly used, and performance varies little among machines using the same chipset, while AGA shows a clear advantage over the ECS. The most striking results, in my opinion, are those for screen-swapping. It appears that a fast-RAM-equipped Amiga can flip from one screen to another at the same rate as the video refresh, e.g. 60 or 70 Hz, depending on the screen mode. This is one of the Amiga's truly unique advantages: you can change from one application to another without having to watch the screen redraw. And the Amiga does this so quickly by simply writing to a couple of hardware registers. It is extremely efficient.
SysSpeed's final tests are, once again, MIPS and MFLOPS. Note that these results are quite different from SysInfo's measurements of supposedly the same units. It serves to reinforce the idea that it is difficult to accurately quantify a computer's capabilities, and that performance will always vary with application. Note that the A500 and A1200 were excluded from MFLOPS measurements for lack of an FPU, and that the 68060's built-in FPU left the 68030's external 68882 extremely far behind, with a difference much greater than that between the chips' integer units.
XOpa is probably known more as a system information and control program, allowing one to remove tasks, close screens, adjust priorities, etc., but it also can perform some speed tests.
The CPU and FPU tests are fairly straightforward. The memory tests are more interesting; chip RAM performance appears to have been largely a function of data "width," with the A500's 16-bit RAM being exceeded by a bit more than a factor of two by the AGA machines' 32-bit memory. The A500/A530's particularly poor performance exemplifies the slow CPU-to-chip-RAM interface I mentioned. Fast RAM was another story, however; unfortunately, I could not test the A500 and A1200, as they have no fast RAM, but the excellent memory design of the A530 showed through, with the A4000T turning in high numbers as well.
Anyway, I was most interested in XOpa's Graphics and Intuition tests, because the user can choose the exact modes it uses. I used a variety of them, and to try to explain my reasoning in doing so, I will list each one here, followed by brief descriptions:
Note that low-resolution DblNTSC modes are not available on the ECS. Also, I didn't use the variety of modes for Intuition tests that I did for Graphics tests, because one will probably not encounter as many different modes in Intuition-based environments as in other types of Graphics environments (e.g. games).
In general, I attempted to explore various limits of the two chipsets, and how they would affect things in the presence of different processors. There are many variables at work here, but the results reveal some interesting things about this interaction among various pieces of Amiga hardware. For instance, if you disregard the capabilities of the blitter, copper, and other special-purpose hardware, you see that much of AGA's advantage is in giving the CPU more time to operate on chip RAM. One might even say that AGA essentially made all of the ECS's modes fast enough to be pleasantly usable, while proceeding to introduce its own new modes, many of which were themselves too slow. XOpa's tests don't seem to use the blitter much, so bus contention and CPU's seem to be the determining factors.
For those who don't know, Scenery Animator is Natural Graphics's landscape generator with extremely user-friendly (albeit simple) keyframe animation capabilities. It relies upon a combination of land elevation data and fractal mathematics in order to do this. It's one of my personal favorites among all the software I've ever used, since it can be a lot of fun, and really showcases the Amiga's animation capabilities.
For our purposes here, I kept things to single frames, and rendered an image of Mount St. Helens on each machine under several different modes. This didn't go as well as I had hoped, because the 2 MB of memory in the A500 and A1200 was not enough space for allocating the buffer needed to render in most of the modes.
To further diversify the testing, I used both the 68000 and 68020/68881 (where possible) versions of Scenery Animator, and I performed all of the tests on the A4000T without CyberPatcher, and then again with it. This should have made no difference with the 68000 version, but I included the results anyway, just to be complete.
The Amiga custom chipset comes into play here once again. Scenery Animator has two ways of rendering: directly into the display, or into a buffer, probably located in fast RAM, if any is available. It doesn't give a choice, per se, but renders standard 16- and 32-color IFF images into the display, and everything else into a buffer, automatically. Naturally, on the Amiga, rendering into the display always exacts a certain speed penalty, and sometimes it can be very severe indeed. The A500 took over two hours to render in 640x400x4-bit, and although the 68000 may be slow, rest assured it would have been much quicker if it hadn't had to deal with the ECS eating up almost all the DMA time just to display the screen. Of course, rendering into a buffer would not have helped much in this case, because the A500 has no fast RAM, and any part of chip RAM would be just as slow as the part being displayed. But keep in mind that under the right circumstances, the 68000 could have turned in a much better time.
Most of the results were probably fairly predictable, with 68060 surpassing 68030 surpassing 68020 surpassing 68000, but the A4000T turned in quite a surprise. It performed best when running the 68000 version, even compared to the 68020/68881 version with CyberPatcher running. I'm not sure why this is. Maybe the 68020/68881 version still includes enough instructions unimplemented in the 68060, beyond what CyberPatcher can catch, to slow things down. Alternatively, word has it that rather than using software floating point (which many such applications do use), Scenery Animator 68000 simply uses integer calculations at reduced accuracy, which may account for this phenomenon. In any case, I know which version I'll be using on the A4000T in the future (and one can imagine that were Scenery Animator optimized directly for the 68060, there would be a noticeable improvement). For the A500/A530, the optimized version yielded better results, although not by a huge margin (certainly well under a factor of two). The possible use of integer calculations within Scenery Animator 68000 may account for this relatively small speedup as well.
This test was probably the simplest and most straightforward of all. I just loaded the "intuition.doc" file used by SysSpeed for various text editor performance evaluations, and had CygnusEd do a global search-and-replace. It was a large file, and the job took a couple of minutes even on the A4000T.
Once again, the Amiga chipset comes into play. "What?", you say? "With a text editor?" Well, CygnusEd scrolls the screen and shows what it's doing while performing a search-and-replace. I tried to minimize the effects by running it in NTSC low-resolution mode, but just considering how much faster fast RAM is than chip RAM, the procedure could probably be performed much more quickly if CygnusEd did all the work within a fast RAM buffer. Of course, it can be useful to see what's being replaced, so I would suggest it as an option within the program rather than an absolute (and conveniently enough, CygnusEd development appears to be back underway...).
Anyway, with the chipset leveling the playing field somewhat, results didn't vary as much among the machines as one might expect, but as with Scenery Animator, there was a definite CPU-related trend.
I hope the tests and test systems I selected provide sufficiently diverse and informative results to give a good idea about the performance of this new A4000T, and how it compares to the other included Amiga systems. I can't say enough times that results will vary depending on how you use your computer. However, this information should give a fairly good idea of what one can expect from different hardware configurations, and under different software uses. Again, if you haven't seen the numbers, check this page of the review for the information on which this discussion is based.