I have an Intel Imac with 2.4 GHZ and 2 GB Ram. I have the latest Avenue 3.1.1 It was working just fine. However in last few gigs, the Output is becoming jerky. When it starts, it works fine but after half an hour the output is not playing smoothly.
It sounds like there indeed is a part of your setup which can't cope with the load. To check how your system is doing, you can use 'activity monitor', found in the utilities folder. That way you can narrow down which part of your system is bearing the brunt of the load.
Also, is your iMac one of the models that is using the shared memory GPU?