Hi spike and welcome to the Forum!
I've never used DVDx, but I downloaded it to have a look.
The best I can suggest is you are limiting the file size, and its making a new file to carry whatevers left over when you created the AVI.
When I loaded a DVD file into it, it showed, in the output settings, that it was going to make a video cd, with resolution of 352x288.
It showed in the 'Volume dont exceed' panel "custom size" .
If I click on custom size, then I see it says 'Volume dont exceed' with value of 730MB, yet the estimated size of the file (in the panel just above that) is 953.3MB. I assume this means that I would get a second file with the remainder which doesnt fit.

If I switch the settings to creating AVI file, as below.. reset the resolution to the same as the original movie (720x576 in this case) you'll now see that it shows the estimated size as unknown.
Its still on Custom size, and the max volume is still 730MB. This will be considerably larger than the VCD file as I've changed the resolution, and its going to have even more space left over.

My suggestion would be to open the drop down box entitled Volume dont exceed, and change that setting to infinite. This means it should create a single large file, I assume.
I hope this helps.
I'd also be curious to know how long it took you to create the file. I have an extremely fast computer with all the good gear, and it looks like it could easily take 12 hours for this conversion to take place. Perhaps I need some help too!! lol
Please let me know how you go, and if you are still having problems, or if it does take a huge amount of time to convert for you too.