Thanks for the suggestions, jensj12! I did a very simple attempt at restricting height as the track grows. It kinda sorta works, but I'll have to think about a more accurate model. The nuclear option would be to take OpenRCT2's code that actually simulates the coaster across the track and measure everything. But the code seems complicated enough, and with the current bruteforce-like approach I'm sure it would be super slow.
In the meantime I converted my Python code to C++ so everything happens in a single step, producing a td6 file that I can test.
I created a large coaster that can do a full lap with some cheating (I changed some flat pieces to boosters). Here is what it looks like. You can see that I also added loops.
Then I added banked turns. To try and get better stats I removed any non-banked turns for now. There are some issues with this, one of them is that I don't restrict banking and sometimes the track wobbles from left to right and back. I'll have to add another state machine that dictates which states are legal, so that once it banks in one direction it commits for at least a few track pieces or until a turn is made, etc.
I generated some super tiny 14x6 coasters, the stats are surprisingly not that bad:
That's it for now, I'll generate more interesting coasters when I have a better model