For optimization of parameters you can go full ham and apply even genetic algorithms.

The problem is actually finding a "legibility" and "quality" metric. Codec ones like PEQ are not good enough.