Or is this can only be obtain on the engine side?

I'd say avoid narrow passes. Regardless of the quality of pathfinding you can only fit a certain number of units per second through any specific gap which means any excess units have to queue behind which leads to them turning around to find a new route and bouncing off other units. It's even worse if friendly units are trying to enter the gap from both sides. Since there is no code to say "stand aside and let the larger group pass" it means a single unit can block an entire army. You'd also want to avoid crossroads where columns of friendly units would regularly cross paths. Once again the code does not allow either group to wait for the other so the result is the groups entangle in the intersection.

Having said that, there are features in the engine like "heatmapping" which provide hints as to whether an area is "busy". Still, I wouldn't rely on that (especially since it can be turned off).

What about the units who have different speeds? I keep finding that the fast one somehow gets stuck behind the group of fast ones.

Another thing I have learned is that however realistic adding some perlin noise to your "flat" areas is, it ruins pathing majorly. Try to keep all playable surfaces free of any high or mid frequency noise. A trick you can do is have 2 heightmaps, one that you use to render the texture (this can have all the noise you want) and one that is the ingame heightmap (strictly smooth surfaces).

ingame select a unit and press F2. (i think first you have to type /cheat /globallos to see whole map)
Ideally passable (green) and unpassable areas (red) have clear borders.
If there are stray, noisy red fields or narrow green fields, pathing will become more problematic.

There is Lua pathfinding ofc, but in general I would say it is too slow. In the future we may have different engine-side pathfinders to choose from, because the interface is now clean enough to allow that.

How slow is that slow? And can it delegate to engine?

There is some simple mod stuff.

Make sure your units are as big as they think they are with equal entries for footprint in movedefs and unitfefs.

Avoid large turn radius or low accelerations unless required.

make "flats" of maps completely flat

make "ramps" completely evenly graded with an even slope width throughout

make "cliff edges" an even slope throughout

frozen fortress paths near-perfectly, this was not by accident... just takes a fair amount of planning and testing