True boxers, I6 and v12 (which is basically two I6s) have the best natural secondary balance . A balancer shaft works at one harmonic only (typically twice crankshaft speed aka second order) , but the piston on crank assembly generates a whole series of forces, not just second order.
i5s and v10s aren't horrible, but they aren't anything like as smooth as an I6 unless they have a balancer shaft. I3s and 90 degree V6s and your usual I4 all create lots of powerful inertial forces.
Roughly speaking an I4 vibrates at 20 g rms at the redline at WOT, an I6 is at 5g rms. 10-1000 Hz.
The disadvantage of I6s and so on is that the crankshaft is so long that you run into torsional and bending problems with the crank.
Friction wise, the fewer cylinders the better for a given capacity , and a balancer shaft also adds friction.