Page 1 of 1

Control Code A/B Testing & Validation Survey

We are researching how engineering teams compare and validate control code algorithms (like balancing, steering, or PID loops) across simulation and physical testing.

This quick 2-minute survey helps us understand your current evaluation workflow, how you handle simulator variance, and what automated statistical tools would make your benchmark testing faster and more reliable. P.S. The questions have been rephrased for clarity, but the content remains unchanged.

On a scale of 1–10, how confident are you that your control code will still work reliably in the real world, even with noise and initial condition changes?

When you make changes to a control loop or policy, what do you use to determine whether the new version (B) is better than the previous version (A)?

On a scale of 1–10, how difficult is it for your team to manually run, export, and compare different versions of your code?

When a new version of your control code performs better in a simulation, how do you check that it will consistently perform better in other runs as well?

On a scale of 1–10, how useful would it be to have a tool that automatically tests different versions of your code under many different conditions and tells you which version performs best?

Control Code A/B Testing & Validation Survey

On a scale of 1–10, how confident are you that your control code will still work reliably in the real world, even with noise and initial condition changes?

When you make changes to a control loop or policy, what do you use to determine whether the new version (B) is better than the previous version (A)?

On a scale of 1–10, how difficult is it for your team to manually run, export, and compare different versions of your code?

When a new version of your control code performs better in a simulation, how do you check that it will consistently perform better in other runs as well?

On a scale of 1–10, how useful would it be to have a tool that automatically tests different versions of your code under many different conditions and tells you which version performs best?

What results would an automated testing tool need to show you for you to trust its conclusion about which version of your code performs better?