This paper describes the development and application of a machine listening system for the automated testing of implementation equivalence in music signal processing effects which contain a high level of randomized time variation. We describe a mathematical model of generalized randomization in audio effects and explore different representations of the effect’s data. We then propose a set of classifiers to reliably determine if two implementations of the same randomized audio effect are functionally equivalent. After testing these classifiers against each other and against a set of human listeners we find the best implementation and determine that it agrees with the judgment of human listeners with an F1-Score of 0.8696.