Has Fractal Audio's amp algorithm ever been verified by "Null test"?

[km 202257]

This is just out of my pure curiosity, so feel free to delete this thread if it's an inappropriate topic here.

I see some of amp modelers are verified by "Null test" for its identicality to the real counterpart.
The idea of "Null test" is to record 1.the signal from the original amp and 2. the signal from the amp simulator emulating the same amp as 1., and then to play both 1. and 2. out of phase. The premise is that 2. will cancel out 1. if 2. is identical enough.

Has Fractal Audio's amp modeling algorithm ever been verified by this "Null test" for its identicality?
Or, is there any reason that such test is not appropriate to verify identicality of an amp simulator?

 

[AJ Vargas]

Those tests are done with amp profilers, not modelers, and usually are done by people who have done the captures/profiles, so they have THE exact/same amp with the same settings, so you can effectively compare your capture devices to the real thing.

To do the same with the Axe FX, you'll need Cliff's/FAS' personal amps, otherwise, parts tolerances would throw your tests off by whatever margin they're off compared to the amp used by Cliff.

 

[Greg Ferguson]

To do the same with the Axe FX, you'll need Cliff's/FAS' personal amps, otherwise, parts tolerances would throw your tests off by whatever margin they're off compared to the amp used by Cliff.

Yep.

Why your amp doesn't sound like Fractal Audio's talks about it.

Cliff periodically will pull one of his reference amps out of their warehouse and compare it to the model to make sure they're behaving correctly. He has the graphs and occasionally will share them.

 

[km 202257]

Those tests are done with amp profilers, not modelers, and usually are done by people who have done the captures/profiles, so they have THE exact/same amp with the same settings, so you can effectively compare your capture devices to the real thing.

To do the same with the Axe FX, you'll need Cliff's/FAS' personal amps, otherwise, parts tolerances would throw your tests off by whatever margin they're off compared to the amp used by Cliff.

Yep.

Why your amp doesn't sound like Fractal Audio's talks about it.

Cliff periodically will pull one of his reference amps out of their warehouse and compare it to the model to make sure they're behaving correctly. He has the graphs and occasionally will share them.

Indeed, if AXE-FX III is null-tested it has to be Cliff's reference amp that is compared with the emulated tone. My question is if FAS does null-test internally with his reference amp.

 

[touch33]

While a “null test” might give you an idea of the devices’ transfer function, it’s never going to tell you what it’s like to actually play through the thing. That elusive “feel factor” that we all spend so much time discussing and searching for is also the “it” factor that we all fork over our $$$$ for — and I’m pretty sure it’s either super-hard or impossible to quantify with a “test”. Maybe @FractalAudio will chime in here with a definitive answer.

 

[rrhoads17]

This is just out of my pure curiosity, so feel free to delete this thread if it's an inappropriate topic here.

I see some of amp modelers are verified by "Null test" for its identicality to the real counterpart.
The idea of "Null test" is to record 1.the signal from the original amp and 2. the signal from the amp simulator emulating the same amp as 1., and then to play both 1. and 2. out of phase. The premise is that 2. will cancel out 1. if 2. is identical enough.

Has Fractal Audio's amp modeling algorithm ever been verified by this "Null test" for its identicality?
Or, is there any reason that such test is not appropriate to verify identicality of an amp simulator?

IMO, comparing a Fractal unit to the amp it’s modeling, especially older handwired amps like Plexis and Bassmans is kind of useless. The main reason is because the model is based on Fractal’s reference amp and may sound a little different than other Plexis for example because Marshall never made all of them exactly the same. A great example is Fractal’s Plexi models are pretty gainy to my ears. But if you play the Amplifire’s Plexi, it’s not very gainy at all, more crunchy. If you hear an amazing Plexi on YouTube or in person and you’re wondering why they sound different, that’s because the Plexi may have slightly different component values, the cab speakers, what kind of wood the cab is made of, the mic, the mic position, etc. Now, in theory, Fractal models of PCB amps like the Friedman stuff should sound nearly identical because quality control is 100x better today and all the amps should sound nearly identical if using the same cab or FRFR. Fractal has done a great job in modeling an amp very accurately and linking it to a great IR/cab. This where a lot of other modelers fall short, their model may sound great, but the IR sounds like crap.

 

[km 202257]

Post in thread 'Axe-Fx III 16.00 Beta 2 "Cygnus" Firmware'
https://forum.fractalaudio.com/threads/axe-fx-iii-16-00-beta-2-cygnus-firmware.169998/post-2045448

Thanks, unfortunately it looks like I don't have the permission to read the post. Is it OK to understand that the post is about reports/results of null-test (or alike)? Then I think it answers my question.
(You don't have to share quote/screenshot of the post; I guess that would be a violation of the rules on those "private" threads.)

 

[km 202257]

IMO, comparing a Fractal unit to the amp it’s modeling, especially older handwired amps like Plexis and Bassmans is kind of useless. The main reason is because the model is based on Fractal’s reference amp and may sound a little different than other Plexis for example because Marshall never made all of them exactly the same. A great example is Fractal’s Plexi models are pretty gainy to my ears. But if you play the Amplifire’s Plexi, it’s not very gainy at all, more crunchy. If you hear an amazing Plexi on YouTube or in person and you’re wondering why they sound different, that’s because the Plexi may have slightly different component values, the cab speakers, what kind of wood the cab is made of, the mic, the mic position, etc. Now, in theory, Fractal models of PCB amps like the Friedman stuff should sound nearly identical because quality control is 100x better today and all the amps should sound nearly identical if using the same cab or FRFR. Fractal has done a great job in modeling an amp very accurately and linking it to a great IR/cab. This where a lot of other modelers fall short, their model may sound great, but the IR sounds like crap.

Again, I know it has to be Cliff's reference amp that is compared with the emulated tone (if AXE-FX III is null-tested). I never intend to null-test the models using the real amps available around me. My question was if FAS does null-test internally with his reference amp (seems to be answered in post #6).

 

[sprint]

Pretty good bet that Fractal tests accuracy to their reference amps scientifically, and beyond "just ears" - what methodology do they use? who knows - a combination of techniques probably (Response Analyzer, Null, latency test, isolated virtual component tests, ... ears...?..). Whatever they do, we know it's rigorous given their established reputation for highly accurate models.

Related side note - I would like to see a picture of all Fractal's real reference amps stacked up in a big mountainous pile.

 

[rrhoads17]

Again, I know it has to be Cliff's reference amp that is compared with the emulated tone (if AXE-FX III is null-tested). I never intend to null-test the models using the real amps available around me. My question was if FAS does null-test internally with his reference amp (seems to be answered in post #6).

I’m sure they do. It’s just good to be aware that what you are hearing is Cliff’s Plexi, which may sound a little different than EVH’s because it’s a known fact that Eddie’s Plexi had some odd stock values that changed the sound a little bit.

 

[Randalljax]

Again, I know it has to be Cliff's reference amp that is compared with the emulated tone (if AXE-FX III is null-tested). I never intend to null-test the models using the real amps available around me. My question was if FAS does null-test internally with his reference amp (seems to be answered in post #6).

The answer is yep for Cygnus and I think Cygnus 2 they actually take ALL of the amps out of storage and re measure that’s why the major releases are in pieces as it takes a lot of time to go through all the amps again

Also if a several users are giving feedback about a specific amp saying it sounds off , Recto or JP2C come to mind , FAS will pull the amp back out and recheck the model sometimes resulting in further improvements and fixes if required

 

[HereToday]

Such a test might have some benefit to the one building the model or to the marketing department perhaps. To the end user, not so much.

 

[lastcaress83]

The null test has some issues as applying to guitar. It’s going to be valid for frequency response and harmonic distortion content but won’t tell us much about the more subtle aspects of distortion characteristics, compression, touch response, things like that. Ideally, it’d be run with an actual guitar being played instead of the test signals often used.
In short, it’s far from perfect comparison but it’s a starting point.

It makes more sense for hifi amps.

 

[IronSean]

First, Cliff definitely has an assortment of audio tests he can run when comparing models to the real amps, and has at different times posted comparison graphs of distortion harmoic content on real amps vs his models. Ultimately there's lots of ways to slice and analyze audio and a null test is one, and Cliff likely has a lot of opinions on which work best in his workflow.

There is one part of the Axe FX models which might make a null test harder to apply. People have mentioned it could only be truly done against Cliff's own amps, but there is also the fact that individual pots have their own tolerences and the value of a given pot at 5 might differ within that tolerence. This means that when Cliff programs in a 2MOhm pot for a given control getting his model and the real amp each to exactly the same resistance value on that pot (which you would need for a truely perfect null test) would be very difficult. And would need to be done for each pot. On a Profile or Capture where the model bakes in the position of the controls this is easy because your Profile is permanently set to exactly match the settings at the time of capture. But in a model where you can adjust the pots yourself it's very hard to get exactly right. (Caputres and Profiles "fake" the EQ pots by putting an EQ at the start or end of a signal chain for their captures, but internally lock all pots in place).

But the real reason I'm commenting is this: Lots of people are talking about "It's not about the frequency response it's about the feel and touch response" and I don't understand that argument. All of those things we attribute to feel, touch response, compression, distortion characteristics are ultimately expressed as waves of pressure captured as the audio signal. If you have the same input signal running into two systems that modify those waves in the exact same way and output an identical wave, that wave should perfectly match and thus perfectly cancel out when inverted. Anything we refer to as "feel" or "touch response" are just aspects of how that wave behaves and a null test would cancel out just the same. If the "feel" was different it would mean that the compression or speed or response or something else differed between the two samples and those differences would fail to cancel on a null test. and result in leftover signal, maybe targetted around the attack of notes. There's no reason a null test can't be testing for all those things.

 

[lastcaress83]

PFirst, Cliff definitely has an assortment of audio tests he can run when comparing models to the real amps, and has at different times posted comparison graphs of distortion harmoic content on real amps vs his models. Ultimately there's lots of ways to slice and analyze audio and a null test is one, and Cliff likely has a lot of opinions on which work best in his workflow.

There is one part of the Axe FX models which might make a null test harder to apply. People have mentioned it could only be truly done against Cliff's own amps, but there is also the fact that individual pots have their own tolerences and the value of a given pot at 5 might differ within that tolerence. This means that when Cliff programs in a 2MOhm pot for a given control getting his model and the real amp each to exactly the same resistance value on that pot (which you would need for a truely perfect null test) would be very difficult. And would need to be done for each pot. On a Profile or Capture where the model bakes in the position of the controls this is easy because your Profile is permanently set to exactly match the settings at the time of capture. But in a model where you can adjust the pots yourself it's very hard to get exactly right. (Caputres and Profiles "fake" the EQ pots by putting an EQ at the start or end of a signal chain for their captures, but internally lock all pots in place).

But the real reason I'm commenting is this: Lots of people are talking about "It's not about the frequency response it's about the feel and touch response" and I don't understand that argument. All of those things we attribute to feel, touch response, compression, distortion characteristics are ultimately expressed as waves of pressure captured as the audio signal. If you have the same input signal running into two systems that modify those waves in the exact same way and output an identical wave, that wave should perfectly match and thus perfectly cancel out when inverted. Anything we refer to as "feel" or "touch response" are just aspects of how that wave behaves and a null test would cancel out just the same. If the "feel" was different it would mean that the compression or speed or response or something else differed between the two samples and those differences would fail to cancel on a null test. and result in leftover signal, maybe targetted around the attack of notes. There's no reason a null test can't be testing for all those things.

It depends on the type of signal applied and how it’s applied. Compression in a cranked guitar amp is not constant, harmonic content is not even constant. That’s why I said if you can null with an actual guitar being played into both inputs simultaneously, that will be much closer than a test signal though I still suspect some elements will remain unaccounted for just because there’s so much going on in a tube amp. That’s why profilers, modelers, etc still aren’t perfect despite decades of effort. There is no one perfect test.

A best case scenario null test would be you plug into both inputs and play every variation you can think of, pinch harmonics, natural harmonics, vibrato, bending, slapping, tapping, scraping, everything… because a real amp is going to respond differently to all of these things at different places on the neck. Also would likely need to use a variety of pickups and guitars since those signals will look different. It may be found that signal is only audible say when palm-muting a high G on the low E string while alternate picking quickly on a telecaster neck pickup with volume at 4 and tone at 7… if so, that’s meaningful.
Additionally, you have to account for the way an amp reacts differently with the speakers (and vice-versa) at varying db levels, which a null test won’t do for you.

 

[IronSean]

It depends on the type of signal applied and how it’s applied. Compression in a cranked guitar amp is not constant, harmonic content is not even constant. That’s why I said if you can null with an actual guitar being played into both inputs simultaneously, that will be much closer than a test signal though I still suspect some elements will remain unaccounted for just because there’s so much going on in a tube amp. That’s why profilers, modelers, etc still aren’t perfect despite decades of effort. There is no one perfect test.

A best case scenario null test would be you plug into both inputs and play every variation you can think of, pinch harmonics, natural harmonics, vibrato, bending, slapping, tapping, scraping, everything… because a real amp is going to respond differently to all of these things at different places on the neck. Also would likely need to use a variety of pickups and guitars since those signals will look different. It may be found that signal is only audible say when palm-muting a high G on the low E string while alternate picking quickly on a telecaster neck pickup with volume at 4 and tone at 7… if so, that’s meaningful.
Additionally, you have to account for the way an amp reacts differently with the speakers (and vice-versa) at varying db levels, which a null test won’t do for you.

I agree with you on one level, there are some inherently random aspects like background noise and thermodynamic effects at the molecular level which can never be perfectly matched because they're inherently random and unrelated to the input signal the amp is affecting. And anyone who has ever needed a noise gate will know those get loud enough to hear in an amp with a few gain stages. Also any of the elements of an amp which are affected by temperature (and there are many) would be very hard to perfectly match as well.

But I would still argue if you ever did have a theoretical perfect guitar amp model that properly accounted for all the behaviours in an amp (including speaker dymaics at a given dB level), it would effectively cancel all the playing related aspects like "feel" and "responsiveness" and "dynamics" when both systems are fed with the same input guitar signal. Which was the point I was trying to make. There's nothing about the "Feel" of an amp that's inherently un-testable with a null test, because there's nothing about the "feel" of an amp that isn't expressed in the sound we're hearing.

But that theoretically perfect amp model doesn't exist, and practically speaking unless you're null testing the specific amp at the specific settings and specific temperature and volume with the same tubes and the same wear and the same exact settings it's very hard to get a match. Meaning this is most viable for a profiling or capturing solution since it just finished baking all of that minutiae info the snapshot of the amp it just created.

You've also made me come around to thinking that even a null test that only leaves the background noise wouldn't be possible, because there are probably some interactions between the random background noise and the audio signal which would not be able to cancel out.

 

[Greg Ferguson]

You've also made me come around to thinking that even a null test that only leaves the background noise wouldn't be possible, because there are probably some interactions between the random background noise and the audio signal which would not be able to cancel out.

Correct. Not everything is modeled, for instance, thermal noise is not, so any interaction in the real amp would vary from the model. Cliff’s goal was to be within 1 dB, which I think is plenty accurate, and it might be even closer now.

https://wiki.fractalaudio.com/wiki/index.php?title=Amp_block#Authentic_or_idealized_modeling and https://wiki.fractalaudio.com/wiki/...le.2C_intermodulation.2C_crossover_distortion are interesting.

 

[[Nathan]]

I'd imagine the results would be similar to null testing 2 of the same off the shelf tube amps

 

[David Peretz]

This is just out of my pure curiosity, so feel free to delete this thread if it's an inappropriate topic here.

I see some of amp modelers are verified by "Null test" for its identicality to the real counterpart.
The idea of "Null test" is to record 1.the signal from the original amp and 2. the signal from the amp simulator emulating the same amp as 1., and then to play both 1. and 2. out of phase. The premise is that 2. will cancel out 1. if 2. is identical enough.

Has Fractal Audio's amp modeling algorithm ever been verified by this "Null test" for its identicality?
Or, is there any reason that such test is not appropriate to verify identicality of an amp simulator?

What good will come from knowing it nulled or not? Does it make the axefx better or worse? Did you ever herd two guitarrs sound identical to null?
I used to consult in making audio products developed for giant audio companies and software. sometimes i would tell them that this plugins sounded wrong! But it “nulls” to 95% the engineer's would claim and i would tell them that as a producer, what do i care about nulling to this degree of a compressor or that?
just bring me something that sounds good and works like it should! the axefx does both - sound good work like it should, so in the words of frank zappa - “Shut Up 'n Play Yer Guitar” if you prefer other more normalised cultural heroes you can listen to clint eastwood - “if you want to play, play! don’t null”

have fun

 

[lastcaress83]

Well said! I love my Axe FX and was absolutely thrown back while cranking the tweed deluxe model last night. It does so much right and some models are just damn good regardless of whatever shortcomings I may complain about from time to time. Thanks Fractal for the continued effort and improvements.