I think the fundamental error regarding the argumentation is that people think that we test lenses but we don't take any pictures.
However, these tests are all based on images that we've taken. Of course, we do so based on a lab scenario but we still attach the lens to a camera and release the shutter just like you do in the field. The images are then fed into a software to get the results. And yes, if the software generates low number this is also directly visible in Photoshop. So testing is really no magic - it is about taking conventional pictures.
As far as sample variations are concerned. According to our metrics we consider a lens to be Ok if
1. the center is perfectly sharp and symmetrical
2. the borders/corners are "reasonably" symmetrical in all 4 corners.
No lens is really absolutely perfect in this respect that there are limits of what we consider to be acceptable.
To put a number on it - only 3 out of 4 lenses are accepted regarding their centering quality or in other words - the defect rate is 25% (according to official figures the manufacturers communicate 3% - just to mention).
Now there are certainly sample issues beyond that. Possibly the factory received lens elements which follow a different spec for instance. This would result in a sharp center but totally blurred borders/corners. Or a mechanical issue of a sample resulting in a wrong positioning of a lens group. Possibly the tested 18-35 has been affected by this, maybe not. We do not know that of course. Testing multiple lenses is, of course, usually no option. Unlike other sites we receive either lenses from readers or we purchase them on the market. We cannot buy 10 lenses and test them just to make sure. That would be absolutely insane.
You may also have observed that we have tested some third-party lenses across the systems. Take for instances the Tamron 17-50/2.8. The results are all very similar apart from scaling variations due to the specific system parameters.
We do test according to a certain parameter set. Technically the test results are also only valid in this scope (we consider the results to be representative from (physical) mid focus to max. focus but not min focus). It may be possible that a lens performs much worse at this specific setup whereas it is great in other scenarios (e.g. the old Nikkor 60mm is supposed to be great at 1:1 but mediocre at infinity). If so this may be bad luck for the lens but even so it would be a mediocre lens then simply because it does not deliver consistent results.
Again, if you think that your sample is substantially better - just provide it for testing. We are always willing to correct our test data if there's evidence.
cheerio
Klaus
However, these tests are all based on images that we've taken. Of course, we do so based on a lab scenario but we still attach the lens to a camera and release the shutter just like you do in the field. The images are then fed into a software to get the results. And yes, if the software generates low number this is also directly visible in Photoshop. So testing is really no magic - it is about taking conventional pictures.
As far as sample variations are concerned. According to our metrics we consider a lens to be Ok if
1. the center is perfectly sharp and symmetrical
2. the borders/corners are "reasonably" symmetrical in all 4 corners.
No lens is really absolutely perfect in this respect that there are limits of what we consider to be acceptable.
To put a number on it - only 3 out of 4 lenses are accepted regarding their centering quality or in other words - the defect rate is 25% (according to official figures the manufacturers communicate 3% - just to mention).
Now there are certainly sample issues beyond that. Possibly the factory received lens elements which follow a different spec for instance. This would result in a sharp center but totally blurred borders/corners. Or a mechanical issue of a sample resulting in a wrong positioning of a lens group. Possibly the tested 18-35 has been affected by this, maybe not. We do not know that of course. Testing multiple lenses is, of course, usually no option. Unlike other sites we receive either lenses from readers or we purchase them on the market. We cannot buy 10 lenses and test them just to make sure. That would be absolutely insane.
You may also have observed that we have tested some third-party lenses across the systems. Take for instances the Tamron 17-50/2.8. The results are all very similar apart from scaling variations due to the specific system parameters.
We do test according to a certain parameter set. Technically the test results are also only valid in this scope (we consider the results to be representative from (physical) mid focus to max. focus but not min focus). It may be possible that a lens performs much worse at this specific setup whereas it is great in other scenarios (e.g. the old Nikkor 60mm is supposed to be great at 1:1 but mediocre at infinity). If so this may be bad luck for the lens but even so it would be a mediocre lens then simply because it does not deliver consistent results.
Again, if you think that your sample is substantially better - just provide it for testing. We are always willing to correct our test data if there's evidence.
cheerio
Klaus