you should use standard deviation instead of variance for consistency. it gives a better representation of how consistent the readings are compared to just the range of what was observed.
Using an icp or other comparable lab based test as your control would’ve be more appropriate that an average of all of them. Further standard deviation would’ve been a better way to analyze your data than simply looking at the range. As an engineer, how you analyze your data is just as important as how you collect it. Nonetheless good work!
What would have been great would be to have sent those same water samples for ICP testing to a couple of different icp companies. Thus helping on the accuracy portion since the baseline could be way off as its just an average and nota definitive fact that its correct values.
A couple comments on the video, first referring to the "accuracy" of a test kit by comparing it to the average is really wrong, like supremely wrong, just look at the Lamotte results and how absurdly high they were which ends up bringing the entire average up quite drastically. A "better" way to do it is to use one of said "costs more than your whole reef" testers which you seem to have (shown in video?), and while that doesn't mean it's more accurate, it does reduce the level of uncertainty related to the test kit itself when comparing lab grade equipment to hobby grade. Second, just as a point of visual emphasis maybe color code the individual people tests, like I know who Randy and Ryan are and they represent the "middle of the road" aquarists but I already forgot the names of the "pro" testers, so maybe as an individual I can maybe give more weight to the numbers that the "pros" got compared to the average person.
I agree. I would also add that measuring consistency among all different users is not the same as measuring consistency among one user performing the tests 10 times. The later being a much more useful and important metric. I love what you guys do, but this might be the first video I actually dislike.
Averaging results to come up with an "accurate" number is such bad science. They could all be over or undershooting the true value. Calibration / Reference solution with known values that all the tests use as a sample would show how close each can get to that known value. Consistency being based of the average of 5 different people doesn't seem like the right word either. Randy using the same Aquaforest, Nyos, Hanna test 5 times and getting exactly the same results means it gives consistent results. Another reefer may be 50-100ppm higher but hits the same number all 5 times as well. In this scenario it just shows the test is maybe a little hard to follow and has easy user error mistakes. The color shift is being interpreted consistently but the exact correct hue is up for debate. I'd hope BRS can redo this Investigates episode sometime in the future.
There are some variables to take into account. 1st the accuracy of each test. Hanna clains +-0.3dkh, redsea +- 0.14 dkh, Salifert +-0.1 dkh. Second you should use a reference certificated solution, consider the kit expiration date, price and of course discuss the importance of the "true value" and what is a tolerance that a hobbist can deal with. To me an alk of 7.0 or 7.3 dkh is the same and is 7is. I would love BRS sell the powders for we make the reference solutions like alk, mg, ca at home that is very simple and only requires an analitical weighter and a graduated flask
It would have been nice to run the same water sample against ICP-OES to see if your base line aligned with ICP testing. How close does mean line match machine analysis?
I know you guys don't sell them and they are often maligned in the hobby, but I would like to see API included in the test kit videos. It would provide valuable information as the bottom of the line in cost/accuracy and what many start with in the hobby.
A video on lab grade units would be helpful too. I realize the cost for such units are much higher than hobby kits, given the overall cost of most enthusiasts reef systems, I believe there is an audience for the information/comparison of whats available to them. The cost of a decent tester is about the same as high end lighting fixtures. So the market is there. I believe the general reef enthusiast doesn't consider $1k to $2k on a lab grade tester not just because of the cost, but more likely because it just isn't a topic of discussion. Personally, I chose to invest in a used Hanna lab multi tester before I invested in an apex controller. I felt the need to have a clear insight into the water quality supersedes the convince of automation. Of course having both is an overall goal.
For alkalinity, Hanna makes a specific checker for fresh and saltwater, so you'll need to use the one that corresponds to the type of water you're testing.
I can't remember exactly, but the corals in the 160 were originally stocked by WWC, Austin Aqua Farms, and BattleCorals, so it's likely from one of those three vendors 🙂
So not only did BRS use tank water and not a standard solution for each test, they didnt even have a lab grade, calibrated piece of equipment to find the true value and compare all the tests to it. This test was ok.... and probably the best I would expect from someone without a heavy chemistry background. But what they should have done was used a standard solution to test the accuracy of the test kits. A standard solution is prepared following a rigourous analytical procedure to make sure that the concentration of the analyte is known. Then this solution is used to determine the accuracy of the test kit. They used tank water and averaged the results of each. The accuracy of all the tests could have been shifted by 100 ppm (not likely, I know) and we would have never known. If they wanted to use tank water instead of a standard solution to give more "real world results," they should have used some form of a control, ie. a piece of testing equipment with known variance and error that has been calibrated using a standard solution as a reference to compare the tests to. The only thing they really tested was the kit's precision. But they really didnt even do that correctly either, because each person only performed the test once. So we dont know how much variance there is within each person performing the test, and thus it messes up the calculation of the overall variance of each test kit. I'm a little disappointed in the video.
Alkalinity, calcium, and magnesium are most commonly tested in a reef tank, but it's not a bad idea to test other parameters too. In particular, nitrates and phosphates.
Good video but the results are misleading. As others have pointed out average doesn’t tell you anything. You need a reference value, either with a very high end kit or by mixing your lab grade salt and using that as a standard or using an off the shelf calibrated standard solution. Standard deviation is really the important thing here because some of these kits provide results that span all over the range and should never be used. That can be useful without knowing the true value of alk ca and mg. Really what we’re seeing is that all these kits suck, some just suck more than others. Also can you use different kits off different batches?
I actually got rid of the trident because of how inaccurate it was it never matched any test kit and the calibration fluid that comes with the reagents is terrible and not accurate at all which makes the trident inaccurate even if you use your tank water to calibrate the trident based off of your test kits and your tank water it still doesn’t match up like it supposed to and it changes overtime the worst $600 purchase I ever made
I believe there are some online tests that you can take. If you have further questions, I would suggest reaching out to an expert on eyesight like an ophthalmologist
Matyas Castro I'm wondering how Lamott could have fared so poorly. I'm inclined to take its methods very seriously. It's interesting, though, that its approach may require a more skillful tester.
We've only ever done a video on a product that we don't carry once and won't do it again. We've looked at the API test kits in the past and our product team decided that they don't fit our good, better, best lineup.
While it's fair to say that a large number of reefers do start out with lower cost kits like the API version, we have only tested a product that we don't carry once and won't do it again. We'd be jerks to publicly review items that we've looked at and decided not to bring in for one reason or another.
@@BRStv You wouldn't be jerks. You'd be respected. What I just learned is that this was just marketing. I love BRS but this didn't feel like it was about us reefers. Felt like it was mainly about your suppliers. If it's about reefing, it has to be more open. Should be more about what we use. Not what you sell. If you sell it and present good reasons you sell an item through comparison, we will throw our money at you. Just sayin.
That's not Accuracy, that's merely the closest to mean. Assuming the average of a variety of test kits is the "correct" answer is dangerous given the ranges of these test results.
It's less about the test kits not working and more about which test kits are more accurate or easier to use than others. The mantra "right tool for the right job" definitely applies here. If you need an ultra accurate reading of a certain parameter, you may choose one test kit vs another. If you need just a general ballpark, you may choose a different kit.
So after watching the video and reading the comments, here is the short answer: No, you can not trust test kits or yourself. Long answer: Find a test that is consistent for YOU and stick with it. More than likely it is wrong but it doesn't really matter that it is wrong IF your tank looks good. A seasoned reefer can look at a tank and see if something isn't quite right. Doesn't matter if the alkalinity is 7 or 12 on the test kit. It could actually be 9 but if you consistently get the same "7" or "12", doesn't matter. For this reason alone, I think there are only a couple things to consider now when buying a test kit. How easy is the endpoint to read (this will mean consistency in your testing) and what is the cheapest of the kits you can properly read.
Usually, I really love your Videos But this one doesn’t make any sense. To use the average as the reference is completely wrong. If you don’t know the exact number of the parameter you are testing, then you can’t say anything about accuracy.
Agreed, I wish that the tests would have been referenced against one or more of the ICP tests
you should use standard deviation instead of variance for consistency. it gives a better representation of how consistent the readings are compared to just the range of what was observed.
a3000gt but then someone would have to take stats101. 😂
That actually kinda triggered me.
Using an icp or other comparable lab based test as your control would’ve be more appropriate that an average of all of them. Further standard deviation would’ve been a better way to analyze your data than simply looking at the range. As an engineer, how you analyze your data is just as important as how you collect it. Nonetheless good work!
Also doing a t test on the data would show if there is any real significance between data sets, although that may be out of scope:)
Anthony Posyton Yeah, then they'd need to understand the concept of a standard error. 😉
Yes! Students t-test, but they didnt even make to the point that the could. Heck if they had a good methodology, I would have run the t-test for them.
What would have been great would be to have sent those same water samples for ICP testing to a couple of different icp companies. Thus helping on the accuracy portion since the baseline could be way off as its just an average and nota definitive fact that its correct values.
A couple comments on the video, first referring to the "accuracy" of a test kit by comparing it to the average is really wrong, like supremely wrong, just look at the Lamotte results and how absurdly high they were which ends up bringing the entire average up quite drastically. A "better" way to do it is to use one of said "costs more than your whole reef" testers which you seem to have (shown in video?), and while that doesn't mean it's more accurate, it does reduce the level of uncertainty related to the test kit itself when comparing lab grade equipment to hobby grade.
Second, just as a point of visual emphasis maybe color code the individual people tests, like I know who Randy and Ryan are and they represent the "middle of the road" aquarists but I already forgot the names of the "pro" testers, so maybe as an individual I can maybe give more weight to the numbers that the "pros" got compared to the average person.
I agree. I would also add that measuring consistency among all different users is not the same as measuring consistency among one user performing the tests 10 times. The later being a much more useful and important metric. I love what you guys do, but this might be the first video I actually dislike.
Averaging results to come up with an "accurate" number is such bad science. They could all be over or undershooting the true value.
Calibration / Reference solution with known values that all the tests use as a sample would show how close each can get to that known value. Consistency being based of the average of 5 different people doesn't seem like the right word either. Randy using the same Aquaforest, Nyos, Hanna test 5 times and getting exactly the same results means it gives consistent results. Another reefer may be 50-100ppm higher but hits the same number all 5 times as well. In this scenario it just shows the test is maybe a little hard to follow and has easy user error mistakes. The color shift is being interpreted consistently but the exact correct hue is up for debate.
I'd hope BRS can redo this Investigates episode sometime in the future.
There are some variables to take into account. 1st the accuracy of each test. Hanna clains +-0.3dkh, redsea +- 0.14 dkh, Salifert +-0.1 dkh. Second you should use a reference certificated solution, consider the kit expiration date, price and of course discuss the importance of the "true value" and what is a tolerance that a hobbist can deal with. To me an alk of 7.0 or 7.3 dkh is the same and is 7is. I would love BRS sell the powders for we make the reference solutions like alk, mg, ca at home that is very simple and only requires an analitical weighter and a graduated flask
The only way this works is if you know the exact levels of what you are testing. Who knows technically the Lamontte could be the most accurate
yup
This interesting information but I really wish that this was performed with a Known Standard rather than "random" tank water.
@@BRStv Tropic marin sells a Multi Standard solution you can use as reference.
@@BRStv then mix it up yourself, use math to make a solution with nice even numbers.
This
It would have been nice to run the same water sample against ICP-OES to see if your base line aligned with ICP testing. How close does mean line match machine analysis?
#1 problem is that most people do not account for the meniscus when measuring the sample water which will skew the results.
This right here
I know you guys don't sell them and they are often maligned in the hobby, but I would like to see API included in the test kit videos. It would provide valuable information as the bottom of the line in cost/accuracy and what many start with in the hobby.
Great testing and experiment! Try it again with a bunch of trident calibration bottles.
A video on lab grade units would be helpful too. I realize the cost for such units are much higher than hobby kits, given the overall cost of most enthusiasts reef systems, I believe there is an audience for the information/comparison of whats available to them.
The cost of a decent tester is about the same as high end lighting fixtures. So the market is there. I believe the general reef enthusiast doesn't consider $1k to $2k on a lab grade tester not just because of the cost, but more likely because it just isn't a topic of discussion.
Personally, I chose to invest in a used Hanna lab multi tester before I invested in an apex controller. I felt the need to have a clear insight into the water quality supersedes the convince of automation. Of course having both is an overall goal.
The Hanna HI931 is a fantastic automatic titrator. I use one in our lab at work and can say that it is the real deal.
Can I use the Hanna alkanity tester for both my freshwater and saltwater fish tanks? Do you really need to buy 2 different ones?
Thanks in advance
For alkalinity, Hanna makes a specific checker for fresh and saltwater, so you'll need to use the one that corresponds to the type of water you're testing.
Please make a new comparison including ICP test and Mastertronic and Reefbot devices.
Where can I get that Stylo at 2:52? Is that a Rainbow Stylo? Wow that;'s nice
I can't remember exactly, but the corals in the 160 were originally stocked by WWC, Austin Aqua Farms, and BattleCorals, so it's likely from one of those three vendors 🙂
Love Red Sea! Thanks for the experiment.
should have definitely compared these to a ICP test.
Honestly bi weekly water changes is doing it for me....But I'm only messing with softies ;)
It says Ryan on the screen when u said u were randy
I’m dyslexic I’m just wondering if there’s a merchene I can just leave in the corner of my set up that records pH serlinaty and outher stuff.
The Apex is a great tool for reading and recording data like pH and salinity if it's within your budget.
So not only did BRS use tank water and not a standard solution for each test, they didnt even have a lab grade, calibrated piece of equipment to find the true value and compare all the tests to it.
This test was ok.... and probably the best I would expect from someone without a heavy chemistry background. But what they should have done was used a standard solution to test the accuracy of the test kits. A standard solution is prepared following a rigourous analytical procedure to make sure that the concentration of the analyte is known. Then this solution is used to determine the accuracy of the test kit. They used tank water and averaged the results of each. The accuracy of all the tests could have been shifted by 100 ppm (not likely, I know) and we would have never known.
If they wanted to use tank water instead of a standard solution to give more "real world results," they should have used some form of a control, ie. a piece of testing equipment with known variance and error that has been calibrated using a standard solution as a reference to compare the tests to.
The only thing they really tested was the kit's precision. But they really didnt even do that correctly either, because each person only performed the test once. So we dont know how much variance there is within each person performing the test, and thus it messes up the calculation of the overall variance of each test kit.
I'm a little disappointed in the video.
Do you use to test other parameters as KH,Ca and Mg?
Alkalinity, calcium, and magnesium are most commonly tested in a reef tank, but it's not a bad idea to test other parameters too. In particular, nitrates and phosphates.
@@BRStv ok Thank you
so are you saying we need to get different brand test kits for different chemical tests
The moral of this story is that you should pick a test kit that is easy for you to perform and one that you can get consistent results with
Be nice to see 5x Trident tests along with these...
Should I get aqua forest kit?
Is there a video on testing to get best results?
Good video but the results are misleading. As others have pointed out average doesn’t tell you anything. You need a reference value, either with a very high end kit or by mixing your lab grade salt and using that as a standard or using an off the shelf calibrated standard solution.
Standard deviation is really the important thing here because some of these kits provide results that span all over the range and should never be used. That can be useful without knowing the true value of alk ca and mg.
Really what we’re seeing is that all these kits suck, some just suck more than others.
Also can you use different kits off different batches?
So which brand should i get as a new saltwater reefer
We use a lot of Red Sea brand kits around here. That said, we do typically use the Hanna checker for alkalinity as well as phosphates.
If it's accurate, would that not make it consistent?
Yo Ryan, what the heck are you doing when testing??. Thanks for the video BRS.
any chance you guys can add some of the other common testers out there like Alkatronics, KH director and KH guardian?
The KH director is the only one of those three that we carry, so while nothing is planned right now, I certainly wouldn't rule it out 🙂
Keep the awesome thumbnails coming. 🤷♂️
I actually got rid of the trident because of how inaccurate it was it never matched any test kit and the calibration fluid that comes with the reagents is terrible and not accurate at all which makes the trident inaccurate even if you use your tank water to calibrate the trident based off of your test kits and your tank water it still doesn’t match up like it supposed to and it changes overtime the worst $600 purchase I ever made
How do you test if you're colorblind?
I believe there are some online tests that you can take. If you have further questions, I would suggest reaching out to an expert on eyesight like an ophthalmologist
*Lamotte has left the chat*
Matyas Castro lol
Matyas Castro I'm wondering how Lamott could have fared so poorly. I'm inclined to take its methods very seriously. It's interesting, though, that its approach may require a more skillful tester.
this feels like a sports race.
dear brs, i see a lot of lab grade products, but where are your lab coats?
Actually, we'd consider all of the test kits in this video to be hobby grade kits 🙂
@@BRStv i mean can you sell us some medical jackets with your logo?
why not do api, while not known for accuracy.. probably the most used in the hobby
We've only ever done a video on a product that we don't carry once and won't do it again. We've looked at the API test kits in the past and our product team decided that they don't fit our good, better, best lineup.
What about Strontium and potassium?
We haven't tested any strontium or potassium kits just yet, but that would be interesting for sure!
Good job
Where was API? Most if not everyone started out with it and many still use it. C'mon BRSTV, think about the beginners too.
While it's fair to say that a large number of reefers do start out with lower cost kits like the API version, we have only tested a product that we don't carry once and won't do it again. We'd be jerks to publicly review items that we've looked at and decided not to bring in for one reason or another.
@@BRStv You wouldn't be jerks. You'd be respected. What I just learned is that this was just marketing. I love BRS but this didn't feel like it was about us reefers. Felt like it was mainly about your suppliers. If it's about reefing, it has to be more open. Should be more about what we use. Not what you sell. If you sell it and present good reasons you sell an item through comparison, we will throw our money at you. Just sayin.
That's not Accuracy, that's merely the closest to mean. Assuming the average of a variety of test kits is the "correct" answer is dangerous given the ranges of these test results.
Why would they make test kits that dont work? Sounds like a lawsuit to me
It's less about the test kits not working and more about which test kits are more accurate or easier to use than others. The mantra "right tool for the right job" definitely applies here. If you need an ultra accurate reading of a certain parameter, you may choose one test kit vs another. If you need just a general ballpark, you may choose a different kit.
Wheres my dam brand API
I think that if we all do better than Aaron, we will all be just fine, hope he’s better at customer service than testing!,
fail, what if lamotte was right? you cant know by guessing.
So after watching the video and reading the comments, here is the short answer: No, you can not trust test kits or yourself.
Long answer: Find a test that is consistent for YOU and stick with it. More than likely it is wrong but it doesn't really matter that it is wrong IF your tank looks good. A seasoned reefer can look at a tank and see if something isn't quite right. Doesn't matter if the alkalinity is 7 or 12 on the test kit. It could actually be 9 but if you consistently get the same "7" or "12", doesn't matter.
For this reason alone, I think there are only a couple things to consider now when buying a test kit. How easy is the endpoint to read (this will mean consistency in your testing) and what is the cheapest of the kits you can properly read.
Usually, I really love your Videos
But this one doesn’t make any sense.
To use the average as the reference is completely wrong. If you don’t know the exact number of the parameter you are testing, then you can’t say anything about accuracy.