And that is the core problem with what happened. There may actually be a grain of truth but now there is a backlash. I'd argue though that the mounds of alternative explanations that weren't followed up on should likely get some priority right now since we know so little about them there is a lot to learn and and we are likely to have a lot of surprises there.
I see this as the same problem with UCT (upper confidence for trees) based algorithms. If you get a few initial random rolls that look positive you end up dumping a lot of wasted resources into that path because the act of looking optimizes the tree of possibilities you are exploring (it was definitely easier to study amyloid lines of research than other ideas because of the efforts put into it). Meanwhile the other possibilities you have been barely exploring slowly become more interesting as you add a few resources to them. Eventually you realize that one of them is actually a lot more promising and ditch the bad rut you were stuck on, but only after a lot of wasted resources. To switch fields, I think something similar happened to alpha-go when it had a game that ended in a draw because it was very confident in a bad move.
Basically, UCT type algorithms prioritize the idea that every roll should optimize the infinite return so it only balances exploration with exploitation. When it comes to research though the value signal is wrong, you need to search the solution space because your goal is not to make every trial find the most effective treatment, it is to eventually find the actual answer and then use that going forward. The trial values did not matter. This means you should balance exploration, exploitation AND surprise. If you do a trial that gives you very different results than you expected then you have shown that you don't know much there and maybe it is worth digging into so even the fact that it may have returned less optimal value than some other path its potential value could be much higher. (Yes I did build this algorithm. Yes it does crush UCT based algorithms. Just use variance as your surprise metric then beat alpha-go.)
People intrinsically understand these two algorithms. In our day to day lives we pretty exclusively optimize exploration and exploitation because we have to put food on the table while still improving, but when we get to school we often take classes that 'surprise' us because we know that the goal at the end is to have gained -some- skill that will help us. Research priorities need to take into account surprise to avoid the UCT rut pitfalls. If they had for the amyloid hypothesis maybe we would have hopped over to other avenues of research faster. 'The last 8 studies showed roughly the same effect, but this other path has varied wildly. Let's look over there a bit more.'
I see this as the same problem with UCT (upper confidence for trees) based algorithms. If you get a few initial random rolls that look positive you end up dumping a lot of wasted resources into that path because the act of looking optimizes the tree of possibilities you are exploring (it was definitely easier to study amyloid lines of research than other ideas because of the efforts put into it). Meanwhile the other possibilities you have been barely exploring slowly become more interesting as you add a few resources to them. Eventually you realize that one of them is actually a lot more promising and ditch the bad rut you were stuck on, but only after a lot of wasted resources. To switch fields, I think something similar happened to alpha-go when it had a game that ended in a draw because it was very confident in a bad move.
Basically, UCT type algorithms prioritize the idea that every roll should optimize the infinite return so it only balances exploration with exploitation. When it comes to research though the value signal is wrong, you need to search the solution space because your goal is not to make every trial find the most effective treatment, it is to eventually find the actual answer and then use that going forward. The trial values did not matter. This means you should balance exploration, exploitation AND surprise. If you do a trial that gives you very different results than you expected then you have shown that you don't know much there and maybe it is worth digging into so even the fact that it may have returned less optimal value than some other path its potential value could be much higher. (Yes I did build this algorithm. Yes it does crush UCT based algorithms. Just use variance as your surprise metric then beat alpha-go.)
People intrinsically understand these two algorithms. In our day to day lives we pretty exclusively optimize exploration and exploitation because we have to put food on the table while still improving, but when we get to school we often take classes that 'surprise' us because we know that the goal at the end is to have gained -some- skill that will help us. Research priorities need to take into account surprise to avoid the UCT rut pitfalls. If they had for the amyloid hypothesis maybe we would have hopped over to other avenues of research faster. 'The last 8 studies showed roughly the same effect, but this other path has varied wildly. Let's look over there a bit more.'