What I was trying to say is that all of this comes down to finding transforms in a very large search space that convert certain inputs to certain outputs. So maybe there are generalized algorithms that can do that better than GAs or NNs or any of the other specific approaches.
Look at all of the effort that has been put into optimizing rasterization in 3D graphics. But meanwhile a student can write a ray tracer in a page of code. I would have preferred that the industry put more effort into the ray tracing side because the abstractions are so much simpler that it would have progressed the state of the art further. Instead we ended up with relatively complex and proprietary implementations of SIMD and that's great and everything but that completely overshadowed the alternatives.
And at the end of the day, users don't care if their 3D framework uses ray tracing or rasterization. All they really see is performance or efficiency under the current paradigm.
So when I see pages and pages of relatively cryptic NN code, I wonder to myself if maybe some other simple curve-fitting or hill-climbing algorithms would produce the same results. Or maybe even spitballing with a GA and letting the computer discover the algorithm would work just as well. It seems like with 10 times the computing power, we could use algorithms that are 10 times simpler. But I'm not really seeing that.
Ok to be a bit more concrete: say you have teams all competing to write the best sorting algorithm. Maybe they all independently derive each of the main ones listed here:
But none of them read the fine print to see that big-O complexity wouldn't be judged, just code size. So the bubble sort team ends up winning with the simplest implementation.
Maybe the judges are running the contest in order to find the smallest code that performs sorting. Maybe they have a special computer with a billion cores that can only hold 256 bytes of code each. But the contestants are still thinking linearly in terms of serial execution so submit solutions that really don't help.
I feel like everyone is focusing on the details of whether to use RNNs or CNNs or any of the other types of NN:
When we should really have "classifier" algorithm that works like a "sort" function in a programming language. The computer should use the best machine learning model for the use case automagically and not trouble us with implementation details. Then we should be able to build up larger constructs out of those basic building blocks.
I'm not articulating this very well. Just trying to express a general frustration I've had with AI from the very beginning and trying to point out alternatives that could get more people involved and bring us artificial general intelligence (AGI) sooner.
Look at all of the effort that has been put into optimizing rasterization in 3D graphics. But meanwhile a student can write a ray tracer in a page of code. I would have preferred that the industry put more effort into the ray tracing side because the abstractions are so much simpler that it would have progressed the state of the art further. Instead we ended up with relatively complex and proprietary implementations of SIMD and that's great and everything but that completely overshadowed the alternatives.
And at the end of the day, users don't care if their 3D framework uses ray tracing or rasterization. All they really see is performance or efficiency under the current paradigm.
So when I see pages and pages of relatively cryptic NN code, I wonder to myself if maybe some other simple curve-fitting or hill-climbing algorithms would produce the same results. Or maybe even spitballing with a GA and letting the computer discover the algorithm would work just as well. It seems like with 10 times the computing power, we could use algorithms that are 10 times simpler. But I'm not really seeing that.
Ok to be a bit more concrete: say you have teams all competing to write the best sorting algorithm. Maybe they all independently derive each of the main ones listed here:
https://en.wikipedia.org/wiki/Sorting_algorithm#Comparison_o...
But none of them read the fine print to see that big-O complexity wouldn't be judged, just code size. So the bubble sort team ends up winning with the simplest implementation.
Maybe the judges are running the contest in order to find the smallest code that performs sorting. Maybe they have a special computer with a billion cores that can only hold 256 bytes of code each. But the contestants are still thinking linearly in terms of serial execution so submit solutions that really don't help.
I feel like everyone is focusing on the details of whether to use RNNs or CNNs or any of the other types of NN:
https://medium.com/@datamonsters/artificial-neural-networks-...
When we should really have "classifier" algorithm that works like a "sort" function in a programming language. The computer should use the best machine learning model for the use case automagically and not trouble us with implementation details. Then we should be able to build up larger constructs out of those basic building blocks.
I'm not articulating this very well. Just trying to express a general frustration I've had with AI from the very beginning and trying to point out alternatives that could get more people involved and bring us artificial general intelligence (AGI) sooner.