They talk about that being from "a neural network trained on places", but it seems to have only one pagoda in mind. Is it poor training, or is it an artefact of earlier layers being prone to isolating features most prominent in pagodas.
This is one of the tricky problems with neural networks that you have to be careful of. The network might not actually be training itself on what you think it ought to be or want it to be learning and perhaps gets "obsessed" with a particular feature - and once over-trained, it then either sees everything in terms of that feature it's obsessed by, or just flatly disregards anything that doesn't match as not important. And the problem is that, because training is basically a feedback loop, "over-training" can become a vicious downward spiral. Providing even more data can just makes it worse, not better.
The classic example of the pitfalls of neural networks was when the US military experimented with using a neural network to identify tanks for an intelligent "friend or foe" system. They had photos of US tanks and of Russian tanks, and trained the neural network up on this data.
All seemed well. When they provided it more photos that it had never seen before, it was correctly identifying the American and Russian tanks. Great. It's learnt the difference. So, let's put the system into a tank and test it in the field (but, obviously, not with any actual armaments, just testing that the "friend or foe" system was functioning properly in real world conditions).
And, suddenly, the results were completely wrong. It was identifying all the American tanks as Russian tanks.
Eventually, they worked out what had gone wrong. Because, of course, the Americans own the American tanks then they have zero problems getting access to them and all the photos of the US tanks were taken in bright sunshine in the middle of the day. But the photos of Russian tanks were taken by spies in secret, when no-one was looking, so those photos tended to be taken at twilight or at night.
Yes, you've guessed it. What the neural network had actually trained itself up on was determining whether a picture was taken in the day or taken at night. If it thought the picture was taken during the day, then it said "US tank" and, if not, it said "Russian tank". And, due to how the photos were obtained, this coincidentally corresponded as being the right answer most of the time.
And when they tested it in the field? Well, they did it in the evening. And, as it got dark, the network determined that this meant "Russian tank", regardless of what you placed in front of it.
When dealing with neural networks, these are the kinds of problems that you have to be keenly aware of to avoid. When it looks at the data, it learns from it. So the problem is that if you give it data that has other incidental patterns in it - the day and night thing - then it could well decide to focus in on that and then, as you provide it more photos of tanks in the day and night which just so happen to fall into the "day = US tank" / "night = Russian tank" pattern, you are unintentionally over-training it to think that this feature is the most important.
Because every time it guesses along these lines, it gets confirmation that it's got the right answer, so - as this seemingly works perfectly - it keeps doing what it's doing and further cements the strong neural connections to the point where they become inflexible and stuck. The network has erroneously become "obsessed" with this particular feature.
Indeed, more input ought to make a network more generic - it's seeing more variants and differences, so it's forming a more generic idea of what it's looking at - but, with "over-training", you end up with the complete opposite. It gets stuck. It gets obsessed. It starts seeing everything in terms of this feature. The more data you provide, the more it sees that feature in everything and makes its "obsession" even worse and it starts to disregard any counter-examples as "not relevant".
What's interesting - because it kind of reflects that famous Jesuit quote: "give me the child for the first seven years, and I'll give you the man" that indoctrination is best applied at the youngest age - is that the initial data you supply has the most influence here.
In that if it picks up the "day / night" distinction within the first few data, then all subsequent training thereafter will have a tendency to just reinforce that further. And if it's over-reinforced, then the pattern can get so "stuck" that it'll take a heck of a lot of counter examples that defy the pattern to build up enough "oomph" to overcome how ingrained and "burnt in" that pattern's become.
But the thing is, supply it the exact same data - but in a different order that makes that "initial picture" more generic and varied - and then subsequent data will tend to reinforce the genericism that you're usually looking for, rather than reinforcing that initial mistake. And it's not different data, it's just the same data in a different order.
Basically, if it gets the wrong idea early then that tends to get in the way of further training, because it sees things in terms of that wrong idea and even if, with further data, you can eventually disabuse it of its misconceptions then, well, you've just spent all that time doing nothing more than correcting its misunderstanding, it's not really helping it too much to understand what you actually want it to understand.
But if it catches onto the right idea early, then more data will just help it refine that understanding.
Same data, different order - can end up with quite interestingly different results, just because early data tends to be more influential than later data. Perhaps, very simply, the first few photos of buildings it was fed were pagodas and, from then on, it was - to horribly anthropomorphise it as one really shouldn't - thinking of everything in terms of "is this a pagoda? How is this like a pagoda? What features does it share in common with a pagoda? This isn't a pagoda - but let's interpret it in terms of how not-pagoda-like it is" (so even when it knows it's not a pagoda, it's still the case that the pagoda is the measure of all things in its universe regardless).
You get the gist. It gets a bit "obsessed" with pagodas and starts interpreting everything in terms of them - because, well, right at the beginning of this process, it knew nothing else but pagodas. It could ONLY interpret what it saw in terms of pagodas, as that's all it, at that point, knew.
But if those first pictures had been bridges, it might have rather gotten obsessed with arches instead and then seen arches in everything. You can see, from how it renders things, that with the "animals" training, it got rather freak-ally obsessed with eyes. Any hint of a curve anywhere and it sees it as an eye.
"Words ought to be a little wild - for they are assaults of thought on the unthinking."
- John Maynard Keynes
I really recommend reading over, theres a few things going on and its all pretty interesting.
Some of it sounds really useful frankly, even if 99% of uses are now going to be cool pictures of a neural net "seeing what could be there" with random things.
This is fascinating stuff. Gives me the same feeling I got when first dabbling with fractals on the Amiga.
Complete faff to get it working it seems (for dunce level dabblers like myself).
I found a web-based version of it but have yet to see it working properly.. deepdreams.zainshah.net/ (hopefully you will have more luck).
This stuff and surrounding possibilities kept me awake last night Not often something comes along that tickles me wetware like that.
I think theres a pretty good chance it is close to what our brain is doing when on some drugs.
Although the question is do LSD and such trigger our brain to "over-extrapolate" by enhancing certain functions, or do they do it by suppressing a filter we have that normally have that shuts this stuff out?
Reminds me of an article I read recently demonstrating that most available neural networks can't tell the difference between a leopard and a couch with leopard-print upholstery. Once it learned to recognize leopard spots, as distinct from other kinds of spots, that's all it focused on. It's a pretty common failing that these networks will latch onto the simplest possible distinctive feature and ignore all others.
Well, thats one of the main use's of Googles idea here; They can run stuff though backwards to see exactly what features the neural net is picking up on.
The example with the dumb bells is particularly good;
As it made it clear the network considered the arm part of it too, as those where the only images it was trained on.
Not often something comes along that tickles me wetware like that.
