I think you're right that for most of antiquity and the middle ages it was very rare for the typical European to see a nonwhite person or have much idea of how such a person might look. Even my mother, who grew up in the Mountain West of the US in the 1950s and 1960s once told me that until she left for the East Coast when she was around 20 years old, she'd only seen nonwhite people once or twice in her life.It was very unusual for an average European to see a non-White person. You are talking about extreme circumstances and the reaction by the Europeans was brutal and lead to a lot of bloodshed as a result. Even then, of then, of the millions of Europeans who lived in Europe from the beginning of time until 1945, that encountered non-Whites in a group large enough to display their own preferences, were tiny, and of those who did, they did not approve of these preferences, such as the Moors being run out of Spain.
My point being that for thousands of years, Europeans never thought of themselves as "White" because it was never a real question posed to them. Technology dictated that most of their life was in the same village and their biggest threat was an invasion by other Whites from a neighboring kingdom and eventually nation. It hasn't been, until the last 30 to 40 years, that enough non-Whites have moved into Europe, to make them setup their own enclaves, to allow the native Europeans to realize both how different they are, and how long term this isn't going to work out. And now they are starting to identify as White. This is a new pattern in Europe, though it has existed in the western hemisphere much longer because there were already numbers of non-Whites forming their own enclaves and the difference was stark.
Europeans once thought of American Whites as just racist backwards uneducated hicks. Now that large sections of European cities have become non-go zones, just like in the USA, and crime is sky rocketing, and the invaders are bankrupting the socialist system by having 5 kids with no job, attitudes are changing quickly.
And that is my simple point. It is all human nature. If 100 Africans move to London, they will be seen as celebrities. People wanting to talk to them, wanting to help them, wanting to welcome them. And most likely these 100 Africans will fit in for the most part and there will be no issues. But when you bring in 100,000 Africans and they form their own enclaves and crime skyrockets and people's grandmothers get mugged and statues of their hero's are torn down to appease them, then you see a whole new outlook and attitude arise.
It makes me think of how liberals and leftists will sometimes lose it about traditional depictions of Jesus as white and European. Leaving aside the fact that I think it's very hard to say exactly what the typical skin tone was in a given region of the world millennia ago, of course Europeans would have depicted Jesus as looking like them because they mostly had at best a vague idea of what non-Europeans might have looked like, if they had any idea at all. Similar to how when Asian Christians depicted Christ in previous centuries he looked, not surprisingly, Asian.