From what you typed, sounds like you didn't re-run the same benchmark. You cannot compare any bench of their against any other.
What was your 3d11 score now? Time Spy and sky diver aren't even remotely close to being the same thing. Run 11 again and see what score you receive.
Yes x as t8me goes on, these get more difficult and score lower in the same hardware.