There is no luminiferous aether but that doesn't mean that space is empty. It is made up something which provides medium to light. Einstein says any such things would slow down the light. Yes it is slowing down the light that is why light is limited to 3,00,000 Km/s.
Speed of a wave depends on the properties of its medium. Space is determining the speed of light and thus the scale of time that is why there is a sequence of events in the universe. In the absence of this limit to the speed of light, all events would have happened instantaneously.