The concept of time is closely intertwined with the existence of matter and the physical universe as we understand it. In the framework of general relativity, which is our current best theory of gravity, matter and energy curve spacetime, and this curvature is what we perceive as the flow of time.
If there were no matter or energy present in the universe, it would mean there are no gravitational effects to curve spacetime, and consequently, the concept of time as we know it would not exist in that context. However, it's important to note that the absence of matter does not imply the complete absence of spacetime itself, which is still considered to be a fundamental fabric of the universe.
Regarding your question about moving at infinite speed through time and space, it's not possible within our current understanding of physics. The theory of relativity, specifically special relativity, imposes limitations on how fast an object can travel through space and time. According to special relativity, as an object approaches the speed of light, its relativistic mass increases, requiring an infinite amount of energy to reach or exceed the speed of light. As a result, it is considered impossible to achieve infinite speed or exceed the speed of light.
Furthermore, time dilation, a phenomenon predicted by special relativity, states that as an object approaches the speed of light, time slows down for that object relative to an observer at rest. However, this does not allow for travel into the past, but rather a difference in the perception of time between the moving object and the observer.
It's worth noting that the hypothetical scenarios involving infinite speed or time travel often lead to paradoxes and logical inconsistencies. They are subjects of scientific and philosophical speculation, but they remain speculative as they go beyond our current understanding of physics.