I guess the question is whether time is orthogonal to space.
Consider just three spatial dimensions. That’s a 3-D space. You have an X, Y, and Z axis and you can perfectly travel along the X axis without also traveling through the Y and Z axis. So if you just add another fourth temporal dimension with a T axis, if that T axis is orthogonal to all three-dimensional axes, then, at least in principle, it is absolutely possible to travel in time without also changing your spatial location.
It’s like a lift. A lift moves up and down without also moving left and right or backwards and forwards. A time-only machine would be like such a lift that instead of moving you up and down, it moves you backwards and forwards in time without changing its location within space.
But if time is not orthogonal to the spatial dimensions, then I agree with you, Chris. In that case it would be impossible to travel in time without it also changing your spatial location.
Thanks for the interesting thought :-)