Light speed time distortion paradox?

All motion is relative and high rates of speed distort time so if one goes half the speed of light ones time is slowed compared to a observer. Yet to you the observer is the one moving so to you his time would be the one that slows down. So my question is how does that work? Because both being slower than the other is impossible.