Thank you for your email. I am sorry to hear your relationship hasn't worked out. It is great to hear you are putting your son's needs first.
If your son's father is keen to have a relationship with his son you should encourage it because as your son grows older he will need his father more and he also has a right to love his father.
While he is still young though it will be unsettling for him to move from home to home as you said and he still needs to be close to you..
It is very important that you and your son's father sit down, with a mediator if necessary, to develop a parenting plan you both are happy with. I can also help you with this.
At this stage it would be best if your son's father could visit your son at your home. However, if you are not comfortable with this it may be best to meet at a public place such as the zoo where you are nearby. It would also be OK for you to drop your son off at his place for a couple of hours say on a Saturday afternoon if your son is happy to separate from you.
Communication is key to building a amicable relationship with your ex moving forward so keep him informed of changes and events in your son's life.
Good luck. let me know how you both are doing.