Yes you can.
The easiest and cheapest way is going to Korea by ship. It will take just 3 hours one way to get to Busan from Fukuoka if you take the Beetle
Beetle English
There is also a normal Ferry that will take longer but is cheaper and more comfortable.
When entering Korea there should not be any visa problems, but when coming back to Japan you will need a ticket going out of Japan in the next 3 months (90days).
Doing this several times in a row might work, but you could also get trouble. Once shouldn´t be a problem.