Vrftsjtryk

As most people, I am not productive 100% of the time I spend at the office, but I try to keep my procrastination time to the minimum (preparing tea, going to the bathroom, answering my messages) and talking to people I want to talk to.

My team leader spends considerable time shopping online, chatting with other co-workers and frequently (once or twice a day) he comes to my place to talk about themes not related to work. This affects my productivity as well as the perception my boss has on my productivity.

From 9 hours, I usually spend 40 minutes on non-work related activities and really don't want to use my time with him because
1. I would have to sacrifice another thing that I have already planned,
2. Reduce my productivity
3. Leave the office later.

He's quite nice but I don't want to encourage a friendship.

I feel uncomfortable asking him to leave or openly showing my disinterest, and usually try to give non verbal cues (I slightly turn my body towards the computer where I'm usually displaying work stuff, I grab the mouse the whole time I talk to him, only lift one headphone, never encourage the conversation) but doesn't seem to work.

There's also the thing about him being my team leader, I feel like I can't use the "I am really busy" line because it would imply that he doesn't have anything to do...

I want to keep a cordial professional relationship. How can I communicate all this without being rude?

Alison covers this very nicely. Two options that I find very useful are saying something when they first interrupt you:

I've only got 5 minutes to talk, and then I need to get back to work.

and

I'm a bit busy right now. But if you come back in 45 minutes, I should have a few minutes free.

In other words, deflect them right off, in a friendly and polite manner. But set limits on the interruptions, so you get back to work.

And yes, you don't want to imply that your boss needs to get back to work. These statements should not be a problem there - it's you who is busy, doing the work they want you to do.

There's also the thing about him being my team leader, I feel like I
can't use the "I am really busy" line because it would imply that he
doesn't have anything to do...

I want to keep a cordial professional relationship. How can I
communicate all this without being rude?

You are being silly avoiding the "I am really busy" line. It's not implying anything, nor is it rude.

Something like "I'd love to chat, but I have to get back to my work." or "Good talk, but now I really need to get back to [whatever task you were doing]" should work if repeated often and consistently.