I dot understand it.

I have an average male, with a longsword.

Longsword has load of 1, so it yealds encumburance of one to this average man who could offset this load with str of +1.

So lightly packed average man with a blade on his belt, does he get a penalty to all his spells?

I mean, I have strength of about -1 or -2, and I can carry blade all day long around, I would get tired when Wielding it, but not holding it.

When to apply encumburance?

It's a little more complicated.

First you must compute your total load.

Then a table (don't have the page ref, sorry, I'm at work) gives you your burden.

If Str > 0, your encumbrance is Burden - Str, otherwise it is equal to your burden.

But in the end, average male cannot carry around a blade bacause it hinders his very life...

Or average mage does not wish to carry a staff or a blade because he gets reduction of 1 to his spell casting rolls!

I check 4th edition how it is described there.

My storytellers explanation:

"If he is holding it in his hand while casting, yes. If he sticks it in the ground, or simply handgs it from his swordbelt, no. So he only ever gets the paenalty if casting spells in melee combat."

Sounds reasonable and logical. Holding a blade incurs encumburance, not having it by the swordbelt as it's not that heavy.