It is believed by linguists that no spoken language is significantly more difficult to learn than any other in absolute terms. Children can learn their mother tongues, whatever they may be, without too much trouble.
However adults already speak one or more languages and generally find it easier to learn a closely-related language than a distantly-related or unrelated one. For example, the least difficult languages for English speakers to learn are Italian, Spanish, French, Dutch and German, in more or less that order.
Each language presents you with a different set of challenges in terms of pronunciation, vocabulary, grammar, spelling and writing system. The closer these are to your native language, the less difficult a language is to learn.
Researchers have found that the brain processes different languages in different ways. A particular study looked at brain activity in native speakers of English and Chinese when listening to their native languages and found that the Chinese speakers used both sides of their brains, whereas the English speakers only used the left side of their brains. The conclusion is that Chinese is more difficult to understand and speak than English.