This is an interesting phenomenon that I think affects a lot of people when talking to speakers of other languages. People do not expect to understand, particularly if they don't speak a second language themselves and particularly if they are not well travelled. This is all the truer when one uses sophisticated or idiomatic peech. If a native had said it, in exactly the same way, it would be understood but if, in this case, a non-French person says the same thing, it is assumed to be unintelligible. What I have found to help is to say everything with great confidence, both in the tone and with one's body language. Look at them as though they should understand and that, if they don't, there must be something wrong with them. I've found that it reduces the problem by about 80%.