Hi Jens
The requirement for the restriction seems to fall more into "application logic" rather than "integration logic". As such I would explore if it's possible for the backend application to raise an application fault which then is propagated back to the calling system. This seems cleaner than raising a mapping exception in PI which results in a system error, and if I'm not mistaken, the full error might not be available back to the calling system (I might be wrong).
You didn't mention how the sender and receiver connectivity is going to be like, and whether async or sync - SOAP to Proxy Synchronous?
If you really want to raise the exception in UDF, both exception types should be fine.
Regards
Eng Swee