I was working at a client who is using the financial industry "standard" format 8583 (which is basically an open format too – see http://en.wikipedia.org/wiki/ISO_8583) and was struggling a little with a way to specify in Xml Schema that the data was of hexadecimal type. The 8583 format includes a few hex fields as well as some open-ended fields that are usually customized by payment processor. The challenge with a hex field is that its field length when coverted to ASCII will have twice the number of characters so it would be easy to overrun a positional flat file thinking all of the fields were just based on an ASCII encoding. In BizTalk you would use the flat-file schema wizard to generate a schema for the file, but the question was how to specify that a field was hexadecimal just based on Xml Schema types.
I did some research and found out about Xsd:HexBinary, which gives us this capability. I found an IBM article which mentioned this in the context of sending attachments as part of (or embedded in) web service calls: http://www-128.ibm.com/developerworks/library/ws-tip-noattach.html. Ironically, this article shows both Java and .NET code examples which is pretty cool. So the Xsd:HexBinary gives us the ability to give an Xml schema information about a hex field, but this is unfortunately not something (that I am aware of) which can be specified during the BizTalk flat-file wizard. This might be a good future feature.