ivan_schnyder

updated

15 years ago

0

2922

October 28th, 2011 01:00

XString: problems with nouns? (like ä,ö,ü)

Hello experts,

we succesfully implemented an interface to an EMC Centera system via XAM. We use the .NET Wrapper Version (1.0.57) and our cluster has CentraStar 4.2.0-3881-0-0 installed. So far so good.

I tested different documents, everything worked fine. But then, depending on the document type I wanted to store on the system, I received an exception. Well not really, becaus the errorToken of the exception was always NULL, so I had no clue what was going wrong. The error only occured when loading a file, it seemed that I always could save the file and I also received a valid XUID (see attachment "save.png").

After some "try and error" I finally found (a bug?): I'm saving various metadata in my xSet, one is the title of the document:


xSet.CreateField(FIELDNAME_TITLE, rowDOK.Titel);

I found out, as soon as my document title contains nouns like ä,ö,ü the system fails to load the file (is it a serializing / encoding) problem? Could you please verify my problem and tell me, if I have to encode or do "something special" myself or if it's a problem in the API/VIM?

Any help would be appreciated,

best regards

Ivan Schnyder

3 Attachments

  • 1340

    0

    Posted November 27th, 2011 23:00

    OK, I see.

    Anyway I tried different aproaches, so far what has worked:

    Using bytearray with  CreateXStream worked perfectly! I was able to read and write back unicode text as bytes array without any issues. This does mean I have to revert to using only XStream instead of XString

    Encoding string with base64 and using this for XString values. Worked perfectly here also (except when encoding Unicode UTF16 string it causes the actual text to be 2 times larger than usual, This can be countered with applying GZipStream afterwards, but I really don't want to go that far, just to be able to store a simple string )

  • 1337

    0

    Posted November 24th, 2011 02:00

    Hi, I have been checking this out, old post, I know, but it seems there is a problem here.

    In XAM documentation it is specified that field name should be of type:

    xam_string application/vnd.snia.xam.string Shall be a UTF-8 encoded Unicode

    string, with a maximum length of 512 bytes

    Each field has a name that is scoped to the XSet. Field names shall be UTF-8 encoded strings (see

    [RFC3629]) with a maximum length of 512 bytes. Field names shall be case sensitive, so

    .xam.org.snia.field

    is distinct from .xam.org.snia.Field. The field names shall not have embedded NULL

    characters. To maximize application interoperability, applications are encouraged to use the IDN profile of

    stringprep

    as defined in [RFC 3491] and [RFC 3454].

    Now I made a simple test case with .NET XAM wrapper to see if this is the case:

    [

    Test]

         

    public virtual void TestUnicodeTextFieldRead()

    {var wrapper = new XAMWrapper(GetConnectionString());

    Trace.WriteLine("Connect with =" + GetConnectionString());

    //CONNECT

    wrapper.Connect();

              

    // WRITE

    //now we can write values - get our writer

    XAMWriter writer = wrapper.WriteXSet();

    //values to test

         

    String stringValue1 = "normal text";

    String fieldName = "öõüpäöõpoõoõzž";

    writer.WriteFieldValue(fieldName, stringValue1); <-- Error here "xam/non-UTF8 parameter"

     

              

    //commit changes

    writer.Commit();

                   

    //READ

    //XUID now exists

    Assert.IsNotNull(writer.XUID);

    //get current field value - should be equal to our initial value

              

    String field1 = writer.GetFieldValue(fieldName);

     

                   

    //delete the xset

    wrapper.DeleteWriter(writer);Trace.WriteLine("Deleted OK, XUID=" + writer.XUID);

    wrapper.Dispose();

              

    Trace.WriteLine("Disposed = OK");

              

    //check the results

    Assert.AreEqual(stringValue1, field1);

    }

    P.S. Your formatting code is really broken.

    I am using my own wrappers on top of this, but basically it calls: 

    XAM_CreateString

  • 1340

    0

    Posted November 24th, 2011 03:00

    Thanks Dmitri for your response!

    It's still a problem and we haven't found a solution yet. But just to clarify: the problem is not special characters in the fieldname, I get my error when I'm using special characteres in the fieldcontent.

    Something like this (adapted to your code):

    String stringValue1 = "äöüaaääöö"

    String fieldName = "Title";
    writer.WriteFieldValue(fieldName, stringValue1);

    Does this work for you? I can save such content, but when I want to load:
    String field1 = writer.GetFieldValue(fieldName);
    I get an exception...

    Thanks, best regards
    Ivan

  • 1340

    0

    Posted November 24th, 2011 04:00

    Yes this works for me (at least it doesn't give an error when reading, but there is an error when I am trying to delete the XSet ! -10214),  here is another test case, if fails in another place (the russian text is incorrect after it was read back also, and it throws an exception when I am trying to delete XSet!  Status -10214)
       
       
       
       
       
       
       
       
       
       
       
       
    [Test]
             
    public virtual void TestUnicodeTextWriteRead()
    {
                  
    var wrapper = new XAMWrapper(GetConnectionString());
    Trace.WriteLine("Connect with =" + GetConnectionString());
    //CONNECT
    wrapper.Connect();
    // WRITE
    //now we can write values - get our writer
    XAMWriter writer = wrapper.WriteXSet();
    //values to test
    String stringValue1 = "normal text";
    String stringValue2 = "hea küll ja teksti on õrn, mahe mägus ka öõüpäöõpoõoõzž";
    String stringValue3 = "Один. Уничтожить вражную надежду на побеждение";
     
    writer.WriteFieldValue(
    "field1", stringValue1);
    writer.WriteFieldValue(
    "field2", stringValue2);
    writer.WriteFieldValue(
    "field3", stringValue3);
    //commit changes
    writer.Commit();
        
    //READ
    //XUID now exists
    Assert.IsNotNull(writer.XUID);
    //get current field value - should be equal to our initial value
    String field1 = writer.GetFieldValue("field1"); //value is "normal text";
    String field2 = writer.GetFieldValue("field2"); //value is "hea küll ja teksti on õrn, mahe mägus ka öõüpäöõpoõoõzž";
        
    String field3 = writer.GetFieldValue("field3"); //value is "?????.??.???.?????????.????" (WRONG!?)
     
    //delete the xset
    wrapper.DeleteWriter(writer); // <-- Exception is thrown here ! (The rights are available, other test cases work, only this one fails)
    Trace.WriteLine("Deleted OK, XUID=" + writer.XUID);
    wrapper.Dispose();
                  
    Trace.WriteLine("Disposed = OK");
     
    //check the results
    Assert.AreEqual(stringValue1, field1);
    Assert.AreEqual(stringValue2, field2);
    Assert.AreEqual(stringValue3, field3);
    }
    I tried different things, here is the actual method:
    DllImport("xam.dll")]
    public static extern int XAM_CreateString (XAMHandle inHandle, string inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, string inValue);
    I can change it to:
    [DllImport("xam.dll")]
    public static extern int XAM_CreateString (XAMHandle inHandle, byte[] inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, byte[] inValue);
    And when passing String values I can use UTF8 encoding to bytes...
    But it seems it doesn't work, I get all kind of errors with this.
    I also tried also using
    [DllImport("xam.dll", CharSet = CharSet.Unicode )]
    public static extern int XAM_CreateString (XAMHandle inHandle, string inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, string inValue);
    But the results I got with this were even more inconsistent, even the normal text returned in chineese (japaneese maybe) after it was stored in english!
    So I really don't know what to do with this right now, if using Unicode UTF8 is not possible, then I have to use alternative approach and encode text with Base64 , can't see any other way right now. Any help anyone ?

    Message was edited by: DmitriKuznetsoiv

  • 1340

    0

    Posted November 24th, 2011 05:00

    Could you send me a set of your versions? I'm working with (64bit):

    Centera VIM: 1.0.57

    XAMSDK.dll: 1.0.0.0

    FPCore.dll: 4.0.660.0
    fpos64.dll: 3.2.281.0
    fpparser64.dll: 3.2.48.0
    FPStreams64:4.0.660.0
    FPUtils64.dll: 4.0.660.0
    FPXML64.dll: 4.0.660.0

    Thanks

  • 1342

    0

    Posted November 24th, 2011 05:00

    ivan.schnyder wrote:

    Could you send me a set of your versions? I'm working with (64bit):

    Centera VIM: 1.0.57

    XAMSDK.dll: 1.0.0.0

    FPCore.dll: 4.0.660.0
    fpos64.dll: 3.2.281.0
    fpparser64.dll: 3.2.48.0
    FPStreams64:4.0.660.0
    FPUtils64.dll: 4.0.660.0
    FPXML64.dll: 4.0.660.0

    Thanks

    My versions are:

    Centera VIM: 1.0.103.0

    XAMSDK.dll: 1.0.0.0

    FPCore.dll: 4.0.698.0
    fpos64.dll: 3.2.281.0
    fpparser.dll: 3.2.48.0
    FPStreams:4.0.698.0
    FPUtils.dll: 4.0.698.0
    FPXML.dll: 4.0.698.0

    I think I am using 32 bit versions here...

  • 1340

    0

    Posted November 24th, 2011 06:00

    Thanks,
    so I updated all my dll to the latest (EMC Centera SDK for XAM 1.0 Patch 3 for Windows (32-bit and 64-bit and EMC Centera SDK for XAM VIM 1.0 Patch) - my VIM Version is now 1.0.115 but still the same error...

  • 1341

    0

    Posted November 28th, 2011 01:00

    Thank you so much Dmitri!

    Your hints took me to the solution: I tried the workaround with the XStream. It didn't work till I set the encoding on the array to UTF32. So after it finally worked with the XStream, I wrote myself two little methods and I can know succesfully write / read a normal  "field" with the following code:

    // Write

    xSet.CreateField(FIELDNAME_TITLE, EncodeToBase64("äöüéàa"))


    // Read

    xSet.GetField(FIELDNAME_FILETYPE, xStringFieldContent);
    string strField = DecodeFromBase64(xStringFieldContent);

    // Methods

    private string EncodeToBase64(string strToEncode)

    {

    byte[] toEncodeAsBytes = UTF32Encoding.Default.GetBytes(strToEncode);

    return Convert.ToBase64String(toEncodeAsBytes);

    }

    private string DecodeFromBase64(string strEncodedData)

    {

    byte[] encodedDataAsBytes = Convert.FromBase64String(strEncodedData);

    return UTF32Encoding.Default.GetString(encodedDataAsBytes);

    }

    Best regards and thank you one more time to analyse my problem!
    Ivan

  • 1340

    0

    Posted November 28th, 2011 02:00

    ivan.schnyder wrote:

    Thank you so much Dmitri!

    Your hints took me to the solution: I tried the workaround with the XStream. It didn't work till I set the encoding on the array to UTF32. So after it finally worked with the XStream, I wrote myself two little methods and I can know succesfully write / read a normal  "field" with the following code:

    // Write

    xSet.CreateField(FIELDNAME_TITLE, EncodeToBase64("äöüéàa"))


    // Read

    xSet.GetField(FIELDNAME_FILETYPE, xStringFieldContent);
    string strField = DecodeFromBase64(xStringFieldContent);

    // Methods

    private string EncodeToBase64(string strToEncode)

    {

    byte[] toEncodeAsBytes = UTF32Encoding.Default.GetBytes(strToEncode);

    return Convert.ToBase64String(toEncodeAsBytes);

    }

    private string DecodeFromBase64(string strEncodedData)

    {

    byte[] encodedDataAsBytes = Convert.FromBase64String(strEncodedData);

    return UTF32Encoding.Default.GetString(encodedDataAsBytes);

    }

    Best regards and thank you one more time to analyse my problem!
    Ivan

    Sure, that's good that it worked for you.

    I see  in this example you are using UTF32, but why ? AFAIK C# strings are UTF16, UTF32 doesn't add anything and just wastes space, you can even use UTF8 for this purpose which should work for most possible cases (and usually less space is needed for storage). And I proposed 2 solutions either base64 or XStream, you don't need to use both of them at the same time, just either one of them is enough.

    For my solution I went with XStream and UTF16 encoded strings (just for the reason to avoid problems when using XString encoded in Base64 limit of 512 bytes, which means 256 characters for UTF16 and 128 for UTF32 in this case, XStream doesn't have this limit)

  • 192

    0

    Posted November 28th, 2011 02:00

    Hi Dmitri

    you're absolutly right what concerns strings in c#, they are UTF16 by default, so UTF32 doesn't make any sense at all. But you also wrote:


    And I proposed 2 solutions either base64 or XStream, you don't need to use both of them at the same time, just either one of them is enough.


    I don't get that one, in my final solution I'm using this method:
    xSet.CreateField(FIELDNAME_TITLE,  "äöüéàa") // second parameter is type of XString,
    So right know I'm just using the method with XString but before I pass the XString, I encode it to a base64 string, I'm not using the XStream method anymore? Maybe my last answer wasn't clear enough.

    Best regards