Recently I had to deal with the problem of displaying and saving Unicode data in a Visual FoxPro form. As we all know, Visual FoxPro does not support Unicode, so this was quite a challenge for me. I would never have solved this problem by myself if Gregory Adam from the UniversalThread would not lend me his generous support and practically solved the problem of saving for me. I want to show the full solution we came to together as this may be a very important topic for the remaining Visual FoxPro developers (and because Koen asked me to write this blog post).

There used to be the classical and very comprehensive article on this topic written by Rick Strahl Using Unicode in Visual FoxPro Web and Desktop Applications. The link to that article now appears to be broken, so I found another shorter article: Unicode to ANSI Translation in VFP 9.0 – Sys(987)

I originally started from the first mentioned article. Our application uses SQL Server to hold data and uses sql pass thru technique to send information from and to SQL Server.

Now, since I am describing the solution I used for the particular application, I will start with the SQL Server table I was using. That table was named Prefs_sl and it was a flat table with 1 record containing all preferences for a sales point. Because of its flat nature it was badly normalized. Say, it had 11 language fields language00 – language10 (nvarchar(100)) to hold name of the language the Sales Point can display and 11 image lngImage00-lngImage10 (varbinary(max)) fields to hold the image of the country flag. So, this was the table I was working with on the SQL Server side.

Now, I wanted to display these 11 languages with their corresponding flags on the form, so the logical choice was to create a class with the textbox for the unicode text and image for the image flag and then instantiate this class on the form.

The first question was what to chose for displaying unicode data? Rick mentioned MS Edit Box control ActiveX, but I didn’t find it in the list of my installed ActiveX. But I found MS Forms2 TextBox ActiveX control and since I also remembered this thread on foxite forum, I decided to try this ActiveX control.
The first problem I encountered with this ActiveX as that it didn’t show up until I clicked on it when I was running the form. Luckily I found the following solution – invoke SetFocus of this control (I called it txtLanguage in my class) and then it was displayed correctly. I am not sure if my solution is the best to handle this ActiveX behavior quirks, but hey, it works.

Ok, so far so good. Now, following Rick’s article I used the following in my form to get the data in the binary format:

LOCAL lcSQL
TEXT TO lcSQL noshow
SELECT pri_key      
      ,defltlang
      ,cast(language00 as varbinary(100)) as language00
      ,lngimage00      
      ,cast(language01 as varbinary(100)) as language01
      ,lngimage01
      ,cast(language02 as varbinary(100)) as language02
      ,lngimage02
      ,cast(language03 as varbinary(100)) as language03
      ,lngimage03
      ,cast(language04 as varbinary(100)) as language04
      ,lngimage04
      ,cast(language05 as varbinary(100)) as language05
      ,lngimage05
      ,cast(language06 as varbinary(100)) as language06
      ,lngimage06
      ,cast(language07 as varbinary(100)) as language07
      ,lngimage07
      ,cast(language08 as varbinary(100)) as language08
      ,lngimage08
      ,cast(language09 as varbinary(100)) as language09
      ,lngimage09
      ,cast(language10 as varbinary(100)) as language10
      ,lngimage10      
  FROM dbo.prefs_sl
endtext  
mysqlexec(m.lcSQL, 'Prefs_sl', program())

mysqlexec here is a wrapper for the sqlexec function.

Now I found the first discrepancy with Rick’s article – I found that I don’t need any conversion after that, I can assign the field’s content to the text property of this ActiveX (or ControlSource property) and the control correctly displays the unicode data!

I was happy – the first part of displaying data turned out to be quite easy!

Here is the code I used to display the data:

FOR EACH loObject IN thisform.Objects foxobject
   IF loObject.Baseclass = 'Container'
      loObject.LanguageField = SUBSTR(loObject.name,4,LEN(loObject.name))
      loObject.ImageField = 'lngImage' + RIGHT(loObject.Name,2)
   ENDIF 
next   

thisform.cntLanguage01.SetFocus()

And in the container itself the following code in the LanguageField assign method:

this.lblLanguage.caption = this.lblLanguage.caption + ' ' + transform(val(right(tLanguageField,2)))
  this.txtLanguage.Enabled = .t.
  this.txtLanguage.TabStop = .t.   
  this.lProgrammaticChange = .t.
  this.txtLanguage.text = evaluate('prefs_sl.' + tLanguageField)
  this.lProgrammaticChange = .f.

Everything was good so far. However, I haven’t expected how hard (for me) it will be the ‘saving’ data part. I was spending days trying various ideas from Rick’s article and was still unable to achieve the desired result. I was about to throw all I had away and try to switch to ADO for data retrieval, but luckily Gregory Adam helped me here and published a working sample.

First of all, we need to make sure there is no UT8 translation to the current code page. For this we need to do the following

COMPROP(this,'UTF8',1)

In the txtLanguage Init method.

We also need the following functions:

&& StringConversion
&& Gregory Adam 2011
*===============================================================================
#define true	.T.
#define false	.F.

*_______________________________________________________________________________
function StringToUTF8(utf8Out, stringIn, codepageIn)
	
	local success
	success = true
	
	do case
	case !m.success
	
	case !StringToUTF16(@m.utf8Out, m.stringIn, m.codepageIn)
		assert false
		success = false
	
	case !StringUTF16ToUTF8(@m.utf8Out, m.utf8Out)
		assert false
		success = false
	
	endcase
	
	return m.success
	
endfunc
*_______________________________________________________________________________


*_______________________________________________________________________________
*_______________________________________________________________________________
#define CP_ACP					0
#define CP_MACCP				2
#define CP_OEMCP				1
#define CP_SYMBOL				42
#define CP_THREAD_ACP			3
#define CP_UTF7					65000
#define CP_UTF8					65001
#define MB_PRECOMPOSED			0x1
#define MB_COMPOSITE			0x2
#define MB_USEGLYPHCHARS		0x4
#define MB_ERR_INVALID_CHARS	0x8

#define WC_DEFAULTCHAR			0x00000040 
#define WC_ERR_INVALID_CHARS	0x00000080 
#define WC_NO_BEST_FIT_CHARS 	0x00000400 
*_______________________________________________________________________________
function StringToUTF16(utf16Out, stringIn, codepageIn)

	local success
	success = true
	
	do case
	case !m.success
	
	case empty(len(m.stringIn))
		utf16Out = ''
		
	otherwise
		local lpWideCharStr, result
		lpWideCharStr = space(len(m.stringIn)*2)
	
		result = MultiByteToWideChar( ;
					evl(m.codepageIn, cpcurrent()), ;
					MB_ERR_INVALID_CHARS, ;
					@m.stringIn, ;
					len(m.stringIn), ;
					@m.lpWideCharStr, ;
					len(m.lpWideCharStr) ;
				)
			
		do case
		case !m.success
		
		case empty(m.result)
			assert false
			success = false
		
		otherwise
			utf16Out = left(m.lpWideCharStr, m.result * 2) 
		
		endcase
		
	endcase
	
	return m.success
	
	
endfunc
*_______________________________________________________________________________
function StringUTF16ToUTF8(utf8Out, utf16In)

	local success
	success = true
	
	do case
	case !m.success
	
	case empty(len(m.utf16In))
		utf8Out = ''
		
	otherwise
	
		local lpMultiByteStr, lpUsedDefaultChar, result
		lpMultiByteStr = space(len(m.utf16In) * 2)
		lpUsedDefaultChar = 0
		
		result = WideCharToMultiByte( ;
					CP_UTF8, ;
					WC_ERR_INVALID_CHARS, ;
					@m.utf16In, ;
					len(m.utf16In)/2, ;
					@m.lpMultiByteStr, ;
					len(m.lpMultiByteStr), ;
					null, ;
					@m.lpUsedDefaultChar ;
				)
		
		do case
		case !m.success
		
		case empty(m.result)			
			assert false
			success = false
			
			
		otherwise
			utf8Out = left(m.lpMultiByteStr, m.result)
		
		endcase
	endcase
	
	return m.success
	
	
endfunc
*_______________________________________________________________________________
function StringUTF8ToUTF16(utf16Out, uft8In)

	return StringToUTF16(@m.utf16Out, uft8In, CP_UTF8)
	
endfunc
*_______________________________________________________________________________

function MultiByteToWideChar
	lparameters codepage, ;
				dwFlags, ;
				lpMultiByteStr, ;
				cbMultiByte, ;
				lpWideCharStr, ;
				cchWideChar

	local success
	success = true

	local result
	
	do case
	case !m.success
	
	otherwise
		try
			declare integer MultiByteToWideChar in Kernel32.dll ;
				long	codepage, ;
				long	dwFlags, ;
				string@	lpMultiByteStr, ;
				integer	cbMultiByte, ;
				string@	lpWideCharStr, ;
				integer	cchWideChar
		
			result = MultiByteToWideChar( ;
					m.codepage, ;
					m.dwFlags, ;
					@m.lpMultiByteStr, ;
					m.cbMultiByte, ;
					@m.lpWideCharStr, ;
					m.cchWideChar ;
				)
		catch
			assert false
			success = false
			
		endtry
	endcase
	
	return iif(m.success, m.result, 0)
	
endfunc
*_______________________________________________________________________________
function WideCharToMultiByte
	lparameters codepage, ;
				dwFlags, ;
				lpWideCharStr, ;
				cchWideChar, ;
				lpMultiByteStr, ;
				cbMultiByte, ;
				lpDefaultChar, ;
				lpUsedDefaultChar

	local success
	success = true

	local result
	
	do case
	case !m.success
	
	otherwise
		try
			declare integer WideCharToMultiByte in Kernel32.dll ;
				long	codepage, ;
				long	dwFlags, ;
				string@	lpWideCharStr, ;
				integer	cchWideChar, ;
				string@	lpMultiByteStr, ;
				integer	cbMultiByte, ;
				string	lpDefaultChar, ;
				integer	@lpUsedDefaultChar
		
			result = WideCharToMultiByte ( ;
					m.codepage, ;
					m.dwFlags, ;
					@m.lpWideCharStr, ;
					m.cchWideChar, ;
					@m.lpMultiByteStr, ;
					m.cbMultiByte, ;
					m.lpDefaultChar, ;
					@m.lpUsedDefaultChar;
				)
		catch
			assert false
			success = false
			
		endtry
	endcase
	
	return iif(m.success, m.result, 0)
	
endfunc

and then in the Change event of the ActiveX textbox I added the following code

IF this.parent.lProgrammaticChange 
    RETURN
ENDIF     
if thisform.IsChanged OR (not this.CurrentValue == this.Text)
   thisform.IsChanged = .t.
   local utf8, utf16Out
   utf8 = this.text
   if not empty(utf8)
      = StringUtf8ToUTF16(@m.utf16Out, m.utf8)
      replace (this.parent.LanguageField) with utf16Out
   else
      replace (this.parent.LanguageField) with ''
   endif
endif

and finally in the Save method of the form

IF THISFORM.IsChanged 
  LOCAL lcSQL

  TEXT TO lcSQL noshow
     UPDATE dbo.prefs_sl 
     SET defltlang = ?prefs_sl.DefltLang,
     language00 = cast(?prefs_sl.language00 as nvarchar(100)),
     lngimage00 = ?prefs_sl.lngimage00,
     language01 = cast(?prefs_sl.language01 as nvarchar(100)),
     lngimage01 = ?prefs_sl.lngimage01,
     language02 = cast(?prefs_sl.language02 as nvarchar(100)),
     lngimage02 = ?prefs_sl.lngimage02,
     language03 = cast(?prefs_sl.language03 as nvarchar(100)),
     lngimage03 = ?prefs_sl.lngimage03,
     language04 = cast(?prefs_sl.language04 as nvarchar(100)),
     lngimage04 = ?prefs_sl.lngimage04,
     language05 = cast(?prefs_sl.language05 as nvarchar(100)),
     lngimage05 = ?prefs_sl.lngimage05,
     language06 = cast(?prefs_sl.language06 as nvarchar(100)),
     lngimage06 = ?prefs_sl.lngimage06,
     language07 = cast(?prefs_sl.language07 as nvarchar(100)),
     lngimage07 = ?prefs_sl.lngimage07,
     language08 = cast(?prefs_sl.language08 as nvarchar(100)),
     lngimage08 = ?prefs_sl.lngimage08,
     language09 = cast(?prefs_sl.language09 as nvarchar(100)),
     lngimage09 = ?prefs_sl.lngimage09,
     language10 = cast(?prefs_sl.language10 as nvarchar(100)),
     lngimage10 = ?prefs_sl.lngimage10
     where pri_key = ?prefs_sl.pri_key
  ENDTEXT
 RETURN mySQLExec(m.lcSQL, '',PROGRAM())
ENDIF

As we can see, we need to convert data back from binary to nvarchar.
With all this code in place, we display the unicode data and save them back to SQL Server.

A big thanks to Gregory for helping me with the code – I don’t know where I would be without his help.

Olaf Doschke showed another way which is even simpler than implemented solution and in accordance to the original Rick’s article.

In the form’s Load we need the following:

Sys(987,.F.)
Sys(3101,65001)

Then, after getting binary data from SQL Server the same way as I show in this blog, we still need to use createbinary, e.g.

this.txtLanguage.text = createbinary(prefs_sl.Language00)

We don’t want to use COMPROP now for the ActiveX.

and then, saving data, we need to follow Rick’s steps:

pcSavedText1 = Strconv(This.Text,12)

*** Must explicitly force to binary – can also use CAST in 9.0
pcSavedText1 = CREATEBINARY(pcSavedText1)

and then convert this value back to nvarchar(max) when saving with sqlexec.

This is how the form with many languages looked like

—————————
I need to also tell, that after I finished the form and was happy to show it to my colleagues, I was up to a big disappointment. It turned out we are not going to support unicode (our main application is not up to support it yet) and so I will be re-doing this form after we will agree on the tables’ structures (they also are going to change). So, what I showed in this blog post we’re not going to use at present in production.

UPDATE. My colleague also re-designed prefs_sl table to one EAV Settings table. We know that EAV design has its own problems, but for saving various application level settings it can be used. I already re-designed the forms we’re going to use.

But in any case, it was a very important exercise and I hope it will help other people who need to display and save Unicode data in their Visual FoxPro applications.