What project is affected?
Deployment script
What language is this in?
What happens?
Special characters :
Prompts:
Plus d'information sur la lettre reçu des éléctions ?
are exported like this
it needs to be - Plus d'information sur la lettre reçu des éléctions ?
What are the steps to reproduce this issue?
Steps to reproduce the behavior:
Create a knowledge base
Use the special characters in the Question or Answer field
Export using the update_cognitive_services.ps1 script
Inspect the resultant in Deployment/Resources/Qna/{language}/nameofknowledgebase.qna
What were you expecting to happen?
Support special characters such as é, ç, ë, à, etc.
Can you share any logs, error output, etc.?
Any screenshots or additional context?
I am reopening this issue, I have the same problem right now even if I have put "-encoding oem $($outJson)"
Thanks @edemehmetcan for reporting this issue. We will review this issue following the repro steps you mentioned.
We will back to you as soon as we have any update for you 😊.
Thank you so much @Batta32 . I am waiting your answer.
Please also check this ? During export, there is an option "qnaFormat" it's a boolean that I put to true in order to have a .qna file in my repository but I have an error "Unexpected Argument: true"
https://www.npmjs.com/package/@microsoft/bf-qnamaker#bf-qnamakerkbexport
Why use that ? I want to export directly the .qna file and not pass by bf qnamaker:convert after to convert json to QnA file.
@edemehmetcan how are you executing the bf qnamaker:kb:export
command with the --qnaFormat
argument?
It should be executed as follows:
bf qnamaker:kb:export --endpoint $qnaEndpoint --environment Prod --kbId $kb.kbId --subscriptionKey $kb.subscriptionKey --qnaFormat
See this example of the unit tests of the botframework-cli repository.
I have tried to export it like this and this is now working, is the syntax correct ?
bf qnamaker:kb:export `
--endpoint $qnaEndpoint `
--environment Prod `
--kbId $kb.kbId `
--subscriptionKey $kb.subscriptionKey `
--qnaFormat | Out-File -Force $($fileDestination)
Yes! That syntax is correct too.
We will be reviewing the special characters issue now, and as soon as we have any update for you, we will back to this thread 😊.
Super thanks!
When I am trying without "Out-File", the export is correct (in the Terminal) with correct characters. Once exporting to a file.. The characters are again changed strangely...
I have updated the botframework-cli and the same script has been launched in Azure DevOps (Azure CLI step), the characters are still changed... @axelsrz @Batta32
After reviewing the file, it's better than before. Now, I just have the problem with "é" and "ï"
@axelsrz - should we transfer this issue to botframework-cli repository? As this appears to be an issue related to the bf-cli tool (bf qnamaker:kb:export
specifically)
@Batta32 Even if I execute the qnamaker:kb:export locally with powershell. I still have the issue of characters. This is not something related to Azure CLI I would say...
@edemehmetcan - appears not be related to the Azure CLI but probably related to the botframework-cli which is the interftace used to manage the Bot Framework bots and related services (see botframework-cli repository), this CLI contains the bf qna:maker:kb:export
command that you are using 😊.
Hi @edemehmetcan,
Can you try changing your PS config to use UTF? See a relevant link below.
https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8
Hello @hcyang thanks for your answer. The powershell is using UTF8.
I have noticed something, after exporting the qna and opened on Notepad++ the encoding is "UCS 2 le BOM" which is very strange...
And it's very ennoying to not have like "luis:export" a property --out for qnamaker:export...
Regards
Hi folks, I did some investigations and this seems not related with the bf-cli. It's more related with the powershell output encoding format issue. I use the cmd to call above export command and the output returns normal result for special chars, but when I use powershell to call the same command, then the output returns unexptected result. So this issue only happens to powershell with Out-File set. I tried some ways to set the default encoding to UTF in powershell, but seems not working. This is same issue with the one https://github.com/microsoft/botframework-cli/issues/1057 @axelsrz pointed above.
As @edemehmetcan suggested, the optional fix is to add --out for kb export command so that it doesn't depend on Out-File
or >>
or >
to output files
Add @munozemilio for more insights.
Created a PR in cli https://github.com/microsoft/botframework-cli/pull/1065 to support output exported qna content to a file with --out option. With this changes in, users can use --out to specify the output file path. Tested that works for powershell.
@feich-ms Thanks for you answer. It's what I was looking for! I am waiting that the PR is merged and I will try.
@munozemilio, can you review @feich-ms PR?
Hello, I just had a try after updated the botframework-cli to version 4.11.1
bf qnamaker:kb:export `
--endpoint $qnaEndpoint `
--environment Prod `
--kbId $kb.kbId `
--qnaFormat `
--out $outQnA `
--subscriptionKey $kb.subscriptionKey `
--force
I have this error: ".Unexpected arguments: --out"
Hi @edemehmetcan, [email protected] was released 8 days ago and doesn't contain the PR microsoft/botframework-cli#1065 as it was merged 3 days ago.
Those changes will be integrated in the next release of the botframework-cli tool 😊.
Hi @Batta32 it's what I have thought about. Thanks for the clarification. When the next release will be scheduled ? :)
@munozemilio & @feich-ms - is there any ETA for the new release of botframework-cli tool?
an update ? @munozemilio & @feich-ms @Batta32
@edemehmetcan sorry for the late response, as far as I know, next release will not be too early. To unblock you, you can install the latest nightly build version which already contains the fix. Just use npm install -g @microsoft/botframework-cli@next. It will install the latest one 4.12.0-dev.20201208.3315f42. Thank you.
@feich-ms thanks a lot, it works with the next release :) waiting for the real release then...
@feich-ms @Batta32 When the next release will be available ?
Hi @edemehmetcan, sorry for the late response. As far as I know, the next release will be mid of Feb. 2/12 is our code completion day and we may push the release one week later after that.
Most helpful comment
Created a PR in cli https://github.com/microsoft/botframework-cli/pull/1065 to support output exported qna content to a file with --out option. With this changes in, users can use --out to specify the output file path. Tested that works for powershell.