Unicode error

tasklib/backends.py、特に p.communicate() の出力をデコードしている execute_command から始め、報告された文字を含むタスクを使って失敗を再現します。その出力に対して期待される処理を特定し、影響を受けるタスクをフィルタリングしても UnicodeDecodeError が発生しなくなることを確認します。関連するテスト箇所が存在する場合は、回帰テストも追加します。

I have a task generated by bugwarrior from a github issue. The githubbody UDA contains this line:

  UDA githubbody:   blah blah blah blhín blah blah blah

Also, another github issue, that contains this: 👏🏽, is being transformed into:

Annotation:        2020-07-24 14:00:31 -- @pontusj - Well done! That was not at all obvious! ꃭ辱ꃭ붿

While using the taskwiki plugin in vim, I got an error:

Error detected while processing function provider#python3#Call:                                                                                                                                                                                                                
line   18:                                                                                                                                                                                                                                                                     
Error invoking 'python_execute_file' on channel 3 (python3-script-host):                                                                                                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                                                                                             
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/main.py", line 666, in <module>                                                                                                                                                                                
    WholeBuffer.update_from_tw()                                                                                                                                                                                                                                               
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/errors.py", line 26, in wrapped_function                                                                                                                                                                       
    original_function(*args, **kwargs)                                                                                                                                                                                                                                         
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/decorators.py", line 10, in wrapped_function                                                                                                                                                                   
    original_function(*args, **kwargs)                                                                                                                                                                                                                                         
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/main.py", line 44, in update_from_tw                                                                                                                                                                           
    c.evaluate_viewports()                                                                                                                                                                                                                                                     
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/cache.py", line 249, in evaluate_viewports                                                                                                                                                                     
    port.sync_with_taskwarrior()                                                                                                                                                                                                                                               
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/viewport.py", line 343, in sync_with_taskwarrior                                                                                                                                                               
    to_add, to_del = self.get_tasks_to_add_and_del()                                                                                                                                                                                                                           
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/viewport.py", line 317, in get_tasks_to_add_and_del                                                                                                                                                            
    matching_tasks = self.matching_tasks                                                                                                                                                                                                                                       
  File "/home/nikos/.local/share/nvim/plugged/taskwiki/taskwiki/viewport.py", line 294, in matching_tasks                                                                                                                                                                      
    return set(                                                                                                                                                                                                                                                                
  File "/usr/lib/python3.8/site-packages/tasklib/task.py", line 476, in __iter__                                                                                                                                                                                               
    self._result_cache = self._execute()                                                                                                                                                                                                                                       
  File "/usr/lib/python3.8/site-packages/tasklib/task.py", line 508, in _execute                                                                                                                                                                                               
    return self.backend.filter_tasks(self.filter_obj)                                                                                                                                                                                                                          
  File "/usr/lib/python3.8/site-packages/tasklib/backends.py", line 334, in filter_tasks                                                                                                                                                                                       
    for line in self.execute_command(args):                                                                                                                                                                                                                                    
  File "/usr/lib/python3.8/site-packages/tasklib/backends.py", line 290, in execute_command                                                                                                                                                                                    
    stdout, stderr = [x.decode('utf-8') for x in p.communicate()]                                                                                                                                                                                                              
  File "/usr/lib/python3.8/site-packages/tasklib/backends.py", line 290, in <listcomp>                                                                                                                                                                                         
    stdout, stderr = [x.decode('utf-8') for x in p.communicate()]                                                                                                                                                                                                              
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 78938: invalid continuation byte                                                                                                                                                                          

Using task edit I managed to pinpoint the problem to these particular taskwarrior tasks. I used vim with task edit and searched for \%xed. The character í was highlighted. That's how I assumed the first task was to blame. Removing the strange character did not solve the issue. To find the second task, I manually loaded each task in taskwiki. Removing the strange characters at the end, solved the issue. So perhaps that was the issue all along.

Perhaps, it is the responsibility of bugwarrior to filter such characters, but given the broad range of sources that can go into taskwarrior, perhaps it would be simpler to fix such an issue in tasklib.