[Python] Python 3.11 cchardet fatal error: longintrepr.h: No such file or directory

문제 발생

기존에 airflow에서 사용하던 Plugin Package(Python3.8)에서 인코딩 detect을 위한 cchardet 사용했다.

하지만 airflow의 Python Version을 3.11로 올리면서 poetry update를 하던 중 에러가 발생했다.

Installing cchardet (2.1.7): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  running bdist_wheel
  running build
  running build_py
  creating build/lib.linux-x86_64-cpython-311/cchardet
  copying src/cchardet/__init__.py -> build/lib.linux-x86_64-cpython-311/cchardet
  copying src/cchardet/version.py -> build/lib.linux-x86_64-cpython-311/cchardet
  running build_ext
  building 'cchardet._cchardet' extension
  creating build/temp.linux-x86_64-cpython-311/src/cchardet
  creating build/temp.linux-x86_64-cpython-311/src/ext/uchardet/src
  creating build/temp.linux-x86_64-cpython-311/src/ext/uchardet/src/LangModels
  g++ -pthread -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isrc/ext/uchardet/src -I/tmp/tmpi1mwq5oy/.venv/include -I/usr/include/python3.11 -c src/cchardet/_cchardet.cpp -o build/temp.linux-x86_64-cpython-311/src/cchardet/_cchardet.o
  src/cchardet/_cchardet.cpp:196:12: fatal error: longintrepr.h: No such file or directory
     #include "longintrepr.h"
              ^~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/g++' failed with exit code 1
  

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with cchardet (2.1.7) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "cchardet (==2.1.7)"'.

 

원인 파악

에러 내용을 보면 cpython에서 longintrepr.h 헤더파일을 찾을 수 없다고 한다.

Python 3.11 docs(https://docs.python.org/3/whatsnew/3.11.html)를 확인해보면 해당 내용을 확인할 수 있다.

제한이 없는 API files들이 include/cpython 디렉토리로 이동되고, eval.h가 없어졌다.

이로 인해 Python.h가 이미 포함되어 있기에 직접사용을 금지한다고 한다.

해결

cchardet의 pypi(https://pypi.org/project/cchardet/)에선 python 3.9까지만 지원한다고 명시되어 있다. 

따라서 cchardet이 대신 charset-normalizer(https://pypi.org/project/charset-normalizer/)로 패키지를 교체해 해결했다.

설치

# pip
pip install charset-normalizer -U
# poetry 
poetry add charset-normalizer

 

구현

기존에는 다음과 같이 사용

import cchardet as chardet

def detect_encoding(file_path, sample_size : int = 10000, confidence_threshold : float = 0.9) -> str:
        """파일의 인코딩을 감지하여 반환

        Returns:
            str: 인코딩
        """
        with open(file_path, 'rb') as f:
            raw_data = f.read(sample_size)
            result = chardet.detect(raw_data)
            confidence = result['confidence']
            encoding = result['encoding']
            if encoding and confidence > confidence_threshold:
                return encoding
            else:
                return DEFAULT_ENCODING

 

패키지 변경 후 

from charset_normalizer import detect

def detect_encoding(file_path, sample_size: int = 10000, confidence_threshold: float = 0.9) -> str:
        """파일 인코딩을 감지하여 반환"""
        with open(file_path, 'rb') as f:
            raw_data = f.read(sample_size)
            result = detect(raw_data)
            if result['encoding'] and result['confidence'] > confidence_threshold:
                return result['encoding']
            else:
                return DEFAULT_ENCODING

 

권장사용방법 (https://charset-normalizer.readthedocs.io/en/stable/user/getstarted.html#basic-usage)

from charset_normalizer import from_bytes, from_path

print(
    str(
        from_bytes(
            my_byte_str
        ).best()
    )
)

# You could also want the same from a file
print(
    str(
        from_path(
            './data/sample.1.ar.srt'
        ).best()
    )
)