SNOW-141545: Request - Reduce package dependencies chaos #284
Description
Please answer these questions before submitting your issue. Thanks!
-
What version of Python are you using (
python --version)?
python 3.7 -
What operating system and processor architecture are you using (
python -c 'import platform; print(platform.platform())')?
Linux-5.3.0-40-generic-x86_64-with-debian-buster-sid -
What are the component versions in the environment (
pip list)?
Package Version
-------------------------- ----------
ansible 2.8.6
asn1crypto 1.2.0
attrs 19.3.0
awscli 1.16.265
azure-common 1.1.23
azure-storage-blob 2.1.0
azure-storage-common 2.1.0
backcall 0.1.0
bleach 3.1.0
blis 0.4.1
bokeh 1.4.0
boto 2.49.0
boto3 1.10.50
botocore 1.13.50
certifi 2019.11.28
cffi 1.13.1
chardet 3.0.4
Click 7.0
cloudpickle 1.2.2
colorama 0.4.1
conda 4.8.0
conda-package-handling 1.6.0
credsman 0.0.1
croniter 0.3.31
cryptography 2.7
cymem 2.0.2
dask 2.6.0
datasketch 1.4.10
dateparser 0.7.0
decorator 4.4.0
defusedxml 0.6.0
distributed 2.6.0
docutils 0.15.2
dxtrack 0.6.1
elasticsearch 7.1.0
entrypoints 0.3
fastjsonschema 2.14.1
fastparquet 0.3.2
fsspec 0.6.2
future 0.18.1
gspread 3.1.0
HeapDict 1.0.1
idna 2.8
ijson 2.5.1
importlib-metadata 0.23
ipykernel 5.1.3
ipython 7.8.0
ipython-genutils 0.2.0
ipywidgets 7.5.1
jedi 0.15.1
jellyfish 0.7.2
Jinja2 2.10.3
jmespath 0.9.4
joblib 0.14.0
json5 0.8.5
jsonschema 3.1.1
jupyter-client 5.3.3
jupyter-core 4.5.0
jupyterlab 1.1.4
jupyterlab-server 1.0.6
llvmlite 0.30.0
locket 0.2.0
MarkupSafe 1.1.1
metaflow 2.0.0
mistune 0.8.4
more-itertools 7.2.0
msgpack 0.6.2
murmurhash 1.0.2
nbconvert 5.6.0
nbformat 4.4.0
nl-core-news-sm 2.2.1
notebook 6.0.1
numba 0.46.0
numpy 1.17.3
oscrypto 1.1.0
packaging 19.2
pandas 0.25.3
pandocfilters 1.4.2
parso 0.5.1
partd 1.0.0
pexpect 4.7.0
pickleshare 0.7.5
Pillow 6.2.1
pip 19.3.1
plac 0.9.6
preshed 3.0.2
prometheus-client 0.7.1
prompt-toolkit 2.0.10
psutil 5.6.5
ptyprocess 0.6.0
pyarrow 0.15.1
pyasn1 0.4.7
PyAthena 1.8.0
pycosat 0.6.3
pycparser 2.19
pycryptodomex 3.9.0
Pygments 2.4.2
PyJWT 1.7.1
pyOpenSSL 19.0.0
pyparsing 2.4.4
pyrsistent 0.15.4
PySocks 1.7.1
python-dateutil 2.8.0
python-rapidjson 0.8.0
pytz 2019.3
PyYAML 5.1.2
pyzmq 18.1.0
recordlinkage 0.14
redis 3.3.11
regex 2019.12.9
requests 2.22.0
requests-aws4auth 0.9
rsa 3.4.2
ruamel-yaml 0.15.71
s3fs 0.4.0
s3transfer 0.2.1
scikit-learn 0.21.3
scipy 1.3.1
Send2Trash 1.5.0
setuptools 41.4.0
six 1.12.0
sklearn 0.0
smart-open 1.9.0
snowconn 3.6.0
snowflake-connector-python 2.1.3
snowflake-sqlalchemy 1.1.17
sortedcontainers 2.1.0
spacy 2.2.1
SQLAlchemy 1.3.10
srsly 0.1.0
tblib 1.5.0
tenacity 6.0.0
terminado 0.8.2
testpath 0.4.2
thinc 7.1.1
thrift 0.11.0
toolz 0.10.0
tornado 6.0.3
tqdm 4.36.1
traitlets 4.3.3
tzlocal 2.0.0
Unidecode 1.1.1
urllib3 1.25.6
wasabi 0.2.2
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.32.3
widgetsnbextension 3.5.1
zict 1.0.0
zipp 0.6.0
- What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
i manage our etl and reporting frameworks so i maintain different snowflake-connector-python versions at all times - What did you expect to see?
docker builds not failling every week because of snowflake-connector-python crazy and nonsensical requirements that should be set up as external optional dependencies - What did you see instead?
package errors when importing snowflake-connector-python - Can you set logging to DEBUG and collect the logs?
since when this happens i have to fix it i cant provide the logs, will do next time it breaks
import logging
import os
for logger_name in ['snowflake.sqlalchemy', 'snowflake.connector', 'botocore']:
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
logger.addHandler(ch)
Basically I think the chaotic behavior of snowflake connector package dependency could be mitigated if most dependencies are moved as optional, such as azure, crypto, etcetera.