From 9a5c183e57fe9d430cd609803ec65ca576be7572 Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Thu, 15 Feb 2024 20:03:00 +0100 Subject: [PATCH 1/8] wip: import xlsx --- README.md | 11 ++++++++++- tests/api/nonsense.xlsx | Bin 0 -> 5693 bytes 2 files changed, 10 insertions(+), 1 deletion(-) create mode 100644 tests/api/nonsense.xlsx diff --git a/README.md b/README.md index b571a4e..44e8f65 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,8 @@
  • How to feed database
  • Rainbow logs with rich
  • Setup user auth
    • +
  • Local development with poetry
    • +
  • Import xlsx files with polars and calamine
    • @@ -47,7 +49,7 @@ ## About The Project Example of [FastAPI](https://fastapi.tiangolo.com/) integration supported by almighty [Pydantic 2.0](https://github.com/pydantic/pydantic) -with [SQLAlchemy ORM](https://www.sqlalchemy.org/) and PostgreSQL +with [SQLAlchemy ORM](https://www.sqlalchemy.org/) and PostgreSQL16 connected via fastest Database Client Library for python/asyncio [asyncpg](https://github.com/MagicStack/asyncpg). Beside of using latest and greatest version of [SQLAlchemy](https://www.sqlalchemy.org/) with it robustness, powerfulness and speed @@ -131,6 +133,12 @@ poetry install ``` Hope you enjoy it. +### Import xlsx files with polars and calamine +Power of Polars Library in data manipulation and analysis. +https://docs.pola.rs/py-polars/html/reference/api/polars.read_excel.html + + + ## Acknowledgments Use this space to list resources you find helpful and would like to give credit to. I've included a few of my favorites to kick things off! @@ -138,6 +146,7 @@ I've included a few of my favorites to kick things off! * [Open Source Shakespeare Dataset](https://github.com/catherinedevlin/opensourceshakespeare) * [SQL Code Generator](https://github.com/agronholm/sqlacodegen) * [Passlib - password hashing library for Python](https://passlib.readthedocs.io/en/stable/) +* [Polars - fast DataFrame library for Rust and Python](https://docs.pola.rs/)

    (back to top)

    diff --git a/tests/api/nonsense.xlsx b/tests/api/nonsense.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..3d2abf05c7b74eaa397bd8be4a5989c68f280dd6 GIT binary patch literal 5693 zcmai21yt1C_NBX#MnXmfh8UzJh7b${hSs5T$N>Rq=^QDCZfWW65|EZI0THBAgdr3K z-=OdP<@5i(@10p|<~M82-LvkwXYYIVfhb|)T)`k9Ai(I>P<@1PDd^G9Znpepj>fK5 zaG2A79t8N@>}=CU^x+*K68z89J7iiPtf2JN@>dK-R6t~l*WpYrqWa0Q-+PZf5V7*a zWU(6nf885|JJ)`>V?yo4u~kWh>p@e!ug^iYXI~Y^DdsGTfYh zZP(KvzmoD0O zik&ka&G)Le_D>r2&R^-lysFyC(;)BB8&Yw@pf$!^(e;yGoh9~Fzk%L*u`kQ>72JIy zHJm`(3WEz4=8A6os3Ghv<0&p=gFz~T4Fd+Xcm9F-VHcpo2=FU$1KRV&eGy37{MvjLCkd>(^Eu5+=gx106QjKs;%~E8}y}K#++wUNbslpphzVy?DtQ6dg zdny-h3eB%wudfZ-N~Nur&N*H$o1UDit@nM334~@(NOQo!<_pNkN)CQXs(73U$r>g0 zWO|JBgix7JBY2Ysed)Kbf|||oUo!<8Wj@ToYTZ`!mC>8yV7G_@ZFomH9m=8NUqZtv zg1I@(ybP9t4J~s-9dOpe9UJLXRQm292_1b;nG0m*Qe*%=07}1|cOt+rTQdZ|p79gy z2BEdsS>B@Mu!=DOJXN0QDAkepzL`ce&t|uXTdy>!P7iWG9iik_tMyXK7Pudv?c>1yKzv(@>aPu!<>2OdOcGf(xqb-|Qo!8k&F7f=x>e zji{;u7-teFw-l-fnf8l5N%b;gs8H)jL@p3hPTs@ga80*W@t`BglMI%1)1+g*XNxca zcI0CoCGiG7V8woJ`#UTVtMgWRPdhKAEDH{EL#$=N5JAD$(E~+o47lWh&xK$%mWGcg zC}>xV`>e%`$x36V_`>wDVC=`(_SuZKZg5nV5L=U^m_=Y8B9I!9i{wLAQ&50X zxgtUWDYzEb00;5NG5OeF-vjxVapdJ6TlCKO?^PFY0&NCUZQH_Tt~M_$!8+YwqD{NS0n;x7^LD&_YwZ>m;5N?KVUDM(=5EjB zx`ztBC#BBbWjK#PwP0;574G)=dU4+i#13j&O7hD@apgKy`UeFp}=!^*0KBel?W34NS(M zv!O58en?s+w@*kb?`u`ea?`Jv`V1_p)E@NS_=P6utN;O4y8h8OFKKq$f`QYfW@hO9 zY5h|CRL=nO7G2UEO2U=E+_XY_n!b%`QU8Ja-kMqD_rDN)+IaEfHMiRNdp>IVJUNjW z5)HWRCz0nZ+lA5y4t~iq-%JJn?MM@YH5$DrCUHe(s1=x7D88e~%_p+-TcyU{SgIu( zs!w<4EV((e|EkVv)Q6`7K}TK*Zl3a>9NzBShUKw5j~)hAoNmXeux9a?NiS2;jXkJY zGw`VRM0KvO@+)p6cPNW39YcigaOkB5q@l(`&dO4=?_5jgabPOEhdKDM@hR z%kIh47~&adX$Y^4HG?EHExy10)b_Kxi_~`~8Fp?RE@0#^9urwWU5sHj`<)JLx*L%v%MC;MPiorjRZsdhG!lbfOBD+#GxXZrbJ16B(8oT-!*`8 z+C9KVW~gui6*PNOPTcFXgdY=3;3Boa8+oOKX&f&*F4rE}yiAJGlGB(u!ZjE;=@&Px z9GPzVxRk-_FlAjW>I7JCnpUjF@(~-BD8WxmCM!8VGjUG4p|Yb+S_V)o-B5Aoe^Q5V zvKD2(<$rU15@F|y;ZjB^Vb9__NwZjp|A`1R>91LP1Lv@tWorDJSz*lqP4tIlBp1;$ z7n*N1a?YpRN9jTfY&xYY_rE6Az{_K44Q3C@X%2NAr+8*8NOkWl%N^doyMMPjLPb#1 zz-r*5j0?XSAM7BD8E!54?7;*6P6(+zfF-V6=RW_YA#BS+)u5~k`nYXYnVX_^SSy^N zZT?MOH6ob&H2&>X`ZuKxGZlM0K6kbIQVRQ-9%c}w==kaRMUExO&mTyej^o$52*Or3 z>_ym*OZEztgJ-81QLM(0xuCF|CpI$va}5UrBXtstqSUrToCt)4fBHj5-VbXYx~n$b z1NORjxgK*H_^*6~WTjkfSg@;+J7U=hJ3vanTE2Y@x#in|39Fiw`Nm`8YC^#Jy{Sr> zO_HMxwfW^OmV~u%ElMF(y6k-|E`vf1F_2OhH&4&zwZ3n&;Qwj zhOv7pn9VaPwFMiiOt<;l4p|v>SGCiH(*>fQlHe=HX4Yee+#TaYWO?np=6W5F4+-;r zTI#IwNW&s-!1GjD=S@9MEh3@L`^59#JTfGP_*C{ z>~eXODu+{@?r0n#)S>WLQ&az~VYs?mdB4e0sYANXOD04mL@wU(jUuG{DaTA`E^MTI z8~gV8lXtpF4bR*u3^>DEOr8@*KV0iGeiurc;`BvX^~U55 zjdUKzn`WMZ0C`m>+jsA~#0Yh2PvGl%<$U1a>R&xG6LOE2=7a$o=dZ-x1M21PWM+Yn z2M1vfupH)|>=qYq@L;!;fnzZ(cv~+U(L%T7^o!JIbVI&+eH(l*3q=+@s0qNxb1m;MR=UI$y~4a;ac(<|0eyX zjL2ouu}EgkqH2F}bA|0nS6VE}PnMZkO}MWHKBq`JMLgp!S1aZV8Xuzeu>{TvDR^A7 zZFnL6xgA)_5qt1RCc79#X_h41CP?=Nbj1x{`4lHd18TSyDWh-lg001Fgl)hB9QDe4 z-$+my@lCZ_I|Q@UO*+HwKC31?ZORyov9ca$<=PhV&0>pL5NLmwVF^K|w^C!>vl4j)1h1#4 z9%tA|WM|V#a>JbBc}!{j;z~7-qpQk>>$;K^=mU9!TxcC#@XY3x*uS})*SmhAS8M4? zon#l{TIaL@VI`DeD&34jDdIp+v3t4$I4kf76j425-fN1S1Eqlz!^ZW6>Qn}4k__b^ ziS}PAuz4111#k>HyNYG!xNa1d#}1JIZbdWSCTtlwRqX8*S5*A6T=1BplKpwh7N+${=p$woeBl`8UPLZR+m_IedW$A z6lgq(gpw<-{+XMBnG|ctK5|H=-Ik`X1&{WzvE^5}1%EJM(21j!j(Y9m6%7UIHlR*8 z$ZI7Q!bQ9)L33?aDbsu*BufAJPoEI!`c(S?`WSBhqg`eEbqv4E!R5pFX%9+}veum- z3I+6ugw9M?#;89COI7AHo6Eze7<+fC;ySzX?CEiu$l`4ZIXZ)|A6?&nj96BkF?#I# zak;hwUm>V)Jg9Bgb#eDX629G}NUE-DBxj)sPc+tDursKbej_;ZRDhUl|FItjVp!c` z*j^!_`ArESWtaoqTPqn= z9d5urkK;=fUZlUFyGW6Lr;GvY5pQ3#)y`Mao1s;$T5Sf^A2tJzBdGY@R0Qhm+a-Aj zQ4C`O5_e9qlZsh2rSrOapR?-tU|$q&JVU$27CIjdXm|DZu7Vi-i(l#bw}X-;^9NDj zj-VZs#j(1Bad>9AzzthcmY22i>@xXQMIF1l>Jh{VF<(*MdBq9tCgg|4Y`~Yjjz*R= zo9&F4)5pzH7{qE&-kX)2MqLk-U2 z_AFQqS)c5hIkca^8doV+ba}r-vmJKm9;-_xkRokDYLoIj#JerG6z7d)g#X)CKup+@ zRgaNG7kWdKaPcnCiO{cPv=m1CT=g&LJadHr literal 0 HcmV?d00001 From cd3ce997d8442c8e849bd32925712ca14040331d Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Thu, 15 Feb 2024 20:04:01 +0100 Subject: [PATCH 2/8] wip: add polars --- poetry.lock | 39 ++++++++++++++++++++++++++++++++++++++- pyproject.toml | 1 + 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/poetry.lock b/poetry.lock index 6864a2b..8614a04 100644 --- a/poetry.lock +++ b/poetry.lock @@ -959,6 +959,43 @@ files = [ dev = ["pre-commit", "tox"] testing = ["pytest", "pytest-benchmark"] +[[package]] +name = "polars" +version = "0.20.8" +description = "Blazingly fast DataFrame library" +optional = false +python-versions = ">=3.8" +files = [ + {file = "polars-0.20.8-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:73f1d369aeddda5f11411b6497f697f2471bbe6ae55fd936677a10a40995c83c"}, + {file = "polars-0.20.8-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:dc3a446fe606095b3ad6df3cf3dddd8ad54be7745f255fedb29f8bdf71a60760"}, + {file = "polars-0.20.8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d3d58ebc7a24d26930535d06b8772e125038a87a6abab4c5dfd87ea19bba61f3"}, + {file = "polars-0.20.8-cp38-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:5b733816ac61156c12bd0edd6d7c1a5e63859830ce0e425b6450b335024f0cd5"}, + {file = "polars-0.20.8-cp38-abi3-win_amd64.whl", hash = "sha256:2300f48ff7120eefe2cac2113990d0b0b5beedad93266b9fedfc8df133e7b13b"}, + {file = "polars-0.20.8.tar.gz", hash = "sha256:a34f6ce1c5469872b291aaf90467e632e81f92dec6c2e18136bc40cd92877411"}, +] + +[package.extras] +adbc = ["adbc_driver_sqlite"] +all = ["polars[adbc,cloudpickle,connectorx,deltalake,fsspec,gevent,numpy,pandas,plot,pyarrow,pydantic,pyiceberg,sqlalchemy,timezone,xlsx2csv,xlsxwriter]"] +cloudpickle = ["cloudpickle"] +connectorx = ["connectorx (>=0.3.2)"] +deltalake = ["deltalake (>=0.14.0)"] +fsspec = ["fsspec"] +gevent = ["gevent"] +matplotlib = ["matplotlib"] +numpy = ["numpy (>=1.16.0)"] +openpyxl = ["openpyxl (>=3.0.0)"] +pandas = ["pandas", "pyarrow (>=7.0.0)"] +plot = ["hvplot (>=0.9.1)"] +pyarrow = ["pyarrow (>=7.0.0)"] +pydantic = ["pydantic"] +pyiceberg = ["pyiceberg (>=0.5.0)"] +pyxlsb = ["pyxlsb (>=1.0)"] +sqlalchemy = ["pandas", "sqlalchemy"] +timezone = ["backports.zoneinfo", "tzdata"] +xlsx2csv = ["xlsx2csv (>=0.8.0)"] +xlsxwriter = ["xlsxwriter"] + [[package]] name = "prompt-toolkit" version = "3.0.43" @@ -2101,4 +2138,4 @@ files = [ [metadata] lock-version = "2.0" python-versions = "^3.12" -content-hash = "c4ed42d2fb1fa0234845ca0a8fe5701c26ef0351e9fa81c3e941e2c5a8e42067" +content-hash = "b2b8464d0d8bae8804ead5aa13137a140b2a22ad23736f3d66243d0b0ea63315" diff --git a/pyproject.toml b/pyproject.toml index ed942bb..56ad247 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,6 +25,7 @@ pyjwt = {version = "2.8.0", extras = ["cryptography"]} redis = "5.0.1" passlib = {version = "^1.7.4", extras = ["bcrypt"]} sourcery = "^1.15.0" +polars = "^0.20.8" [tool.poetry.dev-dependencies] devtools = { extras = ["pygments"], version = "*" } From 101fa05515beb21d3fcb5cf40fb7eb520897c0a8 Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 08:40:49 +0100 Subject: [PATCH 3/8] wip: add import endpoint --- app/api/nonsense.py | 39 ++++++++++++++++++++++++++++++++++++++- app/models/shakespeare.py | 3 +++ app/models/stuff.py | 6 +----- 3 files changed, 42 insertions(+), 6 deletions(-) diff --git a/app/api/nonsense.py b/app/api/nonsense.py index 2da8313..7a90693 100644 --- a/app/api/nonsense.py +++ b/app/api/nonsense.py @@ -1,4 +1,7 @@ -from fastapi import APIRouter, Depends, status +import io +from fastapi import APIRouter, Depends, status, UploadFile, HTTPException +from sqlalchemy.exc import SQLAlchemyError +import polars as pl from sqlalchemy.ext.asyncio import AsyncSession from app.database import get_db @@ -48,3 +51,37 @@ async def merge_nonsense( nonsense = Nonsense(**payload.model_dump()) await nonsense.save_or_update(db_session) return nonsense + + +@router.post( + "/import", + status_code=status.HTTP_201_CREATED, +) +async def import_nonsense( + xlsx: UploadFile, + db_session: AsyncSession = Depends(get_db), +): + file_bytes = await xlsx.read() + + nonsense_data = pl.read_excel( + source=io.BytesIO(file_bytes), + sheet_name="New Nonsense", + engine="calamine", + ) + + try: + nonsense_records = [ + Nonsense( + name=nonsense.get("name"), + description=nonsense.get("description"), + ) + for nonsense in nonsense_data.to_dicts() + ] + db_session.add_all(nonsense_records) + await db_session.commit() + return {"filename": xlsx.filename, "nonsense_records": len(nonsense_records)} + except (SQLAlchemyError, HTTPException) as ex: + await db_session.rollback() + raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=repr(ex)) from ex + finally: + await db_session.close() diff --git a/app/models/shakespeare.py b/app/models/shakespeare.py index b4db11b..411984e 100644 --- a/app/models/shakespeare.py +++ b/app/models/shakespeare.py @@ -63,6 +63,7 @@ class Wordform(Base): - `occurrences` (int): The number of occurrences of the word form. """ + __tablename__ = "wordform" __table_args__ = (PrimaryKeyConstraint("id", name="wordform_pkey"), {"schema": "shakespeare"}) @@ -133,6 +134,7 @@ class Chapter(Base): - `paragraph` (list[Paragraph]): The paragraphs associated with the chapter. """ + __tablename__ = "chapter" __table_args__ = ( ForeignKeyConstraint(["work_id"], ["shakespeare.work.id"], name="chapter_work_id_fkey"), @@ -193,6 +195,7 @@ class Paragraph(Base): - `find(cls, db_session: AsyncSession, character: str) -> List[Paragraph]`: A class method that finds paragraphs associated with a specific character. It takes a database session and the name of the character as arguments, and returns a list of matching paragraphs. """ + __tablename__ = "paragraph" __table_args__ = ( ForeignKeyConstraint(["character_id"], ["shakespeare.character.id"], name="paragraph_character_id_fkey"), diff --git a/app/models/stuff.py b/app/models/stuff.py index 8bf33af..af9b86b 100644 --- a/app/models/stuff.py +++ b/app/models/stuff.py @@ -27,11 +27,7 @@ class Stuff(Base): :param name: :return: """ - stmt = ( - select(cls) - .options(joinedload(cls.nonsense)) - .where(cls.name == name) - ) + stmt = select(cls).options(joinedload(cls.nonsense)).where(cls.name == name) result = await db_session.execute(stmt) instance = result.scalars().first() if instance is None: From a8882084d555950175d0d5bcf6b6796bdcc83636 Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 08:40:56 +0100 Subject: [PATCH 4/8] wip: add import endpoint --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 210e233..bfa465f 100644 --- a/Makefile +++ b/Makefile @@ -35,7 +35,7 @@ safety: ## Check project and dependencies with safety https://github.com/pyupio/ .PHONY: py-upgrade py-upgrade: ## Upgrade project py files with pyupgrade library for python version 3.10 - pyupgrade --py311-plus `find app -name "*.py"` + pyupgrade --py312-plus `find app -name "*.py"` .PHONY: lint lint: ## Lint project code. From 47aaef2bd434bf0022e7670cc61b9a59962ae47c Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 09:47:03 +0100 Subject: [PATCH 5/8] wip: add python multipart --- poetry.lock | 16 +++++++++++++++- pyproject.toml | 1 + 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/poetry.lock b/poetry.lock index 8614a04..d3c5fed 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1358,6 +1358,20 @@ files = [ [package.extras] cli = ["click (>=5.0)"] +[[package]] +name = "python-multipart" +version = "0.0.9" +description = "A streaming multipart parser for Python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "python_multipart-0.0.9-py3-none-any.whl", hash = "sha256:97ca7b8ea7b05f977dc3849c3ba99d51689822fab725c3703af7c866a0c2b215"}, + {file = "python_multipart-0.0.9.tar.gz", hash = "sha256:03f54688c663f1b7977105f021043b0793151e4cb1c1a9d4a11fc13d622c4026"}, +] + +[package.extras] +dev = ["atomicwrites (==1.4.1)", "attrs (==23.2.0)", "coverage (==7.4.1)", "hatch", "invoke (==2.2.0)", "more-itertools (==10.2.0)", "pbr (==6.0.0)", "pluggy (==1.4.0)", "py (==1.11.0)", "pytest (==8.0.0)", "pytest-cov (==4.1.0)", "pytest-timeout (==2.2.0)", "pyyaml (==6.0.1)", "ruff (==0.2.1)"] + [[package]] name = "pyupgrade" version = "3.15.0" @@ -2138,4 +2152,4 @@ files = [ [metadata] lock-version = "2.0" python-versions = "^3.12" -content-hash = "b2b8464d0d8bae8804ead5aa13137a140b2a22ad23736f3d66243d0b0ea63315" +content-hash = "bc15607e221af1a158fd869785e02e017b5b50633d3205d50f1b2bd2bd58d92d" diff --git a/pyproject.toml b/pyproject.toml index 56ad247..0cca430 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ redis = "5.0.1" passlib = {version = "^1.7.4", extras = ["bcrypt"]} sourcery = "^1.15.0" polars = "^0.20.8" +python-multipart = "^0.0.9" [tool.poetry.dev-dependencies] devtools = { extras = ["pygments"], version = "*" } From a95991a6cb044ab238fcdd5d1be5ac7940e084a4 Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 09:47:14 +0100 Subject: [PATCH 6/8] wip: document import endpoint --- app/api/nonsense.py | 23 +++++++++++++++++++++++ tests/api/nonsense.xlsx | Bin 5693 -> 5693 bytes 2 files changed, 23 insertions(+) diff --git a/app/api/nonsense.py b/app/api/nonsense.py index 7a90693..1e52f30 100644 --- a/app/api/nonsense.py +++ b/app/api/nonsense.py @@ -61,8 +61,24 @@ async def import_nonsense( xlsx: UploadFile, db_session: AsyncSession = Depends(get_db), ): + """ + This function is a FastAPI route handler that imports data from an Excel file into a database. + + Args: + xlsx (UploadFile): The Excel file that will be uploaded by the client. + db_session (AsyncSession): A SQLAlchemy session for interacting with the database. + + Returns: + dict: A dictionary containing the filename and the number of imported records. + + Raises: + HTTPException: If an error occurs during the process (either a SQLAlchemy error or an HTTP exception), + the function rolls back the session and raises an HTTP exception with a 422 status code. + """ + # Read the uploaded file into bytes file_bytes = await xlsx.read() + # Use the `polars` library to read the Excel data into a DataFrame nonsense_data = pl.read_excel( source=io.BytesIO(file_bytes), sheet_name="New Nonsense", @@ -70,6 +86,7 @@ async def import_nonsense( ) try: + # Iterate over the DataFrame rows and create a list of `Nonsense` objects nonsense_records = [ Nonsense( name=nonsense.get("name"), @@ -77,11 +94,17 @@ async def import_nonsense( ) for nonsense in nonsense_data.to_dicts() ] + # Add all the `Nonsense` objects to the SQLAlchemy session db_session.add_all(nonsense_records) + # Commit the session to save the objects to the database await db_session.commit() + # Return a JSON response containing the filename and the number of imported records return {"filename": xlsx.filename, "nonsense_records": len(nonsense_records)} except (SQLAlchemyError, HTTPException) as ex: + # If an error occurs, roll back the session await db_session.rollback() + # Raise an HTTP exception with a 422 status code raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=repr(ex)) from ex finally: + # Ensure that the database session is closed, regardless of whether an error occurred or not await db_session.close() diff --git a/tests/api/nonsense.xlsx b/tests/api/nonsense.xlsx index 3d2abf05c7b74eaa397bd8be4a5989c68f280dd6..c090341fd850b3dcb1a90ec6bc277922d75c81cc 100644 GIT binary patch delta 545 zcmdn1vsZ^Vz?+#xgn@&DgW)Sv;6`2#MrI(rIfAi;3Cy_9%m@*A%#y|o7V+UwVgWM> zd3eBTCO7b@)E_bsX!E~d9dWN&*hRa;W5TLcLef`Us~M90J6h!@<~r@DZJ*$>C}+{S zKXdZ!ZvIcZE#)iq!O?Q|<&t>;Okox)-xw~pzdwJosH$R-_Rcb~qN-Mh?MF{Pd-Uh! zgFBrGdwDI_Cap0(*0kcl2ZM6c2pi40<~ENm$w#iREs)e|$S|poRyiqc91~fko>Iqk zcy_fM+2@<}@gd8InO+i3J)--D=<-WwOT{a_97X6JBx#_-OI zk%2*jnSlW%dN#8Q=!0WpvZde)a2%)zl|lsO2s?uWCO?AG8X|sRex-;VnBFB~52pW! eSnvYXMZoQt>?mpgmMIZ+1=D*(&A{|mQ4at#1LR2n delta 545 zcmdn1vsZ^Vz?+#xgn@&DgW*`P|3+R9MrI(rIfAi;3Cy_9%m@*A%#y|o7V+UwVgWM> zd3eBTCO7b@)HfRlwE17KR(LP$JT;7yX^~gx(wtR#x0o%`IKuuo$hA+Jw@;YKBhze! zw*B+j{@>>3S}$HKW}}r}JWVHAta0kPkZ%(7Twk|v7$1zl-rLKf0zB+ zQT|RMaPQGMR~1b(pKD|rFil^_=&e5Wvl~m)FXrRNnl(1Eak$lgK5kL>%uVC;x|S>F z6L+i%5{jE^Be~%r*T)5}3e{^uo0ZR&lv^xbQvPDqhAHQGRE#FbSaI&X@XPJQ2}?J( zACFf?3T#sB-g`-FPW5`u%g^3C$+a>Nn|y4k+R}fS*V zgM>b-{A`Zu{Jnj~detW`MxRp+F8A{YYA)MRTM?IHE8qTc(m98NIijofvWZ0|X8d%z z-(=6(`n@PKJZM|wlVDfj{o3I@1~pHn1ZZ+UwH8%+>%B1betUmxcYJ_1J4au6X_yxy z1A{s<0|QF*Y-ShG2gk-_OTic5I8YHPg$T?Mb_NMdegvg8MEt<~N)bCSy-UO%O#czF d;03CSfZH+IQPcn|QzGgLruT@Nf$6WJ9spAy-dO+u From 45e1c3d5a83d5dd91a81f826f69b0ee78f5b53ed Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 12:12:28 +0100 Subject: [PATCH 7/8] wip: document import endpoint --- app/api/nonsense.py | 19 +++--- poetry.lock | 126 +++++++++++++++++++++++++++++++++++++++- pyproject.toml | 1 + tests/api/nonsense.xlsx | Bin 5693 -> 5696 bytes 4 files changed, 136 insertions(+), 10 deletions(-) diff --git a/app/api/nonsense.py b/app/api/nonsense.py index 1e52f30..7d9e33d 100644 --- a/app/api/nonsense.py +++ b/app/api/nonsense.py @@ -75,17 +75,18 @@ async def import_nonsense( HTTPException: If an error occurs during the process (either a SQLAlchemy error or an HTTP exception), the function rolls back the session and raises an HTTP exception with a 422 status code. """ - # Read the uploaded file into bytes - file_bytes = await xlsx.read() - # Use the `polars` library to read the Excel data into a DataFrame - nonsense_data = pl.read_excel( - source=io.BytesIO(file_bytes), - sheet_name="New Nonsense", - engine="calamine", - ) try: + # Read the uploaded file into bytes + file_bytes = await xlsx.read() + + # Use the `polars` library to read the Excel data into a DataFrame + nonsense_data = pl.read_excel( + source=io.BytesIO(file_bytes), + sheet_name="New Nonsense", + engine="calamine", + ) # Iterate over the DataFrame rows and create a list of `Nonsense` objects nonsense_records = [ Nonsense( @@ -100,7 +101,7 @@ async def import_nonsense( await db_session.commit() # Return a JSON response containing the filename and the number of imported records return {"filename": xlsx.filename, "nonsense_records": len(nonsense_records)} - except (SQLAlchemyError, HTTPException) as ex: + except (SQLAlchemyError, HTTPException, ValueError) as ex: # If an error occurs, roll back the session await db_session.rollback() # Raise an HTTP exception with a 422 status code diff --git a/poetry.lock b/poetry.lock index d3c5fed..15d1461 100644 --- a/poetry.lock +++ b/poetry.lock @@ -474,6 +474,37 @@ typing-extensions = ">=4.8.0" [package.extras] all = ["email-validator (>=2.0.0)", "httpx (>=0.23.0)", "itsdangerous (>=1.1.0)", "jinja2 (>=2.11.2)", "orjson (>=3.2.1)", "pydantic-extra-types (>=2.0.0)", "pydantic-settings (>=2.0.0)", "python-multipart (>=0.0.5)", "pyyaml (>=5.3.1)", "ujson (>=4.0.1,!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0)", "uvicorn[standard] (>=0.12.0)"] +[[package]] +name = "fastexcel" +version = "0.9.0" +description = "" +optional = false +python-versions = ">=3.8" +files = [ + {file = "fastexcel-0.9.0-cp310-cp310-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:a5a35bda90994c63dc5f2c954b20bc1d4fe9467cc62f346fdd2c3646978208b2"}, + {file = "fastexcel-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5006fcf6b619c7c4653c70400ff1ca68103eb253768ea85a4a62883fb72a8d53"}, + {file = "fastexcel-0.9.0-cp310-none-win_amd64.whl", hash = "sha256:f3b47371f650f98e214108034803ffaac9267199c2b13ab1e2fd6885dfa2672b"}, + {file = "fastexcel-0.9.0-cp311-cp311-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:65b93e743f808901cf429fa5f48ecb1f294f2734e98cd49246342c59800d30be"}, + {file = "fastexcel-0.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e55ad5c7b9bc8876201b4a3fb3d325c3cd0a9ae4bd13b4e4aa406234bb10a07e"}, + {file = "fastexcel-0.9.0-cp311-none-win_amd64.whl", hash = "sha256:9c42a5c750c3af2e9dd102bcc18f4326d29189c7573f0fc386aa4f600b61adb0"}, + {file = "fastexcel-0.9.0-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:c9f0a9184b7f7864209bec53d64c0d04ebd51d947501e91a55159288837997b3"}, + {file = "fastexcel-0.9.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c5a026b53e9733a605387fc4d43d5cab2c5a535669e17eab1ec26fe41a4a024"}, + {file = "fastexcel-0.9.0-cp312-none-win_amd64.whl", hash = "sha256:95e50824ae92fd1bf4d02208080cd05e8026cc6b7114574344cd241b3d16e62b"}, + {file = "fastexcel-0.9.0-cp38-cp38-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:b949af5baee79a988589ca7d0e4a7604fcdd708ad1ecd534737dff536cc412c8"}, + {file = "fastexcel-0.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:62054336a956798158baee9c45173d01b510d83dbb38ba13e543d57f8b5585bc"}, + {file = "fastexcel-0.9.0-cp38-none-win_amd64.whl", hash = "sha256:b3757c36efa97572bb667558b6173559fc7611443cbc481d6b3af5e8d479830d"}, + {file = "fastexcel-0.9.0-cp39-cp39-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:5d0c62ef2774040e560fc92b45fe6a091b933ab425293483b22b331daa4ba8c7"}, + {file = "fastexcel-0.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a9403bd6571626f9df048fa7468f699b55053683ce28914028358925a9394b20"}, + {file = "fastexcel-0.9.0-cp39-none-win_amd64.whl", hash = "sha256:b65d6e76423708432885ef609cf1c16042077b567a5334fb0e110f2e411dcd61"}, +] + +[package.dependencies] +pyarrow = ">=8.0.0" + +[package.extras] +pandas = ["pandas (>=1.4.4)"] +polars = ["polars (>=0.16.14)"] + [[package]] name = "greenlet" version = "3.0.3" @@ -881,6 +912,51 @@ files = [ {file = "mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba"}, ] +[[package]] +name = "numpy" +version = "1.26.4" +description = "Fundamental package for array computing in Python" +optional = false +python-versions = ">=3.9" +files = [ + {file = "numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0"}, + {file = "numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a"}, + {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4"}, + {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f"}, + {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a"}, + {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2"}, + {file = "numpy-1.26.4-cp310-cp310-win32.whl", hash = "sha256:bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07"}, + {file = "numpy-1.26.4-cp310-cp310-win_amd64.whl", hash = "sha256:b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5"}, + {file = "numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71"}, + {file = "numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef"}, + {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e"}, + {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5"}, + {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a"}, + {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a"}, + {file = "numpy-1.26.4-cp311-cp311-win32.whl", hash = "sha256:1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20"}, + {file = "numpy-1.26.4-cp311-cp311-win_amd64.whl", hash = "sha256:cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2"}, + {file = "numpy-1.26.4-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218"}, + {file = "numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b"}, + {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b"}, + {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed"}, + {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a"}, + {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0"}, + {file = "numpy-1.26.4-cp312-cp312-win32.whl", hash = "sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110"}, + {file = "numpy-1.26.4-cp312-cp312-win_amd64.whl", hash = "sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818"}, + {file = "numpy-1.26.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:7349ab0fa0c429c82442a27a9673fc802ffdb7c7775fad780226cb234965e53c"}, + {file = "numpy-1.26.4-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:52b8b60467cd7dd1e9ed082188b4e6bb35aa5cdd01777621a1658910745b90be"}, + {file = "numpy-1.26.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d5241e0a80d808d70546c697135da2c613f30e28251ff8307eb72ba696945764"}, + {file = "numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f870204a840a60da0b12273ef34f7051e98c3b5961b61b0c2c1be6dfd64fbcd3"}, + {file = "numpy-1.26.4-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:679b0076f67ecc0138fd2ede3a8fd196dddc2ad3254069bcb9faf9a79b1cebcd"}, + {file = "numpy-1.26.4-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:47711010ad8555514b434df65f7d7b076bb8261df1ca9bb78f53d3b2db02e95c"}, + {file = "numpy-1.26.4-cp39-cp39-win32.whl", hash = "sha256:a354325ee03388678242a4d7ebcd08b5c727033fcff3b2f536aea978e15ee9e6"}, + {file = "numpy-1.26.4-cp39-cp39-win_amd64.whl", hash = "sha256:3373d5d70a5fe74a2c1bb6d2cfd9609ecf686d47a2d7b1d37a8f3b6bf6003aea"}, + {file = "numpy-1.26.4-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:afedb719a9dcfc7eaf2287b839d8198e06dcd4cb5d276a3df279231138e83d30"}, + {file = "numpy-1.26.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:95a7476c59002f2f6c590b9b7b998306fba6a5aa646b1e22ddfeaf8f78c3a29c"}, + {file = "numpy-1.26.4-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:7e50d0a0cc3189f9cb0aeb3a6a6af18c16f59f004b866cd2be1c14b36134a4a0"}, + {file = "numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010"}, +] + [[package]] name = "packaging" version = "21.3" @@ -1116,6 +1192,54 @@ files = [ [package.extras] tests = ["pytest"] +[[package]] +name = "pyarrow" +version = "15.0.0" +description = "Python library for Apache Arrow" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pyarrow-15.0.0-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:0a524532fd6dd482edaa563b686d754c70417c2f72742a8c990b322d4c03a15d"}, + {file = "pyarrow-15.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:60a6bdb314affa9c2e0d5dddf3d9cbb9ef4a8dddaa68669975287d47ece67642"}, + {file = "pyarrow-15.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:66958fd1771a4d4b754cd385835e66a3ef6b12611e001d4e5edfcef5f30391e2"}, + {file = "pyarrow-15.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1f500956a49aadd907eaa21d4fff75f73954605eaa41f61cb94fb008cf2e00c6"}, + {file = "pyarrow-15.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:6f87d9c4f09e049c2cade559643424da84c43a35068f2a1c4653dc5b1408a929"}, + {file = "pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:85239b9f93278e130d86c0e6bb455dcb66fc3fd891398b9d45ace8799a871a1e"}, + {file = "pyarrow-15.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:5b8d43e31ca16aa6e12402fcb1e14352d0d809de70edd185c7650fe80e0769e3"}, + {file = "pyarrow-15.0.0-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:fa7cd198280dbd0c988df525e50e35b5d16873e2cdae2aaaa6363cdb64e3eec5"}, + {file = "pyarrow-15.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8780b1a29d3c8b21ba6b191305a2a607de2e30dab399776ff0aa09131e266340"}, + {file = "pyarrow-15.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fe0ec198ccc680f6c92723fadcb97b74f07c45ff3fdec9dd765deb04955ccf19"}, + {file = "pyarrow-15.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:036a7209c235588c2f07477fe75c07e6caced9b7b61bb897c8d4e52c4b5f9555"}, + {file = "pyarrow-15.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:2bd8a0e5296797faf9a3294e9fa2dc67aa7f10ae2207920dbebb785c77e9dbe5"}, + {file = "pyarrow-15.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:e8ebed6053dbe76883a822d4e8da36860f479d55a762bd9e70d8494aed87113e"}, + {file = "pyarrow-15.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:17d53a9d1b2b5bd7d5e4cd84d018e2a45bc9baaa68f7e6e3ebed45649900ba99"}, + {file = "pyarrow-15.0.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:9950a9c9df24090d3d558b43b97753b8f5867fb8e521f29876aa021c52fda351"}, + {file = "pyarrow-15.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:003d680b5e422d0204e7287bb3fa775b332b3fce2996aa69e9adea23f5c8f970"}, + {file = "pyarrow-15.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f75fce89dad10c95f4bf590b765e3ae98bcc5ba9f6ce75adb828a334e26a3d40"}, + {file = "pyarrow-15.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0ca9cb0039923bec49b4fe23803807e4ef39576a2bec59c32b11296464623dc2"}, + {file = "pyarrow-15.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:9ed5a78ed29d171d0acc26a305a4b7f83c122d54ff5270810ac23c75813585e4"}, + {file = "pyarrow-15.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6eda9e117f0402dfcd3cd6ec9bfee89ac5071c48fc83a84f3075b60efa96747f"}, + {file = "pyarrow-15.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:9a3a6180c0e8f2727e6f1b1c87c72d3254cac909e609f35f22532e4115461177"}, + {file = "pyarrow-15.0.0-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:19a8918045993349b207de72d4576af0191beef03ea655d8bdb13762f0cd6eac"}, + {file = "pyarrow-15.0.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:d0ec076b32bacb6666e8813a22e6e5a7ef1314c8069d4ff345efa6246bc38593"}, + {file = "pyarrow-15.0.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5db1769e5d0a77eb92344c7382d6543bea1164cca3704f84aa44e26c67e320fb"}, + {file = "pyarrow-15.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e2617e3bf9df2a00020dd1c1c6dce5cc343d979efe10bc401c0632b0eef6ef5b"}, + {file = "pyarrow-15.0.0-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:d31c1d45060180131caf10f0f698e3a782db333a422038bf7fe01dace18b3a31"}, + {file = "pyarrow-15.0.0-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:c8c287d1d479de8269398b34282e206844abb3208224dbdd7166d580804674b7"}, + {file = "pyarrow-15.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:07eb7f07dc9ecbb8dace0f58f009d3a29ee58682fcdc91337dfeb51ea618a75b"}, + {file = "pyarrow-15.0.0-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:47af7036f64fce990bb8a5948c04722e4e3ea3e13b1007ef52dfe0aa8f23cf7f"}, + {file = "pyarrow-15.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:93768ccfff85cf044c418bfeeafce9a8bb0cee091bd8fd19011aff91e58de540"}, + {file = "pyarrow-15.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f6ee87fd6892700960d90abb7b17a72a5abb3b64ee0fe8db6c782bcc2d0dc0b4"}, + {file = "pyarrow-15.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:001fca027738c5f6be0b7a3159cc7ba16a5c52486db18160909a0831b063c4e4"}, + {file = "pyarrow-15.0.0-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:d1c48648f64aec09accf44140dccb92f4f94394b8d79976c426a5b79b11d4fa7"}, + {file = "pyarrow-15.0.0-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:972a0141be402bb18e3201448c8ae62958c9c7923dfaa3b3d4530c835ac81aed"}, + {file = "pyarrow-15.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:f01fc5cf49081426429127aa2d427d9d98e1cb94a32cb961d583a70b7c4504e6"}, + {file = "pyarrow-15.0.0.tar.gz", hash = "sha256:876858f549d540898f927eba4ef77cd549ad8d24baa3207cf1b72e5788b50e83"}, +] + +[package.dependencies] +numpy = ">=1.16.6,<2" + [[package]] name = "pydantic" version = "2.6.0" @@ -2152,4 +2276,4 @@ files = [ [metadata] lock-version = "2.0" python-versions = "^3.12" -content-hash = "bc15607e221af1a158fd869785e02e017b5b50633d3205d50f1b2bd2bd58d92d" +content-hash = "c377a22c133bd6006c0cec3730e68cca222f16b67fde065f92be47e2af7190fc" diff --git a/pyproject.toml b/pyproject.toml index 0cca430..f2b4aa0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,6 +27,7 @@ passlib = {version = "^1.7.4", extras = ["bcrypt"]} sourcery = "^1.15.0" polars = "^0.20.8" python-multipart = "^0.0.9" +fastexcel = "^0.9.0" [tool.poetry.dev-dependencies] devtools = { extras = ["pygments"], version = "*" } diff --git a/tests/api/nonsense.xlsx b/tests/api/nonsense.xlsx index c090341fd850b3dcb1a90ec6bc277922d75c81cc..3488785f592f3e74f53830240ff36377c9035b66 100644 GIT binary patch delta 645 zcmdn1b3lhDz?+#xgn@&DgF#txBabH|ld|OIaK=_95PgrC3Cw@QlFkg`dvhqWfan4q zUXc9cdOqd)`&ow-MB3*kyjQrlU2U;+uE(X>hA(_?9{t7;<}bCfzUh`p?eDqEJQuw7 zmEgGe;^(>cf1h&GO)?L%Kh1Hxoa)3G@o>qe<4yl+>h-QoX*sd>$*o9@7otper|JJZ z`t$O^ox%sJq`9Jw-$?0U4Qzgx_)==aj$q6DiX%(yH(jWBV05LP;bO+OSt{?(6r{eK zJK?7#Pk5-;fp+0^X0v;Wawh#5@>y3~CZ_FDpA&d!_k&eCM9%T3m6GXt^7$JVMH{ui13rDBJ7_~$+I^@p40~p4&vQT8N||kQsH3b&K+?{G zPv!Jd`WG_D%y@A7CR+%;1VxRqP#KszTi698`Vc~? zi}-_h6(aT^YNv<;i25yJ$*U|G7y-9%vZJVhJV<1kwdK=eIkCNTdIOFA=%@6Dmi0-_6e zctP@$>-m)Hz4Hzk2(S(LMA z-Jd!6b~pd0-Inr|`rv3e`*O*=0H!dDm2V7}+uxtRSyWZANPB0QSW#80!}g=6pFR5X z^1+?XguT3$Ym?R(A8T51;DbTAX@rgDTyvX8m*gW?*cM1?HPmOAM5~;XHjas`QctPl zIy`l$(1(XdZZ`Cmcd(yTzSZt^HF3erZ=C5f)rOe}4yXMgRvxuS6O$qn__By3~yEZsFaF7r0y2{~i1L6;*e5E(s5AMt isFXZNS8m6a08^k7C0Q64gb`YREb+-AVrp!?U_k&<9O=ve From e65b57e7ad24bdb45edfe953a4f370ab1771598d Mon Sep 17 00:00:00 2001 From: Jakub Miazek Date: Sat, 17 Feb 2024 12:21:55 +0100 Subject: [PATCH 8/8] wip: docs --- README.md | 24 +++++++++++------------- app/api/nonsense.py | 2 -- 2 files changed, 11 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 44e8f65..bba5426 100644 --- a/README.md +++ b/README.md @@ -27,23 +27,18 @@
  • Rainbow logs with rich
  • Setup user auth
  • Local development with poetry
    • -
  • Import xlsx files with polars and calamine
    • +
  • Import xlsx files with polars and calamine
    • - -[//]: # (
  • Usage
    • ) - -[//]: # (
  • Roadmap
    • ) - -[//]: # (
  • Contributing
    • ) - -[//]: # (
  • License
    • ) - -[//]: # (
  • Contact
    • )
  • Acknowledgments
    • +[//]: # (TODO: Usage,Roadmap, Contributing, License, Contact) + + + + ## About The Project @@ -135,9 +130,11 @@ Hope you enjoy it. ### Import xlsx files with polars and calamine Power of Polars Library in data manipulation and analysis. +It uses the polars library to read the Excel data into a DataFrame by passing the bytes to the `pl.read_excel()` function - https://docs.pola.rs/py-polars/html/reference/api/polars.read_excel.html +In `pl.read_excel()` “calamine” engine can be used for reading all major types of Excel Workbook (.xlsx, .xlsb, .xls) and is dramatically faster than the other options, using the fastexcel module to bind calamine. - +

    (back to top)

    ## Acknowledgments Use this space to list resources you find helpful and would like to give credit to. @@ -164,7 +161,8 @@ I've included a few of my favorites to kick things off! - **[JUL 25 2023]** add user authentication with JWT and Redis as token storage :lock: :key: - **[SEP 2 2023]** add passlib and bcrypt for password hashing :lock: :key: - **[OCT 21 2023]** refactor shakespeare models to use sqlalchemy 2.0 :fast_forward: -- **[FEB 1 2024]** bum project to Python 3.12 :fast_forward: +- **[FEB 1 2024]** bump project to Python 3.12 :fast_forward: +- **[MAR 15 2024]** add polars and calamine to project :features:

    (back to top)

    diff --git a/app/api/nonsense.py b/app/api/nonsense.py index 7d9e33d..49422db 100644 --- a/app/api/nonsense.py +++ b/app/api/nonsense.py @@ -75,8 +75,6 @@ async def import_nonsense( HTTPException: If an error occurs during the process (either a SQLAlchemy error or an HTTP exception), the function rolls back the session and raises an HTTP exception with a 422 status code. """ - - try: # Read the uploaded file into bytes file_bytes = await xlsx.read()