GitHunt
QU

QuanBlue/s3-bucket-crud-library

Python Library that interact (CRUD) with S3-compatible object storage

awscrudpythons3-compatibleviettel-cloud

icon
S3 Bucket CRUD Library

Library that can interact with Viettel Cloud, AWS S3 Bucket or any bucket that compatible AWS S3 bucket API

contributors last update forks stars open issues license

Documentation • Report Bug • Request Feature


📖 Table of Contents

⭐ Key features

A Python library that simplifies interactions with S3-compatible object storage services for Viettel Cloud, AWS S3 Bucket or any bucket that compatible AWS S3 bucket API. It provides easy-to-use functions for managing buckets and objects, including creating, listing, uploading, and deleting files.

  • Create and delete S3 buckets
  • List all available buckets
  • Upload files or folders to S3
  • List objects in a bucket
  • Delete objects by prefix
  • Uses boto3 to interact with S3-compatible services

🧰 Getting start

📌 Prerequisites

  • Python3
  • Python Package
    • boto3 >=1.36.0
    • dotenv >=0.9.9

🔑 Environment Variables

To run this project, you need to add the following environment variables to your .env :

  • Configs: Create .env file in ./

    • S3_ENDPOINT_URL: Url that connect to S3 bucket provider, default is viettel_cloud_enpoint
    • S3_ACCESS_KEY_ID: S3 bucket access key
    • S3_SECRET_KEY_ID: S3 bucket secret key id

    Example:

    # .env
# with Viettel Cloud, using https://os.viettelcloud.vn/ endpoint
# with AWS S3 bucket, using https://s3.<region>.amazonaws.com enpoint
S3_ENDPOINT_URL="https://os.viettelcloud.vn/"
S3_ACCESS_KEY_ID="xxxxxxxxxxxxxxxxxxxx"
S3_SECRET_KEY_ID="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

You can also check out the file .env.example to see all required environment variables.

Note: If you want to use this example environment, you need to rename it to .env

🛠️ Installation

Create python environment

python3 -m venv .venv
source .venv/bin/activate

Install packages

pip install -r requirements.txt

📖 Usage

1. Import and Initialize

import os
from dotenv import load_dotenv
from s3_library.s3_utils import S3Utils


# Load environment variables
load_dotenv()
S3_ENDPOINT_URL = os.getenv("S3_ENDPOINT_URL")
S3_ACCESS_KEY_ID = os.getenv("S3_ACCESS_KEY_ID")
S3_SECRET_KEY_ID = os.getenv("S3_SECRET_KEY_ID")

os.environ["AWS_REQUEST_CHECKSUM_CALCULATION"] = "when_required"
os.environ["AWS_RESPONSE_CHECKSUM_VALIDATION"] = "when_required"

s3 = S3Utils(S3_ACCESS_KEY_ID, S3_SECRET_KEY_ID, S3_ENDPOINT_URL)

# [... Your code here ...]

2. Bucket Operations

List all available buckets

# List all available buckets of this s3 bucket
buckets = s3.list_bucket()
print(buckets)

Create a new bucket

# Create a new bucket (ex: my-new-bucket)
s3.create_bucket(bucket_name="my-new-bucket")

Delete a bucket

# Delete a bucket (ex: my-new-bucket)
s3.delete_bucket(bucket_name="my-new-bucket")

Sync 2 buckets

####################################################
# Sync buckets unidirectional (on-way)
# (ex: sync from `my-bucket-1` to `my-bucket-2`)
s3.sync_buckets_unidirectional(
  source_bucket="my-bucket-1", dest_bucket="my-bucket-2"
)

####################################################
# Sync buckets unidirectional (on-way) with specific prefix
# (ex: sync from `my-bucket-1` to `my-bucket-2` with prefix `utils`)
s3.sync_buckets_unidirectional(
  source_bucket="my-bucket-1", dest_bucket="my-bucket-2", prefix="utils/"
)

####################################################
# Sync buckets bidirectional (two-way)
# (ex: sync between `my-bucket-1` and `my-bucket-2`)
s3.sync_buckets_bidirectional(
  bucket1="my-bucket-1", bucket2="my-bucket-2"
)

####################################################
# Sync buckets bidirectional (two-way) with specific prefix
# (ex: sync between `my-bucket-1` and `my-bucket-2` with prefix `utils`)
s3.sync_buckets_bidirectional(
  bucket1="my-bucket-1", bucket2="my-bucket-2", prefix="utils/"
)

3. Object Operations

List objects in bucket

####################################################
# List objects in bucket.
# By default it get first 1000 object (batch=1000)
objects = s3.list_objects_batch(bucket_name="my-bucket")

####################################################
# List first 9999 objects in bucket.
objects = s3.list_objects_batch(bucket_name="my-bucket", batch="9999")

####################################################
# List 1000 first object in folder (ex: foo/boo/)
objects = s3.list_objects_batch(bucket_name="my-bucket", prefix="foo/boo/")

####################################################
# If you wanna create pagination
# use param: "continuation_token" (ex: list 1000 object has index 1001 to 2000)
objects = s3.list_objects_batch(bucket_name="my-bucket", continuation_token="1001")

Print object in tree format

####################################################
# By default it will:
# - Show depth = 3 and 5 item in a folder
# - Only print size of objects (optimize query time)
s3.show_tree(bucket_name="my-bucket")
#
# 📁 my-bucket
# ├── 📁 assets/
# │   └── 📄 s3-bucket-logo.svg                                                    1.10 KB
# ├── 📁 utils/
# │   ├── 📁 __pycache__/
# │   │   ├── 📁 images/
# │   │   │   ├── 📁 temp/
# │   │   │   │   ├── 📁 tmp1/
# │   │   │   │   └── 📄 full_2000005070...a3479bb820b.jpg                       123.95 KB
# │   │   │   ├── 📄 554acfd1-3680-4...ff9a546efc.jpeg                           436.87 KB
# │   │   │   ├── 📄 delete_image.py                                               1.44 KB
# │   │   │   ├── 📄 full_2000005070...a3479bb820b.jpg                           123.95 KB
# ...

####################################################
# Show tree with folder size
# This will query all object of bucket. Be patient, it will take amount of time
s3.show_tree(bucket_name="my-bucket", show_folder_size=True)
#
# 📁 my-bucket                                                                     1.59 MB
# ├── 📁 assets/                                                                   1.10 KB
# │   └── 📄 s3-bucket-logo.svg                                                    1.10 KB
# ├── 📁 utils/                                                                    1.26 MB
# │   ├── 📁 __pycache__/                                                          1.24 MB
# │   │   ├── 📁 images/                                                         811.53 KB
# │   │   │   ├── 📁 temp/                                                       247.89 KB
# │   │   │   │   ├── 📁 tmp1/                                                   123.95 KB
# │   │   │   │   └── 📄 full_2000005070...a3479bb820b.jpg                       123.95 KB
# │   │   │   ├── 📄 554acfd1-3680-4...ff9a546efc.jpeg                           436.87 KB
# │   │   │   ├── 📄 delete_image.py                                               1.44 KB
# │   │   │   ├── 📄 full_2000005070...a3479bb820b.jpg                           123.95 KB
# ...

####################################################
# Show tree with deeper and more file (ex: max_depth=7, max_items_per_level=6)
s3.show_tree(bucket_name="my-bucket", max_depth=7, max_items_per_level=6)
#
# 📁 my-bucket
# ├── 📁 assets/
# │   └── 📄 s3-bucket-logo.svg                                                    1.10 KB
# ├── 📁 utils/
# │   ├── 📁 __pycache__/
# │   │   ├── 📁 images/
# │   │   │   ├── 📁 temp/
# │   │   │   │   ├── 📁 tmp1/
# │   │   │   │   │   └── 📁 12/
# │   │   │   │   │       └── 📄 full_2000005070...a3479bb820b.jpg               123.95 KB
# │   │   │   │   └── 📄 full_2000005070...a3479bb820b.jpg                       123.95 KB
# │   │   │   ├── 📄 554acfd1-3680-4...ff9a546efc.jpeg                           436.87 KB
# │   │   │   ├── 📄 delete_image.py                                               1.44 KB
# │   │   │   ├── 📄 full_2000005070...a3479bb820b.jpg                           123.95 KB
# ...

####################################################
# Show tree start at a folder (prefix) (ex: utils)
s3.show_tree(bucket_name="my-bucket", prefix="utils/")
#
# 📁 my-bucket/utils/
# ├── 📁 __pycache__/
# │   ├── 📁 images/
# │   │   ├── 📁 temp/
# │   │   │   ├── 📁 tmp1/
# │   │   │   │   └── 📁 12/
# │   │   │   └── 📄 full_2000005070...a3479bb820b.jpg                           123.95 KB
# │   │   ├── 📄 554acfd1-3680-4...ff9a546efc.jpeg                               436.87 KB
# │   │   ├── 📄 delete_image.py                                                   1.44 KB
# │   │   ├── 📄 full_2000005070...a3479bb820b.jpg                               123.95 KB
# ...

Download object/folder

####################################################
# Download a object (ex: s3_bucket.py)
# By default it will download at locate that you run script
s3.download_objects(bucket_name="my-bucket", prefix="s3_bucket.py")

####################################################
# Download a folder (download all object have prefix <prefix>)
# It will create and download directory (ex: "utils") at locate that you run script
s3.download_objects(bucket_name="my-bucket", prefix="utils/")

####################################################
# Download a object to different download dir (ex: /Download)
s3.download_objects(bucket_name="my-bucket", prefix="s3_bucket.py", local_dir="/Download")

Upload object/folder/specific objects in csv file

####################################################
# Upload a object (ex: s3_bucket.py)
# By default it will upload at root (/)
s3.upload_to_s3(bucket_name="my-bucket", source_path="s3_bucket.py")

####################################################
# Upload a folder (ex: upload folder utils/)
s3.upload_to_s3(bucket_name="my-bucket", source_path="utils/")

####################################################
# Upload a object to different folder (ex: upload s3_bucket.py to /utils folder)
s3.upload_to_s3(bucket_name="my-bucket", source_path="s3_bucket.py", s3_prefix="utils/")

####################################################
# Upload objects in csv file
# It will upload all objects/folder that listed in csv file
#
# Ex: upload_objects.csv
# (at 1st line, it will upload s3_bucket.py to utils folder.
# At 2nd line, it will upload folder utils to root)
# s3_bucket.py, utils
# utils,
s3.upload_to_s3(bucket_name="my-bucket", csv_file="upload_objects.csv")

Delete object/folder/specific objects in csv file

####################################################
# Delete a object (ex: s3_bucket.py)
s3.delete_objects(bucket_name="my-bucket", prefix="s3_bucket.py")

####################################################
# Delete a folder (delete all object have prefix <prefix>) (ex: utils)
s3.delete_objects(bucket_name="my-bucket", prefix="utils/")

####################################################
# Delete objects in csv file
# It will delete all objects/folder that listed in csv file
#
# Ex: delete_objects.csv
# s3_bucket.py
# utils
s3.delete_objects(bucket_name="my-bucket", csv_file="delete_objects.csv")

🗺️ Roadmap

  • Basic CRUD
    • Bucket
      • List Bucket
      • Create Bucket
      • Delete Bucket
    • Objects
      • List Objects
        • List by batch (default 1000)
        • Pagination
      • Upload Objects
        • Upload single object
        • Upload folder
        • Upload objects by csv file
      • Delete Objects
        • Delete single object
        • Delete folder
        • Delete objects by csv file
    • Print Objects, folder
      • Tree view
      • Object, folder size
  • Sync between 2 Buckets
    • Sync bidirectional
    • Sync unidirectional

👥 Contributors

Contributions are always welcome!

✨ Credits

This software uses the following packages:

  • boto3 - AWS SDK for Python

Emoji and Badges from:

📜 License

Distributed under the MIT License. See LICENSE for more information.

Bento @quanblue  · 
GitHub @QuanBlue  ·  Gmail quannguyenthanh558@gmail.com

Languages

Python100.0%

Contributors

QU
MIT License
Created March 22, 2025
Updated October 3, 2025
QuanBlue/s3-bucket-crud-library | GitHunt