Omnichannel Sales Engine

A complete data engineering portfolio project demonstrating ETL pipelines, dimensional modeling, orchestration, and analytics visualization.

🎯 Project Overview

This project scrapes product data from multiple e-commerce platforms (Amazon, eBay, Shopee), transforms it into a star schema data warehouse, and visualizes insights through an interactive dashboard.

Skills Demonstrated

Area	Technologies
Data Ingestion	Web scraping with BeautifulSoup, API integration
Data Modeling	Star schema (fact/dimension tables), PostgreSQL
ETL Pipeline	Python, SQLAlchemy, data validation
Orchestration	Apache Airflow DAGs
Containerization	Docker, Docker Compose
Visualization	Streamlit, Plotly

🏗️ Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   E-commerce    │     │    Scrapers     │     │   Raw Storage   │
│   Platforms     │────▶│  (Python)       │────▶│   (CSV/JSON)    │
│ Amazon/eBay/... │     │                 │     │   data/raw/     │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                         │
                        ┌─────────────────┐              ▼
                        │   Airflow       │     ┌─────────────────┐
                        │  Orchestration  │────▶│   Processors    │
                        │                 │     │ Validate/Clean  │
                        └─────────────────┘     └────────┬────────┘
                                                         │
┌─────────────────┐     ┌─────────────────┐              ▼
│   Streamlit     │◀────│   PostgreSQL    │◀────┌─────────────────┐
│   Dashboard     │     │  Star Schema    │     │  Transformer    │
│                 │     │                 │     │ Staging → DW    │
└─────────────────┘     └─────────────────┘     └─────────────────┘

📁 Directory Structure

omnichannel-sales-engine/
├── airflow/dags/           # Airflow DAG definitions
├── config/                 # Configuration and settings
├── dashboard/              # Streamlit app (multi-page)
├── database/               # SQLAlchemy models and migrations
├── processors/             # Data validation, transformation, loading
├── scrapers/               # Platform-specific scrapers
├── sql/schema/             # Raw SQL table definitions
├── tests/                  # Unit and integration tests
├── docker-compose.yml      # Full stack configuration
└── Makefile                # Common commands

🚀 Quick Start

Prerequisites

Docker and Docker Compose
Python 3.10+ (for local development)

1. Clone and Setup

git clone https://github.com/syfqfrhnn/omnichannel-sales-engine.git
cd omnichannel-sales-engine
cp .env.example .env  # Edit with your settings

2. Start with Docker

# Start all services
docker-compose up -d

# Wait for services to be healthy
docker-compose ps

# View logs
docker-compose logs -f

3. Access Services

Service	URL	Credentials
Airflow	http://localhost:8080	admin / admin
Dashboard	http://localhost:8501	-
PostgreSQL	localhost:5432	sales_user / sales_password

4. Initialize Database with Sample Data

# Enter the dashboard container
docker-compose exec dashboard bash

# Initialize tables and seed sample data
python database/init_db.py
python database/seed_data.py

🔧 Local Development

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r config/requirements.txt

# Set environment variables
export DATABASE_URL=postgresql://sales_user:sales_password@localhost:5432/sales_analytics

# Run dashboard locally
streamlit run dashboard/app.py

📊 Data Warehouse Schema

Star Schema Design

Dimension Tables:

dim_product - Product attributes (name, brand, URL)
dim_platform - E-commerce platforms
dim_category - Product categories (hierarchical)
dim_date - Date dimension for time analysis

Fact Tables:

fact_price_history - Daily price snapshots
fact_sales - Sales transactions
fact_reviews_daily - Aggregated review metrics

Staging Tables:

stg_products - Raw scraped product data
stg_reviews - Raw scraped reviews

⚙️ Airflow DAGs

DAG	Schedule	Description
`daily_scrape`	6:00 AM UTC	Scrape products from all platforms
`etl_pipeline`	7:30 AM UTC	Transform staging → warehouse
`data_quality`	8:00 AM UTC	Run quality checks

📈 Dashboard Pages

Overview - KPIs, product distribution, price trends
Products - Search and browse products
Price Trends - Historical price analysis, discounts
Platform Comparison - Cross-platform analytics

🧪 Testing

# Run all tests
make test

# Run with coverage
pytest --cov=scrapers --cov=processors tests/

📝 Make Commands

make setup          # Create virtual environment
make install        # Install dependencies
make db-init        # Initialize database
make db-seed        # Seed sample data
make scrape-all     # Run all scrapers
make process        # Run data processing
make dashboard      # Start Streamlit
make docker-up      # Start Docker stack
make docker-down    # Stop Docker stack
make test           # Run tests
make clean          # Clean cache files

🔐 Environment Variables

# Database
DATABASE_URL=postgresql://user:pass@host:5432/db
POSTGRES_USER=sales_user
POSTGRES_PASSWORD=sales_password

# Airflow
AIRFLOW_USER=admin
AIRFLOW_PASSWORD=admin

# Scraper
SCRAPER_USER_AGENT=Mozilla/5.0...
SCRAPER_DELAY=2

🚧 Roadmap

Add more e-commerce platforms (Lazada, Alibaba)
Implement sentiment analysis on reviews
Add price alert notifications
Deploy to cloud (AWS/GCP)
Add real-time streaming with Kafka

📄 License

MIT License - feel free to use for your portfolio!

Built with ❤️ for data engineering portfolios

sofaquitegud/omnichannel-sales-engine