AI Computer Interaction Logger 🖥️🤖

Generate comprehensive datasets for training multi-modal LLMs in autonomous computer control

📋 Table of Contents

About
Features
Installation
Usage
Data Structure
Privacy and Security
Troubleshooting
Contributing
Roadmap
License
Acknowledgments

🔍 About

AI Computer Interaction Logger is a powerful tool designed to capture and log human-computer interactions, creating rich datasets for training multi-modal Language Learning Models (LLMs). By recording various aspects of user interactions, including screen content, mouse movements, keyboard inputs, and audio, this tool enables the development of AI systems capable of understanding and replicating complex computer operations.

Our goal is to provide researchers and developers with high-quality, diverse datasets that can be used to train AI models for tasks such as:

Automated software testing
User experience analysis
Assistive technologies for computer usage
AI-driven task automation

✨ Features

🖼️ High-frequency screenshot capture
- Configurable capture rate (default: 10 fps)
- Supports multiple monitors
- Images saved in PNG format for high quality and compression
🖱️ Precise mouse movement and click logging
- Tracks mouse coordinates (x, y)
- Records left, right, and middle button clicks
- Captures scroll wheel movements
⌨️ Keyboard input recording
- Logs all key presses and releases
- Supports special keys and modifiers (Ctrl, Alt, Shift, etc.)
- Option to mask sensitive data (e.g., passwords)
🎤 Audio environment capture
- Records system audio and microphone input
- Configurable sample rate and bit depth
💾 Efficient data storage and organization
- Structured file hierarchy for easy navigation
- Compressed storage formats to minimize disk usage
🔄 Real-time processing and logging
- Minimal impact on system performance
- Live monitoring of recording status
📊 Structured output for easy ML model ingestion
- CSV logs for events and metadata
- Synchronized timestamps across all data types

🚀 Installation

System Requirements

Python 3.8 or higher
4GB RAM (minimum)
1GB free disk space for the application
Additional disk space for recorded data (varies based on recording duration and quality)

Steps

Clone the repository:

git clone https://github.com/yourusername/AIComputerInteractionLogger.git

Navigate to the project directory:
```
cd AIComputerInteractionLogger
```
Install the required dependencies:
```
pip install -r requirements.txt
```

🖥️ Usage

Basic Usage

To start recording computer interactions:

from src.recorder import DatasetRecorder

recorder = DatasetRecorder(screenshot_freq=5)
recorder.start_recording(duration=60)  # Record for 60 seconds

Advanced Configuration

You can customize various aspects of the recording process:

recorder = DatasetRecorder(
    base_output_dir="custom_dataset",
    screenshot_freq=10,
    audio_channels=2,
    audio_samplerate=48000
)
recorder.start_recording(duration=300)  # Record for 5 minutes

For more detailed usage instructions, please refer to our Usage Guide.

📁 Data Structure

Recorded data is organized in the following structure:

dataset/
└── session_YYYYMMDD_HHMMSS/
    ├── events.csv
    ├── audio.wav
    └── screenshots/
        ├── screenshot_timestamp1.png
        ├── screenshot_timestamp2.png
        └── ...

events.csv: Contains timestamped logs of mouse and keyboard events
audio.wav: Audio recording of the session
screenshots/: Directory containing all captured screenshots

🔒 Privacy and Security

All data is stored locally on your machine
No data is transmitted over the network
Consider implementing additional encryption for sensitive data
Be cautious when recording in environments with confidential information

🛠 Troubleshooting

Common issues and their solutions:

Recording not starting: Ensure you have the necessary permissions for screen capture and audio recording.
High CPU usage: Try lowering the screenshot frequency or reducing the number of monitored events.
Missing events in CSV: Check if any antivirus software is blocking the event hooks.

For more troubleshooting tips, see our FAQ.