Microservice

​

System Architecture

1. Express Server: Handles API requests for manually triggering account processing
2. Cron Job: Automatically processes all connected accounts at regular intervals

​

Technology Stack

​

Core Technologies

• Node.js (v18+): JavaScript runtime environment
• TypeScript (v5.5+): Typed superset of JavaScript for robust code
• Express.js (v4.19+): Fast, unopinionated web framework
• PM2 (v5+): Advanced production process manager with integrated load balancer

​

Web Scraping & Automation

• Puppeteer (v22+): Headless Chrome API for automated web interactions
• Puppeteer-core: Lightweight version with reduced footprint

​

Database & Storage

• Supabase: Open source Firebase alternative
• PostgreSQL: Advanced relational database for structured data storage

​

Queue & Task Management

• Better-Queue: Priority queue with concurrency control
• Multi-Queue System: Load balancing across parallel queues

​

Logging & Monitoring

• Winston: Versatile logging library with multiple transports
• Morgan: HTTP request logger middleware for Express

​

Data Flow Process

​

Implementation Details

​

Authentication Mechanism

// Authentication using storage data from accounts table
export async function getCreatorData(
  userId: string,
  accountId: string,
  username: string,
  userAgent: string,
  localStorage: Json | null,
  sessionStorage: Json | null,
  cookies: Json | null,
  isDebug: boolean = false
) {
  // Initialize browser with collected authentication data
  const browser = await puppeteer.launch({
    headless: !isDebug,
    args: [
      '--disable-extensions',
      '--disable-gpu',
      '--no-sandbox',
      '--disable-setuid-sandbox',
    ]
  });
  
  try {
    const page = await browser.newPage();
    await page.setUserAgent(userAgent);
    
    // Set cookies from stored data
    if (cookies) {
      for (const cookie of Object.values(cookies)) {
        await page.setCookie(cookie);
      }
    }
    
    // Navigate to OnlyFans and inject localStorage/sessionStorage
    await page.goto('https://onlyfans.com/');
    
    if (localStorage) {
      await page.evaluate((storageData) => {
        for (const [key, value] of Object.entries(storageData)) {
          window.localStorage.setItem(key, value as string);
        }
      }, localStorage);
    }
    
    if (sessionStorage) {
      await page.evaluate((storageData) => {
        for (const [key, value] of Object.entries(storageData)) {
          window.sessionStorage.setItem(key, value as string);
        }
      }, sessionStorage);
    }
    
    // Proceed with data extraction
    // ...
  } finally {
    await browser.close();
  }
}

​

Queue Management

const NUM_QUEUES = 3; // Default number of concurrent queues
const queues: Queue[] = [];
let currentQueueIndex = 0; // Round-robin queue selection

// Initialize multiple processing queues
for (let i = 0; i < NUM_QUEUES; i++) {
  const queue = new Queue(async function(accountData, cb) {
    logger.info(`Queue Function called in Queue ${i}`);
    
    getCreatorData(
      accountData.user_id,
      accountData.id.toString(),
      accountData.username,
      accountData.user_agent,
      accountData.web_local_storage ? accountData.web_local_storage : null,
      accountData.web_session_storage ? accountData.web_session_storage : null,
      accountData.web_cookies ? accountData.web_cookies : null,
      false
    )
      .then(() => {
        logger.info(`Finished processing account data for ${accountData.id} in Queue ${i}`);
        cb();
        return true;
      })
      .catch((error) => {
        console.error(
          'Error in account',
          accountData.name,
          accountData.username,
          error.message
        );
        cb(`Error in account: ${accountData.name}, ${accountData.username}, ${error.message}`);
      });
  });
  
  queues.push(queue);
  
  // Log queue statistics daily
  setInterval(
    () => logger.info(`Queue ${i} stats: ${queue.getStats()}`),
    24 * 60 * 60 * 1000
  );
}

​

API Endpoints

​

Environment Variables

# Core configuration
PORT=3001                           # Port for the Express server
NEXT_PUBLIC_SITE_URL=http://localhost:3000/  # URL of the web application

# Security
CRON_SECRET=your_cron_secret        # Secret for securing cron routes
API_SECRET=your_api_secret          # Secret for API authentication

# Supabase connection
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key

​

Development Environment Setup

1. Clone the repository
   Copy

   git clone https://github.com/OnlyAutomator/only-automator-microservice.git
   cd only-automator-microservice
   

2. Install dependencies
   Copy

   npm install
   

3. Configure environment variables Create a .env file with the required variables:
   Copy

   # Core configuration
   PORT=3001
   NEXT_PUBLIC_SITE_URL=http://localhost:3000/
   
   # Supabase connection
   SUPABASE_URL=https://qyzigfweeamoatcrnpgc.supabase.co
   SUPABASE_ANON_KEY=your_anon_key
   SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
   
   # Security
   CRON_SECRET=your_cron_secret
   API_SECRET=your_api_secret
   

4. Generate Supabase type definitions
   Copy

   npm run generate-types
   

5. Build the TypeScript code
   Copy

   npm run build
   

6. Start the service in development mode
   Copy

   npm run dev
   

​

Production Deployment

1. Build the application
   Copy

   npm run build
   

2. Configure PM2 The ecosystem.config.js file contains the PM2 configuration:
   Copy

   module.exports = {
     apps: [{
       name: 'express-server',
       script: 'build/app.js',
       instances: 1,
       autorestart: true,
       watch: false,
       env: {
         SUPABASE_URL: process.env.SUPABASE_URL,
         SUPABASE_ANON_KEY: process.env.SUPABASE_ANON_KEY,
         SUPABASE_SERVICE_ROLE_KEY: process.env.SUPABASE_SERVICE_ROLE_KEY,
         NEXT_PUBLIC_SITE_URL: process.env.NEXT_PUBLIC_SITE_URL,
         PORT: process.env.PORT,
         CRON_SECRET: process.env.CRON_SECRET,
         API_SECRET: process.env.API_SECRET,
       },
     },
     {
       name: 'cron-job',
       script: 'build/cron_job.js',
       cron_restart: '* * * * *', // restart every minute
       // cron_restart: '0 */6 * * *', // every 6 hours (production recommendation)
       env: {
         SUPABASE_URL: process.env.SUPABASE_URL,
         SUPABASE_ANON_KEY: process.env.SUPABASE_ANON_KEY,
         SUPABASE_SERVICE_ROLE_KEY: process.env.SUPABASE_SERVICE_ROLE_KEY,
         NEXT_PUBLIC_SITE_URL: process.env.NEXT_PUBLIC_SITE_URL,
         PORT: process.env.PORT,
         CRON_SECRET: process.env.CRON_SECRET,
         API_SECRET: process.env.API_SECRET,
       },
     }],
   };
   

3. Start the application with PM2
   Copy

   pm2 start ecosystem.config.js
   

4. Configure PM2 to start on system boot
   Copy

   pm2 startup
   pm2 save
   

5. Monitor the application
   Copy

   pm2 status    # Check process status
   pm2 logs      # View logs
   pm2 monit     # Monitor CPU and memory usage
   

​

Cron Job Configuration

{
  name: 'cron-job',
  script: 'build/cron_job.js',
  cron_restart: '* * * * *', // Every minute (for testing)
  // cron_restart: '0 */6 * * *', // Every 6 hours (recommended for production)
}

* * * * *
│ │ │ │ │
│ │ │ │ └─── day of week (0-7, where 0 and 7 are Sunday)
│ │ │ └───── month (1-12)
│ │ └─────── day of month (1-31)
│ └───────── hour (0-23)
└─────────── minute (0-59)

• */30 * * * *: Every 30 minutes
• 0 */6 * * *: Every 6 hours
• 0 0 * * *: Once a day at midnight
• 0 0 * * 0: Once a week on Sunday

​

Scaling Considerations

​

Vertical Scaling

• Increase server resources (CPU/RAM) to handle more concurrent processes
• Adjust Node.js memory limits with --max-old-space-size flag:
  Copy

  NODE_OPTIONS="--max-old-space-size=4096" npm run start
  

​

Horizontal Scaling

• Deploy multiple instances behind a load balancer
• Use PM2 cluster mode for the Express server:
  Copy

  {
    name: 'express-server',
    script: 'build/app.js',
    instances: 'max', // Use maximum available CPUs
    exec_mode: 'cluster',
  }
  

​

Queue Optimization

• Adjust the number of concurrent queues based on server capacity:
  Copy

  const NUM_QUEUES = 3; // Default is 3, increase for higher concurrency
  

• Implement priority queueing for critical accounts:
  Copy

  const queue = new Queue({
    priority: function(task, cb) {
      // Determine task priority (lower number = higher priority)
      if (task.isPremium) return cb(null, 1);
      return cb(null, 10);
    }
  });
  

​

Data Extraction Process

1. User Statistics
   • Subscriber count and growth rate
   • Revenue metrics
   • Content engagement metrics
2. Engagement Metrics
   • Message response rates
   • Fan interaction frequency
   • Content popularity analytics
3. Fan Data
   • Active subscribers list
   • Spending patterns
   • Engagement behavior

// Example of statistics extraction logic
async function extractStatisticsData(page) {
  await page.goto('https://onlyfans.com/my/statistics/');
  await page.waitForSelector('.stats-card');
  
  return await page.evaluate(() => {
    const stats = {};
    
    // Extract subscriber metrics
    const subCount = document.querySelector('.subscribers-count');
    if (subCount) stats.subscriberCount = parseInt(subCount.textContent.trim(), 10);
    
    // Extract revenue metrics
    const revenue = document.querySelector('.revenue-total');
    if (revenue) stats.totalRevenue = parseFloat(revenue.textContent.replace(/[^0-9.]/g, ''));
    
    // Extract other metrics...
    
    return stats;
  });
}

​

Troubleshooting

// Set Winston to debug level
const logger = winston.createLogger({
  level: 'debug', // Change from 'info' to 'debug'
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.printf(({ level, message, timestamp }) => {
      return `${timestamp} ${level}: ${message}`;
    })
  ),
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({ filename: 'logs/error.log', level: 'error' }),
    new winston.transports.File({ filename: 'logs/combined.log' })
  ]
});

​

Security Best Practices

• Environment Variables: All sensitive credentials stored in environment variables
• API Authentication: API endpoints secured with API_SECRET token validation
• CORS Policy: Configured for specific origins in production:
  Copy

  app.use(cors({
    origin: process.env.NODE_ENV === 'production' 
      ? [process.env.NEXT_PUBLIC_SITE_URL] 
      : '*',
    methods: ['GET', 'POST', 'OPTIONS'],
    credentials: true,
    allowedHeaders: ['Content-Type', 'Authorization']
  }));
  

• Helmet.js: Secures Express app with various HTTP headers:
  Copy

  app.use(helmet({
    crossOriginResourcePolicy: { policy: "cross-origin" },
    contentSecurityPolicy: false
  }));
  

• Input Validation: All API inputs validated before processing
• Regular Updates: Dependencies kept updated to patch security vulnerabilities

​

Integration with Other Components

• Web Application: Displays data collected by the microservice through Supabase real-time subscriptions
• Chrome Extension: Provides authentication data (cookies, localStorage, sessionStorage) for the microservice
• Supabase Database: Stores all processed data and enables real-time updates through PostgreSQL change data capture