Skip to main content

Data Flow

Understanding how data flows through the Bellamy Book system. For the big picture (request path, events, workers, data stores), see Understanding the Architecture.

Request Flow​

User Creates a Post​

sequenceDiagram
    participant User
    participant Frontend
    participant API as Backend API
    participant DB as PostgreSQL
    participant Kafka
    participant Worker as Background Workers
    participant WS as WebSocket
    
    User->>Frontend: Create Post
    Frontend->>API: POST /api/Post/users/{userId} (JWT)
    API->>DB: Save Post
    DB-->>API: Post Created
    API->>Kafka: Publish PostCreated Event
    API-->>Frontend: 201 Created
    Frontend-->>User: Post Appears in Feed
    
    Kafka->>Worker: Consume Event
    Worker->>DB: Update Scores/Trends
    Worker->>WS: Notify Friends
    WS-->>Frontend: Real-time Update

Event Flow​

Post Created Event​

graph TB
    API[Backend API] -->|Publish Event| KAFKA[Kafka Topic: post.created]
    
    KAFKA --> SCORE[ScoringWorker<br/>Calculate Relevance]
    KAFKA --> GRAPH[GraphWorker<br/>Update Social Graph]
    KAFKA --> NOTIFY[NotificationWorker<br/>Notify Friends]
    KAFKA --> TREND[TrendingWorker<br/>Update Trends]
    
    SCORE --> PG1[(PostgreSQL<br/>ContentScores)]
    GRAPH --> NEO4J[(Neo4j<br/>Relationship Weights)]
    NOTIFY --> MONGO1[(MongoDB<br/>Notifications)]
    TREND --> PG2[(PostgreSQL<br/>GlobalTrends)]
    
    SCORE -->|May Trigger| KAFKA2[New Events]
    GRAPH -->|May Trigger| KAFKA2

Real-time Updates​

WebSocket Flow​

sequenceDiagram
    participant Client
    participant WS as WebSocket Server
    participant Redis
    participant Worker as Event Worker
    participant DB as Database
    
    Client->>WS: Connect (with JWT)
    WS->>Redis: Store Connection
    WS-->>Client: Connected
    
    Note over DB,Worker: Event Occurs
    DB->>Worker: Trigger Event
    Worker->>Redis: Get User Connections
    Redis-->>Worker: Connection List
    Worker->>WS: Send Update
    WS-->>Client: Real-time Notification
    Client->>Client: Update UI

Caching Flow​

Read Request with Cache​

graph TD
    REQ[API Request] --> CHECK{Check Redis Cache}
    CHECK -->|Cache Hit| HIT[Return Cached Data]
    CHECK -->|Cache Miss| QUERY[Query Database]
    QUERY --> STORE[Store in Redis Cache]
    STORE --> RETURN[Return Data]
    HIT --> RESPONSE[Response to Client]
    RETURN --> RESPONSE
    
    style HIT fill:#90EE90
    style QUERY fill:#FFB6C1

Data Synchronization​

Database Replication​

  • PostgreSQL: Master-slave replication
  • MongoDB: Replica sets
  • Redis: Master-replica setup

Eventual Consistency​

Some operations are eventually consistent:

  • Friend suggestions
  • Trending content
  • Analytics data

Next Steps​