Skip to content

Commit 5fcae4f

Browse files
Rohan-flutterintRohan-flutterint
committed
Initial commit
0 parents  commit 5fcae4f

File tree

5,043 files changed

+699952
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

5,043 files changed

+699952
-0
lines changed

‎.gitattributes‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Auto detect text files and perform LF normalization
2+
* text=auto

‎.gitignore‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Generated by Cargo
2+
# will have compiled files and executables
3+
debug/
4+
target/
5+
6+
# These are backup files generated by rustfmt
7+
**/*.rs.bk
8+
9+
# MSVC Windows builds of rustc generate these, which store debugging information
10+
*.pdb
11+
12+
# RustRover
13+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
14+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
15+
# and can be added to the global gitignore or merged into this file. For a more nuclear
16+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
17+
#.idea/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2025 rohan
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# FlowCore — Streaming Engine
2+
3+
This project is a **minimal** Flink-like streaming MVP implemented in **Rust** for the backend and **React (Vite)** for the frontend.
4+
It demonstrates core streaming concepts:
5+
- Tumbling windows (10s)
6+
- Watermark computation (max event timestamp minus allowed lateness)
7+
- Checkpointing to disk (periodic JSON checkpoint)
8+
- Late-event detection and expose via API
9+
- Simple web UI to view emitted window results and late events
10+
- Built-in event generator (for demo)
11+
12+
## Structure
13+
14+
```
15+
FlowCore/
16+
backend/ # Rust backend (actix-web)
17+
frontend/ # React (Vite) frontend
18+
README.md
19+
```
20+
21+
## Quick start (dev)
22+
23+
### Backend
24+
Requirements:
25+
- Rust toolchain (rustup + cargo)
26+
27+
From `backend/`:
28+
29+
```bash
30+
cd backend
31+
cargo run
32+
```
33+
34+
This will start the server on `http://127.0.0.1:8080`.
35+
36+
- `POST /ingest` accepts JSON events: `{ "id": "...", "ts": 169..., "value": 12.34 }`
37+
- `GET /recent` returns emitted window results (newline-delimited JSON)
38+
- `GET /late` returns recent late events JSON array
39+
40+
Checkpoints are written to `/tmp/flink_rust_mvp_ckpt/checkpoint.json` and emitted window lines to `/tmp/flink_rust_mvp_out/results.log`.
41+
42+
### Frontend
43+
Requirements:
44+
- Node 18+ and npm
45+
46+
From `frontend/`:
47+
48+
```bash
49+
cd frontend
50+
npm install
51+
npm run dev
52+
```
53+
54+
Open the browser at the address shown by Vite (typically `http://localhost:5173`). The frontend proxies calls to the backend when both run on localhost (the UI expects the backend at `/`).
55+
56+
## Notes and limitations
57+
58+
This is a minimal educational MVP, **not** a production-grade replacement for Flink.
59+
What it intentionally keeps simple:
60+
- No distributed runtime (single-process)
61+
- Simple file-based checkpoint + append-only logs
62+
- No fault recovery orchestration
63+
- No exactly-once delivery guarantees
64+
65+
### How it demonstrates Flink features:
66+
- **Watermarks**: computed as `max_event_ts - allowed_lateness_ms`. Windows whose end <= watermark are closed and emitted.
67+
- **Late events**: Events that arrive with timestamp older than watermark + lateness are visible via `/late`.
68+
- **Checkpointing**: current in-memory windows + watermark serialized periodically to `/tmp/.../checkpoint.json`.
69+
70+
## UI
71+
72+
![alt text](image.png)
73+
74+
## Next steps
75+
- Add Dockerfiles + docker-compose for one-command run
76+
- Add persistent storage via RocksDB or sled for state
77+
- Add a proper WebSocket broadcast of results (instead of polling)
78+
- Harden checkpointing + simulated recovery using checkpoint file
79+
- Add support for keyed windows and sliding windows

0 commit comments

Comments
 (0)