Face recognition attendance systems are becoming more common in large enterprises. But integrating with them—especially when you're dealing with thousands of employees and dozens of devices—is far from straightforward. In this post, I’ll walk through how I built a scalable middleware using Node.js and MQTT to interface with smart attendance devices, handling real-time syncing, deletions, and multi-device management.
🧩 The Problem
We were working with face recognition devices that communicate via MQTT. These devices have their own set of protocols for:
Real-time recognition data uploads
Batch user syncing
User deletion
Intruder detection
Device metadata management
Our goal was to:
Sync thousands of employee records across devices.
Handle new employee additions, deletions, and updates efficiently.
Receive and process real-time attendance events.
Scale with increasing devices and users.
🔧 The Tech Stack
Node.js for building the middleware service.
MQTT.js to manage communication with devices.
MySQL for persistent storage of employee and device metadata.
Redis for job queues and temporary state tracking.
Docker + CI/CD for smooth deployments.
🏗 Architecture Overview
plaintext
Copy
Edit
Employee DB/API ↔ Middleware ↔ MQTT Broker ↔ Devices
↕
MySQL, Redis
Key Components:
mqtt.js: Handles connection, reconnection logic, message parsing, and publishing.
syncManager.js: Manages batch sync of employee data to devices.
deletionManager.js: Handles scheduled and manual deletion jobs.
eventHandler.js: Parses and logs attendance and intruder data sent from devices.
🧠 Challenges Faced
- Device Acknowledgments Each device expects an acknowledgment after every message. Missing an ACK could cause it to retry or drop the operation.
Solution: Created a robust messageDispatcher that tracks correlation IDs and ensures timely ACKs with retry logic.
- Bulk Employee Syncing Some devices had memory limits, so syncing 10k users at once caused crashes.
Solution: Introduced batching and throttling using a queue-based worker system:
js
Copy
Edit
const batchSize = 100;
const delay = 200; // ms
Each batch was sent with a delay, and progress was tracked via Redis.
- MQTT Reconnection Logic MQTT connections sometimes dropped silently.
Solution: Added:
Heartbeat topics
Reconnection backoff
On-demand manual reconnect trigger via API
- Multi-device Management Some employees needed to be synced only to specific devices.
Solution: Maintained a many-to-many relationship between employees and devices in MySQL, and dynamically selected the devices for each operation.
🛠 Sample Code Snippet
js
Copy
Edit
mqttClient.on('message', async (topic, message) => {
const parsed = JSON.parse(message.toString());
if (topic.includes('attendance')) {
await handleAttendanceLog(parsed);
} else if (topic.includes('ack')) {
ackTracker.resolve(parsed.requestId, parsed.status);
}
});
📈 Performance
After optimizing:
50+ devices could be managed concurrently.
20,000+ employee records synced without timeouts or failures.
Real-time event processing latency: < 500ms
🧪 Testing Tips
Use an MQTT simulator to mimic device behavior.
Validate ACK timeouts and retry logic with offline devices.
Monitor logs with correlation IDs for each operation.
🚀 Final Thoughts
This project taught me a lot about building real-time, fault-tolerant systems in Node.js. MQTT is powerful but tricky—it requires strict protocol adherence, retry handling, and thoughtful scaling strategies.
Have you worked with IoT devices or MQTT in production? I’d love to hear your experiences!
💬 Let’s Connect:
You can find me on GitHub [@himanshumishir] or check out my portfolio [https://himanshumishir.in].
