Imagine a tester's worst nightmare – and your team's too: users flood support saying pushes aren't arriving, messages aren't getting through, and you can't reproduce the problem. You don't even know how to reproduce it or what push delivery actually depends on.

After scouring resources on testing, I couldn't find a comprehensive guide that provides enough knowledge to truly understand how pushes and notifications work, how to test them, and, importantly, how to investigate and troubleshoot problems when pushes fail to reach users.

I'm Arman and I've been testing mobile apps for the last 5 years. Today I want to tell you about pushes 🌚

❗️ Please note: Some parts of this material will focus specifically on pushes in Android. However, the overall theme of the article should be useful for testing and development on any platform. Some sections should also be helpful for developers looking for solutions related to pushes.

Testing pushes and notifications in apps where they play a key role – like social networks, messengers, banking apps, e-commerce stores, and even games – has become a crucial part of the quality assurance process.

Fast and accurate notification delivery is critical for effectively conveying important messages and encouraging users to return to the app regularly. It's also worth noting that push notifications can replace costly SMS messages, be used for promotions and special offers, and even transmit invisible configuration messages to the app.

Testing helps identify problems with delivery, display, and interaction with notifications, ensuring a flawless user experience.

So, what do you need to know and understand to test pushes and notifications successfully? I'll try to answer that question today by covering the following topics in this article:

What's a Push vs. a Notification, and Why Distinguish Them?

To write proper test scenarios and diagnose issues with pushes and notifications, you need to get the terminology right and understand the difference between a push and a notification. Many mistakenly think they're the same thing, but while related, they can exist independently.

It's important here: I'm not just splitting hairs if someone calls everything "pushes" but understands the difference – that's totally fine. Call it whatever you like, as long as the distinction is clear.

Notification: This is the pop-up banner, the sound, the badge icon – a UI element designed to grab the user's attention. To show a notification, the app doesn't necessarily need anything external. It can display information without an internet connection, remind you about an event, and so on. The trigger for showing a notification can be anything: internal app logic, a timer, or even a push message!

Push: A push is a message! A message in a specific format that needs to be processed on the phone. It arrives via a special intermediary – a push notification service – over the internet. Usually, this service is provided by the OS manufacturer (APNS - Apple, FCM - Google, HMS - Huawei), but alternative push providers can also be used. As a rule, pushes are aimed at showing you a notification, but that's not mandatory. Pushes can be non-displayable (silent) and might be needed by the app simply to update some internal information.

In other words, a push can trigger a notification, but it doesn't have to. A notification can be a push notification, or it can be local (or internal, implemented via the app's logic, bypassing the push server).

Left - notification, Right - push, don't confuse them
Left - notification, Right - push, don't confuse them

The Notification is what we see on the screen, while the Push is the message (often JSON) sent from the push server to the user's device.

Understanding this distinction is crucial because it allows us to correctly diagnose why a particular notification isn't showing up. Also, knowing the specifics and differences helps us create test cases covering both notifications and pushes, as well as their interaction, clearly understanding what's being checked at each stage.

For example, if we know a certain type of notification in our app doesn't use pushes and only appears when the app is opened (interacting directly with our API), we won't bother checking push subscriptions or related parameters.

Conversely, if a feature uses the push mechanism not for displaying notifications but, say, for updating data, we'll immediately know where to look for potential issues.

How Pushes Work

Understanding how pushes work is already 80% of the success in testing them, writing test scenarios, or identifying problems.

As mentioned earlier, pushes have an intermediary, a push service – typically provided by the device or OS manufacturer (Apple Push Notification Service (APNS) – Apple, Firebase Cloud Messaging (FCM) – Google, Huawei Mobile Services (HMS) – Huawei), but could also be a third-party service. Many of these can act as intermediaries for others; for example, FCM and HMS can also send pushes via APNS Such a setup can provide a single entry point on the backend.

Previously, there was also Mi Push services – Xiaomi, but according to recent company statements, they no longer operate outside China. Alternatives like Pushy also exist. I haven't dug deep into their workings, but it seems they either act as a layer between the client and Firebase or maintain their own persistent connection to their server, which could make your app more battery-hungry.

A push service is essentially a cloud-to-device system. On the device, the service maintains a persistent connection to its own cloud and has permission to wake up the app or display a push on its behalf. Developers can use this service for faster and more reliable message delivery because it has high priority within the system.

How does it work?

  1. When the app launches or the user logs in, the app registers with the push notification system and receives a token specific to that user's device.
  2. This token is sent to your server, where it's stored and associated with that user.
  3. Whenever a message needs to be sent to this user (e.g., they received a message), the server takes the user's token and the message content and sends it to the push service's cloud.
  4. The cloud, in turn, uses the token to send the push to the device, where the service receives it and decides what to do (display, delay, or wake up your app to handle the push).

Push Notification Flow Diagram

Besides targeted messages, there are also topic-based (mass) messages. In this case, the app subscribes to a specific message type (topic). Messages sent to that topic are automatically delivered to subscribed devices, and your backend only needs to send messages to the specific topic. See documentation for FCM (Android) and documentation for APNS (works starting iOS 18).

Types of Pushes and Notifications in Your App

Now that we can distinguish pushes from notifications and understand how pushes work, let's figure out the different types of pushes, their differences, and, most importantly, how they change the behavior of your app and the notifications it shows.

❗️ I'll be discussing pushes primarily in the context of FCM with Android. FCM also supports sending pushes to iOS via APNS. The core logic is similar across most push services, but it's recommended to review the specific documentation for the service you use before testing.

Pushes come in several types:

  • Notification Messages (Basic) – These push messages contain a notification field where you can specify only title, body, and image. They are displayed automatically by the system when the app is in the background or closed. If the app is in the foreground, nothing is shown automatically. 
{
       "message": {
         "token": "bk3RNwTe3H0:CI2k_HHwgIpoDKCIZvv...",
         "notification": {
           "title": "Breaking News",
           "body": "Something exciting just happened!",
           "image": "https://example.com/news-image.jpg"
         }
       }
     }
  • Data Messages – These pushes contain only a data field, where you can add custom key-value pairs, without a notification field. When such a message is received, the system wakes up your app to process it. Nothing is displayed automatically. The push is delivered to the app regardless of its state, and the app itself decides how to react – it might show a notification, or it might not.

A feature many don't know about: Such a push will arrive and wake the app even if the user has denied permission to show notifications (because, as mentioned, a push isn't the same as a notification).

{
     "message":{
       "token":"bk3RNwTe3H0:CI2k_HHwgIpoDKCI...",
       "data":{
         "name" : "Arman",
         "body" : "This push will never be shown automatically",
         "silent" : "true",
         "anything" : "Whatever you want",
         ...
       }
     }
    }
  • Hybrid Messages (Notification + Data) – These contain both notification and data fields. The behavior depends on the app's state when the push is received:
    • App in background or closed: The notification is rendered automatically by the system, using only the data from the notification field. The data payload is passed to the app only if the user taps on the notification.
    • App in foreground: FCM delivers the entire payload (both notification and data) directly to the app and does not display anything automatically.

Links to Service Documentation:

Types of Notifications:

While focusing on pushes, it would be wrong not to mention the types of notifications and the logic they might involve:

  • Push Notifications: What we've been discussing above. Triggered by a push message.
  • Local Notifications: The app itself initiates showing the notification based on its internal logic, without contacting the backend.
  • Custom Notification Logic: Could be considered a subtype of local logic. Examples below. Here, the app might or might not contact the backend:
    • Instant display of notifications when the app is active.
    • Displaying notifications during system-scheduled maintenance windows when apps are woken up (the app might contact the backend here).
    • Scheduled notification activity using system services like WorkManager or AlarmManager on Android.
  • Persistent Notifications: A notification that stays visible, often used to keep a connection alive with your backend or inform about ongoing app activity (like music playing or navigation).

If you're unsure which technology, notification type, or push type your app uses, clarify with the developers and document it in your knowledge base.

Understanding the app's logic – not just in the context of pushes – is key to creating effective test checks and finding the root causes of problems.

What Affects Push Delivery

If the system is so cool and well-thought-out, why do problems arise? What actually influences pushes, and why might yours not reach the device?

First and foremost: push services do not guarantee that your push will be delivered! And they certainly don't guarantee it will be delivered on time (even if the user's device is on and connected).

There are several main reasons why a push might not reach the device:

  • Push Priority in FCM: You set this yourself, but FCM reserves the right to lower it. There are two priorities:
    • Normal: Messages that can be delayed (e.g., non-urgent updates).
    • High: Messages that should be delivered as quickly as possible (e.g., chat messages, alerts).
  • Push TTL (Time-To-Live): A parameter defining how long the push is considered relevant and worth delivering. Can range from 0 to 2,419,200 seconds (28 days).
    • If it's too short, the push might simply not have enough time to reach the user due to various reasons (lowered priority, user offline, battery saving, etc.).
    • A value of 0 means the push is only relevant right now (e.g., for a call notification). FCM will try to deliver it immediately, but if it fails, the push is discarded and will never arrive.
  • Non-collapsible Pushes:
    • ℹ️ Collapsible notifications are those that can replace a previous, similar pending notification. It means if the previous one hasn't been delivered yet, it can be dropped, and only the latest one sent.
    • If there are more than 100 pending non-collapsible push messages queued for an app, older ones are discarded, and a special callback is sent to the app (which you need to handle, especially if this scenario is possible for your app).
  • VPN: According to official FCM documentation, VPNs significantly complicate the service's operation. They can break the persistent connection to the server and hide information FCM uses to optimize the connection.
  • FCM Policies: Following its own algorithms and rules, FCM might lower priority independently if it deems your pushes aren't valuable to the user. The FCM concept is that high-priority pushes should result in a user-visible notification. If your high-priority pushes frequently don't lead to notifications (e.g., a large number of silent data pushes), their priority might be reduced. Then, genuinely important pushes might also fail to deliver promptly. Priority also gets lowered if you send high-priority pushes when notifications are disabled for your app.
  • Push Service Limits: For example, FCM has limits like 240 messages/minute and 5,000/hour per device. There's also a project-wide limit ( 600,000 messages/minute, but you can request an increase if needed).
  • Issues with Your Backend and App: Examples include sending an incorrect request to the push service, using an expired (stale) token, or the backend failing to receive the token from the user's app in the first place.

If your users are experiencing issues with receiving push notifications from the push service, these parameters are among the first things to investigate. How to figure out if pushes aren't arriving will be covered below, but first:

Why Push Delivery Doesn't Guarantee Notification Display

After your push successfully reaches the user's phone, a new player enters the game: the device's operating system. Just like the push service, the OS wants to conserve battery life as much as possible, meaning it will actively try to delay waking up the app to process the push and display notifications.

Let's break down the reasons why a push, even after reaching the device, might still not result in a visible notification. ❗️ This section focuses on Android. While many principles are similar across platforms, it's recommended to consult the documentation for the specific platform you're investigating.

  • There is also a priority system here, but for notifications. Android uses Notification Channels, each with its own importance level (priority) that you define. The system uses this to understand how much a certain type of notification should disturb the user.
    • Specific behavior isn't guaranteed. The system considers the priority, user behavior, and other factors to decide when and how (or if) to show the notification.
  • Notification Channel Issues: Errors during channel initialization in your code, or the user manually disabling specific channels in the settings, can prevent notifications from appearing.
  • App Standby and Doze Mode: System features designed to save battery. Only high-priority pushes might wake the app from these deep sleep states, and as we remember, FCM might have lowered your push's priority. If that happens, a backgrounded app will only get brief maintenance windows to perform background tasks, including showing notifications.
    • Some device manufacturers add their own aggressive battery optimization tools that further restrict background activity and notification display.
  • Blocked Ports: Problems with the network ports FCM uses to communicate with the phone can affect notification display. FCM typically uses ports 5228-5230 but can fall back to the standard HTTPS port 443. Firewalls or network configurations might block these.
  • App Force-Stopped: If the user manually force-stops the app via system settings, the OS interprets this as the user explicitly wanting the app to remain inactive, preventing it from starting automatically or showing any notifications until manually launched again.
  • Automatic Permission Revocation: The system might automatically revoke the notification permission if the app hasn't been used for a long time.
  • App Crashes or ANRs: Your app might crash or encounter an "Application Not Responding" (ANR) error while trying to process the push payload, for example, if the data sent by your backend is invalid or unexpected by the app's handling logic.

Before diving deep into why a push isn't reaching the user, always make sure the basics are covered: confirm the user hasn't disabled notifications globally or for specific channels, and ensure that system battery optimizations that might affect background delivery are disabled for your app (or that your app is added to the exclusion list).

There's a helpful website: Don't kill my app! which provides instructions on how to disable various battery optimizations on different devices and OS versions/skins.

How to Know If Pushes Aren't Arriving

To tackle a problem, you first need to know it exists. Yes, you could wait for users to start complaining en masse, but it's far more effective to be proactive and catch issues early. Statistics and monitoring (watchdogs) are your friends here.

Having monitoring systems and statistics allows you to react faster to emerging problems and analyze the effectiveness of push delivery, thereby improving the user experience.

There are two main sources for statistics:

  1. Firebase Console Statistics: Comes with pros and cons. It offers a convenient, aggregated format where you can spot problems on graphs. It also allows exporting data to BigQuery for detailed analysis. However, it can't provide information on notification opens, not all events might be recorded (e.g., if the user opted out of analytics collection), and analyzing specific individual delivery issues can be difficult.

  2. Your Own Analytics System: This involves your app reporting back to your backend when a push is received (and potentially when a notification is shown or opened).

    • Pros: Allows for detailed, custom statistics. Enables end-to-end tracing from server to device. Can be tailored to specific business requirements (e.g., track all notification types, not just pushes; calculate overall message delivery effectiveness regardless of source).
    • Cons: More complex to implement. Won't capture 'notification-only' pushes if they aren't opened (app won't be woken up). You still might not know for sure if a notification was displayed by the system even if the push was received by the app.

Choosing the right solution depends on your requirements and capabilities. If you primarily use data messages or have complex notification logic beyond simple pushes, custom analytics might be necessary. If your needs are basic, Firebase stats may be enough.

What You Can Do to Rescue Your Pushes

We've discussed the problems that can arise with push delivery. Some can be mitigated by following service recommendations:

  • Don't abuse high priorities.
  • Increase TTL if it's too short and causing issues.
  • Use collapsible messages where appropriate.

Others require deep analysis of your app and backend logic, guided by detailed delivery statistics and logs.

However, many factors remain outside our direct control. What do you do if Google or Apple suddenly decides to disrupt push delivery for your app?

This is where backup solutions come in:

  • Implement a Fallback Mechanism: When the app is active (or periodically woken up), have it directly poll your API to fetch pending messages/notifications and display them locally. This bypasses the push service entirely for active users or during sync periods.
  • Use Alternative Push Providers: Consider integrating with additional services (like HMS for Huawei devices) as backups or primary channels for specific device segments.
  • Implement Message Recovery: When the app receives a new push (or becomes active), have it contact your backend to check for any messages that might have been missed previously and display them now. This helps recover notifications lost due to temporary delivery issues.

Conclusion

And so, our fascinating journey into the world of pushes comes to an end. What are the key takeaways and decisions to make if pushes are important to you?

  • Distinguish Pushes and Notifications: Don't confuse them, and tailor your tests based on what you're actually verifying.
  • Understand the Rules: Learn how push services and OS delivery mechanisms work. This makes troubleshooting easier and helps optimize delivery effectiveness.
  • Check Your Settings: Push parameters like TTL and priority significantly impact delivery. Review your current settings and ensure they are justified.
  • Prepare for Surprises: Implement fallback mechanisms and consider multiple delivery paths (alternative providers, fallbacks) to increase resilience.
  • Monitor and Analyze: Use statistics and monitoring (watchdogs) to quickly detect and react to delivery problems.

I hope something I've shared today was useful, and that pushes have become a bit clearer and more transparent for you, despite their inherent complexity. If you have questions or spot any errors, please let me know in the comments. To connect or if you'd like to ask me something directly, you can find me on LinkedIn.