There’s going to be more polish and behind-the-scenes improvements in this version. And hopefully no new regressions! One can dream.
In my continued aspirations to limit the amount of sysadmin work and stress I shoulder, I’ve been casually (and sometimes less casually) searching for a distributed database alternative to BedrockDB. Penguin (the API server) is a Mac app, and it runs on a small cluster of Mac minis. My ideal solution is a database that runs on each API server, but Bedrock dropped macOS support a long while back. FoundationDB is an interesting (if a bit complicated) option, but it warns that macOS support is not for production use.
What then? Well, the Streamie database structure is largely just a key-value store (accessed with SQL) with a little bit of sorting and paging sprinkled in. Looking at database-as-service options, Cloudflare has their Workers KV product. I was dusting off my TypeScript experience, trying to piece together a minimal API sufficient for Penguin to use as a Bedrock alternative when I saw their new(-ish) D1 service, which is (under the hood) SQLite.
They’ve got a pretty intense 100MB database size limit. I pruned the production database down to just my personal account records, used Wrangler to import it to D1, wrote a simple execute-this-sql-and-return-some-json TypeScript function, plugged it all together and … it all worked perfectly.
If D1 pans out for Streamie, then I should be able to fully decommission the Linux servers running Bedrock, which’ll be great for the ambient noise level in Streamie HQ. Although, in the winter, the heat coming off those machines is really nice.
Years ago, even before I was able to work on Streamie full time, T.M. contributed to the engineering efforts of the app, letting me mostly focus on Penguin. App progress was made on a daily basis even while I was working my full time job, which was an amazing first for me. Following a long hiatus, she has re-joined the team!
And talk about geographically diverse: US, India and Turkey.
Recently I started a live stream and it died after 13 hours. I hadn’t enabled logging on that device, so I didn’t learn anything. I enabled logging and restarted the live stream. It finally died after just over 150 hours. Reviewing the logs, I saw that Streamie had reconnected to the camera and when it did so, the presentation times had rolled over. Interestingly, I have code in place for handling that situation for video presentation times, but I hadn’t duplicated that for audio.
Why the reconnect though? Reviewing the logs, connectivity to the camera was lost at 00:10:05. Periodic reconnect attempts continued until 00:12:05 (exactly two minutes??). During that same time period, Streamie was unable to talk to the API server, so this wasn’t a camera issue. The API connection attempt failed with: POSIXErrorCode(rawValue: 50): Network is down. The Apple TV on which this test ran is wired into a switch. An internet failure? Router failure? Switch failure? Hmm.
The point here isn’t to eliminate every way in which a stream can fail (certainly an impossible task), but to make Streamie resilient to failure in general, so that it’ll gracefully recover. In this case, the informed handling of a wrap-around presentation time is a pretty easy thing to account for. And as long as I don’t have to deal with the Int64 rolling over, then all should be well.
Back in v3.13.2 I fixed an ONVIF authentication issue that would occur when a camera’s clock was way off. While I was calculating the time delta to be used during authorization, I wasn’t actually using that delta. Whoops. Anyway, I got that fixed.
Fast forward to yesterday and a long-time user gets in touch that he’s seeing this same error now, where it used to always work just fine. The camera’s clock is set correctly. I have him run some tests and submit some debug logs. I gave up last night and started again this morning.
When performing ONVIF authorization, one component of that authorization is the current time (with respect to what time the camera thinks it is), expressed in GMT. Prior to authorization, an ONVIF client asks the camera for its current time. The camera replies with two times: UTC and Local. It also includes a time zone and a Boolean indication for daylight saving time. An ONVIF client could take the local time, apply the time zone to it and then subtract DST (if applicable) to arrive at GMT and use that time when authenticating … or you could just use the UTC time value which camera also provides.
Unfortunately for this particular Lorex NVR, it seems like it actually calculates the UTC time from the local time setting, and then fails to apply DST. So, when Streamie used that value for authentication, it failed.
The unfortunate solution is to convert the local time to GMT and then compare it to the UTC time value. If they’re off by exactly one hour, then use the local time. Of course, that should never actually happen (except on this one NVR apparently).
I finally grew adequately annoyed at how long it took to load recordings off my slow NAS. Doing a bunch of random reads (to load each snapshot) in quite slow on a spinning disk storage system, particularly when accessed over a network. So….
Internally, this transitions from the old, janky mechanism for getting the camera stream data, to the new method where it simply registers as a sink for audio and video, like all of the other components (camera recording was the last thing remaining to be ported over). This would, for instance, result in a one-line change to record low quality video instead of original quality. Or to record audio / video in a specific codec instead of the source codec. This’ll be key in the next version in supporting custom qualities when recording to an S3-compatible storage service where a user doesn’t have adequate upstream internet bandwidth for the original quality.
Externally, the agonizingly slow thumbnail loading process is a thing of the past. Instead of the thumbnails being stored in the recording file interspersed among the other data, they’re accumulated and stored in the recording file appendix, so they can all be loaded in a single read. Additionally, snapshot size and quality has been reduced. In testing, the appendix for a ten minute recording with snapshots every fifteen seconds was under 150K.
I’m running these latest changes and I’ll produce another video that demonstrates the speed and the event filtering support.
- Started working on a new DataStore implementation for Cloudflare Workers + D1. This'll maybe replace our self-hosted BedrockDB one day. Maybe. Hopefully. My stress level would appreciate this change.
- Updates some of the various StreamingSession-related classes to not use DispatchQueue.global() (which I need to be more consistent with; it's just so handy sometimes), because it'll get "clogged up" and then everything stops working.
- Adds a "Reset Cache" option to the Events tab (iOS-only) that I'm using for testing D1. Rapidly scrolling through events is a great way to trigger rate limiting if anything is going to trigger it. It doesn't really serve any purpose for end users (other than letting them conveniently DOS the servers?).
- Adds keyboard support for controlling pan, tilt and zoom on iOS (and macOS). To pan / tilt use the arrow keys. To zoom, use the up / down arrow keys with the shift key.
- Updates the "downgrade an account" server code, specifically with regards to handling excess account members, to reference the "last seen" history and choose the least recently seen account members for removal from the account. I also updated the old school "unlimited_cameras" purchases to support up to five account members (instead of one). I figure that if you've been using Streamie for 3+ years, your continued use is appreciated.
- Fixes an issue where if you tap on a shared camera link, that shared camera would not automatically be added to your cameras list if you were not allowed to add additional cameras. This is incorrect behavior because shared cameras (much like attached cameras) don't count against your account camera limits, while Nest, ONVIF, RTSP, UniFi cameras _do_ count against your account camera limits. [TM]
- Adds on-device Markdown rendering support for in-line code (denoted with back-ticks). There was existing support for whole lines/paragraphs, but not for substrings in a line. Support for both forms was already in place for the Markdown HTML renderer.
- Adjusts the CMCKrillClient presentation-times-have-rolled-over limits from 4B to 2B, with the expectation that maybe some cameras do 2^31 instead of 2^32?
- Fixes an issue in CMCKrillStreamingSession where if a reconnect occurs and the presentation times roll over, while we were correctly handling that situation for video, we had not duplicated that code for audio. Fixed.
- The default support ticket message when I'm communicating with a user now includes our toll free phone number.
- Adds support for encrypting Event thumbnails, snapshots and recordings. This task was deferred previously due to challenges in loading the thumbnail in the notification service (which shows you the push notification with an image).
- Overhauls the file upload mechanism. You don't care about the details for this one.
- Fixes an issue where nothing would ever happen if you tried to start multiple remote streams at the same time.
- Fixes an expired access token issue with YouTube Live Streaming where it was not automatically refreshed.
- Fixes another ONVIF WSSE authentication issue that can occur when a certain Lorex NVR miscalculates GMT. [CB]
- Overhauls the camera recorder (Record to NAS). You should see considerably faster loading times with all new recordings.
- Fixes a YouTube Live Streaming issue where a seemingly random, no obvious cause, failure of either of the AVAssetWriters (one for audio, one for video) would cause the live stream to fail (obviously). The movie writer will now reset and recover from the error. It'll signal the YouTubeService to reset the other writer. The stream recovers and all is well. You’ll have a “hiccup” in your live stream for like a minute or so.
- Improves remote streaming reliability. Specifically, on the client side, when a connection unexpectedly closes, the reconnection process should be more likely to succeed. One specific issue had to do with the data packet re-order queue. When a reconnect occurred, the server naturally started the sequence over at zero, while the client still expected data packets to continue in the sequence.
- Fixes another audio-related crash where I was improperly accessing unsafe memory. Sigh. I’m a little shocked this doesn’t cause frequent crashes.
Pan, Tilt & Zoom Release Notes YouTube S3 Record to NAS
Streamie provides a best-in-class user experience on your iPhone, iPad, Apple TV and Apple Silicon Mac, with an intuitive user interface that makes it simple to discover, stream, record, monitor and share your HomeKit, Google Nest, Ubiquiti UniFi Protect and ONVIF-compatible IP and RTSP cameras. Streamie keeps you informed with motion event notifications and it works with most cameras using its advanced audio and video codec support. You can watch your cameras from anywhere, record 24/7 to your private NAS, remotely manage multiple locations, device permissions and seamlessly synchronize settings across your devices; configure Hubitat smart home automations, live stream to YouTube and rely on the in-app technical support system when you need help (but you can also reach us by phone). Download Streamie today. Lastly, Streamie is solar powered!