Since the geojson for the timezone data is a static file you can avoid using a database entirely and ship it to your app.
If you'd like to avoid shipping a giant geojson, with the generosity of CDN providers you can actually hit the npm-hosting servers with HTTP Range requests and do this completely on demand from the client: https://github.com/kevmo314/browser-geo-tz
So I guess it's a bit of a lie that it's only geojson :)
The way it works is you load the index file which contains the large regions (in other words, the first level or two of the quad tree) and if the time zone can't be resolved, it relies on querying the data file which is actually a multi-record protobuf file so you can load the specific range of data you're looking for. The algorithm is here: https://github.com/evansiroky/node-geo-tz/blob/master/src/fi...
It seems that your JSON file is way larger than it could be. The current JSON gzips into 140 KB, but there are only 45K quad labels and 32K leaves (where about a half of them are references). And all references are sorted in order so you can use delta coding at some point. So we can safely assume that references are no more than 2 bytes while direct tz indices are almost likely one byte and we should expect about 60 KB instead. Of course you need an additional jump table in order to avoid linear scanning, but that table is only required at higher levels so it is negligible in comparison.
Ack. Like so many managers I asked the question before code review. I see the curTz and getting offset bytes for a proper blob decode. Very legal and very cool.
I'm guessing this does not try to handle things like terra nullius (Bir Tawil, Marie Byrd Land, etc.) or where there might not be a "correct" answer over which timezone is in a place (like https://www.972mag.com/the-worlds-only-ethnic-time-zone/)?
It just uses the map that was generated by the TimeZone Boundary Builder[0]. It does fine for my purposes.
I just figured it might be useful. One reason that I wrote it, was because I couldn't find a decent SaaS that didn't charge eye-watering prices, for a fairly slow response. I wrote it in an afternoon or two. Not really an ambitious project.
I don't usually do "Show HN." Not interested in competing with anyone. If you like it, use it. If you don't like it, don't use it. If there are clear bugs, I'm always open to feedback.
FWIW, I'd found https://www.geocod.io/ to be a good service a couple years back. They sponsored some conferences I went to, and I needed some geocoding->timezone info. IIRC, at the time, it was $5/month which covered everything we needed. Dunno if that's eye-watering or not.
Seems to have changed pricing a bit since. The 'pay as you go' model now has 2500 API calls/day for free, and $1/2000 API calls after that. My needs never grew in to needing much more than what we hit for that $5/month, so I can't vouch for their service at a large scale, but I enjoyed using it.
Co-Founder of Geocodio here — glad we were able to help with your geocoding+timezone needs!
(And glad to hear that conference sponsorships work. A lot of smaller conferences are struggling to find sponsors these days, so if anyone reading is in a position to sponsor, the organizer of your favorite conference would probably appreciate it!)
Thanks for being a sponsor at PHP[Tek] this year. Your stickers are out on a table next to our booth and I've been periodically telling people what you guys do as they walk by.
Super nice! I wanted to do an Arduino-based GPS clock that always shows the correct local time, but got bogged down getting the exact timezone boundaries.
I recently had this issue with an Android app I made, it was surprisingly difficult to resolve. I ended up going with some OpenStreetMap mirror endpoint for my solution
Well, I looked at all kinds of “clever” approaches, and kept coming back to needing to use a map.
I found a project that is under active development, that generates a very precise shapefile, and figured out how to use it.
I load a database, as opposed to doing shapefile parsing on the fly, for performance reasons.
The magic juice is the domain rect. It’s a superfast triage, that gets us into the ballpark, and then I use a fast winding number test, to get precise.
I think that I could probably do a whole lot to optimize it, but PHP/SQL is not my forte, so I kept it simple. It works great for the manner in which I use it.
The unit tests have a lot of points, close together, on either side of a TZ boundary. The edges are where the Devil lives.
I was under the impression that the standard was to use lat/lng, rather than long/lat. Is there such a standard thing, or I am generalizing my tiny corner of the bits?
I don't lose sleep over it. I find that it depends on the implementation. I see both. I believe that I use the most common format, in the declared API.
I suspect that "long, lat" is common, because people usually specify "x, y", and they are kind of synonymous.
Eh. It works, and works fairly well. I looked at all kinds of alternatives, and this was the one that seemed the most practical.
There’s a reason that there’s so little support for this kind of thing out there. It’s deceptively simple.
I’m a Swift app developer, not a backend guy. I just need a system that delivers the data I need, at an acceptable performance, and this does it.
It’s part of a system that aggregates about 35,000 records, from about 50 servers, and scrubs a lot of really filthy data. This part is for adding missing timezone info.
The deal is about “frontloading” all the data scrubbing, so that the system can quickly return an arbitrary amount of uniform data, on demand, and that can run on the cheapest hosting possible, without a need for tecchies.
Since the geojson for the timezone data is a static file you can avoid using a database entirely and ship it to your app.
If you'd like to avoid shipping a giant geojson, with the generosity of CDN providers you can actually hit the npm-hosting servers with HTTP Range requests and do this completely on demand from the client: https://github.com/kevmo314/browser-geo-tz
How should we get right bits by range? Wouldn't we get chunks of JSON that aren't delimited nicely?
Good question, the geojson data is actually broken down into two parts, a true JSON file and a quad tree: https://github.com/evansiroky/node-geo-tz/tree/master/data
So I guess it's a bit of a lie that it's only geojson :)
The way it works is you load the index file which contains the large regions (in other words, the first level or two of the quad tree) and if the time zone can't be resolved, it relies on querying the data file which is actually a multi-record protobuf file so you can load the specific range of data you're looking for. The algorithm is here: https://github.com/evansiroky/node-geo-tz/blob/master/src/fi...
Here's the step that resolves individual tree nodes: https://github.com/kevmo314/browser-geo-tz/blob/main/src/fin...
It seems that your JSON file is way larger than it could be. The current JSON gzips into 140 KB, but there are only 45K quad labels and 32K leaves (where about a half of them are references). And all references are sorted in order so you can use delta coding at some point. So we can safely assume that references are no more than 2 bytes while direct tz indices are almost likely one byte and we should expect about 60 KB instead. Of course you need an additional jump table in order to avoid linear scanning, but that table is only required at higher levels so it is negligible in comparison.
Ack. Like so many managers I asked the question before code review. I see the curTz and getting offset bytes for a proper blob decode. Very legal and very cool.
Look at this library embedding the TZ data, implemented in multiple languages, even providing an HTTP server https://github.com/ringsaturn/tzf
Hi, thanks for mentioning. A list of tzf libraries:
- Go: https://github.com/ringsaturn/tzf - Rust: https://github.com/ringsaturn/tzf-rs - Python: https://github.com/ringsaturn/tzfpy - Swift: https://github.com/ringsaturn/tzf-swift - Wasm(browser only): https://github.com/ringsaturn/tzf-wasm - PostgreSQL extension: https://github.com/ringsaturn/pg-tzf
Or online preview: <https://ringsaturn.github.io/tzf-web/>.
Here is a blog post about the packages history: https://blog.ringsaturn.me/en/posts/2023-01-31-history-of-tz...
I'm guessing this does not try to handle things like terra nullius (Bir Tawil, Marie Byrd Land, etc.) or where there might not be a "correct" answer over which timezone is in a place (like https://www.972mag.com/the-worlds-only-ethnic-time-zone/)?
It just uses the map that was generated by the TimeZone Boundary Builder[0]. It does fine for my purposes.
I just figured it might be useful. One reason that I wrote it, was because I couldn't find a decent SaaS that didn't charge eye-watering prices, for a fairly slow response. I wrote it in an afternoon or two. Not really an ambitious project.
I don't usually do "Show HN." Not interested in competing with anyone. If you like it, use it. If you don't like it, don't use it. If there are clear bugs, I'm always open to feedback.
[0] https://github.com/evansiroky/timezone-boundary-builder
FWIW, I'd found https://www.geocod.io/ to be a good service a couple years back. They sponsored some conferences I went to, and I needed some geocoding->timezone info. IIRC, at the time, it was $5/month which covered everything we needed. Dunno if that's eye-watering or not.
Seems to have changed pricing a bit since. The 'pay as you go' model now has 2500 API calls/day for free, and $1/2000 API calls after that. My needs never grew in to needing much more than what we hit for that $5/month, so I can't vouch for their service at a large scale, but I enjoyed using it.
No affiliation just a happy former customer.
Co-Founder of Geocodio here — glad we were able to help with your geocoding+timezone needs!
(And glad to hear that conference sponsorships work. A lot of smaller conferences are struggling to find sponsors these days, so if anyone reading is in a position to sponsor, the organizer of your favorite conference would probably appreciate it!)
Thanks for being a sponsor at PHP[Tek] this year. Your stickers are out on a table next to our booth and I've been periodically telling people what you guys do as they walk by.
Well the model that I would use, would be about 20,000 or so lookups, every 4 hours.
Might add up.
well... yes, at that rate, it could. :)
thanks for sharing your tool!
Didn't mean to be negative, just wanted to ask about some of the interesting edge-cases that I've heard of.
No worries. Didn’t take it as such.
Thanks for the interesting link!
Super nice! I wanted to do an Arduino-based GPS clock that always shows the correct local time, but got bogged down getting the exact timezone boundaries.
Thanks for bringing up https://github.com/evansiroky/timezone-boundary-builder !
It's a great project. If you are involved, you have my thanks (also, the thanks of a lot of people that have no idea it exists).
TIL that China only has one timezone... I really liked your visualization of all the timezones in the repo!
That's actually a public domain Wikimedia image. There's an SVG version, which can scale nicely: https://commons.wikimedia.org/wiki/File:World_Time_Zones_Map...
i see... regardless, thanks for making me aware of this image LOL
I recently had this issue with an Android app I made, it was surprisingly difficult to resolve. I ended up going with some OpenStreetMap mirror endpoint for my solution
Well, I looked at all kinds of “clever” approaches, and kept coming back to needing to use a map.
I found a project that is under active development, that generates a very precise shapefile, and figured out how to use it.
I load a database, as opposed to doing shapefile parsing on the fly, for performance reasons.
The magic juice is the domain rect. It’s a superfast triage, that gets us into the ballpark, and then I use a fast winding number test, to get precise.
I think that I could probably do a whole lot to optimize it, but PHP/SQL is not my forte, so I kept it simple. It works great for the manner in which I use it.
The unit tests have a lot of points, close together, on either side of a TZ boundary. The edges are where the Devil lives.
I was under the impression that the standard was to use lat/lng, rather than long/lat. Is there such a standard thing, or I am generalizing my tiny corner of the bits?
I don't lose sleep over it. I find that it depends on the implementation. I see both. I believe that I use the most common format, in the declared API.
I suspect that "long, lat" is common, because people usually specify "x, y", and they are kind of synonymous.
A standalone server feels very excessive.
This is static data that only needs to be updated once or twice a year. Why not just make this an npm or py module
Eh. It works, and works fairly well. I looked at all kinds of alternatives, and this was the one that seemed the most practical.
There’s a reason that there’s so little support for this kind of thing out there. It’s deceptively simple.
I’m a Swift app developer, not a backend guy. I just need a system that delivers the data I need, at an acceptable performance, and this does it.
It’s part of a system that aggregates about 35,000 records, from about 50 servers, and scrubs a lot of really filthy data. This part is for adding missing timezone info.
The deal is about “frontloading” all the data scrubbing, so that the system can quickly return an arbitrary amount of uniform data, on demand, and that can run on the cheapest hosting possible, without a need for tecchies.
It works a treat.